Welcome to the Python Generators & MySQL Efficient Data Processing project! This project demonstrates how to:
- Connect to a MySQL database from Python
- Create and seed a table with user data from a CSV file
- Use Python generators to efficiently stream data from the database
| File/Folder | Purpose |
|---|---|
python-generators-0x00/seed.py |
Main script: DB setup, CSV import, and generator logic |
python-generators-0x00/user_data.csv |
Original user data (no user_id) |
python-generators-0x00/user_data_with_id.csv |
User data with unique user_id (generated by update.py) |
python-generators-0x00/update.py |
Script to add unique user_id to each CSV row |
python-generators-0x00/README.md |
Project documentation for the generator project |
README.md |
This root documentation file |
- Connects to MySQL (edit credentials/port in
seed.pyif needed) - Creates a database and a
user_datatable if they don't exist
- Use
update.pyto generateuser_data_with_id.csvwith unique user_ids seed.pyclears the table and inserts all users from the CSV
- (To be implemented) A generator function will yield user records one at a time from the database, saving memory and enabling batch processing.
| user_id (PK) | name | age | |
|---|---|---|---|
| 123e4567-e89b-12d3-a456-426614174000 | John Doe | [email protected] | 30 |
| ... | ... | ... | ... |
- Generate CSV with IDs:
python python-generators-0x00/update.py
- Seed the database:
python python-generators-0x00/seed.py
- (Coming soon) Stream data with a generator:
- Use the generator function in
seed.pyto process users one by one.
- Use the generator function in
- Make sure MySQL is running and accessible on the configured port.
- Update credentials in
seed.pyas needed. - For large datasets, using generators is much more memory efficient!
- Python DB connection & error handling
- Table creation and data seeding
- Unique ID generation for each user
- Efficient data streaming with generators
- Clear, emoji-rich commit messages for every step