Skip to content

Efficient Python generator scripts for database seeding, batch processing, and MySQL integration. πŸš€πŸπŸ—„οΈ

Notifications You must be signed in to change notification settings

enyasystem/alx-backend-python

Repository files navigation

🐍 Python Generators & MySQL Project πŸš€

Welcome to the Python Generators & MySQL Efficient Data Processing project! This project demonstrates how to:

  • Connect to a MySQL database from Python
  • Create and seed a table with user data from a CSV file
  • Use Python generators to efficiently stream data from the database

πŸ“¦ Project Structure

File/Folder Purpose
python-generators-0x00/seed.py Main script: DB setup, CSV import, and generator logic
python-generators-0x00/user_data.csv Original user data (no user_id)
python-generators-0x00/user_data_with_id.csv User data with unique user_id (generated by update.py)
python-generators-0x00/update.py Script to add unique user_id to each CSV row
python-generators-0x00/README.md Project documentation for the generator project
README.md This root documentation file

πŸ› οΈ How It Works

1. Database Setup

  • Connects to MySQL (edit credentials/port in seed.py if needed)
  • Creates a database and a user_data table if they don't exist

2. Seeding Data

  • Use update.py to generate user_data_with_id.csv with unique user_ids
  • seed.py clears the table and inserts all users from the CSV

3. Generator for Efficient Data Streaming

  • (To be implemented) A generator function will yield user records one at a time from the database, saving memory and enabling batch processing.

πŸ§‘β€πŸ’» Example Table Structure

user_id (PK) name email age
123e4567-e89b-12d3-a456-426614174000 John Doe [email protected] 30
... ... ... ...

🚦 Usage

  1. Generate CSV with IDs:
    python python-generators-0x00/update.py
  2. Seed the database:
    python python-generators-0x00/seed.py
  3. (Coming soon) Stream data with a generator:
    • Use the generator function in seed.py to process users one by one.

πŸ“ Notes

  • Make sure MySQL is running and accessible on the configured port.
  • Update credentials in seed.py as needed.
  • For large datasets, using generators is much more memory efficient!

✨ Features & Learning Points

  • Python DB connection & error handling
  • Table creation and data seeding
  • Unique ID generation for each user
  • Efficient data streaming with generators
  • Clear, emoji-rich commit messages for every step

About

Efficient Python generator scripts for database seeding, batch processing, and MySQL integration. πŸš€πŸπŸ—„οΈ

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published