elt-pipeline

DataForge helps data teams write functional transformation pipelines by leveraging software engineering principles

ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

A Pub/Sub for Tables based data integration platform, to discover, publish, modify and consume data effortlessly.

An end-to-end data pipeline which extracts divvy bikeshare data from web loads it into data lake and datawarehouse transforms it using dbt and finally , a dashboard to visualize the data using looker studio, the pipeline is orchestrated using prefect

Frontend & BFF (Backend for frontend) for Olake. This includes the UI code and backend code for storing the configuration of sync and orchestrating it.

A Python package to load complex XML files into a relational database

Decodes, packs, encodes, proves, stores and uploads block-replicas (primarily "block-specimens") produced by EVM or non-EVM byte code based blockchains.

Public DBT instance to aid in data transformation for analytics purposes

Modeling tool like DBT to use SQL Alchemy core with a DataFrame interface like

End-to-end ELT batch pipeline | Medallion Architecture (Bronze → Silver → Gold) | dbt + Snowflake + AWS S3 + Apache Airflow + Docker | 3.5M+ records

A modern data platform implemented on Azure Synapse Analytics using ELT Framework - https://github.com/bennyaustin/elt-framework. Data platform infrastructure provisioned using https://github.com/bennyaustin/iac-synapse-dataplatform

This Repo contains all study, lab and supportive materials for Udemy course on "Google Cloud Professional Data Engineer - A Complete Guide".

A data engineering project with dbt, Docker, Kestra, Terraform, GCP and Looker.

Data engineering projects

A Google Cloud ELT pipeline project with dbt, Cloud Run, Composer, BigQuery, and Dataplex. Includes event-driven automation, data quality checks, and complete setup guidance.

🌄📈📉 A Data Engineering Project 🌈 that implements an ELT data pipeline using Dagster, Docker, Dbt, Polars, Snowflake, PostgreSQL. Data from kaggle website 🔥

💻💛Fundamental Data Engineering Course 2024 Week4 Learn DBT Transform Data with Models, Macro, ELT-Pipeline with Dagster 🌎

Unified ELT pipeline for two apps (service marketplace + ecommerce). PostGIS extraction, Kenyan phone normalisation, cross-platform customer LTV, demand heatmaps. Polars + DuckDB + dbt + Prefect.

An end-to-end real-time data engineering pipeline simulating a music streaming platform, built with Kafka, Spark Streaming, MinIO, ClickHouse, dbt, and Airflow

ELT pipeline (dbt + BigQuery) con medallion architecture, 17 test automatizzati e CI/CD via GitHub Actions. Gestisce dati e-commerce sporchi in modo dichiarato: deduplicazione, integrità referenziale, imputazione statistica tracciabile - nessuna correzione silenziosa.