Skip to content

Add ducklake destination #627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bnjjj wants to merge 29 commits into
base: main
Choose a base branch
from
Open

Add ducklake destination #627

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
bb166af
add POC duckdb
bnjjj Feb 20, 2026
6bea32a
ducklake
bnjjj Feb 23, 2026
78898d3
feat(destinations): add DuckLake s3 config options (#618)
bnjjj Mar 3, 2026
81cf987
Merge branch 'main' of github.com:supabase/etl into bnjjj/ducklake
bnjjj Mar 3, 2026
1afa7ef
improve blocking tasks handling, fix some potential perf issues, clea…
bnjjj Mar 9, 2026
e2d7bbe
make new async
bnjjj Mar 9, 2026
881ce2d
address review comments
bnjjj Mar 9, 2026
c153550
add duckdb extensions vendoring
bnjjj Mar 11, 2026
20c1a53
working but slow AF
bnjjj Mar 13, 2026
8ac3a45
improve connection pooling
bnjjj Mar 13, 2026
3e150d6
add caching
bnjjj Mar 23, 2026
c4d3ff2
improve perfs with inlining
bnjjj Mar 24, 2026
3087dfb
Merge branch 'main' of github.com:supabase/etl into bnjjj/ducklake
bnjjj Mar 24, 2026
2b44fa7
cleaner code
bnjjj Mar 25, 2026
c54031e
add metrics
bnjjj Mar 25, 2026
ed5705a
add maintenance task
bnjjj Mar 26, 2026
f1c739a
add timeout on duckdb connections
bnjjj Mar 26, 2026
69263c7
add metrics for maintenance fails
bnjjj Mar 26, 2026
049126e
refactor
bnjjj Mar 26, 2026
0fe9d3a
add metrics for maintenance tasks
bnjjj Mar 26, 2026
2314a38
cleaning
bnjjj Mar 26, 2026
505d42f
better metrics for ducklake maintenances
bnjjj Mar 26, 2026
09035f3
improve metrics
bnjjj Mar 26, 2026
c521c2e
improve table maintenances
bnjjj Mar 26, 2026
abedf9f
improve inline flush threshold
bnjjj Mar 27, 2026
5a79416
remove config files
bnjjj Mar 27, 2026
b9e9c84
remove config files
bnjjj Mar 27, 2026
c1c5d83
remove duplicated functions for ducklake url
bnjjj Mar 27, 2026
a7f19bf
remove useless logs option for ducklake
bnjjj Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@ logs/
*.log
.DS_Store
Thumbs.db
venv/
venv/
vendor/duckdb/extensions/
14 changes: 12 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,19 @@ target/
site/

# venv files
bin/
lib/
/bin/
/lib/
pyvenv.cfg

# Log files
*.log

# Serena files
.serena

# Optional local DuckDB extension vendoring.
vendor/duckdb/extensions/

# Configuration files
etl-replicator/configuration
configuration
Loading