Add high-performance bulk loading optimizations for large files - #16

Merged

kevinschaper merged 19 commits into

mainfrom

performance-optimizations

Aug 30, 2025

Commits on Aug 13, 2025

Add high-performance bulk loading optimizations for large files

kevinschaper
and
claude
committed
bdc9107
Add required dependencies for performance optimizations

kevinschaper
and
claude
committed
3f2be33
Bump pandas dependency to modern version

kevinschaper
and
claude
committed
003860f
Replace pandas with DuckDB for robust TSV/CSV processing

kevinschaper
and
claude
committed
d7f1ec1
Fix commit function missing Content-Type header

kevinschaper
and
claude
committed
892f60b
Implement true parallel upload processing

kevinschaper
and
claude
committed
9dfb461
Add detailed error logging for Solr configuration failures

kevinschaper
and
claude
committed
df59eb2
Remove CBOR support and fix RAM buffer configuration

kevinschaper
and
claude
committed
551043b
Fix typo in import statement

kevinschaper
and
claude
committed
ad97051
Add timing information to bulkload command

kevinschaper
and
claude
committed
1d6781d
Add granular timing for preprocessing vs upload phases

kevinschaper
and
claude
committed
cff5626
Add comprehensive throughput metrics (docs/sec)

kevinschaper
and
claude
committed
71a6eb7
Add high-performance DuckDB to Solr bulk loading command

kevinschaper
and
claude
committed
7187557
Adjust default performance settings for better balance

kevinschaper
and
claude
committed
0dfe0dc

Commits on Aug 14, 2025

Fix CSV parsing errors with malformed rows

kevinschaper
and
claude
committed
fcddaab

Commits on Aug 15, 2025

Fix field concatenation issue by using TSV format consistently

kevinschaper
and
claude
committed
b344da5

Commits on Aug 22, 2025

add processor option for db bulkload
kevinschaper
committed
f218771
batching improvements
kevinschaper
committed
0f2f324

Commits on Aug 23, 2025

increased request header size
kevinschaper
committed
01985b2