Skip to content

Commit 825d331

Browse files
mceachenmceachen
committed
feat(benchmark): add memory benchmarking tool and scenarios
- Implemented memory-benchmark.ts for testing memory leaks in SQLite drivers. - Created memory-tracker.ts to track memory usage and detect leaks. - Added various memory scenarios to benchmark different operations. - Introduced package.json for benchmark dependencies and scripts. - Developed scenarios.ts to define benchmark scenarios for SQLite operations. - Configured TypeScript settings in tsconfig.json for the benchmark. - Updated root package.json to include benchmark scripts for easy execution.
1 parent 9a2e4c0 commit 825d331

File tree

11 files changed

+2182
-9
lines changed

11 files changed

+2182
-9
lines changed

‎TODO.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -134,10 +134,11 @@ This document tracks the remaining tasks to complete the SQLite extraction from
134134
- [ ] Review and optimize SQLite compile flags
135135
- [ ] Memory allocation strategies
136136
- [ ] I/O optimization settings
137-
- [ ] **Benchmarking suite**
138-
- [ ] Compare against better-sqlite3 and sqlite3
139-
- [ ] Identify performance bottlenecks
140-
- [ ] Profile memory usage patterns
137+
- ✅ **Benchmarking suite** 🆕
138+
- ✅ Compare against better-sqlite3, sqlite3, and node:sqlite
139+
- ✅ Performance benchmarks for all common operations
140+
- ✅ Memory leak detection and profiling
141+
- ✅ Automated benchmark runner with reporting
141142

142143
### Documentation & Examples ✅ **BASIC DOCS COMPLETE**
143144

@@ -261,12 +262,11 @@ This document tracks the remaining tasks to complete the SQLite extraction from
261262

262263
- ✅ **Enhanced location method**: `location(dbName?: string)` for attached databases - Complete with 10 comprehensive tests
263264

264-
10. **🚧 Performance & Compatibility** (Low Priority)
265+
10. **✅ Performance & Compatibility** (COMPLETED!)
265266

266-
- [ ] Benchmark against alternatives
267-
- [ ] Node.js compatibility verification
268-
- [ ] Memory leak testing
269-
- [ ]
267+
- ✅ **Comprehensive benchmark suite** - Performance and memory testing against all major SQLite libraries
268+
- ✅ **Memory leak testing** - Automated detection with linear regression analysis
269+
- [ ] Node.js compatibility verification (when node:sqlite becomes stable)
270270

271271
## Priority Levels
272272

‎benchmark/.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
node_modules/
2+
package-lock.json
3+
*.db
4+
*.db-journal
5+
*.db-wal
6+
*.log
7+
.DS_Store

‎benchmark/README.md‎

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
# SQLite Driver Benchmarks
2+
3+
Performance and memory benchmarks comparing `@photostructure/sqlite` against other popular SQLite libraries for Node.js.
4+
5+
## Summary
6+
7+
The performance of @photostructure/sqlite is quite similar to node:sqlite and better-sqlite3, while significantly faster than async sqlite3.
8+
9+
## Libraries Tested
10+
11+
- **@photostructure/sqlite** - This package
12+
- **better-sqlite3** - Popular synchronous SQLite3 binding
13+
- **sqlite3** - Classic asynchronous SQLite3 binding
14+
- **node:sqlite** - Node.js built-in SQLite (when available)
15+
16+
## Installation
17+
18+
```bash
19+
cd benchmark
20+
npm install
21+
```
22+
23+
## Running Benchmarks
24+
25+
### Performance Benchmarks
26+
27+
```bash
28+
# Run all benchmarks
29+
npm run bench
30+
31+
# Run specific scenario types
32+
npm run bench select # Only SELECT query benchmarks
33+
npm run bench insert # Only INSERT operation benchmarks
34+
npm run bench transaction # Only transaction benchmarks
35+
36+
# Advanced options
37+
tsx index.ts --drivers @photostructure/sqlite,better-sqlite3
38+
tsx index.ts select --drivers @photostructure/sqlite,node:sqlite
39+
tsx index.ts --verbose
40+
tsx index.ts --memory # Include memory usage tracking
41+
```
42+
43+
### Memory Benchmarks
44+
45+
```bash
46+
# Run memory leak detection (requires --expose-gc)
47+
npm run bench:memory
48+
49+
# Or run directly with tsx
50+
tsx --expose-gc memory-benchmark.ts
51+
52+
# Test specific drivers
53+
tsx --expose-gc memory-benchmark.ts --drivers @photostructure/sqlite,better-sqlite3
54+
55+
# Test specific scenarios
56+
tsx --expose-gc memory-benchmark.ts --scenarios prepare-finalize,large-select
57+
58+
# Adjust iterations for leak detection
59+
tsx --expose-gc memory-benchmark.ts --iterations 100
60+
```
61+
62+
### Command Line Options
63+
64+
#### Performance Benchmarks (`index.ts`)
65+
66+
- `--drivers <list>` - Comma-separated list of drivers to test
67+
- `--verbose` - Show detailed output during benchmarking
68+
- `--memory` - Track memory usage during performance tests
69+
- `--help` - Show usage information
70+
71+
#### Memory Benchmarks (`memory-benchmark.ts`)
72+
73+
- `--drivers <list>` - Comma-separated list of drivers to test
74+
- `--scenarios <list>` - Comma-separated list of memory scenarios to run
75+
- `--iterations <n>` - Number of iterations for leak detection (default: 50)
76+
- `--help` - Show usage information
77+
78+
### Example Commands
79+
80+
```bash
81+
# Compare just the sync drivers
82+
npm run bench -- --drivers @photostructure/sqlite,better-sqlite3,node:sqlite
83+
84+
# Test only SELECT performance
85+
npm run bench select
86+
87+
# Run memory tests with more iterations for accuracy
88+
tsx --expose-gc memory-benchmark.ts --iterations 100
89+
90+
# Test specific memory scenario
91+
tsx --expose-gc memory-benchmark.ts --scenarios blob-handling
92+
```
93+
94+
## Benchmark Scenarios
95+
96+
### Performance Scenarios
97+
98+
1. **select-by-id** - Single row retrieval by primary key (143k+ ops/sec)
99+
2. **select-range** - Fetch up to 1k rows with WHERE clause and index (13k+ ops/sec)
100+
3. **select-iterate** - Iterator performance over 1k rows (700+ ops/sec)
101+
4. **insert-simple** - Single row inserts (700+ ops/sec)
102+
5. **insert-transaction** - Bulk inserts (1k rows) in transaction (300+ ops/sec)
103+
6. **select-join** - Complex JOIN with aggregation (1.8k+ ops/sec)
104+
7. **insert-blob** - Binary data handling (10KB blobs) (600+ ops/sec)
105+
8. **update-indexed** - UPDATE operations using indexed columns (700+ ops/sec)
106+
9. **delete-bulk** - Bulk DELETE in transactions (80+ ops/sec)
107+
108+
### Memory Scenarios
109+
110+
1. **prepare-finalize** - Statement lifecycle memory management
111+
2. **large-select** - Memory handling with large result sets
112+
3. **blob-handling** - Binary data memory management
113+
4. **transaction-stress** - Memory usage in large transactions
114+
5. **prepare-cache** - Statement cache stress testing
115+
116+
## Output Format
117+
118+
### Performance Results
119+
120+
The benchmark outputs clean markdown tables that can be directly copied into documentation:
121+
122+
### 📈 Summary
123+
124+
| Scenario | @photostructure/sqlite | better-sqlite3 | node:sqlite | sqlite3 |
125+
|---|---:|---:|---:|---:|
126+
| SELECT by Primary Key | 150,000 ops/s | 150,000 ops/s | 140,000 ops/s | 24,000 ops/s |
127+
| SELECT Range | 13,000 ops/s | 14,000 ops/s | 13,000 ops/s | 5,600 ops/s |
128+
| SELECT with Iterator | 710 ops/s | 860 ops/s | 750 ops/s | 610 ops/s |
129+
| INSERT Single Row | 720 ops/s | 720 ops/s | 740 ops/s | 710 ops/s |
130+
| INSERT in Transaction | 340 ops/s | 350 ops/s | 350 ops/s | 38 ops/s |
131+
| SELECT with JOIN | 1,800 ops/s | 2,000 ops/s | 1,800 ops/s | 1,600 ops/s |
132+
| INSERT with BLOB | 640 ops/s | 660 ops/s | 650 ops/s | 590 ops/s |
133+
| UPDATE with Index | 750 ops/s | 720 ops/s | 740 ops/s | 720 ops/s |
134+
| DELETE Bulk | 89 ops/s | 90 ops/s | 83 ops/s | 25 ops/s |
135+
136+
137+
### 🏆 Overall Performance Ranking
138+
139+
| Rank | Driver | Score |
140+
|---:|---|---:|
141+
| 1 🥇 | better-sqlite3 | 99% |
142+
| 2 🥈 | @photostructure/sqlite | 94% |
143+
| 3 🥉 | node:sqlite | 94% |
144+
| 4 🐌 | sqlite3 | 58% |
145+
146+
Key features:
147+
148+
- **Adaptive iteration counts**: Automatically calibrates to run each scenario for ~2 seconds
149+
- **Markdown-ready output**: Tables can be directly copied into documentation
150+
- **Comma-formatted numbers**: Easy to read large operation counts
151+
- **Overall performance ranking**: Weighted average across all scenarios
152+
153+
### Memory Results
154+
155+
Memory benchmarks also output markdown-ready tables:
156+
157+
```
158+
💾 SQLite Driver Memory Benchmark
159+
160+
📊 Testing @photostructure/sqlite
161+
162+
Statement Prepare/Finalize: Tests for memory leaks in statement lifecycle
163+
✓ No memory leak detected
164+
Heap growth: 0.12 KB/iteration (R²=0.045)
165+
External growth: 0.00 KB/iteration (R²=0.001)
166+
167+
📈 Summary
168+
169+
| Scenario | @photostructure/sqlite | better-sqlite3 | node:sqlite | sqlite3 |
170+
|---|---|---|---|---|
171+
| Statement Prepare/Finalize | ✓ OK | ✓ OK | ✓ OK | ✓ OK |
172+
| Large Result Sets | ✓ OK | ✓ OK | ✓ OK | ✓ OK |
173+
| BLOB Memory Management | ✓ OK | ✓ OK | ✓ OK | ✓ OK |
174+
175+
📋 Memory table generated above - copy/paste ready for documentation!
176+
```
177+
178+
Features:
179+
180+
- **Leak detection**: Automatically identifies potential memory leaks (>1KB/iteration growth)
181+
- **Statistical analysis**: R² correlation values show trend strength
182+
- **Multiple scenarios**: Tests various memory usage patterns
183+
- **Markdown output**: Ready for documentation
184+
185+
## Interpreting Results
186+
187+
### Performance Metrics
188+
189+
- **Ops/sec**: Operations per second (higher is better)
190+
- **Relative**: Performance relative to fastest driver
191+
- **Margin**: Error margin (lower is more consistent)
192+
- **Runs**: Number of benchmark samples collected
193+
194+
### Memory Metrics
195+
196+
- **Heap growth**: Memory growth rate per iteration
197+
- **External growth**: Native memory growth rate
198+
- **R²**: Correlation coefficient (closer to 1 = stronger trend)
199+
- **Leak detection**: Flags potential memory leaks (>1KB/iteration)
200+
201+
## Advanced Features
202+
203+
The benchmark system automatically calibrates iteration counts and scales results based on operation complexity to ensure fair comparisons across all drivers.
204+
205+
## Notes
206+
207+
- **sqlite3 Performance**: The sqlite3 driver shows lower performance in synchronous-style benchmarks because it's inherently asynchronous.
208+
- **Memory Testing**: Always run memory benchmarks with `--expose-gc` for accurate garbage collection control.
209+
- **Real-world Performance**: These benchmarks test specific patterns. Real application performance depends on your specific use case.
210+
211+
## Contributing
212+
213+
To add new benchmark scenarios:
214+
215+
1. Add scenario to `scenarios.js` for performance tests
216+
2. Add scenario to `memory-benchmark.js` for memory tests
217+
3. Follow the existing pattern for setup/run/cleanup
218+
4. Ensure scenarios are fair across all drivers
219+
220+
## License
221+
222+
Same as parent project

0 commit comments

Comments
 (0)