Skip to content

Commit 1071110

Browse files
iemejiaLuciferYang
iemejia
authored and
LuciferYang
committed
[SPARK-56892][SQL] Bulk read optimization for Parquet DELTA_BINARY_PACKED decoding
### What changes were proposed in this pull request? Re-apply the bulk read optimization for `VectorizedDeltaBinaryPackedReader` (reverted in c13302a) with a fix for the INT32 widening bug that caused the CI failure. **Commit 1** — Reapply the original optimization (revert of the revert): - Bulk `readIntegers`/`readLongs` via prefix-sum + `putInts`/`putLongs` - Zero-allocation unsigned long encoding (`encodeUnsignedLongBigEndian`) - `readIntegersAsLongs` and `readIntegersAsDoubles` overrides **Commit 2** — Fix the INT32 widening bug: - The Parquet INT32 delta encoder (`DeltaBinaryPackingValuesWriterForInteger`) computes deltas using Java int arithmetic with modular overflow. The bulk widened readers (`readIntegersAsLongs`, `readIntegersAsDoubles`) were performing the prefix sum in long space and writing raw long results without truncating back to int. When delta overflow occurs (e.g. a sequence containing `Int.MinValue`), the reconstructed long has the wrong sign. - Fix: truncate each prefix-sum result to int before widening to long/double - Add focused low-level tests for the overflow case (single-batch and split reads) - Add benchmark cases for the overflow pattern This is the same content as #55919, which was merged and reverted due to this bug. ### Why are the changes needed? The bulk read path eliminates per-value lambda dispatch overhead and enables the JIT to better vectorize the inner unpacking loop. See #55919 for full benchmark results. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - `ParquetTypeWideningSuite`: IntegerType -> LongType, IntegerType -> DoubleType - `ParquetDeltaEncodingInteger`: new focused tests for modular delta overflow - `ParquetDeltaEncodingInteger`/`Long`: full suites (30 tests) - `ParquetIOSuite`: UINT_64 tests - `VectorizedDeltaReaderBenchmark`: full suite including new overflow cases ### Was this patch authored or co-authored using generative AI tooling? Yes. Assisted-by: GitHub Copilot:claude-opus-4.6 Closes #56543 from iemejia/SPARK-56892-delta-binary-packed-bulk-read-v2. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>
1 parent 0f62d3d commit 1071110

9 files changed

Lines changed: 446 additions & 158 deletions

File tree

‎sql/core/benchmarks/VectorizedDeltaReaderBenchmark-jdk21-results.txt‎

Lines changed: 50 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -2,92 +2,94 @@
22
DELTA_BINARY_PACKED INT32
33
================================================================================================
44

5-
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1011-azure
5+
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
66
AMD EPYC 7763 64-Core Processor
77
DELTA_BINARY_PACKED INT32: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
88
------------------------------------------------------------------------------------------------------------------------
9-
readIntegers, constant 2 2 0 451.8 2.2 1.0X
10-
skipIntegers, constant 3 3 0 415.1 2.4 0.9X
11-
readIntegers, monotonic 3 3 0 369.7 2.7 0.8X
12-
skipIntegers, monotonic 3 3 0 415.6 2.4 0.9X
13-
readIntegers, small-delta random 3 3 0 308.6 3.2 0.7X
14-
skipIntegers, small-delta random 3 3 0 358.6 2.8 0.8X
15-
readIntegers, wide random 4 4 0 249.2 4.0 0.6X
16-
skipIntegers, wide random 4 4 0 281.4 3.6 0.6X
9+
readIntegers, constant 2 2 0 476.2 2.1 1.0X
10+
skipIntegers, constant 2 2 0 561.5 1.8 1.2X
11+
readIntegers, monotonic 2 2 0 480.9 2.1 1.0X
12+
skipIntegers, monotonic 2 2 0 562.4 1.8 1.2X
13+
readIntegers, small-delta random 3 3 0 330.9 3.0 0.7X
14+
skipIntegers, small-delta random 3 3 0 376.2 2.7 0.8X
15+
readIntegers, wide random 4 4 0 268.4 3.7 0.6X
16+
skipIntegers, wide random 4 4 0 293.1 3.4 0.6X
1717

1818

1919
================================================================================================
2020
DELTA_BINARY_PACKED INT64
2121
================================================================================================
2222

23-
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1011-azure
23+
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
2424
AMD EPYC 7763 64-Core Processor
2525
DELTA_BINARY_PACKED INT64: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
2626
------------------------------------------------------------------------------------------------------------------------
27-
readLongs, constant 5 6 0 195.2 5.1 1.0X
28-
skipLongs, constant 6 6 0 175.4 5.7 0.9X
29-
readLongs, monotonic 7 7 0 158.8 6.3 0.8X
30-
skipLongs, monotonic 6 6 0 175.6 5.7 0.9X
31-
readLongs, small-delta random 8 8 0 139.4 7.2 0.7X
32-
skipLongs, small-delta random 7 7 0 152.5 6.6 0.8X
33-
readLongs, wide random 10 10 1 102.9 9.7 0.5X
34-
skipLongs, wide random 6 9 2 185.5 5.4 1.0X
27+
readLongs, constant 2 2 0 518.5 1.9 1.0X
28+
skipLongs, constant 2 2 0 545.8 1.8 1.1X
29+
readLongs, monotonic 2 2 0 517.6 1.9 1.0X
30+
skipLongs, monotonic 2 2 0 559.7 1.8 1.1X
31+
readLongs, small-delta random 3 3 0 357.3 2.8 0.7X
32+
skipLongs, small-delta random 3 3 0 377.0 2.7 0.7X
33+
readLongs, wide random 6 6 0 186.1 5.4 0.4X
34+
skipLongs, wide random 5 6 0 190.7 5.2 0.4X
3535

3636

3737
================================================================================================
3838
DELTA_BYTE_ARRAY
3939
================================================================================================
4040

41-
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1011-azure
41+
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
4242
AMD EPYC 7763 64-Core Processor
4343
DELTA_BYTE_ARRAY: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
4444
------------------------------------------------------------------------------------------------------------------------
45-
readBinary, no overlap, len=16 35 41 4 29.6 33.8 1.0X
46-
skipBinary, no overlap, len=16 41 45 3 25.9 38.6 0.9X
47-
readBinary, half overlap, len=16 41 44 3 25.4 39.3 0.9X
48-
skipBinary, half overlap, len=16 47 50 5 22.3 44.8 0.8X
49-
readBinary, full overlap, len=16 42 44 3 25.1 39.9 0.8X
50-
skipBinary, full overlap, len=16 48 50 3 22.0 45.6 0.7X
51-
readBinary, half overlap, len=64 42 42 1 25.2 39.6 0.9X
52-
skipBinary, half overlap, len=64 47 50 14 22.4 44.6 0.8X
45+
readBinary, no overlap, len=16 27 32 2 38.2 26.2 1.0X
46+
skipBinary, no overlap, len=16 34 36 2 30.8 32.5 0.8X
47+
readBinary, half overlap, len=16 33 34 0 31.5 31.7 0.8X
48+
skipBinary, half overlap, len=16 38 39 1 27.7 36.1 0.7X
49+
readBinary, full overlap, len=16 33 34 1 31.5 31.7 0.8X
50+
skipBinary, full overlap, len=16 38 39 2 27.4 36.5 0.7X
51+
readBinary, half overlap, len=64 33 34 1 31.6 31.7 0.8X
52+
skipBinary, half overlap, len=64 38 39 2 27.9 35.8 0.7X
5353

5454

5555
================================================================================================
5656
DELTA_LENGTH_BYTE_ARRAY
5757
================================================================================================
5858

59-
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1011-azure
59+
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
6060
AMD EPYC 7763 64-Core Processor
6161
DELTA_LENGTH_BYTE_ARRAY: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
6262
------------------------------------------------------------------------------------------------------------------------
63-
readBinary, payloadLen=8 20 21 0 51.7 19.3 1.0X
64-
skipBinary, payloadLen=8 10 10 0 106.3 9.4 2.1X
65-
readBinary, payloadLen=32 16 18 2 63.9 15.7 1.2X
66-
skipBinary, payloadLen=32 6 6 0 175.9 5.7 3.4X
67-
readBinary, payloadLen=128 19 19 0 56.0 17.8 1.1X
68-
skipBinary, payloadLen=128 6 6 0 176.3 5.7 3.4X
69-
readBinary, payloadLen=512 41 43 3 25.6 39.0 0.5X
70-
skipBinary, payloadLen=512 6 6 1 176.4 5.7 3.4X
63+
readBinary, payloadLen=8 16 16 0 65.7 15.2 1.0X
64+
skipBinary, payloadLen=8 6 6 0 184.2 5.4 2.8X
65+
readBinary, payloadLen=32 16 16 1 65.0 15.4 1.0X
66+
skipBinary, payloadLen=32 6 6 0 185.6 5.4 2.8X
67+
readBinary, payloadLen=128 18 19 1 56.9 17.6 0.9X
68+
skipBinary, payloadLen=128 6 6 0 185.6 5.4 2.8X
69+
readBinary, payloadLen=512 39 40 1 27.1 36.9 0.4X
70+
skipBinary, payloadLen=512 6 6 0 185.5 5.4 2.8X
7171

7272

7373
================================================================================================
7474
Variant reads
7575
================================================================================================
7676

77-
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1011-azure
77+
OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
7878
AMD EPYC 7763 64-Core Processor
7979
Variant reads: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
8080
-------------------------------------------------------------------------------------------------------------------------------
81-
readBytes (INT32) 7 7 0 142.3 7.0 1.0X
82-
readShorts (INT32) 7 8 0 143.0 7.0 1.0X
83-
readUnsignedIntegers (INT32 -> Long) 7 7 0 147.1 6.8 1.0X
84-
readUnsignedLongs (INT64 -> Decimal(20,0)) 232 243 23 4.5 221.5 0.0X
85-
skipBytes 4 4 0 285.4 3.5 2.0X
86-
skipShorts 4 4 0 285.4 3.5 2.0X
87-
readByte (INT32 single-value) 13 13 1 80.2 12.5 0.6X
88-
readShort (INT32 single-value) 13 13 1 82.0 12.2 0.6X
89-
readInteger (INT32 single-value) 13 13 0 80.2 12.5 0.6X
90-
readLong (INT64 single-value) 14 14 0 74.9 13.3 0.5X
91-
readBinary(len) (DELTA_BYTE_ARRAY single-value) 64 67 3 16.5 60.6 0.1X
81+
readBytes (INT32) 4 4 0 236.3 4.2 1.0X
82+
readShorts (INT32) 9 9 2 117.6 8.5 0.5X
83+
readUnsignedIntegers (INT32 -> Long) 9 9 1 117.6 8.5 0.5X
84+
readIntegersAsLongs (INT32 -> Long) 4 4 0 286.8 3.5 1.2X
85+
readIntegersAsDoubles (INT32 -> Double) 4 4 0 258.1 3.9 1.1X
86+
readUnsignedLongs (INT64 -> Decimal(20,0)) 28 29 0 36.9 27.1 0.2X
87+
skipBytes 6 6 0 175.1 5.7 0.7X
88+
skipShorts 6 6 1 175.4 5.7 0.7X
89+
readByte (INT32 single-value) 14 14 1 76.5 13.1 0.3X
90+
readShort (INT32 single-value) 14 14 0 76.6 13.1 0.3X
91+
readInteger (INT32 single-value) 14 14 0 76.6 13.1 0.3X
92+
readLong (INT64 single-value) 16 16 0 67.3 14.9 0.3X
93+
readBinary(len) (DELTA_BYTE_ARRAY single-value) 63 65 2 16.6 60.2 0.1X
9294

9395

0 commit comments

Comments
 (0)