Fix overflow when compressing int128 columns with INT128_MIN bound #633
Merged
Conversation
5250166 to
e080484
Compare
Contributor
|
Agreed on avoiding compression for int128_t. Even though science says we should be able to do it, in practice the benefit are small for UUIDs and similar use cases. If you have a follow up commit, please add a test to verify that data compressed with old code can be decompressed after the changes with edge cases taken into account. |
Contributor
|
Doing it in C++ is preferable - so it covers other language bindings too. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses an INT128 bitpacking overflow when a chunk range starts at INT128_MIN. I triggered this when storing UUID nil values in a table because UUIDs are stored with the MSB flipped internally, so I hit a
Here's a test that can go anywhere in the python-api test suite to reproduce:
The test is for UUIDs, but the bug affects any regular INT128 column if it is persisted and has a non-constant chunk with min == INT128_MIN.
The problem is that frame-of-reference INT128 bitpacking can require subtracting INT128_MIN, which overflows. Falling back to uncompressed storage for this edge case should be correct, and addresses the issue without changing UUID representation or storage format
Changes:
The only downside I can think of is that INT128 chunks whose range starts at INT128_MIN now fall back to UNCOMPRESSED, so those rare (?) chunks use more disk space.
DISCLAIMER
I have heavily relied on a friendly LLM to locate the culprit and fix it. I am way above my head with the column compression stuff, so even though I applied my best judgment, I can't claim this is 100% correct. Sorry about that.
PS: I think that UUIDs should use a uint128 representation, now that there is one such type (it was a late addition to Kuzu IIRC), but the changes would be extensive and change the format on disk, requiring db migration, and potentially touching a lot of files. This fix was necessary anyway and solves my problem with UUIDs. So, win-win? :D
PPS: I have a separate PR to python-api with the test above to guard against the UUID bug explicitly, but it's redundant-ish with this fix.