Skip to content

Fix overflow when compressing int128 columns with INT128_MIN bound #633

Merged
adsharma merged 1 commit into
LadybugDB:mainfrom
mdbenito:fix/uuid-uint128
Jun 30, 2026
Merged

Fix overflow when compressing int128 columns with INT128_MIN bound #633
adsharma merged 1 commit into
LadybugDB:mainfrom
mdbenito:fix/uuid-uint128

Conversation

@mdbenito

Copy link
Copy Markdown
Contributor

This PR addresses an INT128 bitpacking overflow when a chunk range starts at INT128_MIN. I triggered this when storing UUID nil values in a table because UUIDs are stored with the MSB flipped internally, so I hit a

RuntimeError: Overflow exception: INT128 is out of range: cannot negate INT128_MIN

Here's a test that can go anywhere in the python-api test suite to reproduce:

def test_copy_from_pandas_uuid_int128_min_persistent(conn_db_empty: ConnDB) -> None:
    conn, _ = conn_db_empty
    conn.execute("CREATE NODE TABLE T(id UUID PRIMARY KEY)")
    df = pd.DataFrame(
        {
            "id": [
                # UUID nil is stored internally as flipped INT128_MIN. Together with
                # a second value, this forces a non-constant persistent chunk and
                # crashes while computing INT128 bitpacking offsets.
                UUID("00000000-0000-0000-0000-000000000000"),
                UUID("00000000-0000-0000-0000-000000000001"),
            ]
        }
    )

    conn.execute("COPY T FROM df")

    result = conn.execute("MATCH (t:T) RETURN t.id ORDER BY t.id")
    assert result.get_all() == [
        [UUID("00000000-0000-0000-0000-000000000000")],
        [UUID("00000000-0000-0000-0000-000000000001")],
    ]

The test is for UUIDs, but the bug affects any regular INT128 column if it is persisted and has a non-constant chunk with min == INT128_MIN.

The problem is that frame-of-reference INT128 bitpacking can require subtracting INT128_MIN, which overflows. Falling back to uncompressed storage for this edge case should be correct, and addresses the issue without changing UUID representation or storage format

Changes:

  • Detect int128_t metadata ranges starting at INT128_MIN and force full-width/no-offset packing info so existing metadata selection falls back to UNCOMPRESSED.
  • Avoid unnecessary signed subtraction when bitpacking offset is zero.
  • Add C++ regression coverage for INT128_MIN compression metadata.

The only downside I can think of is that INT128 chunks whose range starts at INT128_MIN now fall back to UNCOMPRESSED, so those rare (?) chunks use more disk space.

DISCLAIMER

I have heavily relied on a friendly LLM to locate the culprit and fix it. I am way above my head with the column compression stuff, so even though I applied my best judgment, I can't claim this is 100% correct. Sorry about that.

PS: I think that UUIDs should use a uint128 representation, now that there is one such type (it was a late addition to Kuzu IIRC), but the changes would be extensive and change the format on disk, requiring db migration, and potentially touching a lot of files. This fix was necessary anyway and solves my problem with UUIDs. So, win-win? :D

PPS: I have a separate PR to python-api with the test above to guard against the UUID bug explicitly, but it's redundant-ish with this fix.

@adsharma

Copy link
Copy Markdown
Contributor

Agreed on avoiding compression for int128_t. Even though science says we should be able to do it, in practice the benefit are small for UUIDs and similar use cases.

If you have a follow up commit, please add a test to verify that data compressed with old code can be decompressed after the changes with edge cases taken into account.

@adsharma

Copy link
Copy Markdown
Contributor

Doing it in C++ is preferable - so it covers other language bindings too.

@adsharma adsharma merged commit 231a0b8 into LadybugDB:main Jun 30, 2026
4 checks passed
@mdbenito mdbenito deleted the fix/uuid-uint128 branch July 1, 2026 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants