Nessie/Risingwave incompatibility

[duhizjame]

risingwave/src/connector/src/sink/iceberg/mod.rs

Line 667 in 5ef31ee

for (i, (column, transform)) in parse_partition_by_exprs(partition_by.clone())?

Currently, RisingWave enumerates field_ids from 1, while nessie expects field_ids of partitions to be over 1000. This effectively makes creation of partitioned tables in Nessie for the time being impossible.

    CREATE SINK enriched_iceberg FROM rw_sqlmesh_example.enriched_model
    WITH (
        connector = 'iceberg',
        type = 'append-only',
        database.name = 'raw',
        table.name = 'enriched',
        catalog.type = 'rest',
        catalog.uri = 'http://nessie:19120/iceberg',
        catalog.name = 'nessie',
        s3.access.key = 'admin',
        s3.secret.key = 'password',
        s3.path.style.access = 'true',
        s3.endpoint = 'http://nginx:9000',
        s3.region = 'us-west-2',
        partition_by = 'arrival_day',
        create_table_if_not_exists = TRUE,
        primary_key = 'arrival_day'
    );

This config ends up with a error 500 from Nessie.
I have updated this issue on their side as well: projectnessie/nessie#10707
While it is not something that is present in the Iceberg specification docs, it is hardcoded in the implementation:
https://github.com/apache/iceberg/blob/90bfc3dff5f40f9ba886832ba9ccaa7b42298e9a/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L53
It would make sense to align the partition field ids in RW, since every writer compatible with Nessie Rest would have to do this as well

EDIT: it is also in the Iceberg specification:

The field-id property was added for each partition field in v2. In v1, the reference implementation assigned field ids sequentially in each spec starting at 1,000. See Partition Evolution for more details.

The only change from v1 -> v2 is that the field-id is now persisted in the spec, not reassigned.

As such, it should be aligned and ids should be assigned starting from 1000 in RisingWave as well when creating the table.
I might have some time to dedicate to work on this issue after 20th December

[chenzl25]

Thanks @duhizjame . I just fix this issue by the way. #24139