fix(schema): permit opt-in timestamp-precision evolution

[yihua]

Describe the issue this Pull Request addresses

Earlier Hudi versions mishandled long-backed timestamp logical types in AvroInternalSchemaConverter:

1. timestamp-millis and timestamp-micros both collapsed into a single internal TimestampType and were always re-emitted as timestampMicros() on serialize. A source schema declaring timestamp-millis got persisted in the table with the wrong timestamp-micros logical type, while the underlying long values written to parquet remained epoch-millis. Pure logical-type drift.
2. local-timestamp-millis and local-timestamp-micros had no branch at all. They fell through to the bare LongType, and the logical type was dropped from the table schema entirely. Logical-type loss.

In both cases the parquet values are correct; only the logical type on the field is wrong. Current converters recognize all four logical types as distinct, so the writer schema now declares the correct logical type. On every subsequent write the reconcile path compares writer schema against the persisted table schema, finds the logical-type mismatch, and rejects it.

With hoodie.write.set.null.for.missing.columns=false the table schema already self-repairs on the next commit: HoodieSchemaUtils.deduceWriterSchema skips reconcileSchema entirely and lets AvroSchemaCompatibility.checkReaderWriterCompatibility validate. That check is logical-type-blind (both timestamps are long underneath), so it accepts the corrected logical type from the writer schema and the next commit rewrites the table schema's logical type accordingly. No change is needed for this path.

With hoodie.write.set.null.for.missing.columns=true the repair is blocked. HoodieSchemaUtils.deduceWriterSchema instead calls AvroSchemaEvolutionUtils.reconcileSchema, which goes through TableChanges.ColumnUpdateChange.updateColumnType and SchemaChangeUtils.isTypeUpdateAllow. That switch had no case for TIMESTAMP or TIMESTAMP_MILLIS, so any logical-type change fell into default: return false and threw SchemaCompatibilityException. The reconcile path was strictly stricter than the non-reconcile path for the same scenario; this PR fixes only that one path.

Complements the read-side repair from #14161, which handles parquet files carrying the wrong logical type transparently. This PR closes the write-side gap so the table schema itself can be brought into agreement with the writer schema even when set.null.for.missing.columns=true.

Summary and Changelog

Users gain a per-write opt-in to forward-fix tables whose persisted schema carries a wrong or missing timestamp logical type, by allowing the internal-schema reconcile path to update the column's logical type to match the writer schema. The non-reconcile path (set.null.for.missing.columns=false) already repaired the logical type implicitly via the Avro reader/writer compatibility check; this PR brings the reconcile path to parity. Default behavior is unchanged.

Changes:

• New advanced write config hoodie.write.schema.allow.timestamp.precision.evolution on HoodieCommonConfig (default false, sinceVersion("1.3.0")).
• SchemaChangeUtils.isTypeUpdateAllow gains a boolean allowTimestampPrecisionEvolution parameter. When true, the switch permits:
  • timestamp-millis ↔ timestamp-micros (logical-type drift case, both directions)
  • local-timestamp-millis ↔ local-timestamp-micros (precision swap among the recognized variants)
  • long → local-timestamp-millis / long → local-timestamp-micros (logical-type loss case, attach the missing logical type)
• AvroSchemaEvolutionUtils.reconcileSchema gains an overload that threads the flag through to TableChanges.ColumnUpdateChange, which stores it on the change and passes it to isTypeUpdateAllow. Pre-existing reconcileSchema / ColumnUpdateChange.get overloads kept as delegates.
• The four writer-side callers (HoodieSchemaUtils.scala, BaseHoodieWriteClient, HoodieMergeHelper, FileGroupReaderBasedMergeHandle) read the config from the write properties and pass it through to reconcileSchema.
• Tests:
  • TestAvroSchemaEvolutionUtils.testReconcileSchemaTimestampPrecisionEvolution asserts default-strict reject and opt-in permit for all three permitted shapes.
  • testCOWLogicalRepair / testMORLogicalRepair parameterize on both setNullForMissingColumns and allowTimestampPrecisionEvolution; positive variants exercise the gated repair path on the v6/v8/CURRENT logical-repair fixtures from fix(ingest): Repair affected logical timestamp milli tables #14161; a negative variant asserts SchemaCompatibilityException when the reconcile path is on with the gate closed.

Impact

• New advanced write config, opt-in. Default preserves the prior strict behavior; no existing caller sees a change.
• One new overload on AvroSchemaEvolutionUtils.reconcileSchema, one new factory on TableChanges.ColumnUpdateChange.get. Pre-existing overloads kept as delegates, so no public-API breakage.

Risk Level

low

The gate defaults off, so existing writers are unaffected. Test coverage adds positive variants on the existing logical-repair fixtures and a negative variant locking in the default-strict behavior.

Documentation Update

New config documented inline on HoodieCommonConfig.ALLOW_TIMESTAMP_PRECISION_EVOLUTION with sinceVersion("1.3.0"). No website update needed.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-bot]

CI report:

• dc16a7a UNKNOWN
• d4c48a7 UNKNOWN
• 176eb56 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[wombatu-kun]

This omits the third behavior the flag gates: attaching a logical type to a bare long (long -> local-timestamp-millis/micros), the logical-type-loss repair the PR description calls a primary motivation. A user whose 0.x table stored the column as bare long would not learn from "precision-only evolution between these logical types" that this flag applies. Suggest documenting the long -> local-timestamp attach case explicitly.

[yihua]

long -> local-timestamp is backward compatible for readers. The logical type local-timestamp is now supported on master, so we can keep this part out for simplicity.

[voonhous]

Took a pass. Logic looks correct: the gated transitions are exactly the same-zone precision relabels plus long-to-local-timestamp, cross-zone stays rejected, and shouldPromoteType keeps the gate off so canonicalization can't undo a repair. A few things:

1. Main one: this relabels the logical type but never rescales the stored longs. That's right for the 0.x corruption you're targeting, but if someone enables it on a healthy timestamp-micros table and an incoming batch declares millis, every value silently reads 1000x off. The doc only describes what it permits - can we add a clear warning that values aren't rescaled, that it's a one-time migration aid, and that reads of existing files depend on the fix(ingest): Repair affected logical timestamp milli tables #14161 read repair?

2. Related: the gate is bidirectional, but the read-side repair only recovers toward millis (and long-to-local). The reverse directions (toward micros) have no read-side recovery, so old files keep the stale label while new ones don't, giving mixed-precision reads. Since the corruption is always "values are really millis," do we need the micros directions at all? Might be cleaner to only allow micros-to-millis, local-micros-to-local-millis, and long-to-local.

3. Test gap: nothing asserts that with the gate open, cross-zone (UTC vs local) is still rejected - that's the key invariant to pin. Worth an assertThrows in testReconcileSchemaTimestampPrecisionEvolution.

Left a couple smaller things as inline notes on the diff.

[voonhous]

Inlined the specifics from my summary comment so they're easy to find in the diff.

[voonhous]

This relabels the logical type but never rescales the stored longs. Worth spelling that out in the doc: values aren't converted, it's a one-time migration aid, and enabling it on a healthy timestamp-micros table where an incoming batch declares millis will silently read every value 1000x off. Also worth noting correct reads of existing files depend on the #14161 read repair.

[voonhous]

The gate is bidirectional, but the read-side repair only recovers toward millis (and long-to-local). The reverse (toward micros) has no read-side recovery, so old files keep the stale label and you get mixed-precision reads. Since the corruption is always "values are really millis," could we drop the micros directions and only allow micros-to-millis, local-micros-to-local-millis, and long-to-local? A one-line "relabel only, longs not rescaled" comment here would help too.

[voonhous]

Could we add a gate-open assertThrows that cross-zone (UTC vs local) is still rejected, e.g. timestamp-micros vs local-timestamp-micros? That's the key invariant of the feature and nothing pins it right now.

[voonhous]

Two small things here: (a) the description says pre-existing overloads were kept as delegates, but this get(schema, caseSensitive, ...), isTypeUpdateAllow, and reconcileSchema actually changed signature in place - no real breakage since they're all internal.schema.* with no external callers, just worth fixing the wording. (b) the 1-arg get(schema) just below stays hardcoded false, so explicit ALTER TABLE ... ALTER COLUMN TYPE won't honor the flag. Intentional? Fine either way, just undocumented.

[voonhous]

Two more correctness things after a closer look, both about the fix being narrower/less durable than it first looks:

1. Durability across engines. This corrects the table schema, but existing parquet files still carry the old timestamp-micros label with millis values - only Hudi's own reader applies the fix(ingest): Repair affected logical timestamp milli tables #14161 repair. External readers (Trino/Presto, Athena, BigQuery external, Spark-native parquet) will misread those files until they're physically rewritten. So the fix is only durable for non-Hudi engines after the affected base files get rewritten via clustering/compaction under the corrected schema. Worth calling out in the doc, with "rewrite the files" as the second migration step.

2. The flag is ignored on some write paths. Tracing deduceWriterSchema: it's honored on the internal-schema reconcile path (schema-on-read / saveInternalSchema), the MOR merge handles, and the Spark default path only when setNullForMissingColumns=true. It's not consulted for MERGE INTO writes or when hoodie.avro.schema.validate=true (both take the else branch in deduceWriterSchemaWithoutReconcile, which doesn't reconcile), nor on the legacy reconcileSchemasLegacy path (uses shouldPromoteType, hardcoded false). Consistent with the PR's scope, but a user who sets the flag and writes via MERGE INTO will see no effect and no error. Might be worth documenting which paths honor it, and whether MERGE INTO needs covering - the current tests only hit the sync path.

Lower priority, didn't verify: column stats / data-skipping bounds for the relabeled column were computed under the old logical type - worth a sanity check that pruning stays correct after the relabel.