Add supporting of timestamp type in statistics for hive 3 and 4 versions by anton-kutuzov · Pull Request #28296 · trinodb/trino

anton-kutuzov · 2026-02-14T05:45:13Z

Description

Added support for the timestamp type in statistics calculation.
To maintain compatibility with Hive statistics, the computed min and max values are truncated to seconds during write by dividing by MICROSECONDS_PER_SECOND, using floor for min and ceil for max to avoid losing millisecond precision. When reading, values are converted back to Trino format by multiplying by MICROSECONDS_PER_SECOND.
This solution supports both Hive Metastore version 3 and version 4.

Related issues: Add support for lowValue and highValue statistics for TIMESTAMP type in engine #5860, Added support for the timestamp type for statistics in trino-hive connector #26338
Fixes: Add support for Hive min/max statistics for TIMESTAMP statistics in Hive #5859
Fixes Repeated ANALYZE fails on Hive metastore 4.0.x #26214
Fixes Hive4 Partition Overwrite fails for non-managed tables #28330

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

ebyhr

Just skimmed.

ebyhr · 2026-02-16T00:24:50Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/HiveQueryRunner.java

                    .addCoordinatorProperty("http-server.http.port", "8080")
-                    .addHiveProperty("hive.metastore", "glue")
-                    .addHiveProperty("hive.metastore.glue.default-warehouse-dir", "local:///glue")
+                    .addHiveProperty("hive.metastore.uri", "thrift://hive-metastore:9083")


Revert unrelated change. This is HiveGlueQueryRunnerMain.

Thanks. Fixed

ebyhr · 2026-02-16T00:29:50Z

testing/trino-tests/src/test/java/io/trino/tests/tpch/TpchQueryRunner.java

+
+        queryRunner.installPlugin(new HivePlugin());
+        queryRunner.createCatalog("hive", "hive");
+


Revert. When a new plugin is required in the test class, please install it via AbstractTestQueryFramework#createQueryRunner method.

Thanks. Fixed

ebyhr · 2026-02-16T00:30:11Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseTestHiveOnDataLake.java

+                .setCatalogSessionProperty("hive", "insert_existing_partitions_behavior", "APPEND")
+                .setCatalogSessionProperty("hive", "collect_column_statistics_on_write", "true")
+                .build();
+        String testTable = HIVE_TEST_SCHEMA + ".test_tabble_" + randomNameSuffix();


Fix typo > tabble

Thanks. Fixed

findepi · 2026-02-19T10:22:24Z

Related issues: #5860, #26214, #26338

Is this PR "fixes #26214"?

findepi · 2026-02-19T10:24:35Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftMetastoreUtil.java

+        if (timestamp == null) {
+            return OptionalLong.empty();
+        }
+        return OptionalLong.of(LocalDateTime.ofEpochSecond(timestamp.getSecondsSinceEpoch(), 0, ZoneOffset.UTC).toInstant(ZoneOffset.UTC).toEpochMilli() * MICROSECONDS_PER_MILLISECOND);


Could this be done simpler?

Suggested change

return OptionalLong.of(LocalDateTime.ofEpochSecond(timestamp.getSecondsSinceEpoch(), 0, ZoneOffset.UTC).toInstant(ZoneOffset.UTC).toEpochMilli() * MICROSECONDS_PER_MILLISECOND);

return OptionalLong.of(timestamp.getSecondsSinceEpoch() * MICROSECONDS_PER_SECOND);

Thanks. Fixed

findepi · 2026-02-19T10:26:43Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftMetastoreUtil.java

+        LongColumnStatsData data = new LongColumnStatsData();
+        statistics.getIntegerStatistics().ifPresent(timestampStatistics -> {
+            timestampStatistics.getMin().ifPresent(value -> data.setLowValue(floorDiv(value, MICROSECONDS_PER_SECOND)));
+            timestampStatistics.getMax().ifPresent(value -> data.setHighValue(ceilDiv(value, MICROSECONDS_PER_SECOND)));


❤️

Although, stats are only estimates, so ceil div is not strictly necessary.
In fact, it may cause some planning issues (a partition with single value will have min and max different).
My intuition is that, the code as currently written is a better way, so no changes requested.

cc @raunaqmorarka @sopel39

I used floorDiv and ceilDiv because I needed them to implement partition filtering based on statistics in the Hive catalog. With this implementation, I may include two extra partitions, which is not critical, but I won’t miss any.

findepi · 2026-02-19T10:27:29Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftMetastoreUtil.java

            return ImmutableSet.of(MIN_VALUE, MAX_VALUE, NUMBER_OF_DISTINCT_VALUES, NUMBER_OF_NON_NULL_VALUES);
        }
        if (type instanceof TimestampType || type instanceof TimestampWithTimeZoneType) {
-            // TODO (https://github.com/trinodb/trino/issues/5859) Add support for timestamp MIN_VALUE, MAX_VALUE


Please add "fixes #5859" to the PR description.

findepi · 2026-02-19T10:28:26Z

...trino-hive/src/main/java/io/trino/plugin/hive/statistics/AbstractHiveStatisticsProvider.java

+    private static Optional<DoubleRange> createTimestampRange(IntegerStatistics statistics)
+    {
+        if (statistics.getMin().isPresent() && statistics.getMax().isPresent()) {
+            return Optional.of(new DoubleRange(statistics.getMin().getAsLong(), statistics.getMax().getAsLong()));


Shouldn't this multiply by MICROSECONDS_PER_SECOND?

No, we multiply one time, when read from metastore.

findepi · 2026-02-19T10:32:43Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/util/Statistics.java

+        if (block.isNull(0)) {
+            return OptionalLong.empty();
+        }
+        if (block instanceof Fixed12Block) {


Don't inspect the block type. Use the type API to get the value

if (((TimestampType) type).isShort()) { return OptionalLong.of(type.getLong(block, 0)); } else { return OptionalLong.of(((LongTimestamp) type.getObject(block, 0)).getEpochMicros()); }

Yes, I had that option, but I was getting an error class io.trino.spi.block.Fixed12Block cannot be cast to class io.trino.spi.block.LongArrayBlock, so I added a check on Fixed12Block

findepi · 2026-02-19T10:33:15Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/util/Statistics.java

        return Optional.of(LocalDate.ofEpochDay(days));
    }

+    private static OptionalLong getTimestampValue(Type type, Block block)


It doesn't read the timestamp value, only part of it. Call it getTimestampEpochMicro

findepi · 2026-02-19T10:33:41Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/util/Statistics.java

            result.setDateStatistics(new DateStatistics(getDateValue(type, min), getDateValue(type, max)));
        }
+        else if (type instanceof TimestampType) {
+            result.setIntegerStatistics(new IntegerStatistics(getTimestampValue(type, min), getTimestampValue(type, max)));


Don't you need to divide by MICROSECONDS_PER_SECOND?

No, because we do it only before save to metastore
https://github.com/trinodb/trino/blob/master/plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastore.java#L570

findepi · 2026-02-19T10:34:40Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftMetastoreUtil.java

        if (type instanceof TimestampType || type instanceof TimestampWithTimeZoneType) {
-            // TODO (https://github.com/trinodb/trino/issues/5859) Add support for timestamp MIN_VALUE, MAX_VALUE
-            return ImmutableSet.of(NUMBER_OF_DISTINCT_VALUES, NUMBER_OF_NON_NULL_VALUES);
+            return ImmutableSet.of(MIN_VALUE, MAX_VALUE, NUMBER_OF_DISTINCT_VALUES, NUMBER_OF_NON_NULL_VALUES);


This PR should change behavior for type instanceof TimestampType case, but not the TimestampWithTimeZoneType case. Split the if

findepi · 2026-02-19T10:35:59Z

...g/trino-product-tests/src/main/java/io/trino/tests/product/hive/TestHiveTableStatistics.java

                row("c_decimal", null, 2.0, 0.0, null, "345.0", "346.0"),
                row("c_decimal_w_params", null, 2.0, 0.0, null, "345.671", "345.678"),
-                row("c_timestamp", null, 2.0, 0.0, null, null, null),
+                row("c_timestamp", null, 2.0, 0.0, null, "2015-05-10 12:15:31.000", "2015-05-10 12:15:36.000"),


Was this table analyzed by hive?

(How do we verify hive timestamp stats being expressed in seconds in epoch?)

If I understood the question correctly(does hive compute statistics for timestamp in seconds), then yes, it also compute it in seconds: https://github.com/apache/hive/blob/46f27831/ql/src/java/org/apache/hadoop/hive/ql/stats/ColumnStatisticsObjTranslator.java#L303-L310

anton-kutuzov · 2026-02-22T15:50:12Z

Related issues: #5860, #26214, #26338

Is this PR "fixes #26214"?

Yes, the test was added for this case

chenjian2664

Skimmed about the test part

chenjian2664 · 2026-03-13T05:13:01Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseTestHiveOnDataLake.java

+                ")");
+        assertUpdate(session, "INSERT INTO " + testTable + " VALUES(1, TIMESTAMP '2025-09-19 10:00:00', TIMESTAMP '2025-09-19 10:10:00', 2, 3.5, 1, 10.0, 2.0, 12.0)", 1);
+        assertUpdate(session, "ANALYZE " + testTable, 1);
+        assertUpdate(session, "ANALYZE " + testTable, 1);


Is it necessary to analyze twice?

chenjian2664 · 2026-03-13T05:13:20Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseTestHiveOnDataLake.java

    }

+    @Test
+    public void testSupportTimestampStatisticsForHive3andHive4()


"ForHive3andHive4" suffix looks redundant

chenjian2664 · 2026-03-13T05:13:58Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseTestHiveOnDataLake.java

+                "    Tip_amount DOUBLE,\n" +
+                "    Total_amount DOUBLE\n" +
+                ")");
+        assertUpdate(session, "INSERT INTO " + testTable + " VALUES(1, TIMESTAMP '2025-09-19 10:00:00', TIMESTAMP '2025-09-19 10:10:00', 2, 3.5, 1, 10.0, 2.0, 12.0)", 1);


nit: let two timestamp value different(though there is only 1 row)

chenjian2664 · 2026-03-13T05:16:01Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseTestHiveOnDataLake.java

+                .setCatalogSessionProperty("hive", "insert_existing_partitions_behavior", "APPEND")
+                .setCatalogSessionProperty("hive", "collect_column_statistics_on_write", "true")
+                .build();
+        String testTable = HIVE_TEST_SCHEMA + ".test_table_" + randomNameSuffix();


test_table_ -> test_timestamp_statistics

chenjian2664 · 2026-03-13T05:17:35Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseTestHiveOnDataLake.java

+                                    "WITH (partitioned_by=ARRAY['dt'])");
+
+        assertUpdate(session, "INSERT INTO " + testTable + "(history_batch_timestamp, dt) VALUES (CAST(CURRENT_TIMESTAMP AS TIMESTAMP), '2026-02-03')", 1);
+        assertUpdate(session, "INSERT INTO " + testTable + "(history_batch_timestamp, dt) VALUES (CAST(CURRENT_TIMESTAMP AS TIMESTAMP), '2026-02-03')", 1);


Is the second insert operation for testing the overwrite behavior? if so, can we add a test i,e for value count about it?

cla-bot bot added the cla-signed label Feb 14, 2026

github-actions bot added the hive Hive connector label Feb 14, 2026

anton-kutuzov mentioned this pull request Feb 14, 2026

Added support for the timestamp type for statistics in trino-hive connector #26338

Closed

anton-kutuzov requested review from dain, ebyhr, findepi and losipiuk February 14, 2026 06:34

ebyhr reviewed Feb 16, 2026

View reviewed changes

anton-kutuzov force-pushed the add_suporting_tymestamp_in_statistics branch 2 times, most recently from 3789d9e to eebfb46 Compare February 17, 2026 21:34

anton-kutuzov mentioned this pull request Feb 17, 2026

Hive4 Partition Overwrite fails for non-managed tables #28330

Open

findepi reviewed Feb 19, 2026

View reviewed changes

Add supporting of timestamp type in statistics for hive 3 and 4 versions

613db4e

anton-kutuzov force-pushed the add_suporting_tymestamp_in_statistics branch from eebfb46 to 613db4e Compare February 23, 2026 10:28

anton-kutuzov requested a review from findepi February 26, 2026 21:15

findepi requested review from ebyhr, raunaqmorarka and sopel39 March 11, 2026 18:11

ebyhr requested a review from chenjian2664 March 13, 2026 02:36

chenjian2664 reviewed Mar 13, 2026

View reviewed changes


		queryRunner.installPlugin(new HivePlugin());
		queryRunner.createCatalog("hive", "hive");

	return OptionalLong.of(LocalDateTime.ofEpochSecond(timestamp.getSecondsSinceEpoch(), 0, ZoneOffset.UTC).toInstant(ZoneOffset.UTC).toEpochMilli() * MICROSECONDS_PER_MILLISECOND);
	return OptionalLong.of(timestamp.getSecondsSinceEpoch() * MICROSECONDS_PER_SECOND);

Conversation

anton-kutuzov commented Feb 14, 2026 • edited by findepi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release notes

Uh oh!

ebyhr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

findepi commented Feb 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anton-kutuzov commented Feb 22, 2026

Uh oh!

chenjian2664 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

anton-kutuzov commented Feb 14, 2026 •

edited by findepi

Loading