epoch migration command by elliVM · Pull Request #768 · teragrep/pth_10

elliVM · 2025-12-15T10:47:36Z

Description

Implement support for a command to update missing epoch values of S3 object metadata in the archive SQL. Works by running the archive datasource in an epoch migration mode, where epoch value is fetched to the returned schemas _time column.

Updates the pth_03 version to 9.4.0 with teragrep exec migration epoch command support
Adds jooq library for dynamic SQL building
Update the SQL logtime metadata of the S3 objects using epoch migration mode from the archiver
Epoch migraiton mode of PTH_06 provides a calculated epoch value of the first even of the S3 in the _time column. In the _raw column there is a JSON format metadata about the event, this can be used to see how the event was parsed and what was the source of the _time column epoch.

Testing

Included unit and dpl tests

General

I have checked that my test files and functions have meaningful names.
I have checked that each test tests only a single behavior.
I have done happy tests.
I have tested only my own code.
I have tested at least all public methods.

Assertions

I have checked that my tests use assertions and not runtime overhead.
I have checked that my tests end in assertions.
I have checked that there is no comparison statements in assertions.
I have checked that assertions are in tests and not in helper functions.
I have checked that assertions for iterables are outside of for loops and both sides of the iteration blocks.
I have checked that assertions are not tested inside consumers.

Testing Data

I have tested algorithms and anything else with the possibility of unbound growth.
I have checked that all testing data is local and fully replaceable or reproducible or both.
I have checked that all test files are standalone.
I have checked that all test-specific fake objects and classes are in the test directory.
I have checked that my tests do not contain anything related to customers, infrastructure or users.
I have checked that my tests do not contain non-generic information.
I have checked that my tests do not do external requests and are not privately or publicly routable.

Statements

I have checked that my tests do not use throws for exceptions.
I have checked that my tests do not use try-catch statements.
I have checked that my tests do not use if-else statements.

Java

I have checked that my tests for Java uses JUnit library.
I have checked that my tests for Java uses JUnit utilities for parameters.

Other

I have only tested public behavior and not private implementation details.
I have checked that my tests are not (partially) commented out.
I have checked that hand-crafted variables in assertions are used accordingly.
I have tested Object Equality.
I have checked that I do not have any manual tests or I have a valid reason for them and I have explained it in the PR description.

Code Quality

I have checked that my code follows metrics set in Procedure: Class Metrics.
I have checked that my code follows metrics set in Procedure: Method Metrics.
I have checked that my code follows metrics set in Procedure: Object Quality.
I have checked that my code does not have any NULL values.
I have checked my code does not contain FIXME or TODO comments.

elliVM · 2026-02-05T10:03:21Z

+        final String partitionString = row.getString(row.fieldIndex("partition"));
+        final long id = Long.parseLong(partitionString);
+
+        if (!metadata.isSyslog() && LOGGER.isInfoEnabled()) {


Currently non syslog rows are updated to SQL since the epoch migration mode provides the path extracted value in the _time column. This is logged with metadata from the row.

Is this acceptable or should we do more here?

@kortemik what do you think is this fine as it is?

Main point being that should these cases be added to the corrupted logfiles in the archive schema

It must be extracted properly from json in the _raw field. Also the object format should be updated in the same go

Tiihott

Object equality tests should be added for some new classes.
Please rebase.

elliVM · 2026-02-10T10:14:31Z

rebased

Tiihott

Tests fail when run via maven.

elliVM · 2026-02-12T11:15:50Z

rebased

elliVM · 2026-02-24T13:16:29Z

rebased and switched to use new connection pool objects

elliVM · 2026-02-25T13:23:22Z

tests keep failing in actions with connection pool initialization errors

elliVM · 2026-02-26T08:38:44Z

working build now

Tiihott

All tests pass and changes look fine, LGTM.

elliVM · 2026-02-27T12:02:20Z

Doing testing in QA

elliVM · 2026-02-27T12:42:01Z

Issues in QA

Caused by: java.sql.SQLException: No suitable driver executor class path does not have JDBC driver available. FIX: added hikariConfig.setDriverClassName("org.mariadb.jdbc.Driver") to maybe have it load in the executor. if not the executors need to be provided with the driver some other way I think.
archive.url/bloomdb url uses bloomdb database FIX: changed to use archive db credentials

Tiihott

Tests pass and new changes look ok. A new test run in QA should be done to check if the credential and JDBC driver fixes resolve the issues found in QA.

elliVM · 2026-03-02T08:24:53Z

I think the JDBC driver must be added to the spark executor jars

…nTest

…qualified names in the before each SQL setup

…gleton before each in EpochMigrationStepTest

…lean up logic and naming

…ations in EpochMigrationStepTest

…y to try both to parse the json to a result EventMetadata

…ll check _time before parsing JSON

elliVM · 2026-04-09T11:19:02Z

rebased

kortemik · 2026-04-10T08:04:24Z

+
+    @Override
+    public EventMetadata get() {
+        final List<EventMetadata> validEvents = formats


functional programming API is not preferred. please change to object oriented one.

kortemik · 2026-04-10T08:06:34Z

+ */
+package com.teragrep.pth_10.steps.teragrep.migrate;
+
+public interface EventMetadata {


preferred name is ArchiveObjectMetadata

kortemik · 2026-04-10T08:07:30Z

+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+final class EventMetadataFactory implements Supplier<EventMetadata> {


I think this should be just a decorator to EventMetadata which takes in at ctor the list of possible formats and the EventMetadata. when changing to such design the decorator can directly implement the EventMetadata and return the format via format() by running the code which we have here in get()

kortemik · 2026-04-10T08:19:39Z

+
+import java.util.function.Supplier;
+
+public interface CandidateFormat extends Supplier<EventMetadata> {


this interace is unnecessary, please return Stub / illegal argument exception from format() if not matching

kortemik · 2026-04-10T08:24:18Z

+import java.io.StringReader;
+import java.util.Objects;
+
+public final class ParsedJson {


this is a json parsER object. instead it should be implementing EventMetadata directly

kortemik · 2026-04-10T08:26:52Z

+
+    private long resolveFromDatabase(final String objectFormat) {
+        try {
+            // plain SQL since jooq can't insert into a generic table


i am unsure what this comment is about? table is defined on L104 here?

kortemik · 2026-04-10T08:32:25Z

+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+final class ResolvedObjectFormats {


this is unnecessary and error-prone. One should do this as a transaction via inline or as a stored procedure so that database takes care of the proper id selection at the insert.

START TRANSACTION; INSERT INTO format (name) VALUES ('rfc5424-syslog') ON DUPLICATE KEY UPDATE id = LAST_INSERT_ID(id); UPDATE foo (format_id) VALUES (LAST_INSERT_ID()) WHERE id=123; COMMIT;

this uses an upsert pattern.

kortemik · 2026-04-10T08:33:50Z

+
+import jakarta.json.JsonObject;
+
+final class SyslogEvent implements EventMetadata {


is this SyslogArchiveObject? Event is a naming mistake to avoid.

elliVM · 2026-04-27T07:59:59Z

changing to a temp table solution and using a stored procedure to update from temp table

elliVM self-assigned this Dec 15, 2025

elliVM linked an issue Dec 15, 2025 that may be closed by this pull request

Implement a spark job to update missing logfile epoch metadataa #714

Open

elliVM changed the title ~~epoch migration step~~ epoch migration command Dec 15, 2025

elliVM force-pushed the epoch-migration-step branch from 0434986 to 205dfd3 Compare January 28, 2026 09:18

elliVM force-pushed the epoch-migration-step branch 2 times, most recently from 8b6df08 to ccbb108 Compare February 5, 2026 09:19

elliVM commented Feb 5, 2026

View reviewed changes

elliVM marked this pull request as ready for review February 5, 2026 10:07

elliVM requested a review from Tiihott February 5, 2026 10:38

Tiihott requested changes Feb 9, 2026

View reviewed changes

elliVM force-pushed the epoch-migration-step branch from bb9d02d to 5b57d2d Compare February 10, 2026 10:13

elliVM requested a review from Tiihott February 11, 2026 10:51

Tiihott requested changes Feb 12, 2026

View reviewed changes

Comment thread ...a/com/teragrep/pth_10/steps/teragrep/migrate/EpochMigrationForeachPartitionFunctionTest.java

elliVM force-pushed the epoch-migration-step branch from 17823f6 to 9b20d0b Compare February 12, 2026 11:14

elliVM marked this pull request as draft February 18, 2026 11:02

elliVM force-pushed the epoch-migration-step branch from dbc66fb to 481d8b9 Compare February 24, 2026 13:03

elliVM marked this pull request as ready for review February 24, 2026 13:33

elliVM requested a review from Tiihott February 26, 2026 08:38

Tiihott previously approved these changes Feb 27, 2026

View reviewed changes

kortemik requested a review from Tiihott February 27, 2026 15:10

elliVM dismissed Tiihott’s stale review via 493f6eb February 28, 2026 20:37

Tiihott previously approved these changes Mar 2, 2026

View reviewed changes

elliVM added 24 commits April 9, 2026 14:18

use connection pool objects

8a88561

use shared connection source in EpochMigrationForeachPartitionFunctio…

ea5e62d

…nTest

LazyConnectionSource implements Serializable

2aacc20

set h2 db close delay to -1 to not close when connection closes, use …

28f91cc

…qualified names in the before each SQL setup

use unique names for remaining h2 databasees, reset ConnectionPoolSin…

8076b80

…gleton before each in EpochMigrationStepTest

apply spotless

6a98319

close connection after call() finishes sending it back to the pool, c…

fa1260f

…lean up logic and naming

set mariadb jbdc driver in hikariConfig

40c4294

use archive user, password and url config items instead of bloomdb

eb6923f

fix tests after credential changes

e687fca

update config item names for EpochMigrationStepTest

5b30a5e

add ParseJson Object, using jakarta, add mariadb java client

34e3fbd

use ParseJson object in EventMetadataFromString

b0ca693

improve ParsedJsonTest assertions

8b9b132

add CustomResultStep to only show completion message to user

ade5a69

add various timestamps to test data, use qualified names for SQL oper…

71733d7

…ations in EpochMigrationStepTest

use pth_06 version 4.7.1

0ec3dfb

add ResolvedObjectFormats object and tests

3d6fce5

remove equals and hashcode from TeragrepEpochMigrationStep

4b41f90

add support for object_format updates during epoch migration

cdb7d27

add CandidateFormat and SyslogFormat and NonSyslogFormat and a factor…

88f5c36

…y to try both to parse the json to a result EventMetadata

update pth_06 to version 4.8.0, update epochMigrationData.json and nu…

429502a

…ll check _time before parsing JSON

update spotbugs-exclude-filter.xml

8d072d6

use 'unknown' as format for non rfc5424 events

eaa68f9

elliVM force-pushed the epoch-migration-step branch from 3891cc9 to eaa68f9 Compare April 9, 2026 11:18

kortemik requested changes Apr 10, 2026

View reviewed changes

elliVM added 2 commits April 14, 2026 14:12

refactoring of json handling logic - wip

e22c298

finalize refactoring of json formats

8b1ab23


		import java.util.function.Supplier;

		public interface CandidateFormat extends Supplier<EventMetadata> {


		import jakarta.json.JsonObject;

		final class SyslogEvent implements EventMetadata {

Conversation

elliVM commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

General

Assertions

Testing Data

Statements

Java

Other

Code Quality

Uh oh!

elliVM Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kortemik Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tiihott left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elliVM commented Feb 10, 2026

Uh oh!

Tiihott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elliVM commented Feb 12, 2026

Uh oh!

elliVM commented Feb 24, 2026

Uh oh!

elliVM commented Feb 25, 2026

Uh oh!

elliVM commented Feb 26, 2026

Uh oh!

Tiihott left a comment

Choose a reason for hiding this comment

Uh oh!

elliVM commented Feb 27, 2026

Uh oh!

elliVM commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tiihott left a comment

Choose a reason for hiding this comment

Uh oh!

elliVM commented Mar 2, 2026

Uh oh!

elliVM commented Apr 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kortemik Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

elliVM commented Dec 15, 2025 •

edited

Loading

elliVM Feb 5, 2026 •

edited

Loading

kortemik Mar 17, 2026 •

edited

Loading

Tiihott left a comment •

edited

Loading

elliVM commented Feb 27, 2026 •

edited

Loading

kortemik Apr 10, 2026 •

edited

Loading