Skip to content

[fix](fe) Use updated Routine Load CSV parser properties - #67509

Open
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix-routine-load-csv-alter-cache
Open

[fix](fe) Use updated Routine Load CSV parser properties#67509
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix-routine-load-csv-alter-cache

Conversation

@0AyanamiRei

@0AyanamiRei 0AyanamiRei commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: RoutineLoadJob kept CSV enclose and escape values in both persisted jobProperties and separate in-memory byte fields. ALTER ROUTINE LOAD updated only jobProperties, while newly planned Kafka and Kinesis tasks used the stale fields from CREATE ROUTINE LOAD.

This change makes persisted jobProperties the single job-level source of truth. RoutineLoadJob derives the parser bytes through its getters, and each new task receives a snapshot of the current values. ALTER execution, journal replay, and checkpoint loading already preserve jobProperties, so no load-description replay or checkpoint-specific repair is needed.

Release note

ALTER ROUTINE LOAD now applies updated CSV enclose and escape characters to subsequent tasks.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Unit test executed:

DORIS_THIRDPARTY=/path/to/thirdparty-with-thrift-0.24.0 ./run-fe-ut.sh --run org.apache.doris.load.routineload.KafkaRoutineLoadJobTest

Result: 12 tests passed.

The Kafka regression test truncates the table after ALTER and publishes a row that requires the new enclose and escape values while preserving the existing deterministic expected output. It was not run locally because the configured Kafka regression service is unavailable; CI will run it in the Kafka-enabled environment.

  • Behavior changed:

    • No.
    • Yes. New Routine Load tasks use CSV parser properties changed by ALTER.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Routine Load kept CSV enclose and escape values in both the persisted job properties and separate in-memory byte fields. ALTER ROUTINE LOAD updated only the persisted properties, so newly planned tasks continued using the parser configuration from CREATE ROUTINE LOAD. Make jobProperties the single job-level source of truth and derive the parser bytes when creating Kafka and Kinesis task snapshots.

### Release note

ALTER ROUTINE LOAD now applies updated CSV enclose and escape characters to subsequent tasks.

### Check List (For Author)

- Test: Unit Test / Regression test
    - Unit Test: KafkaRoutineLoadJobTest (12 tests passed)
    - Regression test: Extended test_routine_load_property; not run locally because the configured Kafka test service is unavailable
- Behavior changed: Yes. New Routine Load tasks use CSV enclose and escape values changed by ALTER.
- Does this need documentation: No
@0AyanamiRei
0AyanamiRei force-pushed the fix-routine-load-csv-alter-cache branch from db729be to 6e7bc6b Compare September 3, 2026 18:17
@0AyanamiRei 0AyanamiRei changed the title [fix](fe) Refresh Routine Load CSV parser after ALTER [fix](fe) Use updated Routine Load CSV parser properties Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants