[fix](temp-table) Fix CTAS/DROP for temporary tables and unmute 5 P0 cases - #67529
Merged
Conversation
…cases ## Problem Five P0 regression cases were muted on TeamCity. Three of them fail on every master build (they only look green on branch-4.1 because the suite is skipped there), and two are genuinely flaky: | case | failures / last 300 runs | |---|---| | nereids_rules_p0/pkfk/eliminate_inner | 251 | | compaction/test_vertical_compaction_agg_state | 254 | | temp_table_p0/test_temp_table | 250 | | load_p0/routine_load/test_routine_load | 32 | | query_p0/cache/sql_cache_object_type | 10 / 103 | ## Fixes **FE: CREATE TEMPORARY TABLE ... AS SELECT wrongly rejected** A temporary table is stored under `<sessionId>#TEMP#<name>`, but the CTAS existence probe added in apache#66112 (`CreateTableCommand.targetTableExists`) looked up the bare name. Any normal table sharing that name made the statement fail with `Table 'x' already exists`, even though the table it would create does not exist. The probe now mangles the name the same way `InternalCatalog.createTable` does. This is what `temp_table_p0/test_temp_table` hits at line 456. **FE: DROP TEMPORARY TABLE IF EXISTS ignored IF EXISTS** Name resolution falls back to the normal table when the session owns no temporary one, so `table != null` and the earlier `ifExists` branch is skipped; the `mustTemporary` guard then raised `Unknown table` regardless. `IF EXISTS` is now honored there, and the normal table is still never dropped by a `DROP TEMPORARY TABLE`. **Cases and baselines** - `eliminate_inner`: baseline was stale since apache#65264 added `shapeInfo()` overrides to `Cast`, `IsNull` and `Not`, which now keep the table qualifier. Regenerated the 8 affected shape lines. - `test_vertical_compaction_agg_state`: the first assertion compared a literal `collect_set_merge` ordering, but `collect_set` is backed by a `flat_hash_set` whose iteration order is unspecified. Wrapped in `array_sort`, matching the two sibling assertions in the same suite. - `test_routine_load`: the `load_to_single_tablet` section only waited for the job to leave `NEED_SCHEDULE` (i.e. to be scheduled), not for any batch to be committed, and its baseline recorded the empty table that race produced. Added the same data-visibility wait the other nine sections use, and filled in the expected rows. - `sql_cache_object_type`: asserted that a cache entry survived. The FE map holds soft values under a bounded size and the rows live in the BE result cache, so neither is guaranteed to persist. Re-primes the cache instead; the assertion that each setting is served its own result is unchanged. ## Testing Ran on a local cluster; each FE fix was verified with a negative control (revert, rebuild, confirm the new case fails, restore, confirm it passes): - new `test_temp_table_ctas_name_conflict` and `test_drop_temporary_table` - `eliminate_inner`, `test_vertical_compaction_agg_state`, `sql_cache_object_type`, `test_routine_load` (all 11 sections) - `ddl_p0/test_drop_view_nereids`, `ddl_p0/test_truncate_table` for the shared `dropTable` path `temp_table_p0/test_temp_table` cannot be completed locally: it needs S3 credentials for a backup repository at line 213, well before the CTAS this fixes. It still needs a CI run to confirm nothing else is behind line 456. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman
requested review from
924060929,
CalvinKirs,
englefly,
morrySnow and
starocean999
as code owners
September 4, 2026 07:22
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
morrySnow
previously approved these changes
Sep 4, 2026
Contributor
TPC-H: Total hot run time: 16728 ms |
… and fix 2 muted nonConcurrent cases ## Problem The NonConcurrent Regression pipeline (build apache#31074) reports one new failure and six muted cases. Three of the muted ones are stale -- green for the last 40 runs -- and can simply be unmuted. The other three: | case | failures / last runs | |---|---| | job_p0/streaming_job/test_streaming_job_schedule_task_error | 1 / 60 | | compaction/test_mow_compact_multi_segments | 39 / 40 | | query_p0/runtime_filter/rf_bucket_pruning | 90 / 300 | ## Fixes **FE: a streaming job can wedge in RUNNING with no task** `StreamingJobSchedulerTask.handlePendingState()` hands the new task to the scheduler in `createStreamingTask()` and only then writes RUNNING. By that point the scheduler thread may already have failed the task and moved the job to PAUSED (`StreamingTaskScheduler.scheduleTasks`), and the unconditional RUNNING write drops that. Nothing recovers from the resulting state: `handleRunningState()` never creates a task for a TVF source -- `processTimeoutTasks()` returns immediately for anything that is not a `StreamingMultiTblTask` -- and the auto resume handler only runs while PAUSED. The job then holds a canceled task and a failure reason while reporting RUNNING, and makes no progress at all. That is what the build recorded: status RUNNING, canceled task count 2, error `{"code":"INTERNAL_ERR","msg":"debug point StreamingJob.scheduleTask.exception"}`, and five minutes of silence. PENDING -> RUNNING is now a guarded transition (`StreamingInsertJob.updateJobStatusIfCurrent`), so a status another thread wrote survives. It also stops a concurrent DROP/STOP JOB from being revived, which used to raise `IllegalArgumentException`. **Case: test_mow_compact_multi_segments pinned only the cloud policy** The suite runs in both deployments but enabled only `CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets.set_input_rowsets`. On a storage-compute-coupled BE the input rowsets are therefore chosen by size and score: `[2-2]` alone is 263KB (below `cumulative_size_based_compaction_lower_size_mbytes`) with score 4 (below `min_cumulative_compaction_num_singleton_deltas`), so the first compaction is refused with `[E-2000]_input_rowsets is empty`, and the second one merges `[2-2]` with `[3-3]` into `[2-3]`. Only then, one rowset short, does `assertTrue(rowsets.size() >= 3)` fire. The local `SizeBasedCumulativeCompactionPolicy` debug point is now enabled alongside the cloud one, matching `fault_injection_p0/test_ordered_compaction_num_seg_rows`. The two post-compaction segment-count checks also passed `enableAssert = false`, so the wait loop's timeout exit was silent -- that is what hid the first compaction's failure for two months. They now assert. **Case: rf_bucket_pruning did not pin runtime_filter_max_in_num** `nonConcurrent/conf/fe.conf` sets `use_fuzzy_session_variable=true`, and one of the four branches in `SessionVariable.initFuzzyModeVariables()` sets `runtime_filter_max_in_num = 0`. Bucket pruning inverts the IN set and bails out at `hybrid_set->size() > max_in_num`, which a single-value filter already satisfies against 0 -- so no bucket is pruned and the counter stays 0. The suite pinned every other relevant session variable but this one; its sibling `rf_partition_pruning` has pinned it at 1024 since it was written, for the same reason. ## Testing - `StreamingInsertJobStatusTransitionTest` (new, 3 cases): the guarded transition promotes PENDING, and leaves PAUSED and STOPPED alone. - `test_mow_compact_multi_segments` on a local single FE + single BE cluster: both compactions now produce a one-segment rowset, all three rowsets survive, and the closing `delete_bitmap_count == 1` holds -- the part that had never been reached in this deployment. - `rf_bucket_pruning` with `set global runtime_filter_max_in_num = 0`: passes with the pin, and without it reproduces the CI failure verbatim (`single-column HASH distribution should be pruned`, same line). - `test_streaming_job_schedule_task_error` could not be completed locally: it loads from S3 and the available credentials are rejected by the bucket. It needs a CI run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
TPC-DS: Total hot run time: 82322 ms |
Contributor
ClickBench: Total hot run time: 14.62 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 17096 ms |
Contributor
TPC-DS: Total hot run time: 82390 ms |
Contributor
ClickBench: Total hot run time: 14.73 s |
CalvinKirs
approved these changes
Sep 4, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
924060929
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Five P0 regression cases are muted on TeamCity. Three of them fail on every master build — they only look green on
branch-4.1because the suite is skipped there (if (true) { return }at the top oftest_temp_table.groovy), so the mute is hiding persistent master failures rather than flakiness. The other two are genuinely flaky.nereids_rules_p0/pkfk/eliminate_innercompaction/test_vertical_compaction_agg_statetemp_table_p0/test_temp_tableload_p0/routine_load/test_routine_loadquery_p0/cache/sql_cache_object_typeTwo of them turned out to be real FE bugs.
1.
CREATE TEMPORARY TABLE ... AS SELECTwrongly rejectedA temporary table is created under
<sessionId>#TEMP#<name>, but the CTAS existence probe added in #66112 (CreateTableCommand.targetTableExists) looked up the bare name:So any normal table sharing that name makes the statement fail with
Table 'x' already exists, even though the table it would create does not exist. This is user-visible, not just a test problem:The probe now mangles the name exactly the way
InternalCatalog.createTabledoes. PlainCREATE TEMPORARY TABLE t (...)was never affected, because that path goes straight toEnv.createTable, which mangles internally. This is the failuretemp_table_p0/test_temp_tablehits at line 456.2.
DROP TEMPORARY TABLE IF EXISTSignoredIF EXISTSDatabase.getTableNullableresolves the temporary table first and falls back to the normal table, so when a session owns no temporary table of that name,table != nulland the earlierifExistsbranch is skipped. ThemustTemporaryguard then raisedUnknown tableunconditionally:IF EXISTSis now honored there. WithoutIF EXISTSthe statement still reportsUnknown table, and aDROP TEMPORARY TABLEstill never drops the normal table.Case and baseline fixes
eliminate_inner— baseline went stale on 2026-07-16 when [fix](Nereids) Add shapeInfo() override in AggregateFunction to preserve table qualifier in EXPLAIN output #65264 addedshapeInfo()overrides toCast,IsNullandNot, which now preserve the table qualifier (cast(f as ...)→cast(fkt_not_null.f as ...)).branch-4.1has no such override, which is why it stayed green there. Regenerated the 8 affected shape lines.test_vertical_compaction_agg_state— the first assertion compared a literalcollect_set_mergeordering, butcollect_setis backed byflat_hash_set, whose iteration order is unspecified. Wrapped it inarray_sort, matching the two sibling assertions already in the same suite. The set contents were never wrong, only their order.test_routine_load— theload_to_single_tabletsection waited only for the job to leaveNEED_SCHEDULE(i.e. to be scheduled), not for a batch to be committed, and its baseline recorded the empty table that race produced. It now uses the same data-visibility wait as the other nine sections, and the baseline holds the rows that actually load. 9 of the 14 most recent failures of this suite were exactly this tag.sql_cache_object_type— asserted that a cache entry survived. The FE map holds soft values under a bounded size (Config.sql_cache_manage_num) and the rows live in the BE result cache, so neither is guaranteed to persist. It re-primes the cache instead; the assertion that eachreturn_object_data_as_binarysetting is served its own result is unchanged.Release note
Fix
CREATE TEMPORARY TABLE ... AS SELECTfailing with "Table already exists" when a normal table of the same name exists, andDROP TEMPORARY TABLE IF EXISTSraising "Unknown table" in the same situation.Check List
temp_table_p0/test_temp_table_ctas_name_conflictandtemp_table_p0/test_drop_temporary_tableeliminate_inner,test_vertical_compaction_agg_state,sql_cache_object_type,test_routine_load(all 11 sections), plusddl_p0/test_drop_view_nereidsandddl_p0/test_truncate_tablefor the shareddropTablepathNote for reviewers
temp_table_p0/test_temp_tablecannot be completed locally: it creates an S3 backup repository at line 213, long before the CTAS this fixes, and that needs credentials I do not have. A CI run is needed to confirm nothing else sits behind line 456 — note that #65090 previously commented out ashow table statusassertion in that suite instead of fixing it, soSHOW TABLE STATUSstill does not list the current session's temporary tables. That is left untouched here.Happy to split this into separate PRs (FE fixes / baseline updates) if preferred.
🤖 Generated with Claude Code