branch-4.1: [Fix](udf) Key UDF class cache by function ID and enable cleanup in cloud mode (#67046) - #67314
Merged
Conversation
…loud mode (apache#67046) Problem Summary: UDF cache cleanup has two problems: 1. Cloud mode does not register a worker for `CLEAN_UDF_CACHE`, so `DROP FUNCTION` cannot clean the cached UDF classloader. 2. UDF caches are cleaned by function signature. If a function is dropped and recreated with the same signature, a delayed cleanup task may delete the new cache, or the recreated function may reuse the stale cache. The FE removes the function metadata first and then submits `CleanUDFCacheTask` asynchronously. It does not wait for the BE cache cleanup result. In addition, the cleanup task does not report a completion result back to the FE. Therefore, even if the task cannot be submitted or the JNI cache cleanup fails, `DROP FUNCTION` still returns success to the client. ```sql CREATE FUNCTION test_udf(INT) RETURNS INT ...; -- implementation V1 SELECT test_udf(1); -- cache V1 DROP FUNCTION test_udf(INT); -- cache cleanup fails or is delayed CREATE FUNCTION test_udf(INT) RETURNS INT ...; -- implementation V2 SELECT test_udf(1); ```` Before this PR, the last query could reuse V1's cached classloader because V1 and V2 had the same signature. A delayed cleanup task for V1 could also remove V2's cache. - Register the CLEAN_UDF_CACHE worker in cloud mode. - Use the function ID as the key for Java UDF cache lookup, insertion, and cleanup. = Fall back to signature-based cleanup when no valid function ID is provided for compatibility with older FEs. A recreated function receives a new function ID. Therefore, even if the previous function's cache is not successfully removed, the recreated function does not reuse it and can load and execute the correct implementation. A delayed cleanup task also removes only the old function's cache without affecting the recreated function.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Collaborator
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Aug 31, 2026
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick: #67046