feat: implement LOAD_FILE scalar function for Document AI - #1071
Open
yanxf-git wants to merge 2 commits into
Open
feat: implement LOAD_FILE scalar function for Document AI#1071yanxf-git wants to merge 2 commits into
yanxf-git wants to merge 2 commits into
Conversation
Implements BLOB LOAD_FILE(location_name, file_name) that reads a local file via a file:// LOCATION. Registered as T_FUN_SYS_LOAD_FILE=2087. Adapted from OceanBase commit 28c25842 with seekdb-specific changes: - Use POSIX open/pread instead of ObExternalDataAccessDriver (not in seekdb) - Use 1-arg get_tenant_schema_guard (seekdb signature) - Use 2-arg get_location_schema_by_name (seekdb signature) - Drop MultimodeAlloctor (not needed for simple file read) - Use ObString::prefix_match instead of 3-arg compare Local verification: mysqltest ai_funcs/load_file.test PASS
|
|
Member
Document AI & IK Custom Dictionary ScoreDocument AI Functions Score =========================== score: 100.00 / 100 load_file: 50 / 50 ai_split_document: 50 / 50 IK Custom Dictionary Score ========================== score: 0.00 / 100 ik_custom_dict: 0 / 100 FTS Large Benchmark ScoreFTS Large Benchmark Score ========================= score: 0.00 / 100 mean_improvement: -3.34% full_score_improvement: 50.00% build_improvement: -0.08% build_ik_all_sec: baseline=35.2836, current=35.475, improvement=-0.54% build_ik_content_sec: baseline=28.3764, current=28.789, improvement=-1.45% build_beng_en_sec: baseline=14.7578, current=14.5, improvement=1.75% tokenize_improvement: -9.95% tokenize_ik_avg_ms: baseline=0.76478, current=0.8287, improvement=-8.36% tokenize_beng_avg_ms: baseline=0.42262, current=0.4714, improvement=-11.54% query_improvement: 0.00% query_cn_avg_ms: baseline=16.6628, current=16.7131, improvement=-0.30% query_beng_avg_ms: baseline=24.3042, current=24.2318, improvement=0.30% query_mixed_avg_ms: baseline=17.5593, current=17.6115, improvement=-0.30% query_limit_avg_ms: baseline=16.2334, current=16.1842, improvement=0.30% FTS Large Benchmark Report======================================== FTS Large Benchmark Report ======================================== timestamp: 2026-07-16 04:18:34 label: vldb-ci-29469436225-1 git_head: 6df1fe8 git_dirty: 0 rows: 20000 batch: 500 rounds: 3000 query_rounds: 200 samples: 3 warmup: 30 skip_load: 0 ---------------------------------------- select1_avg_ms: 0.2709 select1_stdev_ms: 0.0064 raw_load_sec: 1.546 raw_load_rows_per_sec: 12936.6 build_ik_all_sec: 35.475 build_ik_content_sec: 28.789 build_beng_en_sec: 14.500 build_total_sec: 78.778 ---------------------------------------- tokenize_ik_avg_ms: 0.8287 tokenize_ik_median_ms: 0.8309 tokenize_ik_stdev_ms: 0.0031 tokenize_beng_avg_ms: 0.4714 tokenize_beng_median_ms:0.4703 tokenize_beng_stdev_ms: 0.0039 ---------------------------------------- query_cn_hits: 8001 query_cn_avg_ms: 16.7131 query_cn_stdev_ms: 0.0246 query_beng_hits: 11000 query_beng_avg_ms: 24.2318 query_beng_stdev_ms: 0.0142 query_mixed_hits: 7332 query_mixed_avg_ms: 17.6115 query_mixed_stdev_ms: 0.0467 query_limit_hits: 20 query_limit_avg_ms: 16.1842 query_limit_stdev_ms: 0.0100 ======================================== |
Implements TABLE AI_SPLIT_DOCUMENT(content, params_json) that splits text
or markdown into chunks. Registered as T_AI_SPLIT_DOCUMENT_EXPRESSION=4920.
Supports:
- type: text / markdown (default markdown)
- by: word / sentence (default word, via ICU BreakIterator)
- max: chunks per window (default 256)
- overlap: sliding window overlap (default 0)
Output columns: CHUNK_ID, CHUNK_OFFSET, CHUNK_LENGTH, CHUNK_TEXT.
Architecture (adapted from OceanBase commit 28c25842 for seekdb):
- Core split logic in src/share/ai_split_document/ (L2 layer, passes
module_layer_check; OceanBase original placed it in deps/oblib/src/lib/
which violates seekdb's L0->L2 constraint)
- Include paths: common/json_type/ instead of lib/json_type/
- Tenant-elim adaptation: ObMemAttr("name") without MTL_ID(), drop
set_tenant_id (seekdb removed multi-tenant allocator args)
- Type names: ObONCharType/ObORawType/ObOLobType instead of ObNCharType/
ObRawType/ObLobType (seekdb uses O-prefix for Oracle types)
- ObEvalInfo.evaluated_ = true instead of set_evaluated(true)
- ICU link: libicustubdata.a -> libicudata.a (full Unicode data for
sentence boundary detection)
Local verification: mysqltest ai_funcs/ai_split_document.test PASS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements BLOB LOAD_FILE(location_name, file_name) that reads a local file via a file:// LOCATION. Registered as T_FUN_SYS_LOAD_FILE=2087.
Adapted from OceanBase commit 28c25842 with seekdb-specific changes:
Local verification: mysqltest ai_funcs/load_file.test PASS
Task Description
Solution Description
Passed Regressions
Upgrade Compatibility
Other Information
Release Note