[META] OpenSearch Pluggable Indexing

[mgodwan]

Please describe the end goal of this project

We are planning to introduce components related to enable pluggable indexing #20644, and multiple composable data-formats in OpenSearch. This issue is to track the work related to the feature.

Proposals for various workstreams

Abstractions in Core

• Introduce Lucene agnostic Interfaces (Indexer, CatalogSnapshot, FieldCapability, etc.) Add indexer interface for shard interaction with underlying engines #20675
• Introduce DataFormat Plugin and associated interfaces (DataFormat, IndexingExecutionEngine, etc.) Add indexer interface for shard interaction with underlying engines #20675
• Dataformat Registry Add DataFormatRegistry #21039
• Introduce Lucene agnostic Searcher/Reader interfaces
• Indexer implementation for data formats along with store management Add Data Format aware engine #21181
• Add Merge Handler and Committer interfaces and orchestration logic https://github.com/opensearch-project/OpenSearch/pull/
• Add Catalog Snapshot and File Manager implementation Add catalogSnapshotManager for dataformat aware engine #20982
• Add changes to support mappers to work with new interfaces Updating the FieldMappers to insert call addField of DocumentInput #21001
• Add committer implementation github.com/Add changes for introducing lucene committer in data format aware engine #21103
• Add File Deleter to work with committer File Deletions Support through IndexFileDeleter in CatalogSnapshotManager #21145
• Managing Internal/External Refreshes
• Existing ITs and UTs to wire into new implementation as well.

Composable Data Format Plugin #20909

• Multiplexing implementation
• Add support for Row Id Management for coherence across data formats
• Failure handling across data formats during indexing
• Refresh orchestration
• JMH benchmark for merge flow

Commons for Data formats

• Writer Pool Management to handle concurrency

Parquet Data Format Plugin

• Introduce Arrow based implementation to buffer incoming documents Add parquet-data-format plugin #20930
• arrow-rs based Indexing and native writer management
• JNI Bridge for leveraging arrow-rs
• Field Handlers
• Compaction in parquet add parquet merge support #21079

Lucene Data Format Plugin [TBD]

• Field Handlers
• IndexWriter integration to work with composable data formats Add Lucene engine impl for pluggable data formats #21299

Remote Store Integration

• Update Remote Store to use Catalog Snapshot github.com/Add foundational classes for storage support for multi-format data support #20943/
• Introduce Format Awareness to manage multiple data formats and paths
• Handle Snapshot, Restore, and recovery
• Lucene agnostic data Integrity handling

Supporting References

#20644

Issues

#18416

Related component

Indexing