Skip to content

[META] OpenSearch Pluggable Indexing #20876

@mgodwan

Description

@mgodwan

Please describe the end goal of this project

We are planning to introduce components related to enable pluggable indexing #20644, and multiple composable data-formats in OpenSearch. This issue is to track the work related to the feature.

Proposals for various workstreams

Abstractions in Core

Composable Data Format Plugin #20909

  • Multiplexing implementation
  • Add support for Row Id Management for coherence across data formats
  • Failure handling across data formats during indexing
  • Refresh orchestration
  • JMH benchmark for merge flow

Commons for Data formats

  • Writer Pool Management to handle concurrency

Parquet Data Format Plugin

Lucene Data Format Plugin [TBD]

Remote Store Integration

Supporting References

#20644

Issues

#18416

Related component

Indexing

Metadata

Metadata

Assignees

No one assigned

    Labels

    MetaMeta issue, not directly linked to a PRlucene

    Type

    No type

    Projects

    Status

    New

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions