Introduce MinIO Object Store event based dataprep and retriever using Milvus and LanceDB by dilverse · Pull Request #846 · opea-project/GenAIComps

dilverse · 2024-11-03T06:08:16Z

Description

MinIO is a high performance Object Storage solution which is fully S3 compatible and using which its easier to build highly scalable applications. In this PR adding support for the following

In Dataprep storing all the uploaded documents directly to MinIO
Modularize the dataprep process
Once the document is uploaded to MinIO bucket an event notification is sent to the dataprep service to do the chunking and store the chunked metatdata as msgpack into MinIO Bucket
once the msgpack chunked metadata is stored in MInIO bucket another notification is send to do the embeddings process and store the chunks, metadata and embeddings to to vector database like Milvus and LanceDB.
Also adding support for MinIO LanceDB based retriever

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

List the newly introduced 3rd party dependency if exists.

for more information, see https://pre-commit.ci

mkbhanda · 2024-11-05T08:11:42Z

@lvliang-intel would you kindly review.

mkbhanda

Looks good to me. @chensuyue do we need any tests?

mkbhanda · 2024-11-05T08:40:26Z

+    "parent": ""
+  },
+  {
+    "name": "uploaded_file_2.txt",


would be good to change one of these to a file path/url. What is the name of a file when it is path? id == name?

chensuyue · 2024-11-06T06:34:50Z

Yes, we need some one test for each microservice.
test_retrievers_minio_lancedb_langchain.sh
test_dataprep_minio_lancedb_langchain.sh
test_dataprep_minio_milvus_langchain.sh

chensuyue · 2024-11-06T06:35:26Z

Please refer to https://github.com/opea-project/GenAIComps/tree/main/tests/dataprep

eero-t

Below are few minor comments on the docs and Python code.

As to Dockerfiles, see: opea-project/GenAIExamples#225

Co-authored-by: Eero Tamminen <[email protected]>

for more information, see https://pre-commit.ci

lvliang-intel · 2024-11-12T07:48:58Z

@dilverse,
Please update comps/dataprep/minio/milvus/langchain/Dockerfile, comps/dataprep/minio/lancedb/langchain/Dockerfile and comps/retrievers/minio/lancedb/langchain/Dockerfile to .github/workflows/docker/compose/dataprep-compose.yaml. The yaml is used for release images build.

dilverse · 2024-12-27T01:21:04Z

@dilverse, Please update comps/dataprep/minio/milvus/langchain/Dockerfile, comps/dataprep/minio/lancedb/langchain/Dockerfile and comps/retrievers/minio/lancedb/langchain/Dockerfile to .github/workflows/docker/compose/dataprep-compose.yaml. The yaml is used for release images build.

@lvliang-intel Updated the github workflow

xiguiw · 2025-02-25T08:06:50Z

@dilverse
GenAIComps are refactored in v1.2.
It does not create a micro-service for each DB anymore.
Instead, only one micro-service is needed.
It simplifies the integration of DB.

Please refer to latest code change.

xiguiw · 2025-03-26T01:48:25Z

@dilverse

Remind
Are you still working on this?

dilverse · 2025-04-14T16:53:30Z

@dilverse

Remind Are you still working on this?

Sounds good I will take a look at it and update the PR @xiguiw

xiguiw · 2025-05-30T01:34:12Z

@dilverse

There is conflict, please help to resolve the conflicts.

dilverse and others added 9 commits October 28, 2024 05:59

Initial version of MinIO event based dataprep for Milvus

9950f6f

Merge branch 'opea-project:main' into main

7d76da5

Initial version of MinIO event based dataprep for Milvus

a299782

Update to latest MinIO Image

2d1a561

Add LnaceDB and MinIO event based dataprep support

92e99c6

Add MinIO utils file that parses event notifications from MinIO

d6a5a7e

Merge branch 'opea-project:main' into main

e497710

Update README.md files

2b13f5d

Add MinIO LanceDB retriever support

78064a8

dilverse requested a review from lvliang-intel as a code owner November 3, 2024 06:08

pre-commit-ci Bot and others added 7 commits November 3, 2024 06:08

[pre-commit.ci] auto fixes from pre-commit.com hooks

eb8897d

for more information, see https://pre-commit.ci

Fix README.md paths

616dc63

Fix README.md for Milvus to right path

c05bdcb

[pre-commit.ci] auto fixes from pre-commit.com hooks

a65d3dc

for more information, see https://pre-commit.ci

Fix connection arguments for Milvus

80fe8f6

Remove local file volumes from docker compose

33d38f4

Merge branch 'main' into main

e83f649

mkbhanda reviewed Nov 5, 2024

View reviewed changes

chensuyue reviewed Nov 6, 2024

View reviewed changes

Comment thread comps/dataprep/minio_utils.py

ftian1 reviewed Nov 6, 2024

View reviewed changes

Comment thread comps/dataprep/minio/lancedb/langchain/Dockerfile Outdated

eero-t reviewed Nov 6, 2024

View reviewed changes

dilverse and others added 6 commits November 6, 2024 12:47

Update comps/dataprep/minio/lancedb/langchain/README.md

c4dfea3

Co-authored-by: Eero Tamminen <[email protected]>

Update comps/dataprep/minio/lancedb/langchain/README.md

263aac3

Co-authored-by: Eero Tamminen <[email protected]>

Update comps/dataprep/minio/lancedb/langchain/README.md

5238a85

Co-authored-by: Eero Tamminen <[email protected]>

Update comps/dataprep/minio/lancedb/langchain/README.md

d6142a8

Co-authored-by: Eero Tamminen <[email protected]>

Update comps/dataprep/minio/milvus/langchain/README.md

b66221e

Co-authored-by: Eero Tamminen <[email protected]>

Update comps/dataprep/minio/milvus/langchain/README.md

491cdcd

Co-authored-by: Eero Tamminen <[email protected]>

dilverse and others added 8 commits November 6, 2024 12:49

Update comps/dataprep/minio/milvus/langchain/README.md

94581a7

Co-authored-by: Eero Tamminen <[email protected]>

Update comps/dataprep/minio/milvus/langchain/README.md

535d09a

Co-authored-by: Eero Tamminen <[email protected]>

Update comps/dataprep/minio/milvus/langchain/README.md

e726c57

Co-authored-by: Eero Tamminen <[email protected]>

Update comps/dataprep/minio/milvus/langchain/README.md

fefd9ec

Co-authored-by: Eero Tamminen <[email protected]>

Remove unused code and add appropriate copyrights and minor lint fixex

1efea4d

[pre-commit.ci] auto fixes from pre-commit.com hooks

2952611

for more information, see https://pre-commit.ci

Merge branch 'main' into main

30976f1

Merge branch 'main' into main

7c41fc5

dilverse added 2 commits December 26, 2024 16:49

Merge branch 'main' into main

ce2c768

Added new docker files to github workflow

ac5413f

dilverse added 2 commits December 27, 2024 11:40

Add new lines to the workflows

fba854a

Remove trailing spaces

d526d28

xiguiw added the A3 Maintain label Mar 26, 2025

chensuyue requested review from XinyuYe-Intel, ZePan110, letonghan and lkk12014402 as code owners April 18, 2025 01:59

Conversation

dilverse commented Nov 3, 2024

Description

Issues

Type of change

Dependencies

Uh oh!

mkbhanda commented Nov 5, 2024

Uh oh!

mkbhanda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkbhanda Nov 5, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chensuyue commented Nov 6, 2024

Uh oh!

chensuyue commented Nov 6, 2024

Uh oh!

Uh oh!

eero-t left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lvliang-intel commented Nov 12, 2024

Uh oh!

dilverse commented Dec 27, 2024

Uh oh!

xiguiw commented Feb 25, 2025

Uh oh!

xiguiw commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dilverse commented Apr 14, 2025

Uh oh!

xiguiw commented May 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

xiguiw commented Mar 26, 2025 •

edited

Loading