Skip to content

README.md

Latest commit

 

History

History
87 lines (48 loc) · 2.87 KB

README.md

File metadata and controls

87 lines (48 loc) · 2.87 KB

Dataprep Microservice

The Dataprep Microservice aims to preprocess the data from various sources (either structured or unstructured data) to text data, and convert the text data to embedding vectors then store them in the database.

Install Requirements

apt-get update
apt-get install libreoffice

Use LVM (Large Vision Model) for Summarizing Image Data

Occasionally unstructured data will contain image data, to convert the image data to the text data, LVM can be used to summarize the image. To leverage LVM, please refer to this readme to start the LVM microservice first and then set the below environment variable, before starting any dataprep microservice.

export SUMMARIZE_IMAGE_VIA_LVM=1

Dataprep Microservice with Redis

For details, please refer to this readme

Dataprep Microservice with Milvus

For details, please refer to this readme

Dataprep Microservice with Qdrant

For details, please refer to this readme

Dataprep Microservice with Pinecone

For details, please refer to this readme

Dataprep Microservice with PGVector

For details, please refer to this readme

Dataprep Microservice with VDMS

For details, please refer to this readme

Dataprep Microservice with Multimodal

For details, please refer to this readme

Dataprep Microservice with ElasticSearch

For details, please refer to this readme

Dataprep Microservice with OpenSearch

For details, please refer to this readme

Dataprep Microservice with neo4j

For details, please refer to this readme

Dataprep Microservice for financial domain data

For details, please refer to this readme

Dataprep Microservice with MariaDB Vector

For details, please refer to this readme

Dataprep Microservice with ArangoDB

For details, please refer to this readme

Running in the air gapped environment

The following steps are common for running the dataprep microservice in an air gapped environment (a.k.a. environment with no internet access), for all DB backends.

  1. Download the following models, e.g. huggingface-cli download --cache-dir <model data directory> <model>
  • microsoft/table-transformer-structure-recognition
  • timm/resnet18.a1_in1k
  • unstructuredio/yolo_x_layout
  1. launch the dataprep microservice with the following settings:
  • mount the model data directory as the /data directory within the dataprep container
  • set environment variable HF_HUB_OFFLINE to 1 when launching the dataprep microservice

e.g. docker run -d -v <model data directory>:/data -e HF_HUB_OFFLINE=1 ... ...