TiKV Metadata

[Matts966]

Description

Related Issue

Versions

• Vald Version: v1.7.17
• Go Version: v1.26.0
• Rust Version: v1.93.1
• Docker Version: v29.2.1
• Kubernetes Version: v1.35.1
• Helm Version: v4.1.1
• NGT Version: v2.5.1
• Faiss Version: v1.13.2

Checklist

• I have read the CONTRIBUTING document and completed our CLA agreement.
• I have checked open Pull Requests for the similar feature or fixes?

Special notes for your reviewer

Summary by CodeRabbit

• New Features

  • Meta Gateway: metadata-aware vector operations (search/insert/update/upsert/remove/object retrieval) and REST/gRPC endpoints
  • TiDB/TiKV support: optional cluster manifests and Helm values to enable persistent metadata store
  • Client and e2e support: client-side metadata interfaces and an e2e scenario for metadata CRUD

• Documentation

  • API docs and OpenAPI/Swagger specs updated to include metadata fields and new meta endpoints

[cloudflare-workers-and-pages]

Deploying vald with    Cloudflare Pages

Latest commit:	de82d0e
Status:	✅  Deploy successful!
Preview URL:	https://3fd98ee3.vald.pages.dev
Branch Preview URL:	https://feature-metadata-tikv.vald.pages.dev

View logs

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a new meta gateway feature: proto/grpc REST mappings for "WithMetadata" CRUD APIs, a TiKV-backed metadata client, metadata-aware LB handlers/clients, Helm/TiDB/TiKV deployment assets, Docker/Make/CI updates, and payload/schema/docs changes to carry per-vector metadata.

Changes

Cohort / File(s)	Summary
Meta Gateway CLI & Docker cmd/gateway/meta/main.go, cmd/gateway/meta/doc.go, cmd/gateway/meta/sample.yaml, dockers/gateway/meta/Dockerfile	New meta gateway entrypoint, sample config, and multi-stage Dockerfile.
Meta Gateway Config & Package Layout pkg/gateway/meta/config/config.go, pkg/gateway/meta/config/doc.go, pkg/gateway/meta/handler/doc.go, pkg/gateway/meta/handler/grpc/doc.go	Config loader and package scaffolding for meta gateway with metadata store config.
Helm Chart: Meta Gateway charts/vald/templates/gateway/meta/*, charts/vald/templates/gateway/ing.yaml, charts/vald/templates/gateway/lb/networkpolicy.yaml, charts/vald/values.yaml, charts/vald/values.schema.json, .github/helm/values/values-meta.yaml	Helm templates and values/schema for deploying gateway-meta (Deployment/DaemonSet/HPA/PDB/Svc/NetworkPolicy/ConfigMap/ServiceAccount/priorityclass) and new values for tidbCluster.
Helm Chart: TiDB/TiKV charts/vald/templates/kvs/tikv/*, charts/vald/templates/kvs/tikv/...	Added TiDBCluster, PD, TiKV, discovery, services, ConfigMaps, RBAC and related templates to provision TiDB/TiKV components.
Payload / API Schema & Docs apis/proto/v1/payload/payload.proto, apis/proto/v1/vald/meta.proto, apis/docs/v1/*, apis/docs/v1/payload.md.tmpl	Added bytes metadata fields to Object.Vector/Object.Distance/Update.TimestampRequest; new meta.proto defining WithMetadata services; updated API docs/templates.
gRPC & Swagger Artifacts apis/grpc/v1/vald/vald.go, apis/swagger/v1/vald/meta.swagger.json, other swagger updates/removals	Added ServerWithMetadata/UnimplementedValdServerWithMetadata, client method constants; new Swagger spec for meta endpoints; removed older tikv swagger artifacts.
Internal Clients: Meta internal/client/v1/client/meta/*, internal/client/v1/client/vald/vald.go	New MetadataClient interfaces and TiKV rawKV ManagedMetadataClient implementation; extended internal Vald client with *WithMetadata methods (unary/stream/multi).
LB Gateway Handlers pkg/gateway/lb/handler/grpc/handler.go, pkg/gateway/lb/handler/grpc/option.go, pkg/gateway/lb/handler/grpc/*_meta.go	Added metadataClient field, option setter, and metadata-aware gRPC handlers (search/insert/update/upsert/remove/object) with concurrent metadata fetch/persist and error aggregation.
Removed TiKV proto artifacts apis/proto/v1/tikv/*, apis/swagger/v1/tikv/*	Removed previously checked-in TiKV protos and related swagger files (kvrpcpb, tikvpb, errorpb, README, LICENSE).
Build / Make / CI Makefile, Makefile.d/*, .github/workflows/*, .github/actions/*	Added meta image targets, build/package artifacts, CI workflow for gateway-meta image, e2e meta scenario, and adjusted image detection/waiting actions to include gateway-meta.
Go Modules & Tooling go.mod, hack/go.mod.default, example/client/go.mod*	Bumped Go version to 1.26.1 and updated dependency pins including tikv client and related libraries.
Rust Generated Bindings rust/libs/proto/src/*	Regenerated Rust proto bindings to include meta-related modules (added meta v1, updated vald/tikv-related outputs).
E2E Test Asset tests/v2/e2e/assets/unary_crud_meta.yaml	Added e2e asset for unary CRUD meta scenario.

Sequence Diagram

sequenceDiagram
    participant Client
    participant MetaGateway as Meta Gateway
    participant LBGateway as LB Gateway
    participant TiKV as TiKV Metadata Store
    participant Agent

    Client->>MetaGateway: SearchWithMetadata(request with vector)
    activate MetaGateway
    MetaGateway->>LBGateway: Search(vector)
    activate LBGateway
    LBGateway->>Agent: Search(vector)
    activate Agent
    Agent-->>LBGateway: results
    deactivate Agent
    LBGateway-->>MetaGateway: results
    deactivate LBGateway

    par fetch metadata concurrently
        MetaGateway->>TiKV: Get(id_1)
        MetaGateway->>TiKV: Get(id_2)
        MetaGateway->>TiKV: Get(id_N)
        TiKV-->>MetaGateway: metadata_1
        TiKV-->>MetaGateway: metadata_2
        TiKV-->>MetaGateway: metadata_N
    end

    MetaGateway->>Client: results enriched with metadata
    deactivate MetaGateway

Loading

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly Related PRs

• Add metadata API #3279 — overlapping metadata API additions (payload metadata fields and meta.proto) at protobuf level.
• add TiKV to helm chart #3426 — TiKV/TiDB Helm chart and tidbCluster values additions matching the chart templates here.
• add internal kvs #3281 — TiKV client/error work that relates to the rawKV metadata client implementation.

Suggested Reviewers

• kmrmt
• kpango

"A metadata gateway unfurls its map,
TiKV keeps secrets in a tiny wrap.
Insert, search, update — enriched and set,
Helm deploy hums, the cluster's preset.
Small bytes of meaning, in vectors we wrap."

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/metadata/tikv

[vdaas-ci]

[CHATOPS:HELP] ChatOps commands.

• 🙆‍♀️ /approve - approve
• 🍱 /format - format codes and add licenses
• ✅ /gen-test - generate test codes
• 🏷️ /label - add labels
• 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

[Matts966]

/format

[Matts966]

/format

[vdaas-ci]

[FORMAT] Updating license headers and formatting go codes triggered by Matts966.

[vdaas-ci]

[FORMAT] Updating license headers and formatting go codes triggered by Matts966.

[vdaas-ci]

[FORMAT] Nothing to format.