Skip to content

rotate: add an optional random rotation feature in INT8/INT4 quantization method#483

Merged
zzlin237 merged 84 commits into
alibaba:mainfrom
zzlin237:rotate-int8
Jun 26, 2026
Merged

rotate: add an optional random rotation feature in INT8/INT4 quantization method#483
zzlin237 merged 84 commits into
alibaba:mainfrom
zzlin237:rotate-int8

Conversation

@zzlin237

@zzlin237 zzlin237 commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

New Feature: Added an optional rotation parameter to INT8 quantization, now supporting HNSW, Flat, IVF, and Vamana indexes.
Background: The quantization error in INT8 is directly proportional to the data range (max - min). By applying a random rotation, we can distribute the variance evenly across all vector dimensions. This helps reduce the overall range in certain cases, thereby improving INT8 quantization accuracy.
Benefits: Recall is improved while maintaining the same QPS. Using the cohere-1m dataset as an example:

  1. HNSW: QPS [14,431], Recall [0.9285] -> QPS [14,452], Recall [0.9397] (m=15, ef_construction=180, ef=180)
  2. Flat: Recall [0.9695] -> Recall [0.9881]
  3. Vamana: Recall [0.9576] -> Recall [0.9734]

Known Issues: We encountered issues during IVF testing and discovered pre-existing bugs in the IVF index:

  1. The nprobe parameter at the Python layer is currently ineffective; the index defaults to a fixed scan_ratio=0.1.
  2. When combining IVF with INT8, most points are assigned to a very small number of clusters (e.g., 70% of the data falls into a single cluster), which significantly degrades overall performance.

The test results for INT4 are as follows:

  1. HNSW: Recall [0.2114] -> Recall [0.7117]
  2. Flat: Recall [0.248] -> Recall [0.8146]
  3. Vamana: Recall [0.2415] -> Recall [0.771]

@CLAassistant

CLAassistant commented Jun 9, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@feihongxu0824

Copy link
Copy Markdown
Collaborator
  1. pr title需要能够说明这个pr的作用和目的;2. description里面需要注明一些背景、问题、收益等信息

zzlin237 added 2 commits June 24, 2026 12:28
…class with FhtRotator/MatrixRotator inheritance
…class with FhtRotator/MatrixRotator inheritance
Comment thread tools/core/local_builder.cc
Comment thread .github/workflows/nightly_coverage.yml
Comment thread CMakeLists.txt Outdated
Comment thread src/binding/python/CMakeLists.txt Outdated
Comment thread python/zvec/model/param/__init__.pyi Outdated
Comment thread python/zvec/model/param/__init__.pyi Outdated
Comment thread src/core/quantizer/cosine_reformer.cc Outdated
Comment thread src/core/quantizer/cosine_converter.cc Outdated
Comment thread src/core/quantizer/cosine_converter.cc Outdated
Comment thread src/core/quantizer/rotator/rotator.cc
Comment thread src/core/quantizer/rotator/fht_rotator.cc Outdated
Comment thread src/ailego/math/fht_sse.cc Outdated
Comment thread src/core/quantizer/rotator/rotator.cc Outdated
Comment thread src/core/quantizer/cosine_reformer.cc Outdated
Comment thread src/core/interface/indexes/ivf_index.cc Outdated
Comment thread src/core/interface/indexes/ivf_index.cc Outdated
@zzlin237 zzlin237 changed the title rotate: add an optional random rotation feature in INT8 quantization method rotate: add an optional random rotation feature in INT8/INT4 quantization method Jun 26, 2026
zzlin237 and others added 5 commits June 26, 2026 14:05
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants