Skip to content

Separating Index Parameter Configuration into its Own Class - #41

Draft
BlaiseMuhirwa wants to merge 12 commits into
mainfrom
index-builder
Draft

Separating Index Parameter Configuration into its Own Class#41
BlaiseMuhirwa wants to merge 12 commits into
mainfrom
index-builder

Conversation

@BlaiseMuhirwa

@BlaiseMuhirwa BlaiseMuhirwa commented Feb 24, 2024

Copy link
Copy Markdown
Owner

At a high level, this PR is just adding a IndexParameterConfig class that allows us to configure parameters that can be re-used to create different index objects. The python interface looks like this:

import flatnav
from flatnav.index import create_index, ParameterConfig
import numpy as np

parameter_config = (
    ParameterConfig(dataset_size=10000)
    .with_index_params(
        {
            "max_edges_per_node": 16,
            "ef_construction": 200,
            "num_initializations": 100,
            "num_threads": 4,
            "index_name": "hnsw_16_200"
        }
    )
    .with_reordering(["gorder", "rcm"]) 
)


index = create_index(distance_type="l2", dim=100, parameter_config=parameter_config)
index.add(data=np.random.rand(10000, 100))


# ##########################################
# # second initialization via a MTX file
# ##########################################


outdegree_table = flatnav.util.load_from_mtx_file(filename=hnsw_base_layer_filename)
index.allocate_nodes(data=train_dataset).build_graph_links(outdegree_table=outdegree_table)

I validated that this change doesn't lead to a performance regression in terms of QPS/latency.

@BlaiseMuhirwa
BlaiseMuhirwa marked this pull request as draft February 24, 2024 21:40
Comment thread tools/CMakeLists.txt
Comment on lines +4 to +5
# set(EXAMPLES construct_npy query_npy cereal_tests)
set(EXAMPLES construct_npy query_npy)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a stray comment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants