Skip to content

replace-dead-node.md

Latest commit

 

History

History
296 lines (203 loc) · 12 KB

replace-dead-node.md

File metadata and controls

296 lines (203 loc) · 12 KB
Raw

Replace a Dead Vector Store Node

The replace operation provisions a new Vector Store node in the same availability zone as the dead node, waits for it to fetch all data from ScyllaDB and rebuild its indexes, then decommissions the old node. This is an add-then-remove strategy that ensures zero downtime for vector search queries (assuming other VS nodes in the cluster remain healthy).

The duration of this procedure depends on the data volume and index complexity. Index rebuilding can take anywhere from a few minutes to over an hour.

Prerequisites

Healthy ScyllaDB Cluster

The ScyllaDB cluster must be healthy and reachable. The new Vector Store node needs to connect to a ScyllaDB node via CQL to fetch data and build indexes.

Identify the Dead Node

Collect the following information from the dead Vector Store node (or from your infrastructure records):

  • Private IP address of the dead node
  • Availability zone / rack the node was running in
  • Instance type (e.g., i3.xlarge)
  • Vector Store version (the version of the vector-store binary or AMI)

Collect Configuration from a Healthy Vector Store Node

If you have other healthy Vector Store nodes in the cluster, SSH into one and collect:

# View the current configuration
cat /home/ubuntu/.env

This will show the environment variables used by the Vector Store service, including the CQL username, password file path, ScyllaDB connection URI, and TLS settings. You will replicate this configuration (with appropriate IP changes) on the replacement node.

Collect CQL Credentials

You need the CQL username and password that Vector Store uses to connect to ScyllaDB. On a healthy VS node:

# Username is in the .env file
grep VECTOR_STORE_SCYLLADB_USERNAME /home/ubuntu/.env

# Password is stored on a tmpfs mount
sudo cat /mnt/vector_store/.vector_store_password.txt

Procedure

Step 1: Provision a New Instance

Launch a new VM using the same Vector Store AMI/image as the dead node:

  • Same instance type as the dead node
  • Same availability zone (rack) as the dead node
  • Same VPC and subnet as the rest of the cluster
  • Enable root volume encryption

Note: The instance name convention used by ScyllaDB Cloud is Scylla-Cloud-<CLUSTER_ID>-<DC_NAME>-VectorSearch-<NODE_ID> on AWS, and scylla-cloud-<cluster_id>-vectorsearch-<node_id>-<checksum> on GCP. You can use any naming convention that fits your environment.

Step 2: Configure Firewall Rules

The following network access is required:

Direction From To Port Purpose
Inbound ScyllaDB nodes New VS node TCP 6080 ScyllaDB connects to VS for vector search
Inbound Prometheus / monitoring New VS node TCP 6080 Metrics scraping (/metrics endpoint)
Outbound New VS node A ScyllaDB node in the same rack TCP 9042 CQL native transport
Outbound New VS node A ScyllaDB node in the same rack TCP 9142 CQL native transport (SSL), if encryption is enabled

On AWS, update the relevant security groups. On GCP, update the firewall rules.

Step 3: Configure the Vector Store Node

SSH into the new node and perform the following steps.

3a. Create a tmpfs Mount for the Password File

The CQL password must be stored in memory only (tmpfs) for security:

sudo mkdir -p /mnt/vector_store
sudo mount -t tmpfs -o size=10m tmpfs /mnt/vector_store

3b. Write the CQL Password File

# Replace <PASSWORD> with the actual CQL password collected earlier
echo -n '<PASSWORD>' | sudo tee /mnt/vector_store/.vector_store_password.txt > /dev/null
sudo chown ubuntu:ubuntu /mnt/vector_store/.vector_store_password.txt
sudo chmod 600 /mnt/vector_store/.vector_store_password.txt

3c. Create the .env Configuration File

Create /home/ubuntu/.env with the following content. Replace the placeholder values with your actual configuration:

Without client-to-node encryption:

cat > /home/ubuntu/.env << 'EOF'
export VECTOR_STORE_SCYLLADB_URI=<SCYLLA_PRIVATE_IP>:9042
export VECTOR_STORE_URI=0.0.0.0:6080
export VECTOR_STORE_SCYLLADB_USERNAME=<CQL_USERNAME>
export VECTOR_STORE_SCYLLADB_PASSWORD_FILE=/mnt/vector_store/.vector_store_password.txt
EOF
chmod 600 /home/ubuntu/.env
chown ubuntu:ubuntu /home/ubuntu/.env

With client-to-node encryption (SSL CQL):

cat > /home/ubuntu/.env << 'EOF'
export VECTOR_STORE_SCYLLADB_URI=<SCYLLA_PRIVATE_IP>:9142
export VECTOR_STORE_URI=0.0.0.0:6080
export VECTOR_STORE_SCYLLADB_USERNAME=<CQL_USERNAME>
export VECTOR_STORE_SCYLLADB_PASSWORD_FILE=/mnt/vector_store/.vector_store_password.txt
export VECTOR_STORE_SCYLLADB_CERTIFICATE_FILE=/home/ubuntu/vector_store_client.pem
EOF
chmod 600 /home/ubuntu/.env
chown ubuntu:ubuntu /home/ubuntu/.env

If your deployment also uses HTTPS for the Vector Store API (ScyllaDB-to-VS communication), add these lines:

export VECTOR_STORE_TLS_CERT_PATH=/home/ubuntu/server_cert.pem
export VECTOR_STORE_TLS_KEY_PATH=/home/ubuntu/server_key.pem

And provision the corresponding TLS certificate and key files (signed by the same CA used for client-to-node encryption).

Important: VECTOR_STORE_SCYLLADB_URI should point to the private IP of a single ScyllaDB node in the same rack/AZ as the Vector Store node. The Vector Store connects to one ScyllaDB node, not the entire cluster.

3d. Provision TLS Certificates (if applicable)

If client-to-node encryption is enabled, copy the CA certificate to the new VS node:

# Copy the CA certificate used for client-to-node encryption
scp ca-cert.pem ubuntu@<NEW_VS_NODE_IP>:/home/ubuntu/vector_store_client.pem
ssh ubuntu@<NEW_VS_NODE_IP> 'chmod 600 /home/ubuntu/vector_store_client.pem'

If HTTPS is enabled for the VS API, generate and provision a server certificate signed by the same CA:

# Generate a server certificate for the VS node's IP
# (use your CA to sign it, with the VS node's private IP as the CN/SAN)
scp server_cert.pem ubuntu@<NEW_VS_NODE_IP>:/home/ubuntu/server_cert.pem
scp server_key.pem ubuntu@<NEW_VS_NODE_IP>:/home/ubuntu/server_key.pem
ssh ubuntu@<NEW_VS_NODE_IP> 'chmod 600 /home/ubuntu/server_cert.pem /home/ubuntu/server_key.pem'

If HTTPS is enabled, you must also ensure the CA certificate (truststore) is configured on each ScyllaDB node so that ScyllaDB can verify the VS node's TLS certificate. If this is already set up for the existing VS nodes, no action is needed. If not, upload the CA cert to each ScyllaDB node and configure vector_store_encryption_options in scylla.yaml — this requires a ScyllaDB restart.

Step 4: Start the Vector Store Service

sudo systemctl start vector-store

Verify the service is running:

sudo systemctl status vector-store

The service should show as active (running). If it fails to start, check the logs:

sudo journalctl -u vector-store -f

Step 5: Wait for the Node to Become Ready

After starting, the Vector Store node fetches data from ScyllaDB and builds its indexes. Poll the status endpoint until it returns SERVING:

# Without TLS:
watch -n 30 'curl -s http://localhost:6080/api/v1/status'

# With TLS:
watch -n 30 'curl -sk https://localhost:6080/api/v1/status'

The endpoint returns a plain-text string:

  • While initializing: a status other than SERVING (or a connection error)
  • When ready: SERVING

This can take up to 1 hour or more depending on the data volume and number of indexes. Do not proceed to the next step until the status is SERVING.

Step 6: Update ScyllaDB Configuration

Once the new VS node is ready, update the ScyllaDB nodes so they know about the replacement.

On each ScyllaDB node in the same datacenter, update scylla.yaml with the new Vector Store URIs:

  • vector_store_primary_uri — Comma-separated list of Vector Store node URIs in the same rack as the ScyllaDB node. Example: http://10.0.1.5:6080 (or https://... if TLS is enabled).
  • vector_store_secondary_uri — Comma-separated list of Vector Store node URIs in other racks within the same datacenter.

For example, if you have 3 racks (A, B, C) with one VS node each, a ScyllaDB node in rack A would have:

vector_store_primary_uri: "http://10.0.1.5:6080"         # VS node in rack A
vector_store_secondary_uri: "http://10.0.2.5:6080,http://10.0.3.5:6080"  # VS nodes in racks B and C

Replace the dead node's IP with the new node's IP in these URIs.

After updating scylla.yaml, trigger a configuration reload. Setting vector_store_primary_uri triggers an automatic reload in ScyllaDB — alternatively, you can reload explicitly. No ScyllaDB restart is required for URI changes (a restart is only required if you are setting up vector_store_encryption_options for the first time).

Step 7: Decommission the Dead Node

  1. Terminate the old VM instance (or shut it down and delete it).
  2. Remove the dead node's IP from any security group or firewall rules that referenced it specifically.
  3. Remove from monitoring — delete the old node's scrape target from Prometheus configuration.

Step 8: Update Monitoring

Add the new Vector Store node to your Prometheus scrape configuration:

- targets:
  - '<NEW_VS_NODE_IP>:6080'

Reload Prometheus and verify metrics are being collected from the new node.

Verification

After completing the procedure, verify the replacement was successful:

  1. Vector Store status — The new node's status endpoint returns SERVING:

    curl -s http://<NEW_VS_NODE_IP>:6080/api/v1/status
# Expected: SERVING

  2. Vector search queries — Run a vector search query against your ScyllaDB cluster and verify results are correct.

  3. Monitoring — Check that the new VS node appears in Prometheus/Grafana with UP = 1, and that IndexCount and other metrics match expectations.

  4. Old node removed — Confirm the dead node's IP no longer appears in any scylla.yaml configuration, security group rules, or monitoring targets.

Operational Notes

Hot Configuration Reload

The Vector Store service supports hot configuration reload via SIGHUP. If you need to change the .env file after the service is running (e.g., to update the ScyllaDB IP or credentials), edit the file and then:

sudo systemctl kill -s SIGHUP vector-store

ScyllaDB Node Failure

Each Vector Store node is configured to connect to a single ScyllaDB node in the same rack. If that ScyllaDB node fails, the Vector Store node's configuration must be updated to point to a different ScyllaDB node in the same rack:

  1. Edit /home/ubuntu/.env and change the VECTOR_STORE_SCYLLADB_URI to the IP of another healthy ScyllaDB node in the same rack.
  2. Send SIGHUP: sudo systemctl kill -s SIGHUP vector-store

Port Reference

Port Service Description
6080 Vector Store API, status, and Prometheus metrics endpoint
9042 ScyllaDB CQL native transport
9142 ScyllaDB CQL native transport (SSL)

File Locations on a Vector Store Node

Path Description
/home/ubuntu/.env Vector Store configuration (environment variables)
/mnt/vector_store/.vector_store_password.txt CQL password (on tmpfs, in-memory only)
/home/ubuntu/vector_store_client.pem CA certificate for CQL SSL (if encryption enabled)
/home/ubuntu/server_cert.pem VS API TLS certificate (if HTTPS enabled)
/home/ubuntu/server_key.pem VS API TLS private key (if HTTPS enabled)

Environment Variables Reference

Variable Required Description
VECTOR_STORE_SCYLLADB_URI Yes ScyllaDB CQL endpoint (<IP>:<PORT>)
VECTOR_STORE_URI Yes Listen address (0.0.0.0:6080)
VECTOR_STORE_SCYLLADB_USERNAME Yes CQL username
VECTOR_STORE_SCYLLADB_PASSWORD_FILE Yes Path to CQL password file
VECTOR_STORE_SCYLLADB_CERTIFICATE_FILE No CA cert for CQL SSL connections
VECTOR_STORE_TLS_CERT_PATH No TLS cert for HTTPS API
VECTOR_STORE_TLS_KEY_PATH No TLS key for HTTPS API