disaggregation

Prefill/Decode Disaggregation Deployment Guide

This guide demonstrates how to deploy the Simulator in a Kubernetes cluster using a separated Prefill and Decode (P/D) architecture.

The routing-sidecar runs alongside the Decode service and acts as a reverse proxy: it receives client requests, forwards the prefill phase to a dedicated Prefill service (based on the x-prefiller-host-port header), and then handles the decode phase locally.

This is a standalone simulation setup, intended for testing and validating P/D workflows without requiring the llm-d-inference-scheduler. It uses standard Kubernetes Services for internal communication between components.

Quick Start

1. Deploy the Application Apply the provided manifest (e.g., vllm-sim-pd.yaml) to your Kubernetes cluster:

kubectl apply -f vllm-sim-pd.yaml

This manifest defines two Deployments (vllm-sim-p for Prefill, vllm-sim-d for Decode) and two Services for internal and external communication.

2. Verify Pods Are Ready Check that all pods are running:

kubectl get pods -l 'llm-d.ai/role in (prefill,decode)'

Expected output:

NAME                          READY   STATUS    RESTARTS   AGE
vllm-sim-d-685b57d694-d6qxg   3/3     Running   0          12m
vllm-sim-p-7b768565d9-79j97   2/2     Running   0          12m

Send a Disaggregated Request Using kubectl port-forward

To access both the Decode services from your local machine, use kubectl port-forward to forward their ports to your localhost.

Forward the Decode Service Port

Open a terminal and run:

kubectl port-forward svc/vllm-sim-d-service 8000:8000

This command forwards port 8000 from the vllm-sim-d-service to your local machine's port 8000.

Test the Disaggregated Flow

Now, send a request to the forwarded Decode service port with the necessary headers:

curl -v http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-prefiller-host-port: vllm-sim-p-service:8000" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "Hello from P/D architecture!"}],
    "max_tokens": 32
  }'

Critical Header:

x-prefiller-host-port: vllm-sim-p-service:8000

This header must be provided by the client in standalone mode. It tells the routing-sidecar where to send the prefill request. The value should be a Kubernetes Service name + port (or any resolvable host:port reachable from the sidecar pod).

In production deployments using llm-d-inference-scheduler, this header is typically injected automatically by the scheduler or gateway—but in this standalone simulator, the client must set it explicitly.

Realistic Config

This example already configures non-zero latency parameters to reflect real-world P/D disaggregation behavior:

- "--prefill-time-per-token=200ms"   # ~200ms per input token for prefill computation
- "--prefill-time-std-dev=3ms"       # ±3ms jitter to simulate system noise

Parameter meanings:

• prefill-time-per-token: Average time (e.g., 100ms) to process each prompt token during the prefill phase. Accepts Go duration strings (e.g., 100ms, 1s). Higher values emphasize the cost of large prompts.
• prefill-time-std-dev: Standard deviation for prefill latency (e.g., 3ms), introducing realistic variation across requests. Accepts Go duration strings.