Can I run VLLM with 5090+5070Ti for Llama 70B Q4 (needs approximately 42 GB) inference? Or do I need identical GPUs?

[jayavanth]

I just want ~42GB of GPU VRAM so I don't think I should spend ~$1000 to get 5090 which is already hard to get. Do you think mismatched GPU will work? What kind of time to first token can I expect from this setup?

According to this, I can expect 0.4s on 5090 and 0.5s on 5070Ti for Llama3

[Shyboy0499]

Short answer: with plain --tensor-parallel-size 2 it likely won't fit, because default TP assumes uniform ranks — it shards every layer evenly, so a TP group is only as good as its smallest GPU and the bigger card's extra VRAM goes unused.

Your numbers make it concrete:

• RTX 5090 = 32 GB, RTX 5070 Ti = 16 GB. Even TP across 2 ranks gives each rank ~half the weights → ~21 GB per rank for a 42 GB Q4 70B, which doesn't fit on the 16 GB 5070 Ti. Result: OOM (or a failed load) on the smaller card.
• TP also wants identical architecture + fast interconnect. 5090/5070 Ti are both Blackwell, so architecture is fine, but the group runs at the speed of the slowest rank, and TTFT is bounded by it.

Options that actually work:

1. Uneven TP (experimental): vLLM's --rank-tp-ratio lets you shard unevenly to match per-GPU VRAM, e.g. --rank-tp-ratio 2,1 puts twice the weights on the 32 GB card and 1× on the 16 GB one. It's off by default and not battle-tested — expect rough edges (introduced in PR #50735).
2. Smaller model / lighter quant: a ~30–32 GB model (e.g. Llama 3.1/3.3 32B at Q4) fits entirely on the 5090 — no TP at all. Or serve Q4 70B on the 5090 with CPU/GPU offloading so the remainder spills to system RAM on the 5070 Ti's side.
3. Two identical cards is the clean path if you're buying anyway — 2× 5070 Ti (32 GB total) or 2× 5090.

On your TTFT question: with mismatched cards, time-to-first-token is bounded by the slowest rank plus PCIe transfer overhead between them — the 0.4–0.5 s single-GPU figures you saw won't stack; expect it slower than either card alone in a sane topology. For a 42 GB target, a single 32 GB card + offload, or two matching 32 GB cards, is far simpler than chasing mixed-GPU TP.

See also: vLLM forum — how to run a model on heterogeneous GPUs.