Sporadic permanent GPU wedge on dual Arc Pro B70 (BMG G31): ccs/bcs engine reset + "Fault response: Unsuccessful -ENOENT/-EINVAL" under sustained Level-Zero inference load

[marcorigodanzo]

Pre-submission Checklist

• I am using the latest GPU driver version — currently on distro-provided 26.05.37020.3 (Ubuntu 26.04); latest release is 26.22.38646.4. Happy to retest on the latest release if you expect it to be relevant — the failure is easy for us to reproduce.
• I have searched for similar issues and found none — closest field reports: [Bug]: XPU TP=2 on dual Intel Arc Pro B70 (Battlemage): GP fault + xe BCS engine reset reproduces in intel/vllm:0.17.0-xpu on Ubuntu 24.04 HWE 6.17 vllm-project/vllm#41663 (same dual-B70 hardware, GP fault + bcs engine reset) and [BUG] Tumbleweed + Intel Arc B580: OpenCL LAPLACIAN_ASSEMBLE triggers Xe Engine Reset (GT0: Engine reset: engine_class=ccs) darktable-org/darktable#20257 (same ccs engine-reset signature on Arc B580 via OpenCL, no vLLM/oneCCL involved).

System Information

GPU Hardware

2× Intel Arc Pro B70 (Battlemage G31, PCI ID 8086:e223, ASRock subsystem 6025)

DRI Devices Information

$ ls -ls /dev/dri/*
0 crw-rw----+ 1 root video  226,   0 Jul  4 18:12 /dev/dri/card0
0 crw-rw----+ 1 root video  226,   1 Jul  4 18:12 /dev/dri/card1
0 crw-rw----+ 1 root video  226,   2 Jul  4 18:12 /dev/dri/card2
0 crw-rw----+ 1 root render 226, 128 Jul  4 18:12 /dev/dri/renderD128
0 crw-rw----+ 1 root render 226, 129 Jul  4 18:12 /dev/dri/renderD129

$ ls -la /dev/dri/by-path/
pci-0000:45:00.0-card -> ../card0        (platform framebuffer)
pci-0000:c3:00.0-card -> ../card1
pci-0000:c3:00.0-render -> ../renderD128
pci-0000:c7:00.0-card -> ../card2
pci-0000:c7:00.0-render -> ../renderD129

GPU Detailed Information (lspci output)

$ sudo lspci -vvv -k -s c3:00.0   (c7:00.0 is identical)
c3:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics] (prog-if 00 [VGA controller])
	Subsystem: ASRock Incorporation Device 6025
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Region 0: Memory at 11e00000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at 10800000000 (64-bit, prefetchable) [size=32G]
	Kernel driver in use: xe
	Kernel modules: xe

Host platform: AMD EPYC 7413 (24c). GuC firmware: xe/bmg_guc_70.bin version 70.58.0.

Driver Version

intel-opencl-icd / libze-intel-gpu1 26.05.37020.3-1 (Ubuntu 26.04 distro packages)

Installed GPU Driver Packages

intel-opencl-icd          26.05.37020.3-1
intel-opencl-icd-legacy   24.35.30872.45-1
libze-intel-gpu-dev       26.05.37020.3-1
libze-intel-gpu-legacy1-1 24.35.30872.45-1
libze-intel-gpu1          26.05.37020.3-1
libze1:amd64              1.28.2-2

Driver Installation Details

Distro-provided packages from Ubuntu 26.04 LTS archives (no Intel PPA / no manual install). Workload runs inside Docker containers with --device /dev/dri passthrough; the failure reproduces with two different container userspaces (oneAPI 2025.3.2 and the intel/vllm 0.17 prebuilt image), which points away from a specific container userspace.

Linux Distribution

Other

Other Linux Distribution

Ubuntu 26.04 LTS

Kernel Version & Boot Parameters

$ uname -a
Linux battlematrix 7.0.0-27-generic #27-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 18 19:13:49 UTC 2026 x86_64 GNU/Linux

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.0-27-generic root=UUID=... ro quiet splash crashkernel=2G-4G:320M,...

Issue Description

Actual Behavior

Under sustained multi-request LLM inference (vLLM, tensor-parallel=2 across the two B70s, 8–12 concurrent requests, mixed 4k–32k contexts), roughly once every 2–6 hours one card takes a GPU page fault and the xe driver resets its compute/copy engines:

xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=29, state=0x289
xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL

After the reset the userspace Level-Zero context is permanently wedged: the in-flight SYCL kernel never completes, the worker process never returns, and the serving engine dies (TimeoutError: RPC call to sample_tokens timed out → EngineDeadError). Only a full process/container restart recovers.

Key observations:

• Both cards are affected (events captured on c3:00.0 and c7:00.0) → not a single defective unit.
• 8 application-level failures over July 2–4 each correlate to the second with a dmesg engine-reset event.
• The dying scheduler step can be tiny: one captured crash dump shows 9 requests in decode, 9 total scheduled tokens, KV-cache at 14% — so this is not a batch-size/pressure problem. Incidence appears to scale with the number of kernel submissions over time, not with step size.
• Ruled out on our side: model & quantization path (reproduces with int4/AWQ/FP8 weights), guided-decoding backend, vLLM graph capture (enforce-eager), vLLM async scheduling (disabled), vLLM version (intel/vllm 0.17 image and upstream v0.24 source build), scheduler oversubscription (max-num-seqs capped).
• oneCCL "stable fallback" env from [Bug]: XPU TP=2 on dual Intel Arc Pro B70 (Battlemage): GP fault + xe BCS engine reset reproduces in intel/vllm:0.17.0-xpu on Ubuntu 24.04 HWE 6.17 vllm-project/vllm#41663 already applied in all runs (CCL_ENABLE_SYCL_KERNELS=0, CCL_ATL_TRANSPORT=ofi, CCL_ZE_IPC_EXCHANGE=sockets, CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0, SYCL_UR_USE_LEVEL_ZERO_V2=0) — it removed an earlier crash mode but not this one.
• Same ccs engine-reset signature has been reported without vLLM/oneCCL entirely ([BUG] Tumbleweed + Intel Arc B580: OpenCL LAPLACIAN_ASSEMBLE triggers Xe Engine Reset (GT0: Engine reset: engine_class=ccs) darktable-org/darktable#20257, OpenCL on B580), suggesting driver/firmware level.

Expected Behavior

No GPU page faults / engine resets under sustained compute load; or, if an engine reset occurs, the Level-Zero context should recover (or at least fail fast with an error the application can handle) instead of wedging permanently.

Reproduction Rate

Always reproduces - 100% (within 2–6 hours of sustained load; see steps)

Steps to Reproduce

1. Host as described above (dual Arc Pro B70, xe driver, kernel 7.0.0-27).
2. Serve a ~27B LLM with vLLM v0.24 (XPU build from upstream Dockerfile.xpu, oneAPI 2025.3.2) with --tensor-parallel-size 2 --enforce-eager --max-model-len 32768, oneCCL env as above. (Also reproduces with the intel/vllm:0.17.0-xpu prebuilt image and a 70B AWQ model.)
3. Drive sustained concurrent load: 8–12 parallel /v1/chat/completions requests in a loop for hours (we use JSON-schema constrained outputs, mixed prompt sizes 4k–32k tokens).
4. Within 2–6 hours: dmesg shows the ccs/bcs engine reset + unsuccessful fault response on one of the cards, and the serving engine hangs permanently.

Minimal load generator (python, against the vLLM OpenAI endpoint):

import httpx, json, time, concurrent.futures as cf
SCHEMA = {"type":"object","properties":{"verdict":{"type":"string"},"reasoning":{"type":"string"}},
          "required":["verdict","reasoning"]}
def call(i):
    p = f"Analyze company #{i}. Respond with JSON matching: {json.dumps(SCHEMA)}"
    return httpx.post("http://127.0.0.1:8080/v1/chat/completions", timeout=600, json={
        "model":"local","messages":[{"role":"user","content":p}],
        "max_tokens":300,"response_format":{"type":"json_object"}}).status_code
while True:
    with cf.ThreadPoolExecutor(max_workers=12) as ex:
        print(time.strftime("%T"), list(ex.map(call, range(12))))

Regression Information

Is this a regression?

• Yes, this is a regression — unknown: this dual-B70 configuration has shown the issue on every stack we tried; we have no known-good baseline.

Logs

API Call Logs

Not captured yet — the incident is frequent enough to instrument: we can provide ze_debug/API traces, py-spy/gdb stacks of the hung worker process, or xe debugfs state on request. Please tell us which traces are most useful.

strace Logs

Not captured (available on request, see above).

System Logs / dmesg Output

All engine-reset events captured July 2–4 (timestamps CEST). Every one coincides (±2 s) with an application-level engine death:

[Jul 2 23:40:09] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=43, state=0x289
[Jul 2 23:40:09] xe 0000:c3:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 2 23:53:37] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=48, state=0x289
[Jul 3 00:01:23] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=27, state=0x289
[Jul 3 00:01:23] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=32, state=0x289
[Jul 3 00:12:24] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=26, state=0x289
[Jul 3 00:18:56] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=43, state=0x289
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 4 01:39:33] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL
[Jul 4 05:49:27] xe 0000:c3:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=48, state=0x289
[Jul 4 10:51:14] xe 0000:c3:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=23, state=0x289
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=31, state=0x289
[Jul 4 11:01:17] xe 0000:c7:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL

Backtrace (if crash or hang occurred)

Application-side (vLLM EngineCore parent; the GPU worker itself hangs and never returns — we can attach py-spy/gdb to it at the next occurrence on request):

(EngineCore) ERROR [core.py:1233] EngineCore encountered a fatal error.
  File ".../vllm/v1/executor/multiproc_executor.py", line 387, in get_response
    status, result = mq.dequeue(timeout=dequeue_timeout)
  File ".../vllm/v1/executor/multiproc_executor.py", line 370, in get_response
    raise TimeoutError(f"RPC call to {method} timed out.") from e
TimeoutError: RPC call to sample_tokens timed out.
(APIServer) vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

Captured scheduler state at one crash (shows the dying step was 9 decode tokens across 9 requests, KV 14%):

SchedulerOutput(..., total_num_scheduled_tokens=9, ...)
SchedulerStats(num_running_reqs=9, num_waiting_reqs=0, kv_cache_usage=0.1439...)

Reproducer

Source Code / Reproducer

See "Steps to Reproduce" above — vLLM serve command plus the 15-line python load generator. Full container logs, complete dmesg, and our original forensics write-up are in #946 (filed before we were aware of the template — apologies; this issue supersedes it).

[marcorigodanzo]

Triage datapoint from an independent dual-B70 rig (cross-report from vllm-project/vllm#41663, user Zumbasam, 2026-07-06):

On their stack — Ubuntu 24.04.4, kernel 6.17.0-23, GuC 70.44.1, two B70 on separate root ports — engine_class=ccs resets under SYCL-allreduce load are recoverable: no Fault response lines, the card survives, and with CCL_ENABLE_SYCL_KERNELS=0 (+CCL_ALLREDUCE=ring) they get 2-hour soaks with zero engine resets.

On our stack — kernel 7.0.0-27, GuC 70.58.0, compute-runtime 26.05 — the resets come with Fault response: Unsuccessful -ENOENT/-EINVAL and permanently wedge the Level-Zero context, even with CCL_ENABLE_SYCL_KERNELS=0 applied.

Same GPU model, same workload class → the permanent-wedge mode appears to correlate with the newer KMD / GuC firmware / compute-runtime combination.

We're happy to bisect on our rig (it reproduces within 2–6 h): which component would you like us to vary first — GuC firmware (70.58 → 70.44), kernel xe, or compute-runtime (26.05 → 26.22)? Any debug keys/traces you want captured during the next occurrence?