Add MinerU-Diffusion benchmark harness

[HuXinjing]

Summary

• Add a benchmark harness for OpenAI-compatible MinerU-Diffusion serving and native generation experiments.
• Add PDF-page suite preparation, two-step layout/content comparison tools, layout sampling experiments, and report helpers.
• Add unit tests for benchmark parsing, scoring, scheduling, and CLI behavior.

Experiment report

Full reproducibility package: https://github.com/HuXinjing/mineru-dllm-vllm-experiment-report

Full environment metadata: https://github.com/HuXinjing/mineru-dllm-vllm-experiment-report/blob/main/data/rerun/curated_20260618_pr_repos/environment_20260618.json

The report packages the cleaned PR inputs, rerun artifacts, code snapshots, patches, PDF-page manifest, scripts, GPU/device metadata, Python package versions, and vLLM runtime settings used to reproduce the MinerU-Diffusion vLLM integration measurements.

Cleaned PR inputs used for the rerun:

Component	Commit
vLLM MinerU-Diffusion PR prep	7f6e10e357a46f05572184b53745bf457ccf09e9
MinerU benchmark harness PR prep	b88a65d893b6da86f29fadcec88960e2b0956cda

Hardware and runtime: CUDA_VISIBLE_DEVICES=0 on an NVIDIA GeForce RTX 4090 24GB; host also had a second RTX 4090. Driver 580.126.09, nvidia-smi CUDA 13.0, PyTorch 2.11.0+cu128, torch CUDA runtime 12.8, vLLM V2 model runner, TRITON attention, CUDA graph capture enabled. Dataset: 23-page PDF coverage suite packaged in the report repository.

Layout-only rerun

Variant	OK	Layout total s	Wall s	Mean s	Layout chars/s
DLLM default temp=1	23/23	53.288	14.176	2.317	527.2
DLLM MinerU layout sampling	23/23	48.275	12.404	2.099	591.9
Original MinerU2.5-Pro reference	23/23	26.551	n/a	1.154	1050.6

Layout sampling improved DLLM layout total latency by 1.1038x; mean layout F1 between default and MinerU-style sampling was 0.9586. Against the Pro reference, DLLM MinerU-style layout sampling was still slower, with Pro/DLLM layout speed ratio 0.5500x and mean layout F1 0.8702.

Two-step rerun

The two-step run uses layout first, then content extraction with content_concurrency=4 and dynamic_threshold=0.90.

Run	OK	Total latency s	Mean s	Markdown chars	Markdown chars/s
Original MinerU2.5-Pro baseline	23/23	70.782	3.077	76331	1078.4
Original DLLM t090 conc4	23/23	71.843	3.124	68192	949.2
Rerun DLLM t090 conc4	23/23	76.150	3.311	70506	925.9

Rerun DLLM vs original Pro baseline: total speed ratio Pro/DLLM was 0.9295x, mean markdown similarity was 0.8774, and mean layout F1 was 0.8583. In this harness, DLLM two-step did not outperform the MinerU2.5-Pro baseline.

Throughput supplement

Run	OK	Wall s	Layout wall s	Extract wall s	Pages/s	Markdown chars/s
Rerun DLLM t090 throughput	23/23	45.120	13.921	31.199	0.510	1491.2

Interpretation: the cleaned PR repositories reproduce the earlier conclusion. Text/table markdown remains close to the MinerU2.5-Pro reference, while layout agreement is lower and the measured bottleneck remains the denoising canvas loop and layout stage behavior rather than PDF image rendering or result file I/O.

Test Plan

• python -m pytest -o addopts= tests/benchmarks -q
• Result: 30 passed in 0.73s

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[HuXinjing]

I have read the CLA Document and I hereby sign the CLA

[HuXinjing]

recheck