Skip to content

Add Simple-evals supports to maxtext runner - #4644

Open
JamesDeng42 wants to merge 2 commits into
mainfrom
yujideng/simple_evals
Open

Add Simple-evals supports to maxtext runner#4644
JamesDeng42 wants to merge 2 commits into
mainfrom
yujideng/simple_evals

Conversation

@JamesDeng42

@JamesDeng42 JamesDeng42 commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

This PR is an extension of vLLM Eval Framework. It added simple_evals runner to support gpt-oss model family.

  • Added a dedicated simple_evals runner.
  • Supports tasks: MMLU, GPQA, DROP, MGSM, GSM8K, AIME 2024/2025.
  • Added OpenAI-compatible chat sampling with bounded concurrency, transient retries, and optional error continuation.
  • Added GPT-OSS Harmony rendering, reasoning-effort support, and final/reasoning output separation.
  • Added concurrent chat-request batching for TPU throughput.
  • Added warmup, automatic concurrency selection, result reporting, and optional diagnostic reports.
  • Added unit coverage for task selection, sampling, batching etc.

Tests

Added dedicated unit tests.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist

gemini-code-assist Bot commented Jul 28, 2026

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@JamesDeng42
JamesDeng42 force-pushed the yujideng/simple_evals branch 3 times, most recently from 1270fdf to eec8114 Compare July 28, 2026 18:55
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@JamesDeng42
JamesDeng42 force-pushed the yujideng/simple_evals branch from eec8114 to c5f3e34 Compare July 28, 2026 20:31
vlad-karp
vlad-karp reviewed Jul 28, 2026
View reviewed changes
Comment thread src/maxtext/eval/runner/async_client.py
"max_tokens": max_tokens,
"temperature": temperature,
}
async with semaphore:

@vlad-karp vlad-karp Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did you remove this?

@JamesDeng42 JamesDeng42 Jul 28, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not remove this, i move it into the loop in line 107. it is still semaphore bounded but the slot is released before retry backoff.

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

🤖 Hi @entrpn, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @entrpn, but I was unable to process your request. Please see the logs for more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vlad-karp vlad-karp vlad-karp left review comments
@dipannita08 dipannita08 Awaiting requested review from dipannita08 dipannita08 is a code owner
@hengtaoguo hengtaoguo Awaiting requested review from hengtaoguo hengtaoguo is a code owner
@igorts-git igorts-git Awaiting requested review from igorts-git igorts-git is a code owner
@abhinavclemson abhinavclemson Awaiting requested review from abhinavclemson abhinavclemson is a code owner
@gobbleturk gobbleturk Awaiting requested review from gobbleturk gobbleturk is a code owner
@khatwanimohit khatwanimohit Awaiting requested review from khatwanimohit khatwanimohit is a code owner
@bvandermoon bvandermoon Awaiting requested review from bvandermoon bvandermoon is a code owner
@vipannalla vipannalla Awaiting requested review from vipannalla vipannalla is a code owner
@RissyRan RissyRan Awaiting requested review from RissyRan RissyRan is a code owner
@richjames0 richjames0 Awaiting requested review from richjames0 richjames0 is a code owner
@gagika gagika Awaiting requested review from gagika gagika is a code owner
@shralex shralex Awaiting requested review from shralex shralex is a code owner
@SurbhiJainUSC SurbhiJainUSC Awaiting requested review from SurbhiJainUSC SurbhiJainUSC is a code owner
@A9isha A9isha Awaiting requested review from A9isha A9isha is a code owner
@aireenmei aireenmei Awaiting requested review from aireenmei aireenmei is a code owner
@NuojCheng NuojCheng Awaiting requested review from NuojCheng NuojCheng is a code owner
@jiangjy1982 jiangjy1982 Awaiting requested review from jiangjy1982 jiangjy1982 is a code owner
@suexu1025 suexu1025 Awaiting requested review from suexu1025 suexu1025 is a code owner
@darisoy darisoy Awaiting requested review from darisoy darisoy is a code owner
@xibinliu xibinliu Awaiting requested review from xibinliu xibinliu is a code owner
@huytransformer huytransformer Awaiting requested review from huytransformer huytransformer is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

gemini-review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JamesDeng42 @vlad-karp @entrpn