Skip to content

Commit 09c1ab3

Browse files
committed
test: hold the Groq example count at two so the daily quota fits
Serialising by provider stopped Groq examples competing for the same quota window, but the binding limit is a daily one: the org is capped at 100k tokens per day on this tier, and four Groq examples making several calls each do not fit inside it. No retry policy or backoff addresses that. So the two carrying the least distinct information come out: groq-code-interpreter-js - the same demo as its Python twin, which stays upload-dataset-code-interpreter - a third Groq chart demo groq-code-interpreter-python and mcp-groq-exa-js remain, which still covers both Groq SDK surfaces. Roughly half the token spend for nearly all the coverage. Both dropped examples work - the exclusion note says so explicitly, because "excluded" has meant "broken" for every other entry in that list and these two should go straight back if the tier is raised. They are not being quietly retired. Also fixed a stale reference in the runner's header comment: it still described results.json as feeding updateTestsMd.js, which report.mjs replaced.
1 parent 1cf8bc5 commit 09c1ab3

1 file changed

Lines changed: 16 additions & 10 deletions

File tree

tests/run-examples.ts

Lines changed: 16 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@
88
* e2b SDK: e2b depends on chalk 5 and four other ESM-only packages, and while
99
* Node 22+ resolves those fine from both CJS and ESM, Jest's own runtime does
1010
* not implement `require(esm)`. Rather than fight that, the runner is a plain
11-
* script. It still writes tests/results.json in the shape updateTestsMd.js
12-
* expects, so the reporting and Slack steps are unchanged.
11+
* script. It still writes tests/results.json, which tests/report.mjs turns into
12+
* the artifact table and the Slack payload.
1313
*/
1414
import { promises as fs } from 'fs'
1515
import path from 'path'
@@ -27,12 +27,11 @@ const scripts: {
2727
interpreter: Interpreter
2828
file: string
2929
// Providers that rate-limit per organisation. Examples sharing a provider are
30-
// serialised against each other, which fixed one of the three Groq failures.
31-
// The residual limit is not concurrency though: the Groq org is capped at
32-
// 100k tokens per DAY on this tier ("tokens per day (TPD): Limit 100000,
33-
// Used 99952 ... try again in 10m"), and four Groq examples several calls each
34-
// can exhaust it - especially across repeated runs on one day. No retry policy
35-
// fixes a daily cap; it needs a higher tier or fewer Groq examples per run.
30+
// serialised against each other, so they never compete for the same quota
31+
// window. That is not enough on its own: the Groq org is capped at 100k tokens
32+
// per DAY on this tier ("tokens per day (TPD): Limit 100000, Used 99952"), which
33+
// no retry policy can work around, so the Groq example count is deliberately
34+
// held at two. See the exclusion note below before adding a third.
3635
provider?: string
3736
// What to pass after `uv run` / `poetry run`. Defaults to main.py for uv and
3837
// the `start` console script for poetry. Can be a path, a script name, or
@@ -48,14 +47,12 @@ const scripts: {
4847
{ name: 'codestral-code-interpreter-js', interpreter: 'npm', file: './examples/codestral-code-interpreter-js/' },
4948
{ name: 'gpt-4o-code-interpreter-js', interpreter: 'npm', file: './examples/gpt-4o-js/' },
5049
{ name: 'codestral-code-interpreter-python', interpreter: 'jupyter', file: './examples/codestral-code-interpreter-python/codestral_code_interpreter.ipynb' },
51-
{ name: 'upload-dataset-code-interpreter', provider: 'groq', interpreter: 'jupyter', file: './examples/upload-dataset-code-interpreter/llama_3_code_interpreter_upload_dataset.ipynb' },
5250
{ name: 'hello-world-python', interpreter: 'poetry', file: './examples/hello-world-python/' },
5351
{ name: 'o1-code-interpreter-js', interpreter: 'npm', file: './examples/o1-and-gpt-4-js/' },
5452
{ name: 'gpt-4o-code-interpreter', interpreter: 'jupyter', file: './examples/gpt-4o-python/gpt_4o.ipynb' },
5553
{ name: 'together-ai-code-interpreter-python', interpreter: 'jupyter', file: './examples/together-ai-code-interpreter-python/together_with_e2b_code_interpreter.ipynb' },
5654
{ name: 'langchain-python', interpreter: 'poetry', file: './examples/langchain-python/' },
5755
{ name: 'langgraph-python', interpreter: 'poetry', file: './examples/langgraph-python/' },
58-
{ name: 'groq-code-interpreter-js', provider: 'groq', interpreter: 'npm', file: './examples/groq-code-interpreter-js/' },
5956
{ name: 'claude-code-interpreter-python', interpreter: 'jupyter', file: './examples/claude-code-interpreter-python/claude_code_interpreter.ipynb' },
6057
{ name: 'claude-visualize-website-topics', interpreter: 'jupyter', file: './examples/claude-visualize-website-topics/claude-visualize-website-topics.ipynb' },
6158
{ name: 'mcp-client-js', interpreter: 'npm', file: './examples/mcp-client-js/' },
@@ -98,6 +95,15 @@ const scripts: {
9895
// against the current CLI - either pin Codex in the template or wire its current
9996
// auth - so it is not guessed at here:
10097
// openai-codex-in-sandbox-js, openai-codex-in-sandbox-python
98+
// Dropped to fit the Groq quota, NOT because they are broken - both pass when the
99+
// day's budget allows, and they should be restored if the tier is raised. The org
100+
// is capped at 100k tokens per day and four Groq examples several calls each do not
101+
// fit, so the two carrying the least distinct information come out:
102+
// groq-code-interpreter-js is the same demo as its Python twin, which stays, and
103+
// upload-dataset-code-interpreter is a third Groq chart demo.
104+
// groq-code-interpreter-python and mcp-groq-exa-js remain, covering the two
105+
// different SDK surfaces:
106+
// groq-code-interpreter-js, upload-dataset-code-interpreter
101107
// Calls a model the Fireworks account cannot reach: qwen2p5-coder-32b-instruct
102108
// returns 404 "Model not found, inaccessible, and/or not deployed", which does not
103109
// distinguish a retired model from one this account has not deployed. Needs someone

0 commit comments

Comments
 (0)