Run Inspect AI evals in the cloud

Consolidated model evaluation framework for LLM benchmarking with Ollama

Benchmark for measuring instrumental-convergence behaviour in tool-using LLM agents

ACL Findings benchmark for measuring LLM sycophancy and correction selectivity

Factor(UT): Controlling Untrusted AI Decomposers — AAAI 2026 workshop paper on monitoring untrusted decomposition in code generation workflows.

Release gates for AI agents: replay known incidents and check prompt, model and tool-policy changes before they ship. Runs under Inspect.

A lightweight Inspect AI benchmark for obvious public-facing LLM failures.

Open-source, MCP-native benchmark for whether AI agents reason reliably over fragmented enterprise knowledge — accuracy, provenance, correct refusal.

Real DuckDB Quack infrastructure for multi-agent Werewolf: containerized player nodes, Quack gateway federation, browser runner, and local/hosted LLM evals.

Statistically rigorous, batch-first reliability auditing for LLM judges & reward models — clustered-bootstrap CIs, McNemar, BH-FDR, power/MDE, planted-bias validation. No API key required.

Long-horizon reliability benchmark for industrial edge agents - do self-improving agents get better or corrupt over 1000+ episodes on constrained hardware?

Wellness verification harness for companion AI. Multi-turn adversarial suites grounded in six decades of mental-health research and current clinical standards (988, VERA-MH) and law (SB 243). Point it at any chat endpoint — get an evidence-backed, reproducible report.

Open source pipeline for running evals of chemical security tasks

LLM agent that plays the Wikipedia game, built on AISI Inspect

Inspect AI task pack for financial agent safety and first public LangGraph adapter for Inspect AI.

Security benchmark for LLM agents & their harnesses — offensive, defensive, secure-coding & prompt-injection-safety challenges on live sandboxed targets, with gradated scoring. Built on Inspect AI.

Cross-framework agent regression gate — 9 agent frameworks (OpenAI SDK, PydanticAI, Google ADK, Strands, LangChain, Claude SDK, MAF, CrewAI, Smolagents) evaluated on shared tasks with cross-vendor LLM-as-judge, ordinal pairwise ranking, and multi-axis drift attribution.

Author verifiable eval records through a draft → review → revise → submit loop with server-enforced graders; compile to JSONL/CSV/Inspect/lm-eval via MCP. STDIO or Streamable HTTP.

Open-source psychological-safety benchmark for conversational AI: frozen clinical scenarios scored by an LLM-judge panel into a CI pass/fail gate + a clinician-grade diagnostic.

Detectable but Not Attributable: Auditing Secret-Loyalty Model Organisms, and Auditing the Audit