A collection of evaluations for testing AI model reasoning capabilities across various domains.
This repository contains evaluations designed to test deeper reasoning in AI models - going beyond surface-level responses to probe genuine understanding. Each evaluation targets areas where models were struggling at the time of creation.
A key design principle: all tests are trivial for humans to understand and evaluate quickly, making it easy to assess model performance at a glance.
Tests whether AI models can figure out a simple comment-based code toggling technique using C-style block comments
Much like v1 but toggling between two blocks of code instead of just dealing with one
Each evaluation:
- Presents a clear problem statement
- Specifies exact constraints (e.g., single keypress, no modifier keys)
- Tracks results from various AI models
- Documents notes on model performance
Results are tracked within each evaluation file, showing which models passed or failed, along with observations about their reasoning quality and speed.
MIT