BREAKING AN LLM was a beginner-friendly workshop where we explored the internal mechanics of language models. Instead of treating AI as a "black box," we traced how it reads words, stores context, and predicts the next token.
The session focused on a core principle: models behave like pattern machines rather than thinking systems. Participants conducted live experiments to visualize attention patterns, identify specific circuits, and disable parts of GPT-2 to observe changes in behavior. This repository includes all resources from the technical talk and workshop 'Breaking An LLM', in order to facilitate an intuitive understanding of the concepts covered, as well as some additional content.
Slides — Big Picture & Theory
This section covers the theoretical foundations of transformer interpretability.
What’s Covered:
- Concepts: The "Bomb in the Box" thought experiment and the neuroscience approach to AI.
- Architecture: Transformer anatomy, residual stream mechanics, and attention (QK/OV).
- Mechanics: Logits vs. Softmax, polysemanticity, and the difference between neurons, features, and circuits.
- Analysis: Identifying mechanical failures behind jailbreaks and tracing how output tokens are generated.
Learning Objective: Build a mental model of how transformers process information and understand why behavior emerges from internal circuits rather than intention.
Reference File: Breaking_An_LLM.pptx
Notebook 1 — Induction Heads & Copying Behavior
A practical deep-dive into how models recognize and repeat patterns.
What’s Covered:
- Setup: Tool configuration and defining custom prompts.
- Observation: Tracking loss drops in repeated sequences.
- Visualization: Mapping attention patterns to identify induction heads.
- Experimentation: Activation patching and ablation with control checks.
Learning Objective: Understand how in-context learning works at a circuit level and see how simple copying mechanisms can lead to many-shot jailbreak behavior.
Notebook 2 — IOI Task & Name-Mover Head
Advanced "model surgery" focusing on specific grammatical tasks.
What’s Covered:
- Task: Indirect Object Identification (IOI) prompts.
- Metrics: Logit difference metrics and clean vs. corrupted prompts.
- Localization: Using activation patching heatmaps to find the Name-Mover head.
- Surgery: Mean ablation (disabling the circuit) and result comparison dashboards.
Learning Objective: Causally localize a specific grammatical circuit inside GPT-2 and prove its necessity by disabling it without retraining the model.
Quiz — Quick Fire Concepts
A brief assessment to reinforce the technical vocabulary used in the workshop.
What’s Covered:
- Logits vs. probabilities.
- Residual stream flow and polysemanticity.
- Induction heads and circuit-level reasoning.
Learning Objective: Test conceptual clarity and reinforce core ideas before moving into deeper interpretability work.
- Slides: Review first to understand the underlying philosophy.
- Notebook 1: Run the code to observe copying behavior.
- Quiz: Complete the quiz to check your understanding.
- Notebook 2: Perform advanced ablation and surgery.
- Further Reading: Resources
Break things. Understand them. Then build better ones.
We would like to thank the following individuals for their contributions throughout the workshop:
AI/ML Lead
- Manya Pandey
Core Team
- Chinmayee Parwekar
- Aarja Lohar
- Shreyanshi Singh
- Swarnim Vatsyayan
- Ayush Kanta Bhattacharya
- Simar Walia
- Srijoy Ganguly
- Shourja Banerjee
- Arnav Mahapatra
- Sayan Ghosh
- Ayush Kumar Anand
- Shreya Dubey