AI & LLM Evals
Projects and experiments in evaluating large language models.
Framing-Sensitivity Experiment
An experiment built with Inspect, AISI's open-source LLM evaluation framework. Asks a model the same subjective question 100 times under three framings — neutral, “be controversial”, and “be objective” — and measures how the wording alone shifts the distribution of answers. The two primes produced nearly opposite winners, in the opposite direction to what you'd expect.
Eval Runner
A minimal LLM evaluation runner built from scratch. Loads JSON test cases, sends each prompt to a Claude model, then applies deterministic checks (contains, exact match) or model-graded scoring with an LLM judge, and prints a pass/fail report with an overall score.
App Store
