AIBO, the AI Behavioral Observatory, is an open-source tool for running controlled behavioral experiments on AI systems at scale. Creating and using it over twelve months changed how we work. We went from using AI to build the tool to using AI as an active collaborator in running the research itself, and we shortened the distance between articulating a research question and having the evidence needed to answer it — a shift we examined more closely in a follow-up study.
Cite as:
Shapiro, Dan and Meincke, Lennart and Mollick, Ethan R. and Mollick, Lilach, The AI Behavioral Observatory (July 23, 2026). Available at: https://github.com/wharton-generative-ai-labs/AIBO
Why We Built AIBO
AIBO was developed to answer a deceptively simple question essential to running behavioral experiments on AI systems: “What happens if I change this prompt?” Not once, but thousands of times, as is necessary for scientifically sound research. It had to work across conditions, and the results had to be saved so they could be analyzed.
When we began, the state of AI required us to paste one prompt at a time into ChatGPT or Claude and evaluate each set of answers separately. AIBO let us instead conduct repeatable experiments concurrently, and examine the results efficiently. It let a researcher define a control prompt and a treatment prompt, specify how each response should be scored, run the comparison across many trials, and analyze the results statistically.
That mattered because rigorous claims about AI behavior can’t rest on a handful of chats. A model’s responses vary from one run to the next, so many trials are needed before a real effect rises above the noise. AIBO was built to solve this problem and provide effective facilitation to researchers who don’t code.
The Original Workflow
In early 2025, as we worked on our initial study of whether AI models could be persuaded using classic principles of social influence, AIBO’s user interface was central to our workflow. It was a simple but effective web app that ran experiments directly in the browser. This work coincided with a period when coding assistance from LLMs was rapidly improving, so the tool itself was a product of the moment. AI helped us build a tool for studying AI.
This enabled us to move from idea to experiment with a simple process:
- Write a control prompt.
- Write a treatment prompt
- Specify how the model’s response should be judged.
- Determine desired number of iterations.
- Receive and analyze results.
That first version of AIBO conducted the experiment behind our initial paper: roughly 28,000 conversations, all tested against a single model. AIBO made that scale entirely manageable.
What Changed
By the time we were ready to embark on our follow-up study in early 2026, AI had continued to advance. This included not just the models, but the tools around them. Coding agents like Claude Code and Codex, run from a terminal, were pulling knowledge work away from chat interfaces. So our own use of AIBO changed too. AIBO’s core still did exactly what it always had done: take an experimental design, run it, and analyze the results. But how we drove it changed. Where the first version required clicking through fields in a browser, we now used plain language to describe what we wanted to an AI coding agent.
That shift did more than move the work onto the command line, or occasionally onto our phones letting us work in any location. It enabled us be more rigorous in concrete ways:
- We could run several models in parallel and compare them directly, where the original study had used just one model.
- Testing was done against a larger, more varied set of stimuli rather than a narrow one.
- We leaned on the agent to inspect raw outputs, summarize them, flag inconsistencies, and propose the next round of tests, so more of the analysis got checked rather than accepted on trust.
This increased our sample size well beyond the initial study’s 28,000 conversations on one model, completing 126,000 conversations, across several models. AIBO itself didn’t get smarter. The AI working alongside it did, expanding what a small team could accomplish while maintaining rigor.
Prompting the Experiment Itself
This was the deeper shift. The old workflow looked like this: a researcher has an idea, writes prompts, enters them into a UI, runs the experiment, downloads the results, analyzes them, and decides what to try next. The newer workflow is more conversational. A researcher can say something like: “Come up with five different ways to operationalize scarcity in this experiment. Make them parallel to the control condition. Run a small pilot. Tell me which versions avoid floor and ceiling effects. Then prepare the full run.”
The researcher still makes the scientific decisions, determining what question is worth asking, what counts as a fair comparison, and whether the results are meaningful. But the distance between the question and the evidence gets much shorter. It feels less like operating software and more like talking to a very fast research assistant, in plain English.
Why We’re Sharing It
We’re releasing AIBO because we think more people should be able to do this kind of work. You can try it right now in the live demo — bring your own API key, and note the hosted version is UI-only (no MCP or Skills support). Or you can point an AI coding agent at the full open-source version to generate and execute experiments; it comes with a skill and MCP server included. For students and AI-curious researchers alike, the point is the same: you don’t have to treat AI behavior as magic. You can test it.
Another reason we are open sourcing AIBO is that it may be useful to build and extend on. For example, we primarily use the system to judge AI’s behavior or answers in unambiguous tests (such as answering questions with known rubrics). But an interested researcher may wish to extend AIBO’s capability to do validated scoring of ambiguous or qualitatively rich answers. We hope to see the community build on AIBO or use it as an inspiration for their own projects.