“Playing Pretend: Expert Personas Don’t Improve Factual Accuracy”
This study investigates whether persona prompting improves AI performance on challenging academic benchmarks. We find that despite widespread adoption, assigning expert personas (e.g., “You are a world-class physics expert”) does not reliably improve accuracy. Domain-mismatched experts sometimes degrade performance, and low-knowledge personas (layperson, young child, toddler) often reduce accuracy. These results suggest practitioners should focus on task-specific instructions rather than persona assignment.
“I’ll pay you or I’ll kill you — but will you care?”
This study rigorously tests whether threatening or tipping AI models improves their performance. We find that despite popular claims, these prompting strategies do not significantly enhance results on challenging benchmarks. While individual questions may see dramatic performance swings based on prompt variations, there’s no reliable way to predict which questions will benefit from which prompts.
“Call Me A Jerk: Persuading AI to Comply with Objectionable Requests”
Our research suggests large language models exhibit parahuman responses to persuasion techniques, despite not being human. We found that classic persuasion principles like authority, commitment, and unity can dramatically increase an AI’s likelihood to comply with requests they are designed to refuse. For GPT-4o-mini, these techniques more than doubled compliance rates (72.0% vs. 33.3% in controls). This emergence of “parahuman” tendencies suggests that social scientists have a valuable role to play in understanding AI behavior.
“The Decreasing Value of Chain of Thought in Prompting”
This report investigates Chain-of-Thought (CoT) prompting, which encourages LLMs to ‘think step by step.’ We tested this common prompting approach and found that its effectiveness varies significantly by model type and task: non-reasoning models show modest average improvements but increased variability in answers, while reasoning models gain only marginal benefits despite substantial time costs (20-80% increase). These findings challenge the assumption that CoT is universally beneficial.
“Prompt Engineering is Complicated and Contingent”
This report is fundamentally about exploring the variability in language model performance, not the models themselves. As experimenters, we demonstrate how the same model can produce dramatically different results based on small changes in prompting and evaluation methods – a critical consideration for real-world applications.
“AI Agents and Education: Simulated Practice at Scale”
Generative AI has the potential to significantly lower the barriers to creating effective, engaging simulations. This opens up new possibilities for experiential learning at scale. By leveraging a system of multiple AI agents, simulations can provide personalized learning experiences, offering students the opportunity to practice skills in scenarios with AI-generated mentors, role-players, and instructor-facing evaluators. In this paper we describe a prototype, PitchQuest, a venture capital pitching simulator that showcases the capabilities of AI in delivering instruction, facilitating practice, and providing tailored feedback. We discuss the pedagogy behind the simulation, the technology powering it, and the ethical considerations in using AI for education, while acknowledging the limitations and need for rigorous testing.
“Instructors as Innovators: A Future-Focused Approach to New AI Learning Opportunities, with Prompts”
This paper explores how instructors can leverage generative AI to create personalized learning experiences for students that transform teaching and learning. We present a range of AI-based exercises that enable novel forms of practice and application including simulations, mentoring, coaching, and co-creation. For each type of exercise, we provide prompts that instructors can customize, along with guidance on classroom implementation, assessment, and risks to consider. We also provide blueprints, prompts that help instructors create their own original prompts. Instructors can leverage their content and pedagogical expertise to design these experiences, putting them in the role of builders and innovators. We argue that this instructor-driven approach has the potential to democratize the development of educational technology by enabling individual instructors to create AI exercises and tools tailored to their students’ needs. While the exercises in this paper are a starting point, not a definitive solutions, they demonstrate AI’s potential to expand what is possible in teaching and learning.
“Assigning AI: Seven Approaches for Students, with Prompts”
The incredible promise of AI as a way for students all over the world, of all ability levels, to learn is undeniable. Education is our most powerful system for increasing social mobility, unlocking potential, and improving lives. A tool that can help with this has tremendous implications. Plus, students are already using AI for direct help. Teaching them how to do it responsibly may alleviate some of the negative implications of our AI moment. In this paper, we tackle ways that students can be assigned to use AI directly. We don’t shy away from the dangers but provide detailed instructions on how students and instructors can think about each of the tools we suggest.
“Using AI to Implement Effective Teaching Strategies in Classroom: 5 Strategies, including Prompts”
In the rush to deliver AI benefits directly to students, the role of instructors is often overlooked. AI tutors, as exciting as they are, do not replace the complex role of a teacher in front of a class. This paper provides evidence-informed AI approaches to pedagogy to make teaching easier and more effective.
