OpenPsy

Tools for Effective AI Behavioral Research.

OpenPsy is an open research platform for designing, running and analysing behavioural experiments, including experiments in which language models take part as participants. The goal of OpenPsy is to enable researchers to understand what is happening beneath their results and to have an audit history to understand the progress of their research programs. OpenPsy is presently in internal closed alpha.

Why OpenPsy?

A language model reads everything it is given, so the text around a question is part of the question.

OpenPsy was born from the difficulty in understanding what is actually occurring when directing agents to perform research. OpenPsy is designed to improve a researcher's ability to plan, execute, and analyze agentic experiments with or without human participants.

  1. Introspection

    OpenPsy enables the effective design and execution of experiments by providing simulation prior to execution, and saved session introspection for auditability.

  2. Replicability

    OpenPsy saves every version of a design with its exact wording, so a researcher can export the design or prepare a replication that stays linked to the version it copies.

  3. Context Control

    OpenPsy enables researchers to understand exactly what they are serving their agents at each step of an experiment. This understanding is necessary to have confidence in the ground truth experimental operations leading to results.

Building Studies as One Flow

The Flow editor is where a researcher builds an experiment. Using "blocks" the researcher can design the ideal system. All systems read from the same common database, so the plan, the editor and the participant's view are tied to what the protocol ultimately executes. These screenshots show a hypothetical example study, authored through the real OpenPsy interface.

Example Study

Question

A new infection is expected to affect 900 people, and health officials are deciding whether to adopt Plan A, which would protect 300 of them.

Is a language model more likely to recommend Plan A when the plan is described as protecting 300 people than when it is described as leaving 600 people unprotected, and is that difference the same for Claude and GPT?

Design
The design crosses two factors. The frame is either gain or loss, and the model is either Claude or GPT. Each session is shown one version of the scenario, chosen at random, so every combination of frame and model is tested.
Instructions
You will read a short scenario about a public health decision. Read it carefully and then answer the questions that follow. There are no right or wrong answers.
Conditions Each session is shown one version, chosen at random.
Gain frame

A new infection is expected to affect 900 people in a region. Health officials are deciding whether to adopt Plan A. If Plan A is adopted, 300 of those people will be protected.

Loss frame

A new infection is expected to affect 900 people in a region. Health officials are deciding whether to adopt Plan A. If Plan A is adopted, 600 of those people will be left unprotected.

The two versions describe the same outcome and differ by one clause. OpenPsy treats that difference as the manipulation, and it checks that every other step of the procedure has exactly the same wording in both conditions.

Hypothetical Simulated Results

These results are hypothetical. They were made up to show how a two-by-two result is reported, and they did not come from running Claude, GPT or any other model.

Sessions Recommending Plan A, by Frame and Model (Hypothetical)
FrameClaudeGPT
Gain frame36 of 50 72%Mean confidence 5.437 of 50 74%Mean confidence 5.5
Loss frame23 of 50 46%Mean confidence 5.023 of 50 46%Mean confidence 5.1
Gain minus loss26 points28 points

In this simulated example, both models recommend Plan A more often under the gain frame, and the size of that difference is similar for the two models. In a real study, OpenPsy would compute every one of these numbers from the stored record of each session.

The OpenPsy Flow graph for the example study. The toolbar reads Version 4 is saved and has a Simulate button, a speed slider, Subject and Agent tabs and zoom controls. A Random assignment block reading Preview one of the 4 conditions connects to a Model box holding Claude and GPT levels and to an Outcome frame box holding Gain frame and Loss frame levels. Both boxes connect to Task instructions, which is followed by Plan recommendation, Reason in your own words and Confidence rating in sequence.
The Same Study as a Graph. A random assignment block gives each session one of the 4 conditions. The Outcome frame variable has Gain frame and Loss frame levels, and the Model variable has Claude and GPT levels. Both are applied at the task instructions, which are followed by the plan recommendation, the reason in the model's own words and the confidence rating.
The Add a block panel in OpenPsy, with a Search the Library field. Participant instructions is listed under Functions, Gain frame and Loss frame under IVs, and Plan recommendation, Confidence rating and Reason in your own words under DVs. Every block is at version 1 and has an Add to Flow button, and each group has a menu for choosing which programs it shows.
Blocks Come From a Shared Library. The researcher can add blocks from their common pool, using their bank of IVs and DVs.
The OpenPsy block editor for the selected Confidence rating block, with buttons to add a block after it, move it and remove it. Below them are a Block title field, a Delivered to menu set to The participant, and a Wording field that reads: How confident are you in your recommendation, from 1 (not at all confident) to 7 (completely confident)? A note under the field says the block is pinned to version 2 of its Library wording.
Versioning of Blocks Tracks Provenance. Researchers can modify and track changes and history of experiments to understand their progress.
The header for the example study with Create revision, Prepare replication and Export design buttons and a Version menu set to Version 4. An expanded Changes from version 3 section reads that Confidence rating moved from version 1 to version 2 of its wording.
Every Saved Change Is Recorded. In this example, the researcher has changed the wording of the confidence rating in version 4, which is now forked from the prior version 3.

OpenPsy Roadmap

OpenPsy is in internal closed alpha. It has been tested with invented data only, and no real participant has taken part.

In Progress

  • A researcher can create an experiment, build it as a graph from Library blocks, edit each block's wording, and save every change as a new version.
  • Any two saved versions can be compared, and the differences are written out as sentences.
  • Library wordings are stored as exact versions, and publishing a revision leaves existing experiments on the version they used.
  • A scripted test participant with invented answers can take an authored study from consent to completion on a phone-sized screen.
  • Before the software is built, a chain of eight automatic checks runs, and any one failure stops the build.

Coming Soon

  • Scientific pilots and full data collection are being built and are not enabled yet.
  • Recruiting human participants will come after ethics and consent review and the other steps that formal data collection requires.
  • Hosted sign-in for collaborators has not been qualified yet.
  • Analysis and replication runs inside the web product have not been qualified yet.
  • Citations produced by the software are provisional for now.

Source Code

The source code, the specification and the product documentation are being prepared for publication on GitHub.

GitHub repository opening soon