Does the giver’s tone change confidence?
Vary the task giver’s optimism and compare the handler’s self-rated confidence with what the evaluator scores.
Task × giver optimism × handler trait
Traitrun
Factorial experiments on AI agents
Build a flow of agents, vary their traits and tasks as factors, launch a run across every combination, and read the effects with confidence intervals.
Start with $5 of free credit. No card needed.
Your lab / Experiments / Feedback tone and self-confidence
Feedback tone and self-confidence
V4 · DraftFlowRun designRunsReviewResults
Flow valid525 conditions · 2,100 iterations
How it works
From a flow to findings in five steps
Every experiment follows the same loop, one tab per step.
1 Flow
Build a flow of agents
On a canvas, a task giver hands tasks to a task handler and an evaluator scores each transcript against your rubric. Traits, short prompt texts such as Big Five levels, attach to the agents.
- Traits at five intensities, with a no-text control
- Any trait or task set can become a factor
- See each agent’s compiled prompt
FactorOptimism
Trait on the task giver · 5 levels
- 1Very lowYou expect the task to go badly and say so; you doubt the answers you get.
- 2LowYou are sceptical and point out what might be wrong.
- 3NeutralNo trait text: the control
- 4HighYou are encouraging and expect a good result.
- 5Very highYou are enthusiastic and certain the solver will succeed.
2 Run design
Vary traits and tasks as factors
Traitrun crosses every factor’s levels into conditions and repeats each one. Before you launch, you see the design size and the run’s quote.
- Full factorial or a seeded sample
- A 25-iteration pilot first, if you like
- A quote before every launch
Design size5 × 5 × 21
525 conditions
× 4 repetitions = 2,100 iterations
Quote $4.34 · 86.8% of your $5.00 credit≈ 13.7M tokens, based on Run 13 Runs
Launch a run
Iterations run in parallel on Claude Haiku 5.5. The agents talk, the handler answers a short questionnaire about itself, and the evaluator scores the transcript twice.
- Live progress and transcripts
- Pause or stop at any time
- Charged actual usage, never more than the quote
Run 2RunningFull run · flow v4 · seed 47211,344 of 2,100 iterations
64%
Elapsed4 min 12 sCost so far$1.85Retried3 · 0.2%Flagged314 Review
Rate transcripts blind
Invite people to rate a sample of transcripts with the same rubric, without seeing the condition or the judge’s scores.
- Overlap between raters where you want it
- Agreement as Cohen’s κ and Krippendorff’s α
- Disagreements and flags settled in one place
Task giverAre you sure? Check your answer again.
Task handlerI re-checked every crossing. FINAL ANSWER: 7 crossings
Condition hidden from the raterJudge agreement κ = 0.71
5 Results
Read the results
One regression with every factor at once shows what moves each outcome, with 95% confidence intervals, charts and how well the judge agrees with people.
- Accuracy, confidence and calibration gap
- Effects split by task
- Every iteration as CSV, with a pandas snippet
What moves self-rated confidence
One model with every factor at once. Bars are 95% confidence intervals.
Effect−8.30+8.3Per step · 95% CIGiver optimism+6.1 [+5.8, +6.5]Openness+1.9 [+1.2, +2.5]Conscientiousness+1.2 [+0.5, +1.9]Agreeableness+0.3 [−0.4, +1.0]
Use cases
Questions you can answer
Start from a template or a blank canvas. Any design with agents, traits and tasks fits.
Personality × task difficulty
Give the handler one Big Five trait at five intensities and see which traits keep it working on hard tasks.
Big Five trait × task difficulty
Can you trust the AI judge?
Score every transcript twice with an AI judge, rate a blind sample yourselves and check agreement with Cohen’s κ and Krippendorff’s α.
AI judge × blind human ratings
Built for research
Reproducible
A seed fixes the plan, and every run freezes its flow, design and library.
Blind by design
Raters never see conditions, trait texts or the judge’s scores.
Priced per run
See the quote before you launch. Pay actual usage, never more.
One model for all factors
Effects come from one regression with every factor at once.
Pricing
Pay per run, with credit included
A flat fee for your lab with usage credit included, and a price for every run before it starts.
Free trial
$0
$5 of usage credit to start
No card needed
Lab plan · monthly
$20 / month
Includes $10 of usage each month
Cancel any time
Lab plan · yearly
$200 / year
Includes $100 of usage each year
17% less than paying monthly
Top-ups
$20 · $50 · $100
Each adds the same amount of credit
One-off payments, whenever you need them
Usage is billed at model token cost + 10%. Every run shows its price before launch and is never charged more than that quote. Credit doesn’t expire.
FAQ
Questions and answers
What is a factorial experiment on AI agents?
You choose a few factors, such as the task giver’s optimism, a Big Five trait of the task handler and the task itself. Traitrun runs every combination of their levels several times and estimates how much each factor changes the outcome, with 95% confidence intervals.
Which model do the agents use?
Every agent runs on Claude Haiku 5.5. You set each agent’s instructions, its traits and its effort: low, medium or high.
What do I need to start?
Just an account. A new lab comes with $5 of credit and a sample library: Big Five and other traits, a set of logic puzzles, a scoring rubric and an end questionnaire. Templates get you to a first run in minutes.
Can people review the transcripts?
Yes. Start a rating round and invite raters to your lab. They score transcripts blind, without conditions, trait texts or the judge’s scores, and Results shows how well the AI judge agrees with them.
Can I analyse the data myself?
Yes. Results exports every iteration as CSV, with a pandas snippet to load it. You can also connect an AI assistant such as Claude to Traitrun over MCP with an API token.
Who can see my lab’s data?
Only the members of your lab. Every experiment, run and library item belongs to a lab, and raters see only the transcripts they are asked to rate.
Run your first experiment
Your lab starts with $5 of credit and a sample library: Big Five traits, logic puzzles, a scoring rubric and an end questionnaire.