For small online shops
Refund emails,
handled automatically.
The agent finds the order, applies your policy and drafts the reply. You approve.
How it works
1ReadEmail, any language
2Look upOrder and day counts
3DecidePolicy rules R1–R9
4CheckDraft against facts
5ReviewStaff approve or escalate
Controls
Read-only toolsIdentity check in codeMax 6 stepsAbove 100 € → managerEmail = untrustedAny failure → human
Results
Task success
Share of tickets passing every check. Whiskers: 95% interval.
Staff time per ticket
Minutes, modelled from assumed step times.
Cost per 1,000 tickets, all-in
US dollars: staff time at an assumed $25/h plus model tokens.
Machine time per ticket
Seconds the model takes to produce a draft; mean over tickets.
Scale test
Where each system fails
One cell per ticket. Hover for the failed checks; click to open the ticket in the platform.
✓All checks passed✕At least one check failed?Ambiguous label (hover)
Table view
Demo video
The project
Louan Bardou · HEC Paris
The problem
Small shops answer refund emails by hand: find the order, count days, check the policy, write the reply.
The solution
An agent does the lookup and drafts the reply. A person approves. Above 100 € a manager decides.
How it works
- Read the email
- Look up the order
- Decide by the policy
- Check the draft
- Approve or escalate
Staff hours as tickets pile up
Modelled, not measured.
Pass rate as the tickets are processed
Measured on 500 tickets. Axis zoomed.
Run it
Python 3.10+ and pip install anthropic. Tests and the template baseline need no key.
git clone https://github.com/louanbardou/refund-triage-agent.git cd refund-triage-agent python3 -m unittest discover -s tests -t . # offline checks python3 run_eval.py --systems b0 # offline baseline export ANTHROPIC_API_KEY=... # live runs python3 triage.py T06 # one ticket, full trace python3 run_eval.py --systems b0,b1,b1d,agent_nr,agent --label main
Limits
- Fictional shop, author-written tickets, hand-labelled and cross-checked with a rule-based oracle.
- One small model; deterministic regex checks, no LLM judge.
- Staff time is modelled from assumptions, not measured.
- The platform replays recorded runs and calls no model.