What this is for
Two AI tools have read the same lower-court order and each recommended whether the department should appeal it. We want to know which answer an expert prefers, so we can check whether our automatic scoring agrees with you.
Dashboard
Blind comparison of AI outputs on customs appeal cases.
Your progress
0 of 0 pairs judged
- Remaining
- 0
- With a note
- 0
Annotators
How it works
- Read the case. The lower-court order under review.
- Compare both outputs. Two AI tools answered it, and you won't know which is which.
- Pick the better one. Choose the answer you would rather receive. Add a note if it was close.
Output judgement
Read the case, compare the two outputs, then pick the one you would rather get from a tool.
No pairs have been imported yet.
All pairs judged. You can still review or change any pick. See all judgments
Case under review
Codes in the outputs:
Output 1
Output 2
Classification
Tariff code
Reasoning
Appeal decision
Recommendation
Reasoning
Sources
Cited sources
My judgments
Every pair in the queue. Open one to review it or change your pick.
| # | Case | Overall pick | Parts compared | Note | Updated |
|---|
Guidelines
What you're doing here and how to judge a pair.
How to judge a pair
- Read the case under review. Use find, or the codes the outputs propose, to check what the order says about them. Collapse it once you're done to make room.
- Compare the outputs part by part: the tariff code and its reasoning, the appeal recommendation and its reasoning, and the sources cited. Under each part, mark which output is better or that they are about the same. These are optional and save as you click; click a choice again to clear it.
- Make the overall pick: the output you would rather receive from a tool if you were deciding this appeal. This is what completes the pair.
Keep in mind
- Judge by your own reading of the case. Don't score the answers against the rubric.
- Outputs are shown in random order, and you won't be told which tool wrote which.
- There's no tie. If both are weak, pick the less wrong one and say why in the note.
- Notes are optional, but they help most when the choice was close or hinged on one point.
- You can change any pick later from My judgments.
Keyboard shortcuts
- 1
- Output 1 is better
- 2
- Output 2 is better
- ← →
- Previous or next pair
- /
- Find in the case; Enter and Shift Enter step through matches
- Esc
- Leave the note or find field so the shortcuts work again