Design a repeatable test for a legal AI tool
For legal operations teams and researchers comparing AI-assisted workflows.
What should be fixed before comparing two tools on a legal task?
Fix the task, the test materials, the scoring rule and the decision you will make from the results. Record the version and settings for each tool. Measure how often the workflow gives a usable answer and what happens when the source cannot support one.
Write a test another person could fail
Pick one job with a checkable output, such as finding a notice address and showing the page that supports it. Specify what counts as correct when a document has multiple addresses, an amendment changes one or the relevant clause is missing. Decide in advance which mistakes stop a pilot even if its overall score looks good.
Build a small development set for trying prompts and a separate held-back set for the final comparison. Keep near-identical contracts or amendments from the same family together so a copy in one set cannot reveal the answer to the other. Synthetic or public materials can cover the failure cases without bringing client records into an early test.
Keep the comparison fair
Give both tools the same documents, instructions, permissions and time limits. Record the actual prompt, document versions, model or product identifier, run date, settings and tools that were available. If the provider does not expose a version or randomness control, record that limit. Save the outputs under neutral labels before scoring.
Agree on score fields in advance. A correct address with the wrong page may fail the evidence requirement. Log omitted answers, invented answers and justified refusals separately. Have a second person independently score uncertain examples, then document why the reviewers resolved a disagreement.
Report what will happen if you rerun it
Repeat a defined subset under the same conditions and note answer changes. Include the counts, denominators and examples of the most serious errors. Compare against the current human workflow or another meaningful baseline if that was part of the question. Retain the test materials and rubric where another authorized reviewer can reproduce the comparison.
Worked example
Illustrative scorecard. In 24 held-back synthetic agreements, 18 have a notice address and six intentionally omit it. A tool correctly cites 16 of the 18 addresses. It correctly declines on five of the six omissions and invents an address on the sixth. That is 21 successful cases out of 24, plus one invented address. A preset rule that any invented address stops unsupervised use still applies.
Primary sources
Related work at Agentic Attorney
This evaluation exercise is part of the shared Agentic Attorney resource collection. The main site describes AI consulting and implementation for legal teams. Explore Agentic Attorney’s approach to AI implementation.