What happens when the specification is wrong?
Any tool can turn a clean document into tickets. Regulated teams need to know what it does with the gaps, the contradictions and the missing numbers. So we built a test set full of them and published the results, including the runs that went badly.
Latest run (run 7, 28 October 2026), SprintReady default pipeline with the same-requirement rule added in run 6, tickets written straight to an empty Azure DevOps project. Citations: 32 of 33 correct (97.0%). No false alarms on the trap value.
The test set
Three software requirements specifications written in the style of IEC 62304 / FDA SaMD documentation. The products are fictional; the problems in them are the kind reviewers find in real specs:
- GlucoSense Bolus Advisor v1.0: insulin dosing support app.
- CardioLink ECG Viewer v1.0: clinician-facing ECG review software.
- GlucoSense v1.1: a revision of the first document, sent in afterwards to check that changed requirements update the existing tickets instead of creating duplicates.
Planted in them, with an answer key written before any run:
- 7 missing values: a maximum bolus that is never given, a retry count that is never set, a lockout threshold marked TBD, and similar. The right answer is to flag them, never to guess.
- 2 contradictions: a calculation required "within 2 seconds" in one section and "500 ms" in the performance table; data kept for 30 days in one place and 90 in another.
- A trap value: a number that is stated correctly, to check the tool does not raise false alarms.
- 7 revised requirements in v1.1, which should update the matching v1.0 tickets.
How we scored it
A script reads every ticket back from Azure DevOps and checks it against the answer key: which requirement IDs it cites, whether each planted gap appears as a "CLARIFICATION NEEDED" item, whether any number in the ticket is absent from the source, and whether each v1.1 ticket merged into the right v1.0 ticket or was created new.
Every run, not just the best one
| Measure | Run 1 | Run 2 | Run 4 | Run 5 | Run 6* | Run 7 (current) |
|---|---|---|---|---|---|---|
| Requirements covered | 22 / 27 | 27 / 27 | 27 / 27 | 27 / 27 | 27 / 27 | 27 / 27 |
| Missing values flagged | 4 / 7 | 7 / 7 | 7 / 7 | 7 / 7 | 7 / 7 | 7 / 7 |
| Contradictions flagged | 1 / 2 | 1 / 2 | 2 / 2 | 2 / 2 | 2 / 2 | 2 / 2 |
| Gaps filled with an invented value | 0 | 0 | 0 | 0 | 0 | 0 |
| Tickets with numbers not in the source | 28% | 31% | 0% | 0% | 0% | 0% |
| Citations correct | 95% | 91% | 93% | 97.5% | 94.6% | 97.0% |
| Revisions merged correctly | 2 / 7 | 1 / 7 | 7 / 7 | 7 / 7 | 3 / 7 | 7 / 7 |
Run 3 was an internal run used to find a merge bug and is not scored here. *Run 6 is not a like-for-like repeat of run 5: it ran against an Azure DevOps project that already held copies of these test tickets from an earlier run, so most v1.0 tickets were merged into those older copies, correctly, instead of being created new, and the v1.1 revisions were then checked against the older copies. The scoring assumes an empty backlog, so its duplicate figure measures the test setup more than the duplicate check. Run 7 is the same pipeline as run 6, repeated the same day on an empty project.
What we changed between runs
- Run 1 → 2: long specifications were being cut off. We raised the output limit, and coverage went to 100%.
- Run 2 → 4: revised requirements were being created as new tickets. The duplicate check now anchors on the requirement ID, so a changed GSB-REQ-004 updates the existing GSB-REQ-004 ticket, and Azure DevOps keeps the history.
- Run 2 → 4: about a third of tickets carried numbers we could not find in the source. Every ticket is now checked automatically, and any number not in the source is shown to the customer as unverified.
- Run 4 → 5: the writing rules now require values for the same requirement to be cross-checked across every section, so a conflict is reported with both values and both locations. Run 5 confirmed it: both planted contradictions were reported, including the 2 seconds vs 500 ms conflict that runs 1 and 2 missed. Citation accuracy rose from 93% to 97.5%.
- Run 5 → 6: a new writing rule: when a document states the same requirement twice, the two passages become one ticket citing both, with any difference between them flagged. Run 6 was the first run with it, against a different Azure DevOps project from run 5 (see the note under the table). Coverage, missing values, contradictions and invented numbers were unchanged; citations went from 39 of 40 to 35 of 37 correct.
- Run 6 → 7: no change to the pipeline. We removed the earlier copies of the test tickets from the Azure DevOps project and ran it again. All 7 revisions merged correctly, and citations were 32 of 33 correct (97.0%), against 39 of 40 (97.5%) in run 5: within the variation we expect between runs of a language model.
What this does not show
This is a small, synthetic test set that we wrote ourselves, and each figure comes from a single run of a language model, which can vary between runs. Citation accuracy is 97.0%, not 100%: a reviewer should still check the trace on every ticket. SprintReady drafts tickets and flags problems; it does not replace your design review or your quality system.
We will re-run this benchmark after every change to the ticket-writing pipeline and publish the results here. If you want to run it against your own documents, ask us for the scoring method.
Now tested on a real specification too. We gave SprintReady the public design document for a major Apache Kafka feature, as it stood before the work began, and compared its tickets with what the real team built: see the real-world test. Not affiliated with or endorsed by the Apache Software Foundation.