Why these four?
Each answers a question a small team actually has. Can we trust it? Does it finish the work? Does checking it save time? Could it hurt us? Together they decide whether the platform gives hours back.
1. Verdict agreement
The share of alerts where the platform's verdict matches your analyst's, per category. Record disagreements with a reason, because a disagreement can be the analyst's mistake. A high rate on noisy categories matters more than a high rate on rare ones.
2. Unfinished investigations
The share of alerts the platform hands back without a verdict, or with a verdict it marks as low confidence. Each one is analyst time. Group them by cause: missing data source, unsupported alert type, or unclear logic. Missing sources can often be fixed during the pilot.
3. Review time
How long an analyst takes to check a platform verdict against its evidence, measured on a sample. If review takes nearly as long as doing the investigation, the platform helps coverage but not workload. Use the analyst hours calculator on this site to compare this against your current minutes per investigation.
4. Missed escalations
The number of alerts the platform would have closed, or acted on, that your analysts judged to be real incidents. This is the number that matters most and the one with the smallest sample, so look at every case individually.
What about vendor figures?
Vendors publish headline numbers: Mate says Gamebooks cut MTTR by 93%, Torq says over 90% of cases are remediated completely autonomously, 7AI says it has completed 14M+ investigations. These are the vendors' statements about their own customers, measured in ways you cannot see. They are not targets for your pilot. Your four numbers, on your alerts, are.
How long to measure
Two to four weeks per category is usually enough to see agreement and unfinished rates settle. Missed escalations need longer, which is one more reason to keep sign-off on high-impact actions after the pilot.
Testing beyond your own alerts
Real alerts may not include the attacks you fear most. Some teams replay past incidents or run controlled tests. Industry groups are forming around this: Dropzone AI co-founded the AI Proving Grounds Consortium in June 2026 to test AI defenses before production. Ask each vendor how it supports replaying your past incidents.
Writing it up
One page per category: the four numbers, the main disagreement reasons, and the autonomy step you recommend. That page is your buying case, and later your audit evidence.
Keep the same page format after purchase. Re-measuring the four numbers once a quarter shows whether the platform is holding up as your tools and alert mix change, and it gives you facts for the renewal conversation instead of impressions.