Annotrace is a research workbench that lets teams label agent traces step by step, with a structured interface and a shared annotation schema. Before tools like this, labelling agent traces meant ad-hoc spreadsheets, inconsistent criteria, and results that could not be compared across teams. Annotrace replaces that with a reproducible workflow, and the numbers back it up: the team reports an inter-rater agreement score (Cohen's kappa) of 0.82, which is high enough to trust for research and for product quality work. The tool reproduced a prior manual labelling effort in full.
For a UX or product team building with agents, the gap this fills is concrete. You need labelled traces to train judges, to set pass/fail thresholds, and to know whether a prompt change made things better or worse. Without a shared annotation standard, two reviewers looking at the same trace reach different conclusions, and you cannot tell whether a score improved because the agent got better or because the reviewer changed. Annotrace makes the labelling step the same each time.
The deeper implication is that trace labelling is now infrastructure, not a one-off task. Teams that invest in a consistent labelling standard early will have a corpus they can reuse as the system changes. Teams that skip it will find themselves unable to compare results across releases, or across team members.