Measuring the UX of AI

Back to Devoured - August 04, 2026

DEVOURED

Measuring the UX of AI

Measuring generative AI UX requires shifting from traditional usability testing to frameworks that account for model variability and user trust.

What: Jeff Sauro outlines quantitative approaches to gauging chatbot efficacy, focusing on SUS (System Usability Scale) scores, interaction success rates, and subjective trust metrics.

Why it matters: Standard UX metrics fail to capture the non-deterministic nature of AI, where the same input can yield vastly different, sometimes hallucinated or irrelevant, outputs.

Deep dive

Traditional usability metrics like 'time on task' are less meaningful when the AI's processing time fluctuates significantly.
'Interaction success' must be redefined to account for the truthfulness and relevance of generative responses, not just task completion.
Measuring user trust is critical because a chatbot that provides incorrect information with high confidence can negatively impact product perception.
Behavioral data, such as prompt refinement counts, provides insight into how users struggle with system ambiguity.
Attitudinal surveys should specifically probe the user's perception of AI capability compared to their initial expectations.

Decoder

SUS (System Usability Scale): A 10-item questionnaire used to provide a global view of subjective assessments of usability.
Non-deterministic: A property of a system where the same input does not guarantee the same output, common in stochastic LLM generation.

Original article

Generative AI chatbots' UX can be measured through a mix of behavioral and attitudinal metrics.

Explore all 45 articles in today's digest