Can AI Detect Usability Problems? (7 minute read)

AI tools like ChatGPT can now analyze usability test videos to identify UX problems, but the approach is inconsistent and its reliability compared to human researchers remains unproven.

What: Researchers tested whether AI can detect usability issues by feeding ChatGPT a six-minute OpenTable usability test video, which identified seven specific problems including a search field bug that dropped cuisine selections when users changed location.

Why it matters: This could potentially automate or augment the time-consuming process of analyzing usability test sessions, though the "lossy" nature of video sampling and transcription raises questions about whether AI catches the same issues human researchers would flag.

Decoder

Lossy: In this context, information is lost when AI samples video frames rather than processing every frame continuously, potentially missing important user interactions or subtle usability issues between sampled moments.

Original article

AI can analyze usability test videos by sampling frames, transcribing audio, and generating plausible descriptions of user behavior — though this process is "lossy" and prone to inconsistency. In a test using a six-minute OpenTable usability session, ChatGPT identified seven specific problems, including a search field that dropped cuisine selections when the location was changed and results that surfaced non-sushi restaurants. Whether those findings hold up in terms of accuracy, reliability, and alignment with what human researchers would catch remains an open question to be addressed in follow-up research.