DEVOURED
Stop blaming the model for slow AI. Use Doherty's threshold as a guideline.
Developers should rethink Doherty’s threshold for AI, splitting interactions into immediate acknowledgment and subsequent background processing to mask model latency.
What: By applying Doherty’s 400ms threshold to the system's initial UI response rather than the final AI output, designers can create a perceived performance boost even when model inference is slow.
Why it matters: Latency is a product design challenge, not just a model efficiency problem; decoupling perceived response time from actual compute time is essential for usable AI applications.
Takeaway: Design your UI to provide immediate system feedback or skeleton screens within 400ms of user input, even if the LLM inference takes several seconds.
Deep dive
- Doherty Threshold: A rule stating that human-computer interaction should have a response time of less than 400ms to keep user productivity high.
- Interaction Splitting: Decouple the request acknowledgment (UI) from the execution (LLM inference).
- Skeleton Screens: UI elements that provide a visual placeholder during data loading.
- Streaming Responses: Showing text generation in real-time to bridge the gap between intent and completion.
- Optimistic UI: Updating the interface as if the request succeeded, then handling errors if they occur.
Decoder
- Doherty Threshold: The principle that if a computer responds within 400ms, the user remains focused and productive without feeling a delay.
Original article
Generative AI changes how Doherty's 400-millisecond response threshold should be applied because a request now consists of two separate moments: acknowledging the user and producing the answer. Models may need several seconds to complete their work, but interfaces can immediately acknowledge input, show meaningful progress, and preserve the user's attention while generation continues. Designing this initial response as a strict performance requirement can make slow AI feel substantially faster without changing the model or inference speed.
Explore all 67 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi