Slow first token: latency worth optimizing

What it means

A slow first token usually means upstream queueing or a distant route. Perceived speed depends on it more than on throughput.

Typical symptom:

首字 5 秒以上，之后输出正常

The first token takes more than 5 seconds; after that, output is normal. Nothing errors, so this is a performance problem rather than a failure.

Common causes

How to confirm

Measure time to first token, not only total time. Compare the same prompt across models and sites. If all models are slow, the link is the likely cause. If only large models are slow, the model is the cause. If slowness concentrates in peak hours, queueing is the cause.

Fix

On your side:

1. Choose a low-latency site (use measured data).
2. Shorten the system prompt.
3. Enable streaming so users see text sooner.

On the service side:

1. The site needs to optimize routing and concurrency.

If it still fails

First-token latency affects perceived performance more than total generation time. A fast total time does not help if the user stares at an empty screen for the first several seconds. When choosing a site, prioritize this metric over throughput.