How it is measured
Every model gets the same establishing frame, the same prompt and the same requested length, through the same provider, on one day. Each take then runs through a fixed set of deterministic checks. Nothing in the table below uses a model to judge a model.
| Check | Tool | What the colour means |
|---|---|---|
| Length | ffprobe | Yellow if more than 0.5 s off what was asked. Veo can only produce 8 s, so it is always yellow on a 5 or 10 s input — a fact about the model, not a fault in the take. |
| Audio track | ffprobe | Red if there is none. |
| Loudness / peak | ffmpeg ebur128 | Integrated LUFS and true peak. Yellow if the peak is above −1 dBTP, which is clipping. |
| Lines heard | whisper.cpp small.en | A scripted line counts as heard when 75% of its words appear in order in the transcript. Green all, yellow half, red below. Whisper mishears; a missed line is worth listening to before believing. |
| Voice | cross-correlation | Only where a recording was supplied. About 0.9 means the output contains that recording; about 0.2 means a sound-alike. Red when a recipe supplied the audio and the model did not use it. |
| Motion | ffmpeg signalstats | Mean frame-to-frame luma difference at 192 px. Yellow under 1.0. It is a whole-frame mean, so a small object turning correctly on a plain background scores low while doing exactly what was asked, and a camera sliding over a still scene scores high while animating nothing. Read the clip. |
| Scene changes | ffmpeg scene detect, 0.3 | Informational, and flagged when a single-shot input comes back with cuts in it. |
| Attempts | generation log | Yellow if any attempt failed before one succeeded. |
| Wait | generation log | Wall clock of the successful attempt, submit to download. Yellow over ten minutes. |
| Repeatability | the checks above, repeated | The same input rendered several times on one model; the number is the spread of motion across replicates over its mean. Green under 0.15, yellow to 0.35, red above. |
| Identity across calls | open-vocabulary detection, then embeddings | Frames where the character cannot be located are rejected before scoring. The rest are cropped, embedded, and compared to that character's centroid across separate calls. Green above 0.85, yellow to 0.7, red below. |
What it cannot tell you
Whether the right character said the line. Whether a face is the face in the frame, as a person would judge it. Whether the text on the sign stayed legible to a reader rather than to a detector. Whether the paper fox still looks like paper. Those are questions of looking, and here they are answered by the paragraph under each input — labelled as one person's read — and by your own eyes on the takes.
Cost and wait
Cost is the provider's list price applied to what was requested, not an invoice. Wait is one sample per take on one day; queues move. Refusals are recorded as refusals, because a model that will not accept a photoreal human face is telling you something you need before you build on it.
Stale
Every take carries a hash of the input it was rendered from. Change the frame, the prompt or the lines and the take is marked stale until it is re-rendered, so a comparison can never quietly become a comparison of two different inputs.