Disclosure: this account is run by members of the VIDRAFT team, and the work discussed below is VIDRAFT's own result. It is not an independent review.
TL;DR
"Matches the GPU" is easy to say and hard to show. Here are five checks an evaluator should run on any such claim, applied to VKORE-325M, pretrained on a Rebellions ATOM-Max NPU and on a B200 GPU. It passes four; the fifth (seed variance) is still open.
Check 1: Is there a true twin?
The reference must differ only in hardware. Here initialization, data, batch and schedule were identical.
Check 2: Is the evaluation data really unseen?
Read the sampler, not the split table. The first evaluation used windows from the end of the training file, but training drew random windows from the whole file, so those windows were partly seen. The corrected evaluation uses FineWeb-Edu shard 013, which was never in the training corpus (shards 000 and 001).
Check 3: Is the comparison paired?
Window-to-window variance dwarfs model differences. On identical windows the paired standard error is 0.0007, versus about 0.0049 for each model's mean.
| Model | Loss, unseen (1,024 windows) |
|---|---|
| Rebellions ATOM-Max NPU | 0.7993 |
| B200 GPU | 0.7994 |
Check 4: Can you rerun it?
Both checkpoints and evaluate_unseen.py are public in the model repo. An evaluator can reproduce the table without trusting the authors.
Check 5: Is variance across seeds reported?
Not yet: one run per hardware. A stronger claim would repeat each side with several seeds.
FAQ
What would falsify the claim? A paired difference well outside two standard errors on fresh data, or failure to reproduce with the published files.
Does the claim cover speed? No, only quality.
Top comments (0)