A Better Critic Matters for Distribution Matching Distillation

Overview

Abstract

Distribution Matching Distillation (DMD) trains a few-step student using the difference between a teacher's score and an online critic's estimate of the student distribution's score. We investigate how critic capacity and conditioning affect distillation. Controlled experiments across seven SiT model sizes on ImageNet show that increasing teacher size while keeping a small critic fixed can degrade student quality and cause training collapse. Scaling the critic alongside the teacher, which we call joint scaling, allows the student to benefit from larger teachers. We also introduce a step-aware critic (STAC), which conditions on the student's generation step to estimate step-specific scores. Under a pure distribution-matching objective, joint scaling and STAC together reduce the FID of a 4-step SiT-B student from 13.91 to 6.90 on ImageNet without CFG. We further validate our findings on text-to-image and text-to-video generation with SDXL, FLUX.2, and Wan2.1. Additionally, we explore critic designs and training strategies, with comprehensive results and ablations reported in the appendix.

Findings

Open questions about critic design

We study two independent questions about the online critic. Within each comparison, the deployed student keeps the same size and number of sampling steps.

How large should the critic be?

Scale it together with the teacher.

Scaling the teacher while keeping a small critic fixed can degrade the student and even collapse training. Scaling the critic alongside the teacher, which we call joint scaling, lets the same student benefit from larger teachers, and matched teacher–critic pairs offer a favorable quality–compute tradeoff.

The interactive figure needs JavaScript.

Hover or tap a point to see its run · keyboard: focus the figure, then use the arrow keys

Training compute versus FID. Best FID against the training compute of one student update for every non-collapsed teacher/critic pair of the S-student grids at (a) 1 and (b) 4 steps. Matched pairs are labelled by size and joined in size order, and the FID axis is magnified toward the lowest values. For their compute, matched pairs sit at or near the lowest FID, and most mismatched pairs lie above them. Each point is the run with the lowest FID over training from a learning-rate sweep. (Settings: ImageNet 256×256, w/o CFG; seven SiT sizes from 33M to 951M parameters.)

View the data as a table

What should the critic be conditioned on?

The critic should know the generation step.

A multi-step student produces clean predictions with different characteristics at each generation step. A standard critic sees the noise level t but not the generation step k, so it estimates the score of their mixture. Our step-aware critic (STAC) also conditions on k to estimate each step's own score, with no added inference cost.

Clean predictions from the four steps, k = 1 to 4, of the same SDXL generation: early predictions contain strong artifacts, and later predictions refine the face and the motorcycle wheel. All four images share a prompt and seed. — **Generation quality varies sharply across steps.** Clean predictions of a 4-step SDXL student along one generation path with a shared prompt and seed. The first step already fixes the overall color and layout but still carries significant artifacts; the second step refines the image substantially, yet artifacts remain in detailed regions such as the face and the wheel; the third step is near-perfect apart from residual artifacts around the mouth, which the last step removes.

Results · FLUX.2

Text-to-image results

On FLUX.2, a larger teacher alone gives little gain. Scaling the critic with the teacher, and adding STAC, improve the same 4-step 4B student.

FLUX.2 4B student · 4 steps

Teacher / critic	HPSv3 ↑	DPG-Bench ↑
4B / 4B	10.58	81.58
9B / 4B	10.32	82.08
9B / 9B	10.92	83.70
9B / 9B + STAC	11.86	84.22
32B / 32B	12.10	84.98
32B / 32B + STAC	12.32	85.28

With the 4B critic, the 9B teacher lowers HPSv3 from 10.58 to 10.32. Scaling the critic to 9B as well raises both scores, and STAC raises them further. The 32B teacher and critic give the strongest students: with STAC, the student reaches 12.32 HPSv3 and 85.28 DPG-Bench.

1024×1024. The 4B and 9B teachers and critics are FLUX.2 [klein] base; the 32B ones are FLUX.2 [dev]. Highlighted rows add STAC.

Results · Wan2.1

Text-to-video results

On Wan2.1, joint scaling to a 14B teacher and critic raises the VBench total of the same 4-step 1.3B student, and STAC adds a small further gain.

Wan2.1 1.3B student · 4 steps

Teacher / critic	Total ↑	Quality ↑	Semantic ↑	Dynamic ↑
1.3B / 1.3B	83.96	85.14	79.26	81.39
14B / 1.3B	83.94	84.73	80.79	52.06
14B / 14B	84.26	85.04	81.16	57.52
14B / 14B + STAC	84.32	85.08	81.32	58.06

VBench, 480p. Autoregressive students that share the ODE-initialized student of Causal Forcing and the Self Forcing rollout; 14B / 1.3B is the default DMD recipe of Self Forcing. The highlighted row adds STAC.

The 1.3B / 1.3B pair has the highest Dynamic Degree but the lowest semantic score, with excessive motion and visible artifacts: Dynamic Degree alone does not measure quality.
With teacher, critic, and student all fixed at 1.3B, STAC also leads in VBench total at every evaluated iteration, in both bidirectional and autoregressive training.

Start frame: silhouette of a wolf on a rock under a full moon — Still frames from videos of the 4-step Wan2.1-T2V-1.3B student. (Settings: autoregressive training with the Self Forcing rollout; 480p.)

Frame at two-thirds of the same wolf video — Still frames from videos of the 4-step Wan2.1-T2V-1.3B student. (Settings: autoregressive training with the Self Forcing rollout; 480p.)

Cite

BibTeX

@article{wu2026better,
  title   = {A Better Critic Matters for Distribution Matching Distillation},
  author  = {Wu, Zongze and Wang, Xianbang and Geng, Zhengyang and Cao, Mingdeng and
             Zhang, Zhoutong and Zhang, Richard and Shechtman, Eli and Cao, Ruiming},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}