If you follow image generation, it is easy to feel that the game is already decided: diffusion models are strong, flow matching is clean, autoregressive models are scaling, and everyone has a favorite sampler. So why should we still care about normalizing flows?
My answer is that flows have one unusually honest training objective. A flow says: I will turn a data point into noise, and I will tell you exactly how likely that data point was. No variational bound. No score-matching proxy. No likelihood estimate that needs a separate numerical procedure. The native loss is simply negative log likelihood.
$$ z = f_\theta(x), \qquad z \sim \mathcal N(0, I) $$ $$ \log p_X(x) = \log p_Z(f_\theta(x)) + \log \left|\det \frac{\partial f_\theta}{\partial x}\right|. $$This equation is the reason flows are beautiful. The first term asks whether the encoded variable looks like ordinary Gaussian noise. The second term accounts for how the transformation changes volume. If both terms are tractable, maximum likelihood training is completely direct. This is the classical promise of normalizing flows, from Rezende & Mohamed to NICE, Real NVP, and Glow.
A lot of the older NF literature can be read as different ways to pay this exactness tax. Coupling layers made the Jacobian cheap. Autoregressive flows made the density estimator expressive. Continuous flows such as FFJORD relaxed the discrete-layer view and computed likelihood through neural ODE machinery. The shared theme was always the same: keep exact likelihood, then negotiate with architecture.
By contrast, VAEs usually optimize a lower bound, and diffusion or flow-matching models are not primarily trained by exact likelihood. Those models are extremely powerful, but the likelihood is not their native, most natural supervision signal in the same way.
The same equation also explains why flows have historically been annoying. To compute the likelihood and sample from the model, the map $f_\theta$ must be invertible, and the Jacobian term must be tractable. These are not small constraints. They shape the architecture before the architecture gets to shape the data.
In image generation, this often pushes us toward coupling layers, masking, or autoregressive structure. These designs are clever, but they are not always the architecture we would choose if our only goal were to make a good image generator. In particular, the reverse direction can become causal: generate one patch or group after another, because the inverse depends on previous outputs.
This is the tension I want to keep in mind:
TARFlow was important because it made flows feel alive again. The idea is roughly: take the autoregressive flow family seriously, but parameterize it with Transformers over image patches. Instead of old flow blocks doing all the work, use the representation power of modern Transformers.
That sounds obvious in hindsight, but it mattered. TARFlow showed that a standalone normalizing flow could be competitive on image generation again, while keeping exact maximum-likelihood training. It also introduced practical tricks such as noise augmentation, post-training denoising, and guidance to improve samples.
Then STARFlow scaled this direction further: latent-space modeling, high-resolution synthesis, and text-conditioned generation, while still staying in the normalizing-flow paradigm.
| Model | Params | NFE | FID ↓ | IS ↑ |
|---|---|---|---|---|
| TARFlow-XL/8@pix | 1.3B | AR | 5.56 | - |
| STARFlow-XL/1 | 1.4B | AR | 2.40 | - |
| iTARFlow-XL/2 | 690M | AR | 4.54 | 259.3 |
| BiFlow-B/2 | 133M | 1 | 2.39 | 303.0 |
But the limitation is also clear. If your reverse process is autoregressive, generation is sequential. The model may have a beautiful likelihood, but sampling can become the slow part. More importantly for our story, the reverse architecture is still constrained by the requirement that it is the exact analytic inverse of the forward architecture.
BiFlow asks a small question that changes the design space:
What if the reverse generator does not have to be the exact analytic inverse?
We still keep a normalizing-flow forward model. The data-to-noise direction is trained with the NF objective, so we do not throw away the likelihood view. But for generation, instead of literally running the constrained inverse, we train a separate reverse model that learns to go from noise back to data.
This is also where the one-step direct sampling story belongs. The reverse model is not an iterative sampler bolted onto the end of the method. It is the method: a non-causal generator $\mathcal G_\phi$ that maps one Gaussian draw $z$ to one image $x'$ in a single forward pass.
This reverse model is not just supervised at the final image. It is aligned with the forward model's intermediate hidden states:
$$ \mathcal L_{\text{align}} = \sum_i \mathcal D\bigl(x^i, \varphi_i(h^i)\bigr) \; + \; \mathcal D(x, x'). $$The exact formula is less important than the intent. The forward flow becomes a teacher: it tells the reverse model not only what final image to produce, but also what internal route to follow. The reverse model becomes a student that is allowed to use a more flexible, non-causal architecture. The projection heads $\varphi_i$ are the small but important trick: they let us align trajectories without forcing every intermediate reverse state to live in the input-token space.
| Reverse learning strategy | Attention | FID ↓ |
|---|---|---|
| exact inverse | causal | 44.46 |
| naive distillation | bidirectional | 43.41 |
| hidden distillation | bidirectional | 55.00 |
| hidden alignment | bidirectional | 36.93 |
One mental model I find helpful is what-you-see-is-what-you-get. If the model directly outputs an image $\hat x$, then we can compare it with the real image $x$ in the most literal way. Pixel loss, perceptual loss, feature-space loss, maybe other auxiliary losses: they all become straightforward because both sides of the comparison are actual images.
This is not just a convenience. Image generation is full of objectives that are hard to express through likelihood alone. Sometimes we care about perceptual similarity, high-frequency detail, or whether the sample has the right visual semantics. A strict flow inverse gives us exactness, but not much freedom. A learned reverse model gives us a place to put these losses.
| Distance used to train the reverse model | FID w/o CFG ↓ | FID w/ CFG ↓ |
|---|---|---|
| MSE only | 31.88 | 6.79 |
| + LPIPS | 14.15 | 4.91 |
| + LPIPS + ConvNeXt features | 2.46 | 2.46 |
So BiFlow tries to keep the part of flows that I think is most principled, the NLL-trained data-to-noise model, while making the noise-to-data direction look more like the generators we actually want to use.
You can also squint and see BiFlow as an encoder-decoder system.
This analogy is not perfect. BiFlow is currently two-stage, and the forward model is still a real normalizing flow, not just a generic encoder. But the analogy is useful because it points to a bigger question: can we get the statistical cleanliness of flows and the architectural freedom of encoder-decoder generators at the same time?
The training loop is conceptually short. First train the forward NF with maximum likelihood. Then freeze it and train the reverse model as a supervised inverse: the forward model gives us paired data $(x, z)$ and a trajectory of intermediate states, while the reverse model learns to reconstruct both the trajectory and the final clean image.
# F: frozen forward normalizing flow
# G: learned reverse model
# phi: projection heads for hidden alignment
eps = randn_like(x)
x_tilde = x + noise_level * eps
xs, z = F(x_tilde) # forward trajectory and prior noise
hs, x_prime = G(z) # reverse trajectory and clean reconstruction
hs = [phi_i(h_i) for phi_i, h_i in zip(phi, hs)]
loss_align = mse(xs, hs)
loss_recon = metric(x, x_prime) # MSE / LPIPS / ConvNeXt features / ...
loss = loss_align + loss_recon
After that, sampling is exactly the thing we trained:
@torch.no_grad()
def sample(reverse_model, n, shape, device):
z = torch.randn(n, *shape, device=device)
x_hat = reverse_model(z) # one non-causal forward pass
return x_hat
This is the practical payoff. Compared with a causal reverse process, BiFlow can generate in a single model evaluation. In our measurements against iTARFlow-B/2, the generator itself is about 60x faster on GPU; even after including VAE decoding, the end-to-end per-image speedup is still about 27x. On TPU, the corresponding included-decoder speedup is 42x.
I do not think the moral is "flows beat diffusion" or anything that simple. The more interesting possibility is that these families are starting to borrow the best parts from each other.
TARFlow and STARFlow show that normalizing flows can scale much further than the older folklore suggested. BiFlow suggests that maybe the exact inverse requirement should be treated as a useful training structure, not necessarily as a permanent architectural prison.
There is also a broader recent flow-scaling line that I find interesting. STARFlow pushes latent normalizing flows to high-resolution synthesis. STARFlow2 makes the language-model connection explicit: autoregressive normalizing flows share causal masks and KV-cache structure with LLMs, so they may be a natural bridge for unified text-image generation. Normalizing Trajectory Models take another route, putting exact likelihood back into a few-step generative trajectory.
For language itself, flows have always had an awkward relationship with discreteness. Latent Normalizing Flows for Discrete Sequences is an older example I like because it says the quiet part out loud: flows want continuous variables, language is discrete, and the interesting engineering is often in the bridge between the two.
There are many directions that feel natural from here. Could the two stages become one? Could the reverse model be conditioned in richer ways? Could auxiliary losses be used more aggressively without damaging the likelihood-trained latent space? Could we design better teacher-student objectives than hidden-state matching?
I do not have final answers. What I like about this line of work is that it reopens a classical paradigm without pretending the last decade of generative modeling did not happen. Flows gave us the cleanest likelihood story. Modern image generators gave us flexible architectures and perceptual objectives. BiFlow is one attempt to put those two instincts in the same room.