H3DP: Triply-Hierarchical Diffusion Policy for Visuomotor Learning
Authors: Y. Lu*, Y. Tian*, Z. Yuan*, X. Wang, P. Hua, Z. Xue, H. Xu
Status: ICLR 2026 Poster.
Preprint: arXiv: 2505.07819
The Question H3DP Asks
Robotic manipulation is full of small visual facts that matter a lot. The mug is in front of the bowl, the transparent bottle is behind the shelf edge, the target object is partly occluded, and the action should first move toward the workspace before making a precise local adjustment.
Standard diffusion policies are good at modeling multi-modal action distributions, but the perception-action interface is often too flat. An RGB-D observation is compressed into one representation, and the denoising model is conditioned on that representation throughout the whole reverse process. The result can work well, but it asks one feature vector to carry geometry, semantics, global intent, and local control detail at the same time.
H3DP is our attempt to make that interface more structured. The name says there are three hierarchies: depth, visual representation, and action generation. But the way I like to think about the paper is slightly simpler:
- Depth layering: do the easy geometric discretization before the neural network has to learn it.
- Coarse-to-fine feature-conditioned actions: make the diffusion denoising schedule see visual information at the granularity it naturally needs.
Depth First
Neural networks are excellent at semantic abstraction. They can learn that a shiny curved region is a bottle, that a colored patch is probably a cup, and that a gripper is approaching an object. But metric depth is a different kind of signal. It is numerical, ordered, and strongly tied to foreground-background structure.
If we simply concatenate RGB and depth, we are quietly asking the model to learn two things at once: first, how to use depth as a real-valued measurement; second, how to turn that measurement into the discrete spatial layers that manipulation often cares about. That is possible, but it is a poor use of model capacity, especially when depth discontinuities and occlusions are exactly the places where mistakes matter.
So the first hierarchy is deliberately simple. H3DP partitions the RGB-D input into depth-aware layers. Pixels at different depth ranges are routed into different layer images, and each layer can be encoded separately. This is not meant to be a clever learned module. It is an inductive bias: give the policy an explicit foreground-background decomposition before semantic feature extraction begins.
The Diffusion Lens
The second half of the method comes from a different intuition. Sander Dieleman’s blog post Diffusion is spectral autoregression makes a useful point: diffusion can be understood as doing a soft autoregressive process in a spectral domain. In images, low-frequency structure tends to appear before high-frequency detail.
For robot actions, the same idea suggests a natural design question. If the denoising process first settles the coarse trajectory and later refines precise details, why should every denoising step receive the same visual condition?
H3DP answers by splitting the reverse diffusion process into stages. Early steps are conditioned on coarse visual features, which mostly carry global context: where the workspace is, which object is relevant, and what the rough action mode should be. Later steps receive finer features, which are better suited for contact geometry, local alignment, and small control corrections.
This is not just an analogy imported from image generation. The paper runs a DFT analysis over generated action chunks and finds the same qualitative frequency evolution: low-frequency action components emerge early, while higher-frequency components are introduced later. That observation is the reason the action hierarchy is not merely decorative. It matches how the denoising process already wants to behave.
VAR-Style Features, But For Control
The visual representation hierarchy is what connects the first and second ideas. Once we have depth layers, each layer is encoded into multi-scale visual features. The important part is that these features are not independent thumbnails. They are built in a residual, prefix-like way.
This is where the connection to VAR is useful. VAR reframes visual autoregression as coarse-to-fine next-scale prediction instead of raster-scan next-token prediction. H3DP borrows the spirit of that design, but puts it into a visuomotor policy: represent the scene as a sequence of scales, then let action generation consume those scales over diffusion time.
Concretely, the multi-scale encoder takes a full feature map, projects it to different resolutions, quantizes it, upsamples each scale back, and accumulates the reconstructed pieces as prefixes. After one scale is extracted, its contribution is subtracted from the residual feature before the next scale is formed. This residual algorithm matters because it gives each scale a job:
- The early prefixes should already explain the coarse scene.
- The later prefixes should add the missing local detail.
- The consistency loss keeps every prefix tied back to the original visual feature, so “coarse-to-fine” is supervised rather than just hoped for.
Training Picture
A compact way to describe H3DP is: first organize the observation by depth, then build progressive visual features, then let the diffusion action model use the right feature scale at the right denoising stage.
# I: RGB-D observation
# q: robot proprioception
# a_t: noisy action chunk at diffusion step t
layers = depth_layering(I)
layer_prefixes = []
for layer in layers:
residual = encoder(layer)
prefixes = []
running = 0
for scale in coarse_to_fine_scales:
piece = quantize_and_lift(residual, scale)
running = running + piece
residual = residual - piece
prefixes.append(running)
layer_prefixes.append(prefixes)
features_by_scale = transpose(layer_prefixes)
for t in reversed(diffusion_steps):
k = stage_index(t)
eps = denoiser(a_t, features_by_scale[k], q)
a_t = scheduler_step(a_t, eps, t)
The actual training objective combines the diffusion loss with the feature consistency loss. The diffusion loss teaches the policy to predict denoising noise for actions. The consistency loss pushes the multi-scale visual prefixes to remain faithful to the original feature. Together, they make the architecture’s story trainable: the model should not only have multiple scales, it should make those scales useful for staged action refinement.
Why It Helps
The payoff is clearest in cluttered or depth-sensitive scenes. A point-cloud policy such as DP3 can work very well when segmentation and depth sensing are clean, but that is a fragile assumption in real systems. H3DP operates directly on raw RGB-D input, uses depth layering to expose geometric structure, and then lets multi-scale features guide the action generator.
The ablations are useful because they separate the three pieces. Removing depth layering, hierarchical action conditioning, or multi-scale representation all hurts performance. In the main ablation suite, the full model averages 59.6, while the variants without one of the three hierarchies land around 46.5 to 49.0.
| Method | Average success | What is missing |
|---|---|---|
| H3DP | 59.6 | full hierarchy |
| w/o depth layering | 46.5 | no explicit depth planes |
| w/o hierarchical action | 49.0 | same condition across denoising |
| w/o multi-scale representation | 48.7 | no progressive visual prefixes |
| DP (w/ depth) | 42.1 | flat RGB-D conditioning |
At the larger scale, H3DP obtains a $+27.5\%$ average relative improvement across 44 simulation tasks. In the real-world evaluation, it also improves over Diffusion Policy by $+32.3\%$ on four bimanual manipulation tasks, including cluttered and long-horizon setups such as cleaning a fridge, pouring juice, placing a bottle, and sweeping trash.
Loose Thoughts
What I like about H3DP is that it does not try to solve everything by making the policy larger. The paper instead asks where structure should enter the system.
Depth layering says: some geometric distinctions are cheap and reliable enough to impose before learning. Multi-scale feature construction says: representation should have an order, not only a resolution. Hierarchical action conditioning says: diffusion time is meaningful, so the condition should change as denoising moves from coarse action structure to fine action detail.
The broader lesson is that visuomotor learning is not only about better perception or better action models in isolation. The important part is the coupling between the two. H3DP tries to make that coupling explicit: depth organizes what the model sees, residual multi-scale features organize what the model knows, and diffusion stages organize how the model acts.
