Persona: Senior Research Scientist specializing in Deep Learning Architectures and 3D Reconstruction.
Abstract:
This presentation introduces FAST-HMR (an acronym derived from the proposed methods: Fast Acceleration via Sparse Transformation), a novel framework designed to significantly accelerate Human Mesh Recovery (HMR) by addressing two primary computational redundancies in existing transformer-based HMR pipelines.
The first redundancy—the excessive depth of transformer stacks (e.g., 32 layers in ViT-Huge variants)—is mitigated using Error Constraint Layer Merging (ECLM). ECLM iteratively merges adjacent layers by averaging their weights, provided the resulting increase in mean positional error remains below a predefined threshold ($\tau \approx 0.1 \text{ mm}$).
The second redundancy involves the inefficient processing of background tokens alongside information-rich person tokens. This is resolved via Mass-Guided Token Merging (MassToMy). This operation selectively masks similarity comparisons between person-to-person tokens (to preserve accuracy) and prioritizes merging background tokens based on high token similarity scores, thereby reducing the computational load within the initial transformer blocks.
To maintain or enhance output accuracy despite these speed optimizations, FAST-HMR integrates a lightweight diffusion decoder utilizing a single denoising step, replacing traditional linear heads. This decoder is pre-trained using a VAE on large-scale mocap data to enforce physically plausible pose priors. Quantitatively, FAST-HMR demonstrates up to $2.3\times$ speedup over HMR2 and $1.9\times$ speedup over Camera-HMR baselines while achieving slight performance enhancements on the 3DPW and AMDP benchmarks, notably achieving this using only one noise sample per inference step.
Reviewing Groups and Summary:
The primary audience for this work comprises Deep Learning Engineers focused on 3D Computer Vision, Machine Learning Systems Architects, and Researchers specializing in Efficient Neural Network Inference.
FAST-HMR: Accelerating Human Mesh Recovery via Layer and Token Optimization
00:00:01 Introduction & Problem Definition: The presentation introduces FAST-HMR, a model accelerating Human Mesh Recovery (HMR) through layer merging, token merging, and diffusion decoding.
00:00:09 Computational Redundancies: Current transformer-based HMR methods suffer from two inefficiencies: (1) large stacks of transformer layers (e.g., 32 layers in ViT-Huge) where adjacent layers show high kernel similarity, and (2) processing information-rich person tokens identically to less informative background tokens across all layers.
00:00:49 ECLM (Error Constraint Layer Merging): Addresses layer redundancy. The process calculates the baseline mean positional error, then iteratively tests merging adjacent layers (by averaging weights) starting from the end of the stack. Merging is accepted only if the error increase is below a threshold ($\tau = 0.1 \text{ mm}$).
00:01:53 Mass-Guided Token Merging (MassToMy): Addresses token redundancy by merging background tokens in the initial transformer blocks.
00:02:33 Token Merging Mechanism: Similarity comparison matrices are masked by replacing person-to-person interaction scores with negative infinity. This forces the merging operation to prioritize high-similarity background tokens over information-rich person tokens.
00:03:39 Lightweight Diffusion Decoder: To compensate for accuracy loss from merging, a single-step diffusion decoder replaces the baseline linear head for pose output, using a VAE pre-trained on mocap data to impose strong, physically plausible priors.
00:04:08 Training Strategy: The diffusion decoder is only substituted for the pose output head, while shape and camera parameters still utilize the pre-trained linear heads. Velocity prediction is used instead of standard noise prediction loss during this phase.
00:04:24 Performance Metrics: FAST-HMR achieves a $2.3\times$ speedup on the HMR2 baseline and $1.9\times$ on the Camera-HMR baseline.
00:04:34 Accuracy Enhancement: The method slightly enhances performance on the 3DPW and AMDP benchmarks compared to the baselines.
00:04:41 Inference Efficiency: The proposed method is highly efficient, utilizing a single denoising step and only one noise sample at inference time, contrasting favorably with methods like Score Hypo which require many samples.