VGGT vs. VGGT-Ω (VGGT-Omega): A Complete Guide to Feed-Forward 3D Reconstruction

Most 3D reconstruction systems take minutes or hours per scene, running expensive optimization loops that assume the world holds still. Feed-forward reconstruction flips this model: train a single transformer once, then reconstruct any scene (static or dynamic) in one forward pass, in under a second. No bundle adjustment, no per-scene optimization, no rigidity assumption.

VGGT (Visual Geometry Grounded Transformer) won the Best Paper Award at CVPR 2025 as demonstrating that a single feed-forward transformer could match or surpass classical pipelines like COLMAP on standard benchmarks, while running orders of magnitude faster. Less than a year after that award, its successor arrived: VGGT-Ω (VGGT-Omega), itself a CVPR 2026 oral and one of the conference’s Best Paper Finalists. It scales the idea significantly, training on 15× more supervised data and 18 Million Self Supervised Videos, while using about 30% of VGGT’s memory. It also introduces register attention, supports dynamic scenes, and reports that reconstruction quality follows approximately power-law curves as model and data size grow. The Ω in the name is a personal touch from lead author Jianyuan Wang; it marks his final PhD paper.

In this blog we walk through how both models work, what changed between them and why, and then benchmark them head-to-head on the same 4K UHD video running on an NVIDIA RTX 5090, comparing inference speed, GPU memory, depth quality, scaling behavior, and visual reconstruction output.

Table of Contents

1. Introduction: What Is Feed-Forward 3D Reconstruction?
2. VGGT: The Breakthrough (CVPR 2025 Best Paper)
3. VGGT-Ω: What Changed and Why (CVPR 2026)
4. Architecture Comparison
5. Head-to-Head Benchmark: 4K UHD Video
6. Inference Speed Comparison
7. GPU Memory, Scalability, and the bf16 Discovery
8. Depth and Reconstruction Quality
9. The Visual Quality Trade-Off: An Honest Assessment
10. Scaling Behavior: Power Laws
11. Beyond Reconstruction: Registers
12. Quick Start: Running VGGT-Ω
13. Controlled Four-Way Comparison (Matched Conditions)
14. Practical Recommendations
15. Conclusion
16. References
17. Frequently Asked Questions

1. Introduction: What Is Feed-Forward 3D Reconstruction?

Given a set of ordinary 2D images, 3D reconstruction recovers two things: the 3D geometry of the scene (how far away every surface is) and the camera parameters for each frame – its pose/extrinsics (where the camera was and how it was oriented), and its lens/intrinsics (focal length, optical center, and similar). These are deeply coupled: knowing one helps you solve the other, yet in practice you begin with neither.

The Classical Approach

The classical pipeline (COLMAP) works by detecting features, matching them across images, then running bundle adjustment, a large nonlinear optimization that jointly refines all cameras and 3D points. It is accurate (sub-degree camera angles) but slow (minutes to hours), brittle to low-texture regions, and assumes a rigid, static world.

The Feed-Forward Alternative

Feed-forward reconstruction replaces that entire pipeline with a single neural network. The network learns geometric priors from data and applies them at inference in under a second. It is faster, more robust, handles dynamic scenes, and (as VGGT-Ω demonstrates) learns representations useful far beyond reconstruction.

Key Concepts

• Camera extrinsics, position (translation) and orientation (rotation quaternions) in the world.
• Camera intrinsics, lens properties, chiefly field of view (2-vector: horizontal, vertical).
• Depth map, per-pixel distance from camera to surface (H×W, continuous values).
• Point map, per-pixel 3D world coordinates; derivable from depth + camera via unprojection.
• Parallax, apparent shift of objects as the camera translates; the signal that gives you depth.

Related Topics:

2. VGGT: The Breakthrough (CVPR 2025 Best Paper)

VGGT is a ~ 1.2B-parameter transformer from Oxford / Meta AI that processes multiple images jointly and predicts cameras, depth, 3D points, and tracking features in one forward pass.

Architecture

Tokenization: DINOv2 ViT (patch size 14) produces a grid of patch tokens per image. Per-frame camera tokens and register tokens are appended.

Alternating attention: Frame attention (within one image) alternates with global attention (across all images). The model handles any number of frames with no frame-index embeddings.

Four output headsdecode the shared backbone tokens:

Head	Output	Time (35f)	Notes
Camera	Rotation + translation + FOV	0.0071s	Nearly free
Depth	Per-pixel depth + confidence	0.1118s	DPT decoder
Point	Per-pixel 3D coordinates + conf	0.1116s	Direct (X,Y,Z) prediction
Track	2D point tracks across frames	0.2406s	Iterative refinement, heaviest

The shared backbone (aggregator) costs 0.8413 s for 35 frames. Backbone + track head = 82.46% of the 1.312 s total forward pass.

3. VGGT-Ω: What Changed and Why (CVPR 2026)

VGGT-Ω asks: does feed-forward reconstruction scale like language models? To answer this, it needed to train far bigger models on far more data, which required making the architecture dramatically cheaper.

1. A Leaner Architecture

Register attention: 25% of global-attention layers are replaced with a cheaper layer where only the 16 register tokens per frame attend across images. Saves ~23% FLOPs and ~16% backbone memory with no accuracy loss.

Lightweight depth head: The high-resolution conv layers in the DPT decoder are the hidden memory monsters. It has huge activations, but very few parameters. VGGT-Ω replaces them with an MLP + pixel-shuffle operator.

Single dense head + multi-task losses: Only one dense head (depth) and one sparse head (cameras) remain. Point maps and matching are still supervised via losses but have no output heads at inference.

2. Massively More Data (15× Supervised, 18 Million Self-supervised Videos)

A 6-stage annotation pipeline processes ~40M raw internet videos, retaining ~0.8M with clean camera + depth labels. A teacher–student self-supervised stage adds 18M unlabeled videos.

3. Dynamic Scene Support

Predicting only depth + cameras (no motion masks or dynamic outputs) keeps camera and scene motion separable. This unlocks internet video (where nearly everything moves) as training data.

4. Registers as a Reusable Representation

The registers carry near-semantic scene information. Frozen scene tokens improve a VLA robotics model (LIBERO: 97.1% → 98.5% success rate), and a learnable language token reading only the registers achieves 76.8% top-1 scene-to-text retrieval.

4. Architecture Comparison

Aspect	VGGT	VGGT-Ω
Backbone	DINOv2, patch 14	DINOv3, patch 16
Parameters	1.257 B	1.144 B
Cross-frame mechanism	Global attention only	Global + register attention (25% replaced)
Registers	Present (auxiliary, discarded)	16/frame, reused for VLA + language
Dense output heads	4 DPT heads	1 depth head (MLP + pixel-shuffle) + 1 camera
Camera prediction	Iterative refinement	Single pass
Outputs at inference	Depth, points, tracks, cameras	Depth + cameras only
Dynamic scenes	Not designed for	Fully supported
Self-supervision	None	Teacher–student on 18M videos
Training data	~260K sequences	~4M
Training memory	Baseline	~30% of VGGT
Model sizes	1B only	200M / 500M / 1B / 10B
Inference precision	bf16 autocast (backbone)	bf16 autocast (backbone); heads fp32
Weight storage	fp32 (~5.1 GB)	fp32 default (4.575 GB); bf16 option (2.28 GB)

5. Head-to-Head Benchmark: 4K UHD Video

Both models were run on the exact same video, on the same GPU, with inference times measured using torch.cuda.synchronize() before and after each forward pass.

1. Test Setup

Property	Value
Video	3840×2160 UHD 4K, H.264, 29.97 fps, 35.5 s, 1065 frames, 100.8 MB
GPU	NVIDIA RTX 5090 (Blackwell, sm_120, 32 GB VRAM, 33.669 GB as torch reports it in decimal GB)
CUDA	12.8 (cu128)
PyTorch	VGGT: 2.11.0+cu128 | VGGT-Ω: 2.8.0+cu128
System	134.8 GB RAM, 32 CPU cores, Linux 6.8.0

2. How Each Model Handles 4K Input

Important: 4K resolution gives neither model extra detail at default settings

Both models downscale 4K to their internal resolution before the network sees it. VGGT center-crops to 294×518 (~152K pixels/frame, ~777 patch tokens). VGGT-Ω scales to 384×688 (~264K pixels/frame, ~1,032 tokens) preserving the 16:9 aspect ratio. A 1080p input would produce the same internal resolution. 4K only makes frame decoding more expensive.

	VGGT	VGGT-Ω
Internal resolution	294×518 (center-crop)	384×688 (aspect-preserving)
Pixels per frame	152,292	264,192
Patch tokens per frame	~777 (patch 14)	~1,032 (patch 16)

6. Inference Speed Comparison

1. At 35 Frames (Both Models’ Full Reconstruction Run)

Metric	VGGT(default)	VGGT (bf16 weights)	VGGT-Ω (default)	VGGT-Ω (bf16 weights)
Frames	35	35	35	35
Neural inference (s)	1.312	1.257	1.286	1.178
Per-frame latency (s)	0.037	0.036	0.037	0.034
Pixels processed per frame	152K	152K	264K (+73%)	264K (+73%)
Heads run	camera_head, depth_head, point_head, track_head	camera_head, depth_head, point_head, track_head	camera_head, depth_head	camera_head, depth_head
Peak GPU Memory (GB)	13.430	8.807	10.699	6.567

The headline finding

Both models already run bf16 compute in their backbones, so the precision is matched. At default weights the two models take almost the same time at 35 frames (VGGT 1.312 s vs VGGT-Ω 1.286 s) while VGGT-Ω processes 73% more pixels per frame, its extra pixel cost offset by its eliminated point and track heads, and VGGT-Ω already uses 20% less GPU memory (10.699 GB vs 13.430 GB). Casting VGGT-Ω’s backbone to bf16 with a one-line change with 0.22% depth difference, and an option equally available to VGGT, drops it further to 1.178 s and 6.567 GB. (VGGT runs on PyTorch 2.11.0 and VGGT-Ω on 2.8.0 here, each pinned by its public repo, see 13 for a version-matched rerun in a single shared harness.)

2. At Lower Frame Counts (Where VGGT-Ω Pulls Ahead)

At lower frame counts (the practical sweet spot for interactive use) VGGT-Ω is significantly faster:

VGGT (s)	VGGT-Ω (s)	VGGT-Ω Advantage
0.287 (5f)	0.095 (4f)	~2.4× per-frame (57.5 vs 23.6 ms)
0.394 (10f)	0.192 (8f)	~1.6× per-frame (39.4 vs 24.0 ms)

3. Where VGGT’s Time Goes: Per-Head Breakdown (35 Frames)

Component	Time (s)	% of Total
Aggregator (backbone)	0.8413	64.1%
Camera head	0.0071	0.5%
Depth head	0.1118	8.5%
Point head	0.1116	8.5%
Track head	0.2406	18.3%
Total	1.312	100%

The track head is the key

VGGT’s track head (0.2406 s) plus point head (0.1116 s) together cost 0.3522 s, time that VGGT-Ω simply doesn’t spend, because it has no point or track heads. At low frame counts this savings dominates; at high frame counts the quadratic attention on VGGT-Ω’s larger token count catches up, producing the convergence seen at 35 frames.

7. GPU Memory, Scalability, and the bf16 Discovery

1. Practical frame ceiling (32 GB)

Metric	VGGT	VGGT(bf16 weights)	VGGT-Ω (default)	VGGT-Ω (bf16 weights)
Practical frame ceiling (32 GB)	177 frames	189 frames	280 frames	331 frames

Key takeaway

With bf16 backbone weights, VGGT-Ω’s peak GPU memory drops from 10.699 GB to 6.567 GB at essentially zero quality cost (0.22% mean depth difference, 0 NaN), and its frame ceiling rises from ~280 to ~331. At matched default weights, VGGT-Ω already uses 20% less memory than VGGT (10.699 vs 13.430 GB) and handles ~280 frames to VGGT’s ~177. Enable bf16 with one line: model.aggregator.bfloat16() after loading weights.

2. VGGT Frame-Count Scaling (294×518)

Frames	Inference (s)	ms/frame	fps	Peak GPU Memory (GB)
5	0.287	57.5	17.4	8.827
10	0.394	39.4	25.4	9.212
20	0.714	35.7	28.0	10.297
40	1.527	38.2	26.2	14.133

3. VGGT-Ω Frame-Count Scaling (384×688)

Frames	Inference (s)	ms/frame	fps	Peak GPU Memory (GB)
1	0.036	35.8	28.0	6.830
4	0.095	23.6	42.4	8.203
8	0.192	24.0	41.7	8.633
12	0.313	26.1	38.3	8.834
25	0.827	33.1	30.2	9.863
50	2.029	40.6	24.6	11.998
100	6.094	60.9	16.4	16.497
150	12.085	80.6	12.4	21.172
200	19.690	98.5	10.2	24.518

VGGT-Ω hits the 32 GB limit at 280 frames (directly measured, not extrapolated); the table here runs to 200 frames, at 24.518 GB. Time grows super-linearly, from ~N¹·³ at low frame counts toward ~N¹·⁷ (the quadratic cross-frame attention) at high frame counts. Throughput peaks around 4 frames (~42 fps). One note on comparing the two tables above: 7.2 and 7.3 report GPU Memory totals (which include the ~2.5–3 GB CUDA context, pytorch’s reserved cache and driver allocation).

4. The bf16 Precision Discovery

An important finding from our benchmarking: VGGT-Ω already runs its backbone (79% of parameters) in bf16 via torch.autocast by default, so the compute precision was always matched with VGGT. Only the weight storage was fp32, keeping an unnecessary full-precision copy of weights that are immediately cast to bf16 for every MatMul.

Converting the backbone weights to bf16 (one line: model.aggregator.bfloat16(), keeping the camera/depth heads in fp32) produces dramatic savings at negligible quality cost:

Metric	fp32 weights (default)	bf16 backbone weights	Δ
Model Weights	4.575 GB	2.288 GB	−49.98%
Peak GPU Memory (35 frames)	10.699 GB	6.567 GB	−38.62%
Inference (35 frames)	1.286 s	1.178 s	1.092× faster
Depth vs fp32	—		mean Δ 0.0021 m (0.22%)

Verdict: Because the MatMuls already run in bf16 under autocast, storing the backbone weights in bf16 just drops the redundant fp32 master copy. That cuts peak memory by more than a third at no measurable quality cost: 98.8% of pixels stay within 1% of fp32. Casting the whole model to bf16 (model.bfloat16(), heads included) also holds up: in the controlled four-way comparison (13) it produced sub-0.3% mean depth error and sub-0.06° mean camera drift for both models, with no NaN, and the smallest memory footprint of the four configurations. The backbone-only cast is the conservative default; full bf16 is the most memory-efficient option, with long-trajectory and large-scene behavior still worth spot-checking.

8. Depth and Reconstruction Quality

1. Depth Statistics (35 Frames)

Metric	VGGT	VGGT-Ω
Depth type	float32	float32
Depth range	0.53 – 2.45 m	0.225 – 5.78 m
Depth median	~0.88 m	~0.63 m
Depth mean	1.074 m	1.088 m
Confidence mean	4.277	16.894 (high)
Confidence max	12.496	57.221
Scene type	point cloud + per-frame camera frustums	point cloud + per-frame camera frustums

Both models identify this as a shallow, close-range scene. VGGT-Ω’s depth scale appears roughly metric for this clip and it reports substantially higher confidence mean i.e. 16.894 with a max of nearly 57, indicating the model is very sure about most pixels.

2. Reconstruction Output

Property	VGGT	VGGT-Ω
Frames	35	35
Candidates (all pixels)	35×294×518 = 5,330,220	35×384×688 = 9,246,720
After 50% confidence	2,665,110	4,623,360
After cap	1,500,000 (capped)	1,500,000 (capped)
Output file	31 MB gltf	32.05 MB gltf
Inference time	1.312s	1.286s (default) / 1.178 s (bf16 wt)

3. Integrating BA from COLMAP (VGGT Only, 12 Frames)

Mode	Time	3D Points	Reproj. Error	Notes
Feedforward	22.9 s	100,000 (capped)	—	conf_thres=1.5 (default 5.0 over-filters)
With BA	70.1 s	29,548 (tracked)	1.48 → 0.25 px	Converged in 11 iters (~1.5 s for BA solver)

Section 8.3 pairs VGGT with COLMAP, the classical Structure-from-Motion pipeline, to see how much a refinement step sharpens VGGT’s feed-forward output. COLMAP relies on bundle adjustment (BA), the optimization that jointly nudges every camera pose and 3D point to minimize reprojection error, the pixel gap between where each 3D point projects back into an image and where its feature was actually observed. The slow part of classical SfM is solving this from nothing, but VGGT’s predicted cameras hand BA a strong pose initialization, so it mostly has to polish rather than solve from scratch, while LightGlue supplies the cross-frame feature tracks BA optimizes over.

The payoff is in the table: BA cut reprojection error from 1.48 to 0.25 pixels (~5.9×) and converged in just 11 iterations, with the solver itself costing only ~1.5 s while most of the 70.1 s went to LightGlue feature tracking (~47 s) on this naturally SfM-friendly scene.

9. The Visual Quality Trade-Off: An Honest Assessment

From our earlier matched comparison on the Neuschwanstein Castle clip, we found that VGGT consistently produces a visually denser and more spatially coherent point cloud, even with matched hyperparameters. This carries over to the 4K benchmark, and the cause is structural.

Root cause is architectural: VGGT has a dedicated point-map head: a full DPT neural network trained specifically to predict 3D world coordinates (X, Y, Z) per pixel. As a result it learns to produce spatially smooth surfaces directly. VGGT-Ω instead derives 3D points by unprojecting depth through the predicted cameras, a two-step chain where small errors in either depth or camera get multiplied and amplified into 3D position noise.

Increasing max_points doesn’t help because the issue is per-point spatial quality, not point quantity. VGGT-Ω dropped the dedicated point head as part of the architectural simplification (single dense head + MLP/pixel-shuffle depth decoder + register attention) that together cut training memory by ~70%.

10. Scaling Behavior: Power Laws

VGGT-Ω’s central scientific contribution is demonstrating that feed-forward reconstruction scales predictably:

Axis	Range	Point Error (lower = better)
Model size	0.2B → 10B parameters	0.107 → 0.046
Data size	~2K → ~2M sequences	0.275 → 0.073

Both curves follow approximately power-law trajectories; the paper is careful to say their shape suggests a power law rather than to assert one definitively. Making the architecture cheap enough to train is what made this study possible. The frame-count scaling we measured on the RTX 5090 fits VGGT-Ω’s design goals: it handles ~280 frames where VGGT tops out at ~177 on this aspect ratio (rising to ~331 with bf16 backbone weights), and memory grows linearly rather than running away.

11. Beyond Reconstruction: Registers

VGGT-Ω’s register tokens, the 16 scene tokens per frame, carry cross-frame information in the 25% of layers where register attention replaces global attention (the other 75% stay full global attention). Without any explicit supervision, these tokens turn out to encode rich, near-semantic scene information.

Robotics

Frozen scene tokens concatenated with OpenVLA-OFT’s inputs improve LIBERO success rate from 97.1% to 98.5% across all task categories. No fine-tuning of VGGT-Ω needed.

Language Alignment

A learnable language token reading only the registers (never image patches) achieves 76.8% top-1 / 97.0% top-3 scene-to-text retrieval after just 10K iterations. Even zero-shot transfer to a text-only LLM embedding yields 47.5% top-1. This is the capability shipped in the released VGGT-Omega-1B-256-Text-Alignment checkpoint, the text-aligned model the FAQ refers to, enabled via VGGTOmega(enable_alignment=True).

12. Quick Start: Running VGGT-Ω

The official VGGT-Ω release ships two checkpoints, a minimal inference API, and a Gradio demo. The two models VGGT-Omega-1B-512 and VGGT-Omega-1B-256-Text-Alignment can be downloaded from this link. The checkpoints are gated and licensed CC-BY-NC-4.0 (research/non-commercial use only), so you’ll need to request access before downloading, and commercial use isn’t covered by this license. The snippets below reproduce the project README so the workflow benchmarked above is easy to follow.

Clone and Install

git clone [email protected]:facebookresearch/vggt-omega.git
cd vggt-omega
pip install -r requirements.txt
pip install -e .

Run the Model

A few lines of code load a checkpoint, preprocess the input images, and read back cameras, depth, and the register tokens:

import torch

from vggt_omega.models import VGGTOmega
from vggt_omega.utils.load_fn import load_and_preprocess_images
from vggt_omega.utils.pose_enc import encoding_to_camera

checkpoint_path = "path/to/vggt_omega_1b_512.pt"
image_names = ["path/to/imageA.png", "path/to/imageB.png", "path/to/imageC.png"]

model = VGGTOmega().to("cuda").eval()
model.load_state_dict(torch.load(checkpoint_path, map_location="cpu"))

images = load_and_preprocess_images(image_names, image_resolution=512).to("cuda")

with torch.inference_mode():
    predictions = model(images)

extrinsics, intrinsics = encoding_to_camera(
    predictions["pose_enc"],
    predictions["images"].shape[-2:],
)

depth = predictions["depth"]
depth_conf = predictions["depth_conf"]
camera_and_register_tokens = predictions["camera_and_register_tokens"]
camera_tokens = camera_and_register_tokens[:, :, :1]
registers = camera_and_register_tokens[:, :, 1:]

For the text-aligned checkpoint, use VGGTOmega(enable_alignment=True) with image_resolution=256 and read predictions[“text_alignment_embedding”].

Interactive Demo

Install the demo dependencies:

pip install -r requirements_demo.txt

Launch the Gradio demo with a local checkpoint path:

python demo_gradio.py \
    --checkpoint checkpoints/VGGT-Omega-1B-512/model.pt \
    --image-resolution 512

The demo accepts uploaded images or a video, runs camera and depth inference, and visualizes the depth-unprojected point cloud and predicted cameras as a GLB scene.

13. Controlled Four-Way Comparison (Matched Conditions)

Earlier sections compared the two models across runs that differed in frame count, torch version, and which heads were active. To remove those confounds, all four configurations (VGGT and VGGT-Ω, each in its shipping mixed precision and in full bf16) now run inside a single shared harness on the same 35 frames, with GPU clocks locked and TF32/cuDNN settings matched across all four. Torch is pinned at 2.8.0+cu128 (CUDA 12.8, the same toolchain used throughout). The only differences left between runs are the model itself and the bf16 cast.

What’s held constant

Held identical across all four runs: the same 4K source video and the same 35 uniformly-sampled frame PNGs (seed 0), a 50% confidence threshold, a 1.5M-point cap, depth-unprojection reconstruction, the glTF export, the timing protocol (1 warm-up + 3 timed + 1 captured pass), locked GPU clocks, and matched TF32/cuDNN flags, all on one RTX 5090 under torch 2.8.0+cu128.

Four properties are intrinsic to the two models and cannot be equalized: input resolution (VGGT 518 → 294×518 at patch 14; VGGT-Ω 512 → 384×688 at patch 16), parameter count (VGGT 1.26B vs VGGT-Ω 1.14B), head architecture (VGGT’s separate depth head vs VGGT-Ω’s single dense head), and the weights themselves (facebook/VGGT-1B vs a local VGGT-Ω checkpoint). Precision labels: mixed is the shipping pipeline (bf16-autocast aggregator + fp32 geometry heads); bf16 casts the whole model to bf16, including the output heads. Because the heads are cast too, the memory figures in this section match the backbone-only bf16 numbers reported earlier in the post (for instance, VGGT-Ω bf16 shows 6.567 GB Memory in both).

Time

Metric	VGGT mixed	VGGT bf16	VGGT-Ω mixed	VGGT-Ω bf16
Forward total (s)	1.312	1.257	1.286	1.178
— aggregator (backbone, s)	0.8413	0.8669	1.1884	1.1249
— camera head (s)	0.0071	0.0031	0.0059	0.0020
— depth / dense head (s)	0.1118	0.0741	0.0913	0.0511
—point head	0.1116	0.0729	-NA	-NA
—track head	0.2406	0.2400	-NA	-NA

Mean of the timed passes. bf16 speedup over each model’s own mixed baseline: VGGT 1.04×, VGGT-Ω 1.09×. The aggregator barely moves (it already runs bf16 under autocast) so the gain comes from the heads (camera head ~2.3–3.0×, depth/dense head ~1.5–1.8× faster in bf16).

Accuracy (precision robustness)

Metric (bf16 vs that model’s fp32-head reference)	VGGT	VGGT-Ω
Camera rotation, mean (°)	0.0494	0.0579
Camera rotation, max (°)	0.1938	0.2374
Camera translation L2, mean	0.001	0.00052
Depth rel-error, mean (%)	0.220	0.232
Depth rel-error, median (%)	0.148	0.159
Depth MAE	0.00276	0.00290
Point drift, mean (% of scene extent)	0.194	0.128
Point drift, p95 (world units)	0.0113	0.0127

This measures how far bf16 deviates from each model’s own fp32-head output, precision robustness, not absolute accuracy. Without ground-truth geometry for this clip, absolute cross-architecture accuracy can’t be graded, which is why the figures are framed per-architecture.

Bottom line

bf16-only wins on resources for both models, ~34–39% lower peak GPU memory, for a ~0.2–0.3% depth error and sub-0.06° mean camera drift. The glTFs are visually indistinguishable from their mixed-precision references.

VGGT-Ω has the slower backbone here (1.19 s aggregator vs VGGT’s 0.84 s), because it processes a higher-resolution grid (688 vs 518 wide). The higher resolution makes the backbone slower without using more memory: VGGT-Ω has fewer parameters and a single output head, so its dense head is faster than VGGT’s depth head, and in bf16 it has the smallest footprint of the four (6.567 GB Memory).

Speedup is modest (1.04–1.09×) because the aggregator dominates the forward pass and already ran bf16 in the mixed baseline; the precision change mainly accelerates the heads (~1.5–2×). The bf16 runs even fall below full GPU utilization, confirming they are no longer compute-bound.

Note on the matched task:both models here run only their shared depth and camera outputs. Earlier in the post VGGT-Ω looked faster at low frame counts, but that let VGGT also run its point and tracking heads, which slowed it down. With the heads matched, VGGT’s lighter, lower-resolution backbone is the faster one, and VGGT-Ω’s advantage on this task is its smaller memory use and better scaling.

14. Practical Recommendations

For VGGT Users

• Input aspect ratio matters more than source resolution. This 16:9 4K clip costs less per frame (~777 tokens) than a 1:1 portrait clip (~1369 tokens). Budget frames accordingly.
• Use conf_thres ~1.5 for square-padded clips, the default 5.0 over-filters because padding lowers depth confidence.
• Frame ceiling on 32 GB: ~177 for 16:9, ~85 for square crops.
• COLMAP BA is fast on SfM-friendly scenes (11 iters, 0.25 px error) but can hit iteration caps on fast-motion vertical shorts.

For VGGT-Ω Users

• Enable bf16 backbone weights immediately
• Don’t pay for 4K on the model. At image_resolution=512 the 4K source is downscaled to 384×688. Feed 1080p or lower, the network result is identical, but frame decode is much faster.
• Sweet spot is ≤ ~50 frames for interactive use (≤ 2 s). Use more for coverage, but note the super-linear time growth.
• For large clips, chunk frames (≤ 100/segment) and keep the model resident.

15. Conclusion

VGGT established that a single transformer could replace entire SfM pipelines. VGGT-Ω shows this approach scales predictably, bigger models and more data yield reliably better reconstruction, following the same power-law curves that drove the language-model revolution.

Our head-to-head benchmark on the same 4K video makes the trade-offs concrete. Because both backbones already compute in bf16, the models are precision-matched going in, which is what makes the rest of the comparison meaningful. Run each model’s full default reconstruction at 35 frames and the two land within a few hundredths of a second of each other, even though VGGT-Ω’s backbone is processing substantially more pixels every frame; it makes up the difference by skipping the point and track heads entirely, and it still holds a real memory advantage over VGGT, using roughly a fifth less at default.

That advantage widens considerably once the backbone weights are stored in bf16 rather than fp32, a change that costs essentially nothing in output quality. Drop the frame count toward the interactive range and VGGT-Ω’s edge widens sharply, to roughly 2× faster, since it isn’t carrying VGGT’s point and track heads at all; restrict both models to only the outputs they share (depth and cameras), though, and the comparison flips, VGGT’s lighter, lower-resolution backbone comes out slightly ahead. The practical upshot for hardware headroom: at default settings a 32 GB card holds roughly a hundred more VGGT-Ω frames than VGGT, and storing the backbone weights in bf16 stretches that ceiling further still.

Visually, VGGT produces a denser point cloud thanks to its dedicated point-map head. Quantitatively, VGGT-Ω scores higher on benchmarks (camera and depth accuracy), handles dynamic scenes, and yields a representation (the registers) reusable for robotics and language. On the paper’s own headline numbers the gap is large: it reports camera accuracy improving on the prior best by 77% on Sintel, at roughly 50× the speed of optimization-based methods like MegaSaM. Both can be true at once: if you need the cleanest, densest point cloud, use VGGT; for speed, memory headroom, dynamic scenes, or reusable geometry features, VGGT-Ω is the better default.

16. References

VGGT Paper:Wang et al., “VGGT: Visual Geometry Grounded Transformer,” CVPR 2025 (Best Paper Award).

VGGT-Ω Paper:Wang et al., “VGGT-Ω,” CVPR 2026 (oral; Best Paper Award candidate); arXiv:2605.15195, May 2026.

Project page:https://vggt-omega.github.io/

VGGT GitHub:https://github.com/facebookresearch/vggt

VGGT-Ω GitHub:https://github.com/facebookresearch/vggt-omega

VGGT-Ω demo (Hugging Face):https://huggingface.co/spaces/facebook/vggt-omega

VGGT-Ω checkpoints (gated, request access – CC-BY-NC-4.0):https://huggingface.co/facebook/VGGT-Omega

Hardware:NVIDIA RTX 5090 (sm_120), CUDA 12.8, 32 GB (33.669 GB in torch’s decimal reporting). Driver 595.71.05.

Frequently Asked Questions

Post Views: 28