Plann3r: Predicting Planning Costs Grounded in 3D

Abstract

Planning paths or trajectories for robot navigation requires understanding scene geometry and traversability. Classic approaches relied on accurate 3D maps to define occupancy-based planning costs. Learning-based alternatives predict planning costs either in terms of distance to goal or temporal distance between images. The former overfits to scene layout and the latter lacks geometric understanding. Most of these methods estimate image-level scalar costs, which are not sufficient to guide the robot. We propose Plann3r, a 3D-grounded method that predicts pixel-level planning costs in terms of geodesic distances for any given set of images with an arbitrary subgoal pixel. We integrate Plann3r within a navigation pipeline, VGGT-Nav, in two ways. The offline mapping cum global-planning phase iteratively uses Plann3r to generate subgoals and reference-image global costmaps. In the execution phase, Plann3r performs simultaneous localization and local planning to generate planning costmaps that directly condition a learnt control policy. Plann3r and VGGT-Nav outperform baselines on the planning and navigation tasks of an existing benchmark, and we show real-world demonstrations of sim-to-real deployment.

Key ideas

What's new

Plann3r predicts planning costs as pixel-level geodesic distances from a query image to an arbitrary subgoal pixel within a submap. It uses a pre-trained VGGT backbone with a learnable goal token and a lightweight MLP cost decoder.
VGGT-Nav is a single-camera visual navigation pipeline with a 3D-grounded backend for mapping, localization, and planning. It improves control prediction and navigation success rates across the four benchmark tasks.
Simultaneous localization and planning. The query costmap arises from query-to-submap cross-attention, which removes the need for explicit pairwise feature matching that is more prone to failure.

Method

Plann3r architecture

Plann3r builds on a pre-trained VGGT model, fine-tuned to predict geometrically consistent planning costs as geodesic distances from an image pixel to a (sub)goal pixel within a submap of images. A learnable goal token designates the subgoal, and a custom MLP head projects each patch embedding to a scalar cost to produce a coarse H/P x W/P costmap.

The VGGT-Nav pipeline

VGGT-Nav pipeline: offline mapping/global planning and online localization/local planning.

The same Plann3r module runs iteratively in two modes. Offline, it walks backward through overlapping submaps and propagates low-cost connectors from the goal through the reference traverse, which produces precomputed reference costmaps. Online, the robot localizes from a monocular RGB frame, retrieves a local submap, predicts a query costmap, and converts it into egocentric waypoints for the controller.

Offline: mapping and global planning

Functional map of costs

From the goal pixel, Plann3r predicts local costmaps over overlapping submaps and recursively back-propagates a cost-to-go offset through the traverse. The map stores each pixel's planning cost toward the goal directly, instead of geometry or a graph.

Online: localization and local planning

Cost-conditioned control

The query image, submap, and a subgoal pixel pass through Plann3r together. Global attention localizes the query against the submap and the MLP head emits a dense query costmap. A learnt controller (inherited from ObjectReact) turns this costmap into 5 future BEV waypoints.

Why geodesic

Dense geodesic costmaps

Blue regions are cheap because they sit on navigable space; red regions become expensive when the robot must route around walls, furniture, or missing connectivity. The cost is geodesic rather than Euclidean image distance, so it reflects what the robot can actually reach. When the goal is outside the room, the near side of the wall correctly receives a high cost in both the ground-truth and Plann3r costmaps, whereas a 3D-Euclidean baseline assigns it a low cost and ObjectReact misses the gradient.

Qualitative comparison of planning costmaps: Ground Truth, Plann3r, Euclidean, ObjectReact.

Qualitative comparison of planning costmaps. The goal lies outside the room. Plann3r reproduces the ground-truth geodesic structure; 3D-Euclidean distance and ObjectReact do not.

Try it

Demo

Pick a scene and goal, then scrub the traverse to see Plann3r's predicted geodesic costmap per frame. Blue is cheap to reach the goal, red is expensive. We also ran the same model on Sekiro, video game it never saw during training, with no fine-tuning, which shows the planner generalizes beyond its indoor training scenes.

Cost to goal along the traverse

Overlay

Loading demo data...

Each goal is a pixel selected in one reference frame (white-and-red marker). The costmaps come straight from Plann3r's offline propagation pass over this scene. Global normalizes color across the whole traverse (absolute cost to goal); relative normalizes each frame on its own, which brings out local structure even far from the goal.

Demonstrations

Mapping and execution in action

Real recordings from the VGGT-Nav pipeline.

Offline mapping

Mapping to an alternate goal

Costs propagate through overlapping local submaps; the global cost field follows the floor plan instead of cutting through obstacles.

Execution view

Query, submap, costmap, and rollout

The live query, retrieved submap, and predicted costmap run alongside the real-world robot rollout toward the alternate goal.

Offline mapping

Mapping to the final goal

The same functional-map representation transfers to a different target with no change to the module.

Real-world rollout

Reaching the alternate goal

Closed-loop, cost-conditioned waypoints drive the physical robot to the alternate goal.

Results

Navigation performance

Method	Imitate		Alt Goal		Shortcut		Reverse		Average
Method	SPL	SSPL	SPL	SSPL	SPL	SSPL	SPL	SSPL	SPL	SSPL
GNM	78.79	82.95	8.70	15.44	15.38	31.74	3.33	6.11	26.55	34.56
GNM (HM3D)	81.82	86.38	0.00	10.91	15.38	24.57	13.28	20.77	27.62	35.66
PixNav	42.42	46.75	26.09	31.66	7.69	22.29	16.16	25.56	23.09	31.57
RoboHop	57.56	64.99	30.43	38.23	30.77	40.87	9.98	16.92	32.19	40.25
ObjectReact	60.60	68.51	21.74	26.68	23.08	39.64	30.00	42.01	33.36	44.71
GT Costs + Control (Ours)	64.24	81.76	61.15	67.36	30.67	44.20	9.12	28.81	41.29	55.53
Plann3r + Control (Ours)	78.67	89.71	58.98	67.43	45.10	65.83	9.20	22.34	47.98	61.32

State-of-the-art comparison on four navigation tasks (HM3D / IIN-val). We report SPL and Soft-SPL. Boldface marks the best score per column; the two shaded rows are ours. Plann3r conditions control on dense predicted geodesic costmaps and reaches the best average SPL and SSPL, along with the best scores on Alt Goal and Shortcut. The GT-cost ablation shows that most of the remaining gap is in control and subgoal selection rather than cost prediction.

Costmap quality

Costmap Source	All Pixels Below Threshold					Within Cost Range
Costmap Source	5%	15%	30%	50%	100%	0-5%	5-15%	15-30%	30-50%	50-100%
Euclidean Distance	2.284	2.533	2.617	2.561	2.232	2.236	2.403	2.412	2.296	2.645
ObjectReact	3.650	3.600	3.483	3.260	2.837	3.731	3.312	2.975	2.619	3.304
Plann3r (Ours)	2.398	2.424	2.369	2.299	1.986	2.387	2.262	2.091	2.043	2.318

Mean Absolute Rank Difference (MARD) between ground-truth and predicted costs (lower is better). Euclidean distance is most accurate only in the immediate low-cost region around the subgoal; Plann3r is consistently better once reasoning extends beyond nearby space.

Qualitative navigation overview

Qualitative overview of the navigation pipeline: submap and query costmaps, subgoal/goal pixels, predicted waypoints, and a top-down view.

Qualitative overview of the navigation pipeline. The query costmaps remain geodesically consistent with the submap costmaps and correctly direct the robot at both timesteps, regardless of whether the final goal is visible in the current image.

BibTeX

@inproceedings{plann3r2026,
  title     = {Plann3r: Predicting Planning Costs Grounded in 3D},
  author    = {Anonymous Authors},
  booktitle = {Under review at the Conference on Robot Learning (CoRL)},
  year      = {2026},
  note      = {Project page: https://plann3r.github.io/}
}