Robots are increasingly integrated into industrial and daily settings, but navigating complex indoor environments remains a challenge. Traditional systems rely on multiple, rule-based modules for localization and path planning, often struggling in repetitive or feature-scarce spaces.
ByteDance has developed Astra, a dual-model architecture designed to overcome these limitations. Detailed in the paper "Astra: Toward General-Purpose Mobile Robots via Hierarchical Multimodal Learning," the system follows the System 1/System 2 paradigm.[reference:1] It features two primary sub-models:
- Astra-Global: A multimodal large language model (MLLM) that handles low-frequency tasks like self-localization and target localization. It uses a hybrid topological-semantic graph as contextual input, allowing it to locate positions based on query images or text prompts.[reference:2]
- Astra-Local: A multi-task network that manages high-frequency tasks such as local path planning and odometry estimation.[reference:3]
Astra-Global's localization capabilities are built on a coarse-to-fine two-stage process for visual-language localization. The coarse stage detects landmarks and filters candidates, while the fine stage samples reference map nodes to predict the pose.[reference:4] For target localization, the model interprets natural language instructions, identifies relevant landmarks, and retrieves target images and 6-DoF poses.
Training Astra-Global involved supervised fine-tuning (SFT) on diverse datasets and a GRPO phase with a rule-based reward function. Experiments showed that GRPO significantly improved zero-shot generalization, achieving 99.9% localization accuracy in unseen home environments, surpassing SFT-only methods.
Astra-Local comprises three core components: a 4D spatio-temporal encoder, a planning head, and an odometry head. The 4D encoder processes omnidirectional images through a Vision Transformer and Lift-Splat-Shoot to convert 2D features into 3D voxel features, then predicts future states.[reference:5] The planning head generates executable trajectories using Transformer-based flow matching and incorporates a masked Euclidean Signed Distance Field (ESDF) loss to reduce collisions. The odometry head fuses data from multiple sensors (IMU, wheel data) using a Transformer to predict relative pose.
Extensive experiments in warehouses, offices, and homes validated Astra's performance. Astra-Global demonstrated significant advantages over traditional Visual Place Recognition (VPR) methods, capturing fine details like room numbers and maintaining stable localization even with large camera angle changes. It achieved over 30% improvement in pose accuracy in warehouse environments. Astra-Local's planning head outperformed methods like ACT and diffusion policies in collision rate and overall score on out-of-distribution datasets. Integrating IMU data reduced overall trajectory error to approximately 2%.
Astra holds promise for deployment in complex indoor environments like shopping malls, hospitals, and libraries. Future work includes improving map compression methods, adding active exploration mechanisms for robust localization, and integrating instruction-following capabilities for more natural human-robot interaction.