LingBot-Depth
Transforming flawed depth sensors into precision 3D perception through masked depth modeling.
LingBot‑Depth delivers metric‑scale depth estimation in challenging scenarios where conventional sensors fail—including transparent objects, glass surfaces, and highly reflective materials. Beyond single‑frame accuracy, our model demonstrates strong temporal consistency, producing stable and coherent depth sequences for video inputs without explicit temporal modeling.
LingBot‑Depth delivers metric‑scale depth estimation in challenging scenarios where conventional sensors fail—including transparent objects, glass surfaces, and highly reflective materials. Beyond single‑frame accuracy, our model demonstrates strong temporal consistency, producing stable and coherent depth sequences for video inputs without explicit temporal modeling.
LingBot‑Depth provides a powerful foundation for downstream spatial perception tasks. By refining noisy sensor depth into clean, complete measurements, our model enables (1) accurate structured indoor scene mapping with improved camera trajectory estimation, and (2) reliable 4D point tracking for robot learning, where both static geometry and dynamic object motion are captured in metric scale.
LingBot‑Depth provides a powerful foundation for downstream spatial perception tasks. By refining noisy sensor depth into clean, complete measurements, our model enables (1) accurate structured indoor scene mapping with improved camera trajectory estimation, and (2) reliable 4D point tracking for robot learning, where both static geometry and dynamic object motion are captured in metric scale.
By jointly aligning RGB appearance and depth geometry in a unified latent space, LingBot‑Depth enables robust robotic manipulation in previously intractable scenarios. We train a diffusion‑based policy for grasp pose generation that leverages our refined depth and aligned features, achieving high success rates on transparent cups, reflective steel containers, and other challenging objects where raw sensor depth completely fails.
By jointly aligning RGB appearance and depth geometry in a unified latent space, LingBot‑Depth enables robust robotic manipulation in previously intractable scenarios. We train a diffusion‑based policy for grasp pose generation that leverages our refined depth and aligned features, achieving high success rates on transparent cups, reflective steel containers, and other challenging objects where raw sensor depth completely fails.
Existing stereo‑based depth cameras work like human eyes—two lenses capture the same scene from slightly different angles, and the system calculates depth by matching corresponding points between the two views. But this approach has a fundamental weakness: on textureless regions, transparent surfaces like glass, or highly reflective materials like mirrors and metal, the matching algorithm fails because both views look identical or distorted. The result? The camera returns blank spots exactly where you need measurements most.
Our approach, called Masked Depth Modeling, works similarly. We train the model on millions of RGB‑depth image pairs, but with a twist: we intentionally hide portions of the depth data and ask the model to predict what's missing based on the color image. Over time, the model learns the relationship between what things 'look like' and how far away they 'are'.
The key insight is that sensor failures aren't random—they happen predictably on certain materials and lighting conditions. Instead of treating these failures as noise to filter out, we treat them as learning opportunities. The model learns: "When I see this glass‑like appearance with these reflections, the depth should be approximately this value."
After training on 10 million images spanning homes, offices, gyms, and outdoor environments, LingBot‑Depth can take a corrupted depth map and fill in the gaps with accurate, physically plausible measurements.
1. Camera Depths as the Natural Mask for Masked Depth Modeling
Unlike standard masked autoencoders that randomly hide image patches, we leverage the naturally occurring "holes" in depth sensor outputs as our masking signal. These missing regions aren't random—they precisely indicate where geometric reasoning is most challenging. By training the model to reconstruct depth in these sensor‑failed regions using RGB context, we teach it to solve exactly the problems that matter in real‑world deployment.
2. Large Scale Real&Sim Data Curation
We built a scalable data curation pipeline combining real‑world captures and realistic simulations, collecting 3 million RGB‑depth pairs (2M real + 1M synthetic). Our real data spans diverse environments—homes, offices, gyms, lobbies, and outdoor scenes—captured with multiple commercial depth cameras. Our synthetic pipeline renders photorealistic scenes with simulated stereo matching artifacts, mimicking real sensor failures. For ground‑truth supervision, we compute high‑fidelity depth from stereo IR pairs (real) or perfect renders (sim), providing clean targets even where sensors typically fail.
3. Alignments of Explicitly Geometry and Implicit Feature Representations
LingBot‑Depth learns a unified latent space where RGB appearance and depth geometry are tightly coupled. The model not only outputs accurate metric depth maps (explicit geometry) but also produces aligned feature representations across modalities (implicit features). This dual alignment enables powerful downstream applications: the explicit depth feeds robotic manipulation pipelines, while the aligned features support tasks like 3D tracking and scene understanding.
LingBot‑Depth achieves state‑of‑the‑art results across multiple depth perception benchmarks and real‑world robotics applications:
Depth Completion
Outperforms Industry‑Grade Hardware
Robotic Manipulation
Foundation Model Capabilities
LingBot‑Depth enables robust 3D point tracking by providing complete, metric‑scale depth for off‑the‑shelf trackers. Our refined depth eliminates holes from sensor failures, allowing accurate camera trajectory estimation and dynamic object tracking in challenging environments with glass, mirrors, and reflective surfaces.
LingBot‑Depth transforms robotic manipulation of transparent and reflective objects—previously impossible due to sensor failures. By reconstructing accurate geometry from corrupted depth, our model enables diffusion‑based grasp policies to generate precise hand poses, achieving reli able grasps on glass cups, metal containers, and translucent boxes.