Learning Heterogeneous Tactile Representation for General Dexterous Manipulation
Bridging heterogeneous tactile signals and real-world dexterous manipulation
Overview
LingBot-Touch bridges heterogeneous tactile sensor signals and real-world dexterous robotic manipulation, enabling tactile experience to transfer across hardware platforms and tasks. It is a JEPA-based tactile representation model built around a shared Haptic Transformer and latent representation prediction from spatiotemporal tactile context, pretrained on nearly 5,000 hours of piezoresistive, magnetic and visuotactile recordings from tactile gloves and robot hands.
A canonical geometric interface and modality-specific tokenizers feed the shared Haptic Transformer, which learns unified, reusable representations of contact physics across sensors and heterogeneous layouts. LingBot-Touch thus enables a rich understanding of spatiotemporal tactile signals across sensing modalities while supporting high-frequency inference and cross-task transfer.
Unified
Heterogeneous tactile data from multiple sources mutually enhance representation learning within a unified latent space.
Transferable
Even when pretrained on entirely different tactile sensors, LingBot-Touch transfers to unseen tactile sensors and real-world dexterous robotic hands.
Efficient
Even with its smallest backbone, LingBot-Touch supports high-frequency downstream inference with only 5.46M parameters in the HaT-Tiny configuration.
Scalable
Scaling the amount of tactile training data yields continued performance improvements on both downstream tactile tasks and real-world robotic manipulation tasks.
Touch at Scale
The pretraining corpus comprises 4,983 hours of recordings, 464,431 interaction episodes and roughly 1.33 billion frames from piezoresistive, magnetic and visuotactile sensors.
01Real-World Manipulation
Plug-and-play gains on 26 real-world dexterous tasks
Mean success rate over all 26 tasks on both dexterous hands, 20 trials per task.
Spotlight: contact-rich deployments
These tasks where success hinges highly on touch — feeling a key seat in the lock, metering toothpaste by pressure, and sealing a bag along its track. Shown as qualitative deployment demos of LingBot-VLA + LingBot-Touch: real-time inference on real robots, played at 1× speed.
Benchmark tasks
The 26 tasks behind the success rates above, each evaluated over 20 trials: policies post-trained with frozen LingBot-Touch features on Sharpa Wave and XHand, covering tactile-guided fine manipulation, fragile and deformable object handling, and bimanual coordination.
All clips are real-time inference on real robots, played at 1× speed
02Simulated Reinforcement Learning
More efficient and stable policy learning in simulation
Learned policies in simulation
Reinforcement-learning policies observing LingBot-Touch latents, ordered as in the tech report (a–l).
03Scaling
Performance keeps improving with data
LingBot-VLA mean success rate on real-world tasks as the tactile pretraining corpus grows from 0 to 5K hours.
04Contact Visualization
Recorded tactile signals (top) and probe read-outs from the frozen encoder: force reconstruction, contact area estimation and future force prediction for fingertips.