TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types
Abstract
Dexterous teleoperation plays a crucial role in robotic manipulation for real-world data collection and remote robot control. Previous dexterous teleoperation mostly relies on hand retargeting to closely mimic human hand postures. However, these approaches may fail to fully leverage the inherent dexterity of dexterous hands, which can execute unique actions through their structural advantages compared to human hands. To address this limitation, we propose TypeTele, a type-guided dexterous teleoperation system, which enables dexterous hands to perform actions that are not constrained by human motion patterns. This is achieved by introducing dexterous manipulation types into the teleoperation system, allowing operators to employ appropriate types to complete specific tasks. To support this system, we build an extensible dexterous manipulation type library to cover comprehensive dexterous postures used in manipulation tasks. During teleoperation, we employ a MLLM (Multi-modality Large Language Model)-assisted type retrieval module to identify the most suitable manipulation type based on the specific task and operator commands. Extensive experiments of real-world teleoperation and imitation learning demonstrate that the incorporation of manipulation types significantly takes full advantage of the dexterous robot's ability to perform diverse and complex tasks with higher success rates.
Method
The overview of our system is shown in the figure. First, we construct a dexterous manipulation type library, which covers the types required for various manipulation tasks. Then, we propose a MLLM-assisted type retrieval module to select the most appropriate manipulation type based on the current task. Finally, for the teleoperation process, we design an interpolation mapping strategy to control the dexterous action of specific type by human hand motion.
Inspired by existing human grasp taxonomies that classify hand postures into distinct types to encompass most human manipulations, we design a Dexterous Manipulation Type Library, comprising diverse dexterous types to guide dexterous postures across a wide range of teleoperation tasks. The library is built upon recent taxonomies and augmented with postures specially designed for dexterous hands, which are extracted from a variety of dexterous manipulation tasks. Overall, our library is consisted with 4 sub-categories and 30 types.
Experiments
The hardware of our system involves the hand motion capture device, robot arm, dexterous hand and camera. For motion capture device in teleoperation, we employ Rokoko Gloves to capture 3 DOF position of each finger and employ the controller of Meta Quest 3 VR to capture wrist 6 DOF pose. For the robot, we employ two Kinova arms (6 DOF and 7 DOF, respectively) and two LEAP dexterous hands (16 DOF each). For vision data collection, we employ a Realsense L515 LiDAR Camera to capture a single-view RGB-D observation of the scene.
We evaluate our teleoperation system and imitation learning policy through diverse manipulation tasks. For teleoperation, we measure success rate \(Suc\), total task completion time \(T_{all}\), and average demonstration duration \(T_{single}\). For imitation learning, we assess policy performance when trained on the collected demonstrations.
Performance Demonstrations
Intresting Tasks
Comparison With Baselines
We compared our TypeTele with retargeting-based teleoperation systems(including vision-based and glove-based ones) across multiple manipulation tasks. Our approach achieved leading performance in both operation success rate and time consumption.
ARCap
AnyTeleop
TypeTele
Autonomous Policy
To evaluate the impact of teleoperation quality, we train separate policies on datasets collected from the retargeting-based and our systems, using the same number of demonstrations and policy hyper-parameters. The results demonstrate the higher quality of data collected by TypeTele. Below we present the performance of the policy trained with TypeTele-collected data.
Pick and Place
Collect and Store
Handover
Pouring from Pan
Use Scissors
Spray Water
Use a Heavy Kettle
Open a Large Box
Grasp Two Objects
Type Adjustment
Our system supports type adjustment to further enhance its versatility, while our type library can already cover most common tasks, and each type generalizes well across objects with similar geo- metric characteristics. To enable such adjustment, the system allows users to explicitly apply offsets to the position or orientation of specific fingertips.
TypeTele applied on Inspire Hand
TypeTele is applicable to various dexterous robotic hands. We conduct real-world manipulation experiments using the Inspire Hand.
Avp (Retargeting-based method)
Avp (Type strategy)
Conclusion
We believe that achieving effective teleoperation for the data collection of delicate dexterous manipulation task is important in the robotic learning communities. In this paper, we propose Typetele, a novel dexterous teleoperation system with the insight that introducing types into teleoperation. To support this system, we build a dexterous manipulation library, comprising various types required for common dexterous tasks. During the teleoperation, a MLLM-assisted type retrieval module is proposed to select the suitable type for current task. And a interpolation mapping is used to control the dexterous hand by human hand motion. The extensive experiments show that our system not only enables tasks previously unachievable by teleoperation, but also greatly improves data collection efficiency and quality, thereby enhancing imitation learning and autonomous policy performance.
BibTeX
@misc{lin2025typetelereleasingdexterityteleoperation,
title={TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types},
author={Yuhao Lin and Yi-Lin Wei and Haoran Liao and Mu Lin and Chengyi Xing and Hao Li and Dandan Zhang and Mark Cutkosky and Wei-Shi Zheng},
year={2025},
eprint={2507.01857},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2507.01857},
}