3D-ViTac: placing touch in three-dimensional space
A spatial view of where contact happens, beyond adding a tactile image.
INTERACT ROBOTICS · Research Notes · Reviewed September 7, 2026
Where touch happens gives it meaning.
Fingertip pressure changes need to be interpreted alongside where the object is held. 3D-ViTac combines visual point clouds and touch in a 3D representation to learn fine manipulation policies.
Full-task success by representation
| Input representation | Cooking an egg | Collecting hex keys |
|---|---|---|
| RGB only | 50% | 45% |
| Visual point cloud only | 55% | 65% |
| Point cloud + tactile image | 70% | 50% |
| Point cloud + tactile points / Proposed method | 85% | 90% |
Original paper, Table 1 / 30 demonstrations and 20 evaluations per task. Full-task success is distinguished from success at individual stages.
What 90 versus 45 doesn’t tell you.
The RGB baseline and proposed method differ in visual representation as well as touch. To assess tactile contribution, also read the visual point-cloud comparison. The result supports this representation under these conditions; it does not show that all tactile-image encoders are ineffective.
Space and physics, together.
Our next question is how to represent force direction, change, slip, and their relationship to action alongside contact location. What matters is preserving the physical information a task needs. Spatial alignment and sensor calibration are as important as representation learning.
References
- 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing ↗
2024 / Table 1. Two of the external study’s four tasks, presented with the same baselines.