UniTac-VAE

A Depth-Aligned Latent Space for Sensor-Agnostic Tactile Representation

Jian Hou, Adam J. Spiers
Manipulation and Touch Lab, Imperial College London

IEEE Robotics and Automation Letters (RA-L), 2026

Paper Video Code Dataset BibTeX

The same touch looks completely different to every tactile sensor. UniTac-VAE maps all of them into one latent space, grounded in the geometry of the contact.

One space, four sensorsVision-based (GelSight Mini, DIGIT) and taxel-based (PapillArray, uSPa 46) sensors share one latent space.
Modular by designEach sensor is trained only against depth maps, so a new sensor is added without retraining the others.
Train on geometry, run on touchA model trained only on synthetic depth maps runs zero-shot on real sensors.

Abstract

Tactile sensing enables great benefits for robotic manipulation, yet the algorithms used to interpret tactile-sensor data often fail to generalize across different sensor types because of incompatible signal formats and distributions. We present UniTac-VAE, a unified tactile representation framework that aligns heterogeneous tactile modalities to a shared, geometry-grounded latent space using local contact depth maps as a sensor-agnostic common language. UniTac-VAE first trains a depth-map Variational Autoencoder (VAE) to define a target latent distribution. Each tactile sensor is then modeled by a sensor-specific VAE that jointly reconstructs its native signals and aligns its latent distribution to the paired depth-map latent distribution via a frozen depth encoder. This avoids sensor-to-sensor paired data and allows new sensors to be added without retraining existing modules.

Benchmarking across vision-based and taxel-based sensors shows that UniTac-VAE maintains high reconstruction accuracy while establishing a more sensor-agnostic latent space. We further validate the representation on a downstream shape-recognition and peg insertion task, where a single model trained on synthetic depth maps is deployed across vision-based and taxel-based sensors, operating on real-world tactile inputs. We have open-sourced the collected dataset, pairing each contact across four commercial sensors with a depth map derived from YCB meshes.

Video

Method

Whatever the transduction mechanism, every deformation-based tactile signal starts as the sensor surface deforming around a contact. UniTac-VAE uses the local contact depth map as the common language.

System overview: a pre-trained depth-map VAE defines the latent space; each sensor has its own VAE aligned to it; downstream models trained on depth maps run on real tactile input.

System overview (paper Fig. 1).

  1. Depth-map VAE. A VAE trained on 8,000 synthetic 50 × 50 contact depth maps defines a 64-dimensional, geometry-grounded latent space.
  2. Sensor-specific VAEs. Each sensor gets its own VAE. It reconstructs the sensor's native signal while a KL term aligns its latent distribution with the frozen depth encoder's output for the paired depth map.
  3. Any sensor in, any modality out. Encode with the source sensor's encoder and decode with any other decoder: sensor-to-sensor translation, depth estimation from touch, and depth-to-touch synthesis.

Because each sensor is only ever paired with depth maps, no sensor-to-sensor paired data is needed, and a new sensor plugs in without retraining the existing modules.

Training: (A) depth-map VAE pretraining; (B) sensor-specific VAE trained with reconstruction loss and KL alignment to the frozen depth encoder.

Training (paper Fig. 2). (A) Depth-map VAE pretraining. (B) Sensor-specific VAE: L1 reconstruction of its own signal plus KL alignment to the frozen depth encoder.

Dataset

Four commercial tactile sensors press the same contact points on 3D-printed replicas of four YCB objects. Every press is paired with a contact depth map computed from the object mesh and the robot pose.

4
commercial sensors
4
YCB object replicas
618
contact points
1,854
paired presses per sensor
3
force levels (5, 7.5, 10 N)
8,000
synthetic depth maps
GelSight Mini
GelSight Mini
vision · 18.6 × 14.3 mm
DIGIT
DIGIT
vision · 19.0 × 16.0 mm
Contactile PapillArray
Contactile PapillArray
taxel (optical), 9 pillars · 24 × 24 mm
XELA uSkin uSPa 46
XELA uSkin uSPa 46
taxel (magnetic), 24 taxels · 30 × 40 mm

Data collection (20× speed). Each sensor in turn is mounted on a UR5e and pressed normally onto the replicas at a 10 mm grid of contact points, at three approach angles, up to 10 N.

(A) The YCB objects; (B) the 3D-printed replicas with mounting plates.

(A) Windex bottle, hammer, scissors and large marker from the YCB set. (B) 1:1 PLA replicas printed from the high-resolution YCB meshes, with mounting bases.

(A) Contact points and approach angles; (B) depth map extraction from the TCP pose and object point cloud.

From robot pose to depth map (paper Fig. 4). The local object geometry under the sensor is cropped from the mesh (40 × 40 mm) and rasterised into a 50 × 50 contact depth map.

Download the dataset

Translate touch between sensors

Pick an object, click one of the marked contact points, and choose the sensor that measured. UniTac-VAE encodes that one signal and decodes it into every other sensor or depth map.

Object
Source sensor
Contact depth map
Pressed at
Drag
Depth map: 0 to 4 mm indentation Taxels: size and colour = normal force, arrow = shear

Touch a CAD model with a sensor that is not there

A virtual sensor slides on the scissors. At every step the contact depth map is rendered from the depthmap, and UniTac-VAE generate it into all four sensors reading.

Drag on the scissors
Depth map
rendered from the mesh
GelSight Mini
decoded
DIGIT
decoded
PapillArray
decoded
uSPa 46
decoded

Five modalities, one space

Each point is the latent code of one held-out press as encoded by one modality, projected to 2-D with t-SNE. In a sensor-agnostic space the colours mix instead of forming one cluster per sensor.

Colour by

BibTeX

@article{hou2026unitacvae,
  title   = {UniTac-VAE: A Depth-Aligned Latent Space for Sensor-Agnostic Tactile Representation},
  author  = {Hou, Jian and Spiers, Adam J.},
  journal = {IEEE Robotics and Automation Letters},
  volume  = {11},
  number  = {10},
  pages   = {11545--11552},
  year    = {2026},
  doi     = {10.1109/LRA.2026.3726336}
}

This work was supported in part by The Royal Society under Grant RGS/R2/222382 and in part by Imperial College London internal funds.