IROS 2025 Vision-Tactile Sensing

3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception

A curved robotic fingertip that splits one optical field into complementary RGB and near-infrared observations, then reconstructs contact geometry with neural photometric stereo.

Yuankai Lin Xiaofan Lu Jiahui Chen Hua Yang^†

State Key Laboratory of Digital Manufacturing Equipment and Technology
Huazhong University of Science and Technology

IEEE Paper arXiv Watch demo

Fig. 01

Three-finger robotic gripper equipped with GelSplitter3D sensors reconstructing screwdriver contact into dense point clouds

Three curved fingertips Dense contact geometry

Trace the light

01 / Premise

Beyond planar touch

Curved sensing changes the optics.

Anthropomorphic fingertips offer continuous contact over complex object geometries, but their rapidly changing surface gradients make uniform illumination and faithful reconstruction difficult.

Split the field

A prism redirects the same tactile image plane to RGB and NIR cameras, adding complementary observations without fragmenting the contact view.

Learn the normals

A photometric stereo neural network maps RGB-NIR intensities and positional encoding to a dense surface-normal field.

Anchor the surface

CAD-derived boundary depth priors suppress cumulative integration drift on the non-planar sensing surface.

02 / Sensor & Optics

One contact. Two spectra.

A prism turns missing light into another view.

The optical path preserves a common field of view while directing visible and near-infrared signals toward dedicated cameras. Four embedded light sources create complementary photometric cues across the fingertip.

Diagram of prism-mediated RGB and NIR optical paths inside GelSplitter3D — 02.1 Prism-mediated light-field redirection

Shared image plane

Both cameras observe the same curved contact region, keeping spectral evidence spatially aligned.

Complementary cues

Near-infrared illumination recovers shape and texture evidence that can be ambiguous in visible channels alone.

Curvature-aware design

The optical assembly is integrated around a finger-like gel contact module instead of treating curvature as an afterthought.

Exploded view, illumination arrangement, and cross section of the GelSplitter3D sensor — Exploded architecture and optical assembly

Designed as a fingertip

Mechanics, optics, and compliant material meet in one form.

Camera mounts, illumination, prism, acrylic support, and the gel contact module are co-designed around the sensing geometry. The result can be mounted directly on a dexterous three-finger gripper.

Imaging: RGB + NIR cameras
Redirection: Cube beam-splitting prism
Contact: Curved reflective elastomer

03 / Fabrication

From mold to sensing skin

Fabrication of the Compliant Sensing Surface

The finger-shaped elastomer is cast in a multi-part mold, cured, coated, and assembled with the internal acrylic structure to form a continuous tactile surface.

White-background fabrication process for mold assembly, casting, curing, and elastomer release — 01 Mold assembly, casting, curing, and release

04 / Method

Pixels to contact geometry

Learn locally.
Integrate globally.

Calibration begins with CAD-derived supervision, then moves from aligned RGB-NIR observations to surface normals and a boundary-anchored depth map.

GelSplitter3D pipeline from RGB-NIR acquisition and CAD calibration through PSNN normal prediction and depth-prior normal integration

CAD-grounded calibration

A spherical probe samples contact positions while the known sensor geometry supplies dense normal supervision.

RGB-NIR PSNN

Pixel intensities and positional encoding are mapped to normal vectors through a compact multilayer perceptron.

III

Depth-prior integration

Boundary depths from the CAD model constrain normal integration and prevent global drift across the curved surface.

Robot-assisted spherical probe calibration and CAD-derived GelSplitter3D geometry — Calibration apparatus
Robot-controlled sampling links physical contact locations to CAD-derived geometric supervision.

05 / Results

Fine structure, recovered

NIR resolves what RGB alone leaves uncertain.

Across a watch screwdriver, fingerprint, grid texture, and color-ring resistor, RGB-NIR reconstruction preserves sharper contours and microscopic relief.

0.0130 mm / pixel

RGB-NIR total gradient MAE

40^%

lower gradient MAE than RGB-only PSNN

0.0406 mm

depth reconstruction MAE

0.0072 / 0.0058 mm / pixel

G_x / G_y MAE

Comparison of RGB-only and RGB-NIR normal maps and reconstructed point clouds for four fine-grained objects

Normal integration

Geometry priors stop error from accumulating.

Classical planar boundary assumptions fail on the curved fingertip. Adding flexible boundary handling helps; anchoring the integration with CAD-derived edge depth reduces the error further.

Fast Poisson0.579 mm

Flexible boundary0.1255 mm

With depth prior0.0406 mm

06 / Demo

The sensing loop, in motion

See RGB, NIR, normals and point clouds respond together.

GelSplitter3D hardware and synchronized tactile reconstruction. 00:52

07 / Code

Implementation

Code coming soon.

08 / Citation

Use this work

Cite GelSplitter3D.

Published in the 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems.

DOI ↗ arXiv ↗

BibTeX

@inproceedings{lin2025gelsplitter3d,
  title     = {3D Vision-tactile Reconstruction from
               Infrared and Visible Images for Robotic
               Fine-grained Tactile Perception},
  author    = {Lin, Yuankai and Lu, Xiaofan and
               Chen, Jiahui and Yang, Hua},
  booktitle = {2025 IEEE/RSJ International Conference
               on Intelligent Robots and Systems (IROS)},
  pages     = {16023--16029},
  year      = {2025},
  doi       = {10.1109/IROS60139.2025.11246893}
}