SMF-PSNN

IEEE/ASME Transactions on Mechatronics · 2025

High-Precision RGB-NIR Vision-Tactile Sensor via Neural Inverse Rendering for Robotic Perception

Yuankai Lin Xiaofan Lu Hua Yang

Huazhong University of Science and Technology

IEEE Paper Citation
Overview of the RGB-NIR tactile sensing, geometry reconstruction, illumination calibration, image rendering, and downstream perception pipeline.

01 · Overview

One calibrated model, multiple tactile capabilities.

SMF-PSNN unifies surface geometry reconstruction, multi-LED illumination self-calibration, neural inverse rendering, and force-aware robotic perception within a single RGB-NIR vision-tactile sensing framework.

Geometry

High-fidelity surface normals

RGB-NIR observations and position-aware photometric reasoning recover contact geometry over the sensor's compliant surface.

Optics

Self-calibrated near-field lighting

A spatially varying multi-LED model estimates illumination and reflectance parameters for reliable reconstruction and rendering.

Perception

Transferable tactile images

Neural inverse rendering produces tactile observations that support force estimation, pose estimation, fingerprint recognition, and slip detection.

02 · Sensor system

RGB-NIR multispectral sensing through a compact beam-splitting architecture.

GelSplitter couples a compliant tactile interface with visible and near-infrared imaging. The complementary observations improve photometric conditioning while retaining the spatial coverage required for robotic contact tasks.

The calibration process jointly identifies geometric, optical, and normal-force parameters, linking camera observations to surface shape and physical interaction.

GelSplitter sensor structure and RGB-NIR beam-splitting optical system.
GelSplitter structure and RGB-NIR optical design.

03 · Method

SMF-PSNN combines photometric stereo with neural inverse rendering.

The method estimates surface normals under near-field, multi-source illumination and reconstructs tactile images through learned inverse rendering. Its shared calibration links geometry, lighting, reflectance, and normal force.

SMF-PSNN method diagram including near-field multi-LED modeling, normal estimation, and neural inverse rendering.
01RGB-NIR acquisition
02Illumination calibration
03Normal reconstruction
04Inverse rendering

04 · Core results

Precision gains across reconstruction and rendering.

Evaluation covers surface-normal recovery, depth reconstruction, tactile image rendering, and force estimation on the sensor surface.

28.2%normal error reduction
4.926°surface-normal MAE
0.05 mmdepth MAE
30.22 dBrendering PSNR
Surface normal and depth reconstruction results across evaluated objects.
Surface-normal and depth reconstruction results.
Tactile image rendering pipeline and qualitative rendering comparison.
Neural inverse rendering reaches 30.22 dB PSNR, 6.33 dB above the single-light-source baseline.
Normal force reconstruction results and comparisons.
Normal-force reconstruction under calibrated contact.
Rendered tactile pressure patterns and contact details.
Rendered contact appearance and fine surface detail.

05 · Applications

From synthetic tactile data to contact-rich perception.

The calibrated rendering model supports three representative downstream tasks in contact-rich robotic perception.

Needle eye, needle tip, and knot pose estimation using synthetic tactile samples.

Pose estimation

Accurate learning from 200 synthetic samples.

Needle eye / tip
0.995
Knot
0.944

mAP50–95

Fingerprint recognition examples and classification results from tactile images.

Fingerprint recognition

Fine tactile texture remains identifiable.

Accuracy
0.810
False positive rate
0.032
Slip detection sequences and performance for robotic manipulation tasks.

Slip detection

Reliable temporal contact-state recognition.

Average F1
94.3%
Needle-task F1
96.0%

06 · Citation

Cite the published work.

The published paper is available from the official IEEE DOI page.

Open the IEEE paper 
@article{lin2025highprecision,
  title={High-Precision RGB-NIR Vision-Tactile Sensor via Neural Inverse Rendering for Robotic Perception},
  author={Lin, Yuankai and Lu, Xiaofan and Yang, Hua},
  journal={IEEE/ASME Transactions on Mechatronics},
  pages={1--12},
  year={2025},
  doi={10.1109/TMECH.2025.3640680}
}