Spray-On Smart Skin! AI-Powered Rapid Hand Task Recognition | Yunqi Capital Science Snack

In recent years, wearable biosensors have found widespread use in health monitoring, human-computer interfaces, and other fields. We've all seen it in sci-fi movies — people typing on invisible keyboards, identifying objects through touch alone, controlling apps with simple hand gestures...

Now, a new type of smart skin has made these scenarios a reality.

In this edition of "Yunqi Science Snap", we're sharing the latest on wearable devices. Enjoy~

➤➤➤ Recently, in a paper published in Nature Electronics, a research team from Stanford University, KAIST, and Seoul National University developed a stretchable, biocompatible material that can be sprayed onto the back of the hand like sunscreen. Its embedded microelectronic network senses skin stretching and bending.

(Source: Nature Electronics)

The team says they can use AI to link hand movements and gestures to various everyday tasks, potentially extending this smart skin to gaming, sports, telemedicine, robotics, and other industries.

In previous research, scientists typically used wristbands or wearable gloves to measure muscle electrical activity in order to capture hand movements and gestures. However, these devices are bulky and require multiple sensory components to precisely locate each joint's motion. Additionally, training an algorithm demanded collecting massive amounts of data for each user and task. For these reasons, such devices have been difficult to integrate into everyday electronics.

In this study, Professor Zhenan Bao and colleagues' new smart skin is highly practical — visually streamlined, functionally adaptable, and essentially universal across users, even with limited data.

The new smart skin also employs a leaner software approach that learns faster. This precision could be key for virtual reality applications, enabling fine-grained motion transmission for more realistic experiences.

According to the paper, the smart skin takes the form of a sprayable, electrically sensitive mesh network embedded in polyurethane, composed of millions of gold-plated silver nanowires. It exhibits excellent electroactivity, biocompatibility, and breathability; conforms closely to finger wrinkles and creases; and is difficult to dislodge. A lightweight Bluetooth module connected to the mesh enables wireless signal transmission.

Bao explains, "When fingers bend and twist, the nanowires in the mesh are squeezed together and stretched, changing the network's conductivity. These changes can be precisely correlated with hand, finger, and joint movements."

The direct-on-skin spray application ensures the mesh is supported without a substrate. This key engineering decision eliminates unnecessary motion artifacts and allows the team to generate information about multiple finger joints from a single conductive mesh.

This spray-on characteristic lets the smart skin adapt to any hand size or shape, and could potentially be used on the face to capture subtle emotional cues — possibly spawning new methods for computer animation or enabling virtual meetings with more realistic facial expressions and gestures.

Additionally, the computer monitors patterns of conductivity changes and maps them to specific physical tasks and gestures. For example, when typing an X on a keyboard, the algorithm learns to recognize this task from the conductivity change pattern. Once properly trained, the physical keyboard becomes unnecessary. The same principle can be applied to recognizing sign language, or even identifying objects by tracing their external surfaces.

"With just a few quick tests, the smart skin can rapidly recognize any new hand task and user," said co-first author Dr. Kyun Kyu Kim. "Because the nanomesh captures fine-grained details in its signals, we can achieve faster computational processing with less data."

Reference link: https://www.nature.com/articles/s41928-022-00888-7