Ultrasound imaging turns a robot hand into a skillful mimic

Our hands are the nimblest parts of our bodies, coordinating 34 muscles, 27 joints, and over 100 tendons and ligaments to perform countless nuanced movements and gestures. So far, robots have been notoriously bad at mimicking that dexterity, in part because researchers struggle to capture what is actually going on under our skin in order to reproduce it. Now MIT researchers are pioneering a promising new approach.

Mechanical engineering professor Xuanhe Zhao and colleagues at the Institute and the University of Southern California have designed a wristband equipped with an ultrasound “sticker”—a miniaturized version of the transducers used in medical offices, paired with a hydrogel that can safely adhere to the skin. As the wearer’s hand moves, the device produces ultrasound images of the wrist’s muscles, tendons, and ligaments. Then an artificial-intelligence algorithm, trained on ultrasound images meticulously labeled by humans, continuously translates the images into the corresponding positions of the five fingers and the palm. “The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers,” says Gengxi Lu, a former MIT postdoc and one of the lead authors of a paper on the work. “So the idea is: Each time you take a picture of the state of the strings, you’ll know the state of the hand.”

In demonstrations, the team has shown that a person wearing the wristband can wirelessly control a robotic hand. As the person gestures or points, the robot does the same. In a sort of wireless marionette interaction, the wearer can manipulate the robot to play a simple tune on the piano and shoot a mini basketball into a desktop hoop. With the same wristband, a wearer can also manipulate objects on a computer screen—for instance, pinching the fingers together to enlarge and minimize a virtual object.

The researchers are planning to further miniaturize the wristband’s hardware, which is currently similar in size to a cell phone. They also hope to train the AI software on movements from more volunteers with a wider variety of hand sizes, finger shapes, and gestures. 

They envision building a large data set of hand motions that can be plumbed, for instance, to train humanoid robots in delicate tasks such as surgical procedures. The ultrasound band could also be used to let people grasp, manipulate, and interact with objects in design applications, video games, or other virtual settings. Ultimately, the team is building toward a wearable hand tracker that anyone can use to wirelessly manipulate humanoid robots or virtual objects with high dexterity in real time.

“We believe this is the most advanced way to track dexterous hand motion—through wearable imaging of the wrist,” Zhao says. “We think these wearable ultrasound bands can provide intuitive and versatile controls for virtual reality and robotic hands.”