New LightTok chip converts light directly into 'tokens' for AI — slashing energy use in drones and other autonomous machines

Researchers in China have developed a sensor that could reduce the amount of energy that drones and other autonomous hardware use to process visual data.

In a study published Aug. 19 in the journal Nature Sensors , scientists detailed the architecture of a new 2D chip dubbed "LightTok," which can convert raw light into tokens used by artificial intelligence (AI) models on the chip itself. The device skips several energy-intensive steps that conventional sensor systems depend on, the scientists said.

"Our design idea was to move token generation onto the sensor itself, allowing the chip to directly produce tokens that AI models can process once light reaches the sensor," Miao Feng , director of Nanjing University's Institute of Brain-Inspired Intelligence, said in a statement . "These tokens contain complete image information."

In conventional visual perception, light signals have to go through multiple stages. A sensor captures the signals, and an analog-to-digital converter turns the light into pixels. That data is then stored temporarily before it's shuttled to a separate chip, where the image is cut into square "compartments" — much like dividing a photo into a grid of tiles. Each compartment is then converted into a token for an AI model to read.

One study widely cited by other academics found that an analog-to-digital converter is responsible for 66% of an image sensor's energy consumption, on average. Moving visual data processing off the chip and into the cloud can increase overall energy consumption further.

LightTok's solution to the energy problem is to collapse the five stages into one by building the sensing, memory and computation processes into the same pixel. The researchers achieved this through an array based on a technology called a single-layer molybdenum disulfide floating-gate phototransistor that can sense light, remember what it sensed, and then factor that into a calculation.

Molybdenum disulfide is a 2D material that reacts well to light and can be grown in sheets one atom thick. The phototransistor converts incoming photons — particles of light — into an electrical current, while the floating gate is an isolated component inside the phototransistor that can trap and hold an electrical charge, rather than disappearing after the light goes away.

Get the world’s most fascinating discoveries delivered straight to your inbox.