This 'Mind-Reading' AI Can Recreate Images Seen By The Human Brain

Scientists have developed a new artificial intelligence tool that can recreate the images a person is looking at by reading their brain scans, Popular Mechanics reported, adding that experts and tech enthusiasts were stunned by an unprecedented level of visual detail and accuracy of the tool. The system, called "Brain-IT", was created by researchers led by Professor Michal Irani at the Weizmann Institute of Science in Israel. The technology analyses activity recorded during functional magnetic resonance imaging (fMRI) scans and translates those neural signals back into visual images.

Also read | 15-Year Conservation Effort Saves World's Heaviest Insect Species

How the technology works

When a person looks at an object, different regions of their visual cortex light up. fMRI machines track these subtle changes in blood oxygen levels across the brain.

It was previously not possible as earlier, the decoding methods could often guess the general category of an image, for example, it would guess that 'a person was looking at an animal or a vehicle'. But, more or less, they failed to recreate the actual layout, shape, or colours with stunning detail.

But when it comes to Brain-IT, the report noted that it solves this by using a two-part AI decoder. It maps out the layout, shapes, position, and colours in the scene. And then it decodes the meaning and subject matter of the image. The information together is then combined by a specialised image-generation AI to reconstruct a picture that is similar to the subject the volunteer was looking at.

Also read | 194-Year-Old Giant Tortoise Shares Genetic Secrets To Long Life

"There exist nowadays models that translate brain activity into images, and they can even produce impressive reconstructions that preserve the semantic meaning of the image reasonably well," Irani said in the statement.

"However, they tend to make mistakes in basic features such as composition and colour. The new model we developed outperforms them in reconstructing both the content of the image and its details. What's more, while every other model requires dozens of hours of brain scans to learn to 'read' a new person, our model needs only one hour."

There were several issues with the research, including the fact that it was expensive. AI models require up to 40 hours of brain scan data from a single individual just to learn how that specific person's brain processes visual information.

Photo Credit: www.weizmann.ac.il

"None of us can afford 40 hours of imaging for a new subject," Tommy Sprague, a neuroscientist at the University of California, Santa Barbara, said as quoted by the New York Post. "It's something like $600 to $1,000 an hour."

To solve this, the Weizmann Institute team managed to build a "universal brain encoder" trained on thousands of brain scans. The model maps out common patterns shared across human brains, grouping them into 128 functional visual regions. As a result, the AI can adapt to a completely new person using just one hour of setup data.

While the technology appears promising, it is not flawless. The report says that during testing, the system occasionally made humorous mistakes, such as when the searcher said that it misinterpreted a dog sitting in a bathtub as a goat in a similar setting; it also turned a slice of cake into a stack of sandwiches.

The breakthrough has also sparked debates around mental privacy.

"What remains especially challenging is decoding video - for example, during dreaming," Irani explains. "Dozens of images change every second, while an fMRI scan takes about two seconds. If we overcome all these obstacles, it's possible that in the future we may even be able to read dreams."