Mental Image Reconstruction from Human Brain Activity

Individual image files from the original publication are available here.

Can people share their imagined images with others?

In this study, we present a machine learning method for visualizing subjective images in the human mind based on brain activity. Although many previous studies have demonstrated that images observed by humans can be reconstructed from their brain activity, the visualization (externalization) of mental imagery remains a challenge. To achieve this long-standing milestone, we combined a previous visual image reconstruction method with recently developed neural network technology into a single Bayesian estimation framework. The results demonstrated that our proposed framework successfully reconstructed both seen and imagined images from brain activity, which would provide a unique tool for directly investigating the subjective contents of the brain such as illusions, hallucinations, and dreams.

Enjoy our mental image reconstruction algorithm with Google Colab!

We provide our demo code on Google Colab here. You can reproduce our results without any cumbersome installation.

Iterative image reconstruction process shown on YouTube

Our proposed algorithm iteratively (gradually) updates images so that they would have contents and features similar to the imagined ones. This iterative process can be seen on YouTube.

Several articles explaining  our current work

• Mental image reconstruction using generative AI — Decoding brain signals into images: New technology developed by QST

• AI-generated imaging using brain activity hits record 75% accuracy: Japan research

• Japan Succeeds in Image Reconstruction from Brain Signal Data

• Japan scientists create world's 1st mental images with AI tech

Acknowledgements

We want to express our special thanks to

• VQGAN by Patrick Esser, Robin Rombach, and  Björn Ommer

• CLIP by Alec Radford, Jong Wook Kim, Ilya Sutskever, et al.

• text-to-image using VQGAN+CLIP by Sagar Budhathoki Magar

• Demo code creation supported by Ryoga Otake (大竹 遼河) of CJS Inc. (株式会社知能情報システム)

Copyright

The content on this website and the original research paper are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).