FAQ (English)

Q1: What are the novel contributions of Koide-Majima et al. (2024)?

A1: Koide-Majima et al. (2024) made two significant contributions. Firstly, they demonstrated that arbitrary natural images imagined in the human mind can be reconstructed from brain activity. While many previous studies have focused on reconstructing seen images (i.e., images observed by the human eye), externalizing mental imagery remains a challenge. Although a few studies have reported successful visualization of mental imagery, their visualizable images have been limited to specific domains such as alphabetical letters or geometric shapes (Senden et al., Brain Structure & Function, 2019; Shen et al., PLOS Computational Biology, 2019; Lee & Kuhl, Journal of Neuroscience, 2016). In contrast, Koide-Majima et al. (2024) successfully reconstructed arbitrary natural images imagined in the mind without limiting the type of images.

Secondly, Koide-Majima et al. (2024) proposed a new machine learning algorithm (image reconstruction method) that introduces generative AI into brain decoding technology, enhancing the previous method proposed by Shen et al. (2019). For reconstructing mental imagery, Koide-Majima et al. (2024) first estimated millions of visual features characterizing the imagined image from brain activity. Then, they reconstructed the image using a generative AI by synthesizing an image that matched the estimated visual features. To improve the quality of this image reconstruction process, they introduced mathematical estimation methods called Bayesian estimation and the Langevin dynamics algorithm. The effectiveness of Bayesian estimation was quantitatively demonstrated in Figure 6 of Koide-Majima et al. (2024).

Q2: What are the novel contributions of Koide-Majima et al. (2024) compared to Shen et al. (2019)?

A2: While the method of Shen et al. (2019) can reconstruct simple geometric shapes like "X" and "+" imagined in the mind, it struggled to produce meaningful images for brain activity associated with imagining natural images such as animals and objects. Koide-Majima et al. (2024) achieved this using their newly proposed image reconstruction method. Since they validated its reconstruction performance using the same data as Shen et al. (2019), the successful reconstruction can be attributed to this new method.

Q3: How was the quality of the reconstructed images evaluated?

A3: To examine whether a reconstructed image preserved information about the original image (i.e., the target image to be imagined), we evaluated how accurately the original image could be identified among multiple candidates using the reconstructed image. If the reconstructed image has visual features similar to those in the original image, the original image can be identified. The mean identification accuracy was evaluated using a binary classification paradigm and a simple classification algorithm based on an image similarity metric, as frequently performed in previous studies (Lee & Kuhl, Journal of Neuroscience, 2016). As a result, the original images were identified with an accuracy of 75.3% using the reconstructed images obtained by our proposed method. In contrast, the accuracy was 50.3%, almost at chance level (i.e., random guess), for the reconstructed images obtained by the conventional method (Shen et al., 2019).

Since the purpose of Koide-Majima et al. (2024) is to reconstruct arbitrary natural images, geometric shapes, which can be easily distinguished from natural images, were excluded in the evaluation. For a fair comparison between different image reconstruction methods, for each reconstruction method, the image similarity metric the reconstruction method was based on was used in this evaluation procedure. 

Additionally, as part of a multifaceted evaluation, the naturalness of reconstructed images was assessed using another metric, the Inception Score, in Koide-Majima et al. (2024).

Q4: Was manual evaluation by humans (human judgment) used for evaluating the quality of the reconstructed images?

A4: Manual evaluation has the problem of varying results depending on the characteristics of the evaluators. If the evaluation is conducted by highly focused individuals, the accuracy will be high; if it is done by infants who cannot understand the instructions, it will be at chance level. Therefore, taking into account reproducibility issues, manual evaluation methods were not adopted in Koide-Majima et al. (2024). However, this does not mean that we deny the value of manual evaluation methods; we believe that a multifaceted evaluation is desirable. In fact, we had prepared for a manual evaluation, but since it was not required at all during the peer review process, the current paper does not include the results of such evaluations. The five reviewers involved in the review of this paper seemed to agree that the evaluation results currently presented in the paper sufficiently guarantee the reconstruction accuracy.

Q5: Was the method of Shen et al. (2019), which was used for comparison, properly reproduced in Koide-Majima et al. (2024)?

A5: Yes, it should be. To faithfully reproduce the method and results of Shen et al. (2019), Koide-Majima et al. (2024) used the officially released code and decoded features provided by the Kamitani Laboratory at Kyoto University. This fact is also clearly stated in the methods section of the Koide-Majima et al. (2024) paper. For more details about the code and results of Shen et al. (2019), please refer to the following GitHub website: 

https://github.com/KamitaniLab/DeepImageReconstruction

Q6: Where can I see the video of the process of reconstructing mental images?

A6: You can watch it at the following link: Mental Image Reconstruction from Human Brain Activity

Q7: I would like to introduce this research to children and students who might be interested in it. Is it possible to use the images and videos as teaching materials?

A7: Yes. The contents of this paper and this website are published under the CC BY license. Therefore, as long as you provide attribution, you are free to use the figures and videos, including modification and commercial use (you do not need to ask for permission). Please be sure to include the attribution.

Attribution examples:

Koide-Majima et al., Neural Networks. 2024.

Koide-Majima, Nishimoto, Majima. Neural Networks. 2024.