I hope all is well. Are you still planning to submit your review this week as we discussed earlier? It would be great to have your opinion, but if this is no longer possible for you, we’ll need to proceed with a decision based on what has already been submitted.
Thanks again for your time.
Sincerely,
I hope all is well. Could you please let me know when you will be able to submit your review? Your expert opinion is very much needed to make a decision (we do not have enough reviews for this manuscript at the moment).
Looking forward to hearing from you.
Best regards,
I would like to kindly remind you of the review of the below paper. The paper is quite a bit past due date, and your timely review would be very helpful in finalizing the decision of this paper. Could you indicate when it will be possible to submit your review?
With kind regards,
Dear X:
I am wondering if you can give some indication as to when you can provide the review for this revised manuscript. As the expert in the field and the reviewer for the original submission, I would really appreciate receiving your opinion on this revision to see if your prior concerns were addressed,
Dear Dr. X,
Thanks for your willingness to review the referenced manuscript. I was able to obtain a third review elsewhere. Since the three reviews agree and your review isn't expected until May 4, I will just go ahead and make a recommendation to the EIC. I hope that you haven't spent too much time already on the manuscript. If you have written anything, then I can forward the additional comments (anonymized, of course) to the authors. But please don't feel any obligation at all.
Thanks!
Six reviewers have evaluated this submission. Unfortunately, none has expressed enthusiasm over a manuscript that is written in very poor English (Reviewers 3 and 6). One weakness that has been reported by most reviewers (Reviewers 2, 3, 4, and 5) is the lack of direct comparisons with state-of-the-art methods. Another major issue is related to the derivative-truncated gamma transformation that the authors use as a preprocessing step to enhance images. They present this transformation as the major innovation of their work; yet, according to Reviewers 2, 4, and 6, this step brings only a marginal gain. Moreover, as evoked by Reviewers 3 and 6, it is applied inconsistently so that the use of image enhancement during inference does not match its non-use during training. The other authors' contribution, namely, false-negative mining, is found by Reviewers 2 and 4 to be not novel enough to justify publication in the IEEE Transactions of Medical Imaging. Finally, Reviewer 6 regrets that the claims of significance are not substantiated by statistical analysis. Moreover, the split of the data in training-validation-testing sets is very much confusing: the numbers do not add up."
Associate Editor:
Comments to the Author:
The authors have done considerable effort to address the reviewers' comments. I think this work has promise and several reviewers have identified that. However, in the revised version, some additional aspects have been raised particularly regarding whether it is fair to not compare with SoTA and whether the manuscript has shown it is contribution.
I would like to encourage the authors to address these comments in a brief revision. Below I offer my own interpretation and suggestions on the basis of these comments.
Specifically,
* R3 has minor comments/suggestions
* R1 has not appreciated the contribution of this paper. My suggestion here is to better illustrate in the response (as the previous response was too terse). Also this is a key element of the paper (which powers the "paradigm" and the algorithm). So i would also suggest you elaborate on this further in the introduction.
* R2 has some concerns regarding the comparisons and whether it is necessary to use SoTA models. I agree that nnU-net makes comparisons harder given its design adaptations. I would encourage the authors to include material within the paper that justifies that U-net used is as a fair baseline. I agree that the authors propose a paradigm and hence it shouldnt matter what is the "basebone" used. However, I would encourage the authors to comment whether there are particular design choices that nnU-net makes that may make the pocketNet paradigm less/more impactful.
* R2 also raises again a remark regarding the 1.5% data experiment (Fig. 5). I agree with R2 that this point was not well addressed. Hopefully you can elaborate better in the paper and in your response why this might be happening. You remark mostly on "speed" (you say learns faster) but i wonder whether there is a regularisation imposed by the paradigm itself which would be welcome. It is quite possible that it encourages ever more invariance, if one is to believe in the emergence of invariance due to the information bottleneck (see Achille & Soatto, Emergence of ...).
My own additional comments:
* Could you please better specify how you run the statistical tests? What is the null? Are you testing for equality etc? In Table I including the p-value (of the paired Wilcoxon text) is somewhat redundant. I would simply use *, **, *** etc to indicate significance at 0.1, 0.05, 0.01 level or use N.S. for non-significance, <0.05, <0.1, <0.001 etc. As is with the exponents it is harder to understand.
* The mean+/-std results on dice (Table I) appear very close. Yet the p-values show significance in some cases (e.g. ResNet NFBS, and in general in the NFBS data). If this is a paired test it implies that in some cases PocketNet does better in others less so. It would be interesting to have some analysis/discussion why that might be the case.
---------- Forwarded message ---------
Hi Leslie and Rutao,
Let me first respond to some of the authors' main rebuttal points, and then I'll follow up with my own opinions of this paper at the end.
> This work is a substantial extension of an ISBI 2022 paper, where it received
> highly favorable reviews.
While I can understand the authors' expectations after the ISBI review and their frustration and surprise at the TMI decision, receiving positive reviews at ISBI does not mean that the work will have a positive outcome at TMI. The standards are different, editor/reviewer expertise is different, and editor/reviewer effort is also different. Discrepancies between conference reviews and journal reviews are relatively common.
> The Editor comments seem to be mostly reviewer-style comments to authors
> to address rather than a substantial concern on either novelty or impact.
I don't feel like this is a fair characterization of the decision letter. The decision letter clearly expresses the opinion that there is a lack of innovation, a lack of practical impact, and a combination of insufficient theoretical insight with insufficient experimental validation. These are all high-level concerns that relate directly to novelty and impact.
That said, even if the decision letter had included "reviewer-style comments," why would that be a problem? It is important for the authors to keep in mind that, regardless of how comments are phrased, editorial rejections are designed in part to save the authors time if it is felt that a full review is unlikely to have a positive outcome. TMI allows at most one major revision, so it is especially important that the first submission is very strong. In this case, there is a clear sentiment that the submission is not strong enough and the chances of eventual acceptance are too low to warrant a full review. While it may be natural for authors to be frustrated and react defensively, the bright side is that the authors get to receive quick feedback that allows them to strengthen their manuscript without having to wait for the conclusion of a lengthy review process and without burning a major revision.
> I do not think the choice of distance metric in the proposed formulation is a
> major concern. We chose several standard metrics to show that the effect of
> the specific distance metric has a marginal effect on performance.
I agree with the authors (and disagree with the decision letter) on this point -- the choice of distance metric is not a major concern for me. But the choice of distance metrics was just one of many factors used to justify the previous rejection. The previous evaluation would still be quite negative if we strike the comments about distance metrics.
> I do not agree that training the network is time-consuming. In fact, we have
> provided the runtimes in the discussion section that show it can be as low as
> a few minutes depending on the training type compared to several tens of
> hours for regular network training.
The authors are supporting their argument with an unfair comparison. Regular network training only has to be done once, and the per-scan cost amortizes to zero if the same network is used routinely. The proposed approach requires substantial per-scan training cost. "A few minutes" is still substantially slower than methods that have negligible per-scan cost. I do not believe that long computation times should warrant a rejection, but am also concerned at the authors' unbalanced representation of computation time in both the paper and the rebuttal.
Moving from the rebuttal to my own thoughts about the submission: While I'm not as negative about the work as the previous decision letter, it is probably better for all parties if the authors would spend some effort to strengthen the manuscript before it is sent to reviewers. Otherwise, I do not think that one round of major revisions will suffice, and the paper is unlikely to ever be accepted by TMI. If the authors do not want to revise based on the feedback they've recieved, then I think an Immediate Reject decision is appropriate for the current version of the manuscript.
In addition to the issues mentioned in the previous decision letter, I have the following concerns:
1. The paper has not done a good job of reviewing the related literature, which makes the significance of the contribution harder to evaluate. Scan-specific networks like RAKI and LORAKI have things in common with the proposed scan-specific training approach, but these methods are not discussed at all. The proposed approach is also quite similar to and potentially derivative of common transfer learning approaches, and discussion of this is needed to make the novelty of the contribution clearer.
2. If I interpret correctly, the retrospective simulations include something unrealistic that may artificially enhance performance. It appears that coil sensitivity maps are estimated from fully sampled data, and then that fully sampled data is retrospectively undersampled and passed through reconstruction. This way of doing things is not a realistic representation of real MRI experiments, and will cause the sensitivity maps to possess information about the missing k-space samples that they would not normally have access to. It would make more sense and be more realistic to use only data from the undersampled measurements for sensitivity estimation.
3. The validation study is limited to just one database of knee scans, which makes it difficult to have confidence in the generalizability of the results. The results also seem incremental, with only small improvements in PSNR. It is not clear that these minor advantages would still exist for different databases, and it is not clear that the minor advantages are significant enough for TMI without more compelling theoretical insights or more rigorous empirical evaluation.
4. I would expect that using a small number of local neighbors will make the method prone to overfitting. Proper choice of k seems like it will be essential for the method to work well, but I did not see anything about this issue in the manuscript. This is another example of the need for stronger empirical verification.
> -----Original Message-----
> From: Rutao Yao <rutao.yao@ieee.org>
> Sent: Wednesday, June 22, 2022 9:40 PM
> Subject: Second look of an Immediately Rejected paper
>
> Dear ,
>
> Would you review the attached manuscript and provide your opinion? This
> paper received an Immediate Reject decision from us and the authors
> appealed. According to our review protocol, we ask a second AE to take
> another look at the manuscript. Both the rejection comments and the
> authors' arguments are included in the message below.
>
> Your assessment is greatly appreciated.
>
> Rutao
>
> Rutao Yao, PhD
> Managing Editor, IEEE Transactions on Medical Imaging Professor,
> Department of Nuclear Medicine, State University of New York at Buffalo
>
>
> ---------- Forwarded message ---------
> From:
> Date: Wed, Jun 22, 2022 at 9:07 PM
> Subject: Re: Decision on IEEE TMI-2022-1038
> To: leslieying@ieee.org <mailto:leslieying@ieee.org> <leslieying@ieee.org
> <mailto:rutao.yao@ieee.org> <rutao.yao@ieee.org
>
>
>
> Hi Leslie,
>
> Thanks for the email.
>
> I am very surprised by this decision, especially based on the comments
> included.
>
>
> This work is a substantial extension of an ISBI 2022 paper, where it received
> highly favorable reviews.
>
> The Editor comments seem to be mostly reviewer-style comments to authors
> to address rather than a substantial concern on either novelty or impact.
> I do not think the choice of distance metric in the proposed formulation is a
> major concern. We chose several standard metrics to show that the effect of
> the specific distance metric has a marginal effect on performance.
>
> I do not agree that training the network is time-consuming. In fact, we have
> provided the runtimes in the discussion section that show it can be as low as
> a few minutes depending on the training type compared to several tens of
> hours for regular network training.
>
> The bilevel optimization framework is by itself a very novel perspective in
> deep learning based imaging. Given that the problem is NP-hard and highly
> challenging, we showed empirical convergence results for the proposed
> method, which could be expanded if needed.
>
> I believe the paper offers novelty, insight, and impact, and is a good fit for
> consideration in TMI. The editor's comments do not seem to doubt this.
>
>
> I would like to hear your thoughts more.
>
> Thanks,
>
>
> ________________________________
>
> From: IEEE Transactions on Medical Imaging
> <onbehalfof@manuscriptcentral.com
> <mailto:onbehalfof@manuscriptcentral.com> >
> Sent: Wednesday, June 22, 2022 8:50 PM
> To:
> Subject: Decision on IEEE TMI-2022-
>
> ... Reconstruction from Undersampled Data - TMI-2022-
>
> *********
> Please note that by selecting “Reply All”, your message will automatically
> include the email addresses of the Editor in Chief and the Managing Editor.
> *********
>
> Dear Colleague,
>
> The Editorial Board has carefully considered the information presented in
> your manuscript and has decided not to send it out for external peer review.
> Our consensus is that your manuscript is not suitable for publication in the
> IEEE Transactions on Medical Imaging. Such an expedited decision is for the
> benefit of the authors who may submit the manuscript to another journal
> sooner.
>
> COMMENTS FROM THE EDITORIAL BOARD:
>
> How to adaptively select training samples that are consistent with the
> distribution of test data is an interesting topic. The authors claimed this study
> has six novel aspects. However, there are some major concerns:
> This paper only selected the common L1 norm, L2 norm and cross-
> correlation as metrics to search for training samples with a consistent
> distribution, which lacks in-depth explanations and innovation. Although the
> paper shows the advantages of local training, however, as shown in
> Algorithm 1, it is very time-consuming to train a neural network for each test
> data, so such methods are not currently applicable in practice. Also, the
> convergence of the bi-level programming has only limited experimental
> verification (and experiments cannot verify whether it is globally converged),
> and there is no theoretical analysis.
>
> Sincerely,
> Dr Leslie Ying
> Editor-in-Chief, IEEE Transactions on Medical Imaging
Thank you for your thoughtful reviews of the manuscript above. I noticed in your last review you recommended the authors to add citations to 5 papers. I agree that the topic of contrastive learning could have been better discussed in the manuscript. However, instead of providing specific references, could you give suggestions of what subjects or techniques the manuscript lacks in discussion or literature review? This will help the authors improve the manuscript while allowing TMI to maintain the highest standards of academic integrity and quality in the publication process.
If I pressed the right button, I think you should be able to edit your review in the system now. Let us know if that is not the case.
Thanks again for your contribution!
The re-revised manuscript was sent back to two of the original four reviewers. Although both reviewers appreciated the effort the authors put into the revision and both appreciated the concept of the proposed method, one reviewer still had major reservations about publishing this work in TMI. Several new concerns were raised, and the reviewer clearly put a lot of thought and effort into this review (more than usual for a third-round review).
In my view, the most significant concern appears to be that the newly added super-resolution simulations may lack realism and that the results may not be clinically acceptable because important features of the original images have changed in clinically relevant ways when using HA-GAN. (After looking closely at this issue myself, the reviewer's concern about clinical acceptability appears to be valid, and I don't see corresponding caveats in the manuscript. In addition, I was not able to find a detailed description of exactly how the low-resolution MRI and CT data was simulated. Realistic simulations of low-resolution CT and MRI data need to account for imaging physics, and each of these modalities will require the use of point-spread functions with different characteristics. Unrealistic simulations are a longstanding and problematic issue in this field, and a clear description of the data acquisition assumptions is important for journals like TMI, but I do not see this in the manuscript. TMI often rejects manuscripts on super-resolution for having unrealistic/undescribed simulation setups. In addition, the MRI images are clearly skull-stripped, which would not be the case for practical data, but this issue is not discussed.)
This is a tricky situation with all reviewers having some positive things to say about the work, but with some substantial valid concerns still remaining. Even if some of the other concerns could be addressed with minor revisions or left unresolved, major revisions appear to be necessary to resolve the substantial concerns about the new super-resolution experiments as described above. Unfortunately, TMI policy prevents a third major revision decision.