Research Reality Check

** This page is taken from Professor Koji Yatani's website from the University of Tokyo ** Please refer to http://iis-lab.org/misc/realitycheck/ for the original post.

This page presents a check list which we use when we start a new project. This Reality Check has bee produced through collaboration and interaction with students. This checklist demonstrates our way of thinking: “Think radically. Execute logically.” It may be a little biased toward HCI research, but I hope it could be useful for other fields. Please feel free to revise, and let me know when you build you own!

Reality Check

There are a couple of things you need to keep in your mind for this Reality Check.

§ Write down your answers in a paper. It is quite common that you may think it works but you notice it doesn’t when you write down. This check list also serves as an exercise for writing. I know it’s a tough exercise, but that’s something I want you to learn.

§ Write your answers in as much detail as you can. Make sure that you include sufficient information so that other people can understand your idea without you.

§ Ask people around you for double-check. Once you are done, get someone to go through your answers and ask feedback. Any answer where your description is vague, your plan is unclear, or your explanation is not logical? Uncovering such weak points is the whole purpose of this Reality Check.

Think radically: Idea Overview

Let’s start sorting out your current idea. We are going to clarify what new ideas or thoughts your idea contains.

§ What is your goal?

• What do you want to achieve or uncover through this research?
• What will you build? What will you improve? What will you survey?
• Do you describe enough details about your system so that others can also understand what you are going to do?
• How do you objectively demonstrate that you have achieved or uncovered?

§ How radical?

• What in your idea makes people say like “Can you really do it!?” or “That’s pretty crazy!”?
• How can you describe what makes your idea so unique in one sentence?

§ Who cares?

• Who gets benefits from your idea?
• Who are the target users and under what contexts/scenarios?
• How many are such users? How often or likely are the target users under the context/scenario you are thinking of? Isn’t your target user population too narrow?

When you have answered all questions, please move on to Execution Strategy.

Execute Logically: Execution Strategy

Now you are asked to think about how you can turn your idea into reality.

§ How will you implement your system and/or run your study?

• How will you turn your idea into reality?
• What existing technology can you utilize?
• What novel technology will you have to develop?
• What methods will you use for your survey? And why?

§ What will you need to build your system and/or run your study?

• What equipment and facilities will you need to build your system?
• What services will you use for your survey? How will you collect data?

§ What are the classics?

• What are typical or common solutions?
• What pros and cons do they have?

§ Who are competitors?

• What are competitors’ technologies?
• What are latest related technologies?
• What is strongly relevant research?
• What pros and cons do they have?

§ What are your advantages?

• What merits do your idea have compared to existing technologies?
• If you say “efficient/efficiency,” what efficiency do you mean? Can you really measure?
• If you say “optimal or optimized,” what exactly will you optimize and with what metrics can you confirm? Can you show that what you get is really “optimal” (i.e., nothing else is better).
• If you say “best”, with what metrics can you confirm that what you get is best? Can you show that what you get is really “best” (i.e., nothing else is better).
• If you say “intuitive”, how quantitatively can you show such intuitiveness?

§ How will you confirm your advantages?

• Through what experiments can you show advantages of your idea?
• What can you hypothesize that would be improved and how?
• What will you measure? Can you measure them objectively?
• What conditions do you think to be considered? How would you narrow them down? Why?
• Are conditions fair? How do you justify?
• Who are your participants? Aren’t they different from your target users?

§ How do you make this happen?

§ Describe your plan so that you can finish everything one month before the final deadline (e.g., thesis submission).

§ Is there any other important deadline or milestone in your plan? If so, what and when?

§ Itemize 5-7 major action items and when you will start and end for each.

§ Can you really follow your plan? If there is a stretch, which goals should you re-consider?

When you have answered all questions, please move on to Devil’s Review.

Criticize Deeply: Devil’s Review

Now you are criticizing your idea by playing a Devil’s advocate. Please answer the following questions and revise your answers above if necessary. Do not provide simple answers like “shouldn’t be an issue.” You must provide justifications and explanations for your argument of “no issue”.

§ Why is your idea so important?

• How important is the problem your idea is trying to solve?
• What impact would you offer if you make your idea happen.
• What new insight would you bring after your survey?

§ Aren’t you just building toys?

• Aren’t we creating something that just looks cool but totally useless?
• Are there really people who would be willing to use your system? If so, how many?
• What are applications that bring benefits in the real world? If so, what is that? If not, why are we building this technology after all?
• Is there anything only this system or method enables? If so, what is that? How much happier would it make users if your idea happens?

§ Aren’t you solving toy problems?

• Is the problem you are going to solve really important? Aren’t you just making up a problem for research?
• If you solve that problem, who would be happier? And how much?

§ Isn’t your system just all known?

• Isn’t your system just a sum of existing technologies?
• Can you create more than effect from the sum of technologies? If so, why do you think so?

§ Aren’t you working on a small delta?

• Is your work just a small delta from prior work?
• If you can improve speed or accuracy, how much would it impact on users?
• What convinces people that your work is innovative, not incremental?
• Would the utility of your system really bring impact to the society?

§ Are you really the right person to work on this?

• Is your idea really what we should do? Any other research group who may be working on similar ideas?
• What is our unique advantage that other people do not have?
• Do we have necessary knowledge for execution? If not, what will we have to learn?

§ Aren’t you missing important related work?

• Can you list up a few names who work on relevant projects?
• Have you checked publications at recent conferences?
• Have you checked classic publications?
• What is the oldest paper related to your idea?

§ Do your limitations make your system useless in reality?

• Does your system or experiment work only under a limited condition?
• Can your system be deployable?

§ Aren’t you just designing your system however you like?

• Can you justify your system design rationales?
• How can you justify your interface design?
• How would you determine parameters for your system and/or experiment? How can your decision convince other researchers?

§ Aren’t you cheating in your evaluations?

• Aren’t you using metrics that are in favor for you?
• What are other possible metrics? Why don’t you use them?
• Aren’t you setting conditions and task that are in favor for you?
• What are other possible tasks and conditions? If so, why don’t you use them?
• If you conduct user studies, do you pick up participants who are biased toward you? Is there any discrepancy between them and your target users?

When you have answered all questions, please move on to Executive Summary.

State Concisely: Executive Summary

As a final exercise, let’s explain your idea concisely and precisely.

§ Explain your research in 30 seconds.

• Consider how you can explain your research to others and convince them your project is very cool.
• Explain the five following points in 1 or 2 sentences and make the paragraph 7-8 sentences.
• What are you going to do?
• What problems will you solve?
• Why is your problem so important?
• What is unique in your research?
• What merits would your idea offer?

§ Explain your research in one sentence.

• Can you explain your idea in a single sentence?
• Does your description contain all important keywords?
• Make sure you are not over-claiming what exactly you are going to do.
• Isn’t your description too generic? Can similar projects use the same description?
• Isn’t your description too narrow?

This is the end of the reality check. Great job! Please take a look at the last section and create a paper draft!

Congratulations!

That’s all for the reality Check. Now, you can create a paper draft with making minor revisions on your answers. You can roughly map between what you have written and what you need to put in your paper draft as follows.

§ Title

• Explain your research in one sentence.

§ Abstract

• Explain your research in 30 seconds.

§ Introduction

• Why is your idea so important?
• Who cares?
• What is your goal?

§ Contributions

• How radical?

§ Related Work

• What are the classics?
• Who are competitors?
• What are your advantages?

§ Implementation

• How will you implement your system and/or run your study?
• What will you need to build your system and/or run your study?

§ Experimental Design

• How will you confirm your advantages?