Working at a hearing aid company, I look at methods to evaluate hearing aids in real life. This can be done by providing study participants with hearing aids and have them answer a retrospective questionnaire when they come back a few weeks later. While this is representative of real life, we do not know to what extent results may be influenced by memory bias. Also, it is unclear how different experiences with the hearing aids are combined into one rating. Maybe they are influenced particularly by the most recent or the worst experience. And finally, we can ask for a situation of for example music listening, but we still do not know the exact acoustic parameters of the listening situation like the sound pressure level and what features have been active at that moment inside the hearing aid.
A more detailed way to look at this is Ecological Momentary Assessment. This consists of repeated surveys on the current situation. Hence there should be very little memory bias. Nowadays this is often done via smartphone and in addition to the self-reports objective data from the hearing aid can be collected as well.
One of my interests with EMA are potential biases and how to improve the method. Answering surveys is not in all situations equally appropriate. It is for example unsafe when driving. More importantly many participants, particularly older ones, consider mobile phone usage in social situations impolite and indeed we could show in a study with 20 German participants, that social situations and speech in noise situations were underrepresented in the surveys. Ways to make it easier for the participant to also report on social situations are to only use a very short questionnaire or to allow reporting shortly after the situation. Social norms of course depend strongly on the subject sample. In Singapore for example, mobile phone usage is more widespread than in Germany, and in a repetition of that study with Singapore participants the effect was reduced.
In our first study, we paid participants per survey and later I have been asked often, whether that would not encourage them to answer carelessly just to get the money. However, when searching for careless responses, we did not find any. Here you can read the corresponding paper here or watch a presentation. This study included highly motivated participants that we knew personally. Possibly, careless responses are a bigger issue with large scale studies where participants simply sign up and download the app. However, even then it is easier to skip a questionnaire completely than to answer it carelessly.