When we interact with objects and environments, we gather knowledge that form mental representations. We refer to these internal representations of what we interact with in the external world as mental models.
Mental models are memory representations of systems or situations that we store mental models in long term memory and access in working memory. We develop and modify our mental models over time, which we can base off of actual or simulated interactions.
When you apply it to human computer interaction, a mental model is what the user believes about the system at hand. So if you are building a website, you would hope that the users beliefs are closely related to the facts, especially as they derive their predictions and future actions based on their mental model. Knowing this, a human factors/UX researcher can become involved to assess the gap between designers' and users' mental models.
A common method that we utilize is called the think aloud method. To conduct this method, a researcher would recruit representative users and ask them to continuously think out loud by encouraging them to verbalize their thoughts and actions while completing a specific task within a system.
You can use this method to compile a users' mental model as they verbalize what they are thinking, believing, and predicting, while they complete a task.
Furlough, C. S., & Gillan, D. J. (2018). Mental Models: Structural Differences and the Role of
Experience. Journal of Cognitive Engineering and Decision Making, 12(4).
Nielsen, J. (2012). Thinking Aloud: The #1 Usability Tool. Retrieved from
https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/