Dr Ahmed Kharrufa

Newcastle University

Threshold Concepts in Human-Computer Interaction

Part of this work is now published as

Kharrufa, A.; Gray, C. (2020). Threshold Concepts in HCI Education. In proceedings of the 2nd annual symposioum on HCI Education (EduCHI). DOI: 10.13140/RG.2.2.11854.69444

Introduction

(Sanders & McCartney, 2016) described a threshold concept as “a core concept within a particular discipline, and the transformed view that results from its understanding is specific to (and characteristic of) that discipline.” With (Meyer & Land, 2003) likening it to “a portal, opening up a new and previously inaccessible way of thinking about something.” Examples include ‘complex numbers’ and ‘limits’ in mathematics, ‘signification’ and ‘deconstruction’ in literary and cultural studies, ‘opportunity cost’ in economics (Meyer & Land, 2003), and Fourier Transform in electrical engineering (Reeping. D et al., 2017). As such, threshold concepts are used as lens to investigate the scholarship of teaching and learning and to inform curriculum design of many disciplines (Webb, 2016) with computing science being no exception (Boustedt et al., 2007; Falkner, Vivian, & Falkner, 2013; Sanders & McCartney, 2016).

Investigations of threshold concepts in computing started in 2006 with Eckerdal et al (Eckerdal et al., 2006) with many questions being raised about them in computing at these early stages (e.g. “Threshold Concepts in Computer Science: Do they exist and are they useful?” (Boustedt et al., 2007)) and up till current times (see “Threshold Concepts in Computing: Past, Present, and Future” (Sanders & McCartney, 2016). However, threshold concepts identified in computing are mainly about programming such as ‘abstraction’, ‘pointers’, ‘classes’, ‘recursion’, and ‘polymorphism’ among others and include nothing at all in relation to human-computer interaction. The one identified exception being the example given by (Sanders et al., 2012) ‘user-centered design’ being a potential threshold concept: “the notion that software is designed for other people to use. Introductory students don't generally think in this way (and judging by the software on the market, not all computing professionals do either). The idea, once grasped, transforms the way students view their profession; it affects many aspects of software design, development, and testing; and because it is such a fundamental shift in perspective, it is unlikely to be reversed.” This was followed by the question “Are there threshold concepts (human-centered design, for example) that are not bounded within a single discipline?” (Sanders & McCartney, 2016).

The aim of this work is to 1) start a discussion among HCI educators around threshold concepts in HCI and 2) propose a starting set of potential threshold concepts identified through looking at HCI students’ reflective logs and interviews with graduate HCI students and HCI educators.

Characteristics of Threshold Concepts

The key characteristics of a threshold concept as stated by (Meyer & Land, 2003)

1. Transformative (essential): they change the way a student looks at things in the discipline.

2. Irreversible (probably): they are difficult for the student to unlearn.

3. Integrative (optional): they tie together concepts in ways that were previously unknown to the student.

4. Bounded (possibly often - though not necessarily always): they indicate the limits of a conceptual area or the discipline itself.

Threshold Concepts in HCI literature

Searching for the term “threshold concept” in ‘all fields’ in CHI proceedings (using the ACM Digital Library dl.acm.org search facility) returns zero results )as of 25/07/2019(ِ. Expanding this to include ACM guide to computing literature, returns 57 results. Examining the titles and checking the abstract, when necessary, of these 57 results to exclude those not related to computing education or to the term ‘threshold concept’ in the educational sense (e.g. “threshold concept lattice”), left only 26 papers. After examining these papers to see if they refer to any HCI related threshold concepts only two papers were left with the same authors. These do not go much beyond asking if there are threshold concepts in HCI (Sanders & McCartney, 2016) and suggest, with no evidence provided, that ‘user-cenered design’ is a potential threshold concept in this discipline (Sanders et al., 2012). This survey shows that while the field of computing has been busy discussing threshold concepts in programming, the HCI community has completely overlooked this concept and its potential benefits to the teaching and learning of the discipline.

With the discussion and dispute still going on with regards to what is a threshold concept in programming and what is not (with object-oriented programming frequently gaining and losing this status for example), this work does not claim to suggest a core set of threshold concepts in HCI with this first attempt. The aim is mainly to start the discussion and propose a number of concepts that have the potential to be examined and scrutinized towards being identified as threshold concepts.

Methodology

Students’ reflective logs

The first phase of this research was identifying topics that can be candidates to being considered as threshold concepts by examining students’ reflective logs submitted as part of their coursework for the two HCI modules that I teach: Introduction to HCI (IHCI) and Advanced Interaction Design (AID). Both are optional modules taught to 3^rd year undergraduate students in the School of Computing at Newcastle University.

The reflective logs for the Introduction to HCI module are for the following required readings

• Human Computer Interaction - brief intro, John M Carroll, The Encyclopedia of Human-Computer Interaction, 2nd Ed. | Interaction Design Foundation

• Personas, Lene Nielsen, The Encyclopedia of Human-Computer Interaction, 2nd Ed. | Interaction Design Foundation

• Scenarios and stories, Lene Nielsen, The Encyclopedia of Human-Computer Interaction, 2nd Ed. | Interaction Design Foundation

• "Designing for usability: key principles and what designers think." Gould, John D., and Clayton Lewis. Communications of the ACM 28.3 (1985): 300-311.

• "Affordance, conventions, and design." Norman, Donald A. interactions 6.3 (1999): 38-43.

• Paper prototyping, videos from https://speckyboy.com/10-effective-video-examples-of-paper-prototyping/

• The 4 questions to ask in a cognitive walkthrough, from https://www.userfocus.co.uk/articles/cogwalk.html

• "Interaction design gone wild: striving for wild theory." Rogers, Yvonne. interactions 18.4 (2011): 58-62.

For Advanced Interaction Design, the students were asked to select 10 topics to write on from 10 lectures and 10 seminar sessions covering the following main topics

• Introduction to Interaction Design

• Fieldwork, ethnography and ethnomethodology

• More on analysing fieldwork data: Distributed Cognition and Activity Theory

• Designing for user experience

• Tangible User Interfaces

• Designing for social interactions

• Data-driven life

• Designing for play

• Persuasive systems

• Participatory design

• Designing for usability

• Design workbooks

• Affordances in HCI

• Designing for collective action

For Introduction to HCI, I went through the reflective logs of a total of 80 students (40 out of 67 students who submitted reflective logs for 2017, and 40 out of 98 students who submitted reflective logs for 2018). For Advanced Interaction Design, I went through all the logs of the 38 students in 2017 and all the logs of the 66 students for 2019. Apart from the Advanced Interaction Design students for 2019, students where only asked to write reflective logs (averaging 200 words/entry) without any prompts. For Advanced Interaction Design in 2019, and since I started looking into the topic of Threshold Concept, students where given the following questions as indirect prompt to help in the data analysis.

• 1. State briefly what you see as the main concept/idea discussed in this lecture/seminar (i.e., understanding post-attending the lecture/seminar)

  2. Think about whether this differs from your previous understanding (i.e., understanding pre-attending the lecture/seminar) and if so, whether, and how, it changed how you view/design products in the future?

  3. Do you think that what you have learned in this lecture/seminar is something that you may easily forget like many other topics you've come across?

  4. Can you link what you have learned here to other concepts that you already know, or did it affect your understanding of other things?

As such, there were more entries that can be used as indicators for threshold concepts. For that reason, I ended up excluding some accounts based on the following criteria

• Accounts that, even when saying a topic was new to them, reported an incremental rather than transformative, knowledge.

• Accounts that, while saying a topic is new, just report on it being ‘interesting, or ‘very interesting’ but again do not demonstrate a transformative view.

• Accounts from students who listed everything, or almost everything, as ‘changed how I view/do x.

• Very short entries <100 words (no deep reflection)

For the general reflective logs, I looked at accounts that reflect a transformative, irreversible, and potentially troublesome. Identifying integrative and bounded is more challenging so these were not part of the criteria.

Interviews with HCI educators

The second phase of this work is conducting interviews with HCI educators. The exact method of such interviews depended on context. So far I have carried two interviews with international academic: The first was an opportunistic informal interview at CHI 2019 where notes were taken during the interview, then were shared through email with the academic for validation. The second was a more formal semi-structured interview overs skype. The interview was recorded and then listened to repeatedly to identify and record key points.

Findings

Findings from the analysis of the reflective logs

It is worth noting that students were evidently honest in their reflective logs especially when it came to whether the concepts/information are forgettable or not (e.g. on designing for play “I will likely forgot the content of this lecture as it was very content heavy, with many new concepts and theories to remember, and also because I have no prior knowledge to link it too.” (AID,2019); on technology as experience “This article and seminar are very interesting, but if to consider about will I forget this complex concept easily, I think I will due to the vague understanding of this as I do not thoroughly understand the core idea of the article” (AID,2019), same student saying “I believe what I learned in this lecture could be things I could easily forget” about analysis frameworks (AT & DC) but then saying “I believe what I read will stick with me because this is an area that I want to continuously focus on.” About the ‘technology as experience’ seminar (AID, 2019)

It comes as no surprised that the themes below (table 1) reflect very closely a subset of the topics covered in the lectures and seminars. It is important to note that I am not making the claim that these are threshold concepts, rather, they are themes for candidate concepts after triangulating with more academics interviews and graduate input but the help get the discussion going. For each main identified theme below, I will list a number of sub-themes, and some students’ sample quotes that helped identify the corresponding theme. Due to the length of the original quotes, only the most significant bits are used, but it is important to realize that these quotes are part of a longer essay.

Human-Computer Interaction as a discipline.

This was an interesting findings as students comments indicate that students viewed the discipline itself can be considered as a threshold concept. The main identified sub themes are:

HCI’s evolving and non-static nature: “Something I didn’t initially think about, but the text exposed me to, was about how human computer interaction is always developing and never constant…”, “To me HCI is constantly evolving it does not just understanding and critically evaluating the interactive technologies people use and experience. But it is also about how those interactions evolve as people are more familiar with technology.”

HCI’s wide scope: “I had no idea that the use of HCI spreads to cover such a vast array of fields…”, “Before reading this paper, I didn’t realise that HCI was the culmination of many other fields outside of computer science…”, “From someone who came from very little knowledge on the field, I feel that a new world has been opened up to me, in this field of computer science, and I am very excited to learn more.”

Thinking beyond usability: “One sentence that resonated with me was “Usability now often subsumes qualities like fun, well-being, collective efficacy, aesthetic tension, enhanced creativity, joy, support for human development, and others.”, “I had never actually thought that designers have such intricate jobs that go into such depths in order to create something that people will find easy and pleasant to use.”, “Before reading this, I was narrow-minded about what usability means…”

User-centred design

As suggested by (Sanders et al., 2012), students did flag user-centeredness as a strong candidate to being a threshold concept. “It [cognitive walkthroughs] made me realise that your target audience is really important when designing things and be careful about making assumptions on the user.”, “This [the design process] differs from my previous knowledge in that I was not aware it was defined from the 'user's perspective' … It has changed how I will view future products as I will always consider the person who will be using the product, instead of only visualising how I would use a potential interactive design.”, “When critiquing the examples of interaction design I realised how different users could be, as something that I thought was intuitive was confusing to someone else.”

Fieldwork

The idea of carrying out fieldwork to inform or evaluate a design was also flagged up repeatedly by students. While the main highlighted point doing fieldwork and its ‘in-the-wild’ nature, other sub-themes particularly ethnography as well as the qualitative nature of fieldwork were also flagged up.

In the wild design and evaluation: “A subtle yet major flaw in HCI has been looking at users in a lab environment as opposed to 'wild' which I had never really considered until reading this article.”, “The 'Designing in the Wild' section of this weeks reading was the part that stood out the most to me…This made me re-evaluate what the most important aspects of designing a system is..”, It was also talked about as a ‘troublesome’ concept "This is a very different concept from ones previously studying, and to me it seems a bit confusing, but I can understand how it can be beneficial."

Ethnography: “[Ethnography] changed my previous understanding as it highlights an area of research which I've been ignoring. This allows for easier gathering of natural user interaction which will be valuable in future developments. I will not forget what I learned this lecture...”, “I think the concept of studying the target user group in a sense of working under their context is relatively new to me, so if really want to say is it affects my overall understand about conducting fieldwork. I think what I have learned in this seminar is not something that can easily forget", “For me, this was extremely revealing as to how the intuitive data collection strategy of observing an environment and reporting on it has a far more scientific basis which I previously wasn't aware of and how the planning methodology of ethnography can have a severe impact on the quality of results.”

The ‘qualitative’ nature of fieldwork: “[different fieldwork techniques such as focus groups, diaries, and cultural probes] highlighted to me the value of qualitative, rich research… and made me question my previous approaches to so called user centred designs that weren't that user centred”.

The presence of different data analysis theories and methods

During the advanced interaction design course, students are presented with different theories and methods that can help in observation and analysis of fieldwork such as Distributed Cognition, Activity Theory, and Ethnomethodolgy. There was probably more focus on distributed cognition, but the underlying message was the different insights these different methods bring into focus. “The most interseting part of this lecture for me was learning about the different methods of analysing the same data, this was interesting as I had never preivously thought about how from a small amount of data you could produce alot of differnt types of information simply by using different analytical methods to alanalys it.”, “The idea of analytical perspective and its base in design theory was something I had never properly considered prior to this lecture and has informed my understanding and interpretation of fieldwork going forwards.”, "I definitely won't easily forget this topic [analysis using distributed cognition and activity theory] since this is one of the most interesting topics that I have understood throughout this semester in Advanced Interaction Design.", "I am unlikely to forget the slide on looking at events and information flow... I had never thought of looking at events across time or space…", “This lecture really changed how I view or design a product, especially with the analytic perspective of distributed cognition.”

Personas and scenarios

While for other topics, students either indicate how useful the topic is, or just ignore it, the topics of personas and scenarios was different. Students were split between the very enthusiastic and the very sceptic. Yet, many of the sceptics had a change of heart after using them in their coursework. “In reality I now realise that they’re [personas] a lot more than some nonsense made up figure; instead they are changeable (through evaluation and updates) and useful tools for making design decisions.”, “I found the concept of personas very confusing at first, especially when writing them for my first piece of coursework. I couldn’t get my head around the fact that the person was entirely fictitious and that I had to make up all of the information about them. I kept thinking it would just be simpler to use a file of an actual person and use their needs to help with the design process instead and use real data. However, I did find using personas useful for designing the prototype”, “Before this reading, I did not the goal of making a persona. However, owing to this text, my point of view has completely changed. Indeed, thanks to a persona, it can help designers to maintain the perspective of users-end.”

Usability

Another generic HCI principle that was repeatedly flagged by students is usability not only in terms of designing for usability, but the idea that there are design principles for usability and the process of iterative design to improve usability. While all seems common sense to HCI practitioners, they seem to be interesting new concepts to students.

Designing for usability: “Personally, I had never stopped to consider the usability of common household objects,...I feel that, as a result, I became more aware of how interaction design can affect even the smallest of computer systems,..”, “The most interesting part of the lecture was the usability goals topic. This taught me the goals I need to target when designing a product such as learnability, efficiency, memorability etc. As I have had previous experience designing products I had never thought of these goals, which opened a new perspective for me”

Following design principles: “I am one of the probably many people who, when faced with design rules etc. thought the same as is suggested in this article: that these are pretty much just common sense…despite many considering design principles to be just common sense, not a great deal of developers actually design with them in mind…In the future I would like to keep this article in mind when developing things because I do realise more now that a lot of the things that I use are not actually very intuitive and took quite a lot of learning to use”, “This paper in particular has highlighted my own oversight and persuaded me of the dangers of assigning things to intuition.”

Iterative design: “I particular I loved the idea of iterative design. Having programmed for a little over two years now, I followed other models of design, however never focused solely on the design, test, modify, test, modify, … cycle. I am looking into variations/extensions of this methodology to use in my dissertation.”, “I become aware of as a designer, I can’t do my designer job right at the first time. And it is not a shame. It is a part of the job and it is absolutely normal to fail the design at the first time according to the end-user.”

Affordances

Affordances was another topic that split opinions. However, many still reported at is a transformative concept. “This is the article [on affordances] I’ve found most interesting so far and has resulted in me starting to read his book The Design of Everyday Things.”, “I believe that this article will be one of the most useful thus far in forming the base of my future career in Human-Computer Interaction related fields.” One emerging subtheme was

Affordances and culture: “This article details the cultural constraints on modern HCI designs, this is not something I had previously considered...I find this concept very interesting, it’s strange that a culture forms around the concept of HCI”, “as someone who has lived her whole life in Britain, it’s never occurred to me that others on first sight of these slightly different objects may have to try perceiving how they’re used.”, “I found it interesting how the use of cultural groups affect design, as it’s something I’ve never really considered before, and makes me wonder how the differences between ethnic cultural groups may differ”

Cognitive walkthroughs

While there was a dedicated reading topic about cognitive walkthroughs, the strength and number of logs that included evidence to this being a possible threshold concept came as a surprise to me. Students particularly commented on the idea of identifying and designing for the ‘happy path’. “I did not know that there was a real process in order to evaluate a prototype. For me, the evaluation depends to the feeling.”, “This cognitive walkthrough is simple genius.”, “I was intrigued by the point this article raised that sometimes just determining the best course of action - the ‘happy path’ with no real problems – is enough to highlight interface issues in itself by the number of steps required. This has made me consider going back to double check my own walkthroughs…”, "Creating the happy path was the thing I found the most interesting in this week's reading”

Participatory design

This importance of this concept is best described by the first quote: ”I did not know of participatory design beforehand, but now I will consider it whenever implementing the design phase of a product…I doubt I will forget this lesson after hearing about how the users are ‘given a voice’ in a process of mutual learning for themselves and the designers…This lecture has turned design on its head for me, I always thought of it as a process undertaken by those who are creating the product, not those that are using it." , “[PD] has changed my previous understanding of designing systems in the way that I used to believe that designing and building software systems is only the work of developers and designers and the actual users do not come into play at any point of this process until the system is fully developed, which is the exact opposite of what was taught in this lecture…I will definitely remember this topic as it brings into focus a completely different design approach to what I used to follow, "

“This was very useful information as it was an idea which I had never considered before”, “Before this lecture I knew nothing about PD but now I see the benefits of it…This is not a concept I think I will forget about as it is very important to use the end users in the design process as they are the ones to use it.”

User experience

Beyond usability, students were intrigued by the notion of designing for experience, what that actually means, and the theories and models supporting it. It is perhaps one of the topics with the largest number of comments indicating it threshold concept characteristics. “This session taught me that what I originally assumed UX is, is not nearly the whole picture, and that you must be very careful and consider a lot of different avenues when designing a UX”, “One particular feature of UX design which stood out for me was the disparity between objective and subjective experience, prompting me to look beyond the objective function of my designs.”, “I enjoyed this lecture as it introduced me to a new way of thinking about experience. I originally thought that experience was just how you felt when doing a task. This lecture made me realise that there was a lot that influenced how a user would experience something.”, “Before this lecture I thought the present experience was the main thing that mattered but now I feel that the remembered experience is the main point to focus on. I think some of the concepts discussed will be easily remembered...While discussing this topic I kept linking it to how I interact with objects. How I have felt during certain experiences and applied the areas of discussion to those.”, “We had limited our view to functionality and usability. This 3rd methodology is important…The designing route of functionality -> usability -> the experience, will stick with me”, “Things I have learnt by reading this research paper are completely conflicting to what I used to believe up till now - which was that technology is a standalone field. In reality, technology is integrating more now than ever before with the human cultures and experiences.”

Designing for play

While a rather niche topics, any design can be a playful design and students seemed to particularly appreciate this concept. “Previously, I had never considered elements of play in the field of interaction design.”, “I was definitely inspired by this lecture as the content that was covered in this lecture was completely different to my previous understanding of 'play'. I have never thought about how important is 'play', or its influence, and not to mention why it is important for interaction design.”, “it was eye opening to see how many different stages of play there actually are and this gave me alot of insight into some of the things i need to consider when designing for play”, “It [designing for play] was new to make that the user experience can be improved through play, and the different types of play. Making interfaces fun with noises and movement is something I wouldn't normally consider”

Persuasive technologies

As with designing for user experience, this topic was extensively discussed by students. Many students expressed surprise that services many services they use has a persuasive elements (e.g. when recommending media to listen to or watch) "but media services such as Netflix and mpg meters in cars are both examples of persuasive technologies which I assumed did not fall into this category. This has taught me to look at the bigger picture and look at systems that influence the user socially", “I was interested by this seminar; I'd never heard of this kind of design and how technology can be deemed as persuasive.” Also, as one may expect, some students found the concept ‘unsettling’ “This lecture made me realise how many persuasive technologies I had used without realizing it. I do have to admit that it was slightly unsettling as well, as it wasn't a nice feeling to have the impression that one may not have performed in a certain way out of free-will alone. That thread of thought is enough for an existential crisis though, admittedly. I think in the future I will be more open to this, and conscious of how my actions are being influenced…”, “It really gets me to question the ethicality of these types of technologies as well, as when we start designing products to manipulate the user, it seems like a downward slope…I doubt I'll forget this concept quickly”

Design workbooks

The surprising significance of this topic to students is best summarized by the first comment “I gained the understanding of Design Workbooks far more in depth than previously understood…My biggest takeaway was the ability for a Design Workbook to be structured medium in which one can create a design space, identify key turning points and project futures, rather than just sketching ideas. I doubt this knowledge will fade, as it is a fundamental concept to understand and implement in interaction design, at least in the methodology I wish to employ in my career.”, “I have changed my view on design in a significant way after learning about design workbooks significance. It is much more important than I have previously believed it to be”, “I feel like it is something that I would not easily forget because I already had an understanding of the topic before attending the lecture, but now I will be more likely to use this to assist me in designing my prototype for this module and for any other future projects", “after the seminar it changed the way I design a product. A lot of time, once I have a concept or idea for a new product, I tend to stay with the first idea that I came up with, workbook prevent this from happening. Personally I do not think what I have learned from seminar is something that I might forgot easily”, “It will change my way of designing by collecting all the possible design ideas, in order to keep a broad range of possibilities. Instead of narrowing down into a single design path… Resulting in something that I won't forget about as quickly as I might do with some more involved design methods.”

Tangible user interfaces

“ I'd say the seminar has affected my understanding of user interfaces as a whole, as I had never thought of tangible interactions before…Tangible interactions are definitely something I'm not likely to forget quickly…”, “This lecture introduces new and exciting ideas about tangible interfaces. It changed my opinion on them by providing the answering machine example. I hadn't realised how meaningful and representative a tangible interface can be. I think this is one of the topics from the course that I will remember for a while since I found it really interesting to cover”, “it allowed me to look differently at various interfaces around me: now I am able to analyse why their creators made their decisions on everything they implemented”, “The information learned in this seminar has changed my view on tangible aspects, and it was interesting to see how far the technology which provides user feedback to interactions has advanced, and I would say it links well to a lot of my thoughts in how gaming is progression.”

Interviews with HCI Educators

Interview 1: An academic teaching HCI for 3^rd year students as an optional module. This was an informal interview based on the main question of ‘what do you think are some of the key threshold concepts in HCI, with some or all of the characteristics of being transformative, irreversible, integrative, bounded and potentially troublesome?’. The main identified topics were

• Affordances and specifically the example of ‘Norman Doors’. That is, affordances as discussed by D. Norman.

• Usability guidelines

• Taking the ‘users’ perspective’: Understanding that “the user is not the designer and vice versa”

• Confirmation bias in design: overcoming students’ belief that “I designed it, so it is good”. This was described as a troublesome concept that also affects students’ usability tests.

• Iterative Design Cycle: In relation to the ‘I designed it so it is good’, ‘iterative design cycle’ is another threshold concept and that designs are ever-changing and they are never right in the first time.

• Sketchbooks

With respect to advanced topics in Interaction design,

• At a high level, doing ‘research’, ‘design’, and ‘evaluation’ are all transformative threshold concepts for the students.

• Applying theory to practice. For example, actually doing ‘participatory design’ and understanding the deep level politics involved in participatory design. Also understanding ‘when to apply’ and ‘how to fit this’ in the user-centred design cycle.

References

Boustedt, J., Eckerdal, A., McCartney, R., Moström, J. E., Ratcliffe, M., Sanders, K., & Zander, C. (2007). Threshold Concepts in Computer Science: Do They Exist and Are They Useful? SIGCSE Bull., 39(1), 504–508. https://doi.org/10.1145/1227504.1227482

Eckerdal, A., McCartney, R., Moström, J. E., Ratcliffe, M., Sanders, K., & Zander, C. (2006). Putting Threshold Concepts into Context in Computer Science Education. SIGCSE Bull., 38(3), 103–107. https://doi.org/10.1145/1140123.1140154

Falkner, N. J. G., Vivian, R. J., & Falkner, K. E. (2013). Computer Science Education: The First Threshold Concept. In Proceedings of the 2013 Learning and Teaching in Computing and Engineering (pp. 39–46). Washington, DC, USA: IEEE Computer Society. https://doi.org/10.1109/LaTiCE.2013.32

Kharrufa, A.; Gray, C. (2020). Threshold Concepts in HCI Education. In proceedings of the 2nd annual symposioum on HCI Education (EduCHI). DOI: 10.13140/RG.2.2.11854.69444

Meyer, J. H. F., & Land, R. (2003). Threshold concepts and troublesome knowledge: linkages to ways of thinking and practicing. (J. H. F. Meyer & R. Land, Eds.), in Rust, C. (ed.), Improving Student Learning: Improving Student Learning Theory and Practice – Ten Years On. Oxford: Oxford Centre for Staff and Learning Development. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.476.3389

Reeping. D, McNair, L., Harrison, S., Knapp, R., Lester, L., Martin, T., … Wisnioski, M. (2017). How are Threshold Concepts Applied? A Review of the Literature. In ASEE Annual Conference & Exposition. Retrieved from https://peer.asee.org/board-97-how-are-threshold-concepts-applied-a-review-of-the-literature

Sanders, K., Boustedt, J., Eckerdal, A., McCartney, R., Moström, J. E., Thomas, L., & Zander, C. (2012). Threshold concepts and threshold skills in computing. In Proceedings of the ninth annual international conference on International computing education research - ICER ’12 (p. 23). New York, New York, USA: ACM Press. https://doi.org/10.1145/2361276.2361283

Sanders, K., & McCartney, R. (2016). Threshold concepts in computing: Past, present, and future, 91–100.

Webb, A. S. (2016). Threshold concepts and the scholarship of teaching and learning. In and M. T. F. Land, Ray, Jan HF Meyer (Ed.), Threshold Concepts in Practice (pp. 299–308). Brill. Retrieved from https://brill.com/downloadpdf/book/edcoll/9789463005128/BP000023.pdf