Isa Jahnke, Nathan Riedel, Kanupriya Singh, & Joi Moore (2021). Advancing Sociotechnical-Pedagogical Heuristics for the Usability Evaluation of Online Courses for Adult Learners. In: Online Learning Journal (OLJ), Vol 25, No 4 (2021), https://olj.onlinelearningconsortium.org/index.php/olj/article/view/2439 Open access.
This website contains all 195 items used to create the sociotechnical-pedagogical heuristics for evaluating online courses.
1.1 Employ a conversational tempo and not speak instructions too quickly. (Van der Meij and Van der Meij, 2013)
1.2 Use a conversational tone in the narration. (Mayer, 2002)
1.3 Learners are asked to introduce themselves to the class. (Quality matters, 2018)
1.4 The course provides learners with opportunities to access extended feedback from instructors, experts, peers, or others through e-mail or other Internet communications. (Reeves et al., 2002)
1.5 The self-introduction by the instructor is professional and is available online. (Quality matters, 2018)
1.6. Instructor plays different roles. (e.g. expert, mentor, coach, learning-companion) (Jahnke, 2015)
1.7 Authentic stories, anecdotes, emotion, or human conflict are used to engage learners and show real-world relevance when appropriate. (Nokeleinen, 2006)
1.8 Requirements and/or policies for student interaction and responses to instructor-initiated interaction are clearly stated and provides a basis for the instructor to evaluate learner performance. (Jahnke, 2015)
1.9 Etiquette expectations (sometimes called ‘netiquette’) for online discussions, email, and other forms of communication are stated clearly if the course involves interaction and communication between the instructor and/or among students. (Quality Matters, 2018)
2.1 Learning activities are active and facilitate engagement via learner-content, learner-learner, and learner-instructor interactions. (Jahnke, 2015)
2.2 Activities and interactions engage learners by “doing” something and allow students to produce a deliverable. (Jahnke, 2015)
2.3 Activities engage students in higher-level thinking skills, including critical and creative thinking, analysis, and problem-solving. (Jahnke, 2015)
2.4 A wide variety of learning strategies may have to be employed, including memorization, direct instruction, drill-and-practice, deduction, and induction. (Reeves, 1994)
2.5 Activities promote active reflection, collaboration, discussion, and real-world engagement. (Reeves, 1994)
2.6 Activities are designed to align with course/module objectives by incorporating interaction, which promotes learner achievement of the stated objectives by actively engaging the learner with course content. (Jahnke, 2015)
2.7 Learners can make decisions about what sections/ content to study through interactive material. (Reeves, 1994)
2.8 The course supports various modes of learning – asynchronous, synchronous, hybrid for flexible learning. (Dringus & Cohen, 2005)
2.9 Follow the user’s mental plan in describing an action sequence. E.g. Instructions are presented with goals/ purpose and tell the user about actions and states that lead to goal achievement. (Van der Meij and Van der Meij, 2013)
2.10 Course tools promote student engagement and active learning by facilitating interactions with instructor, course materials, and other learners. (Jahnke, 2015)
2.11 The purpose of the activities are clearly stated so students understand how they tie into course objectives. (Jahnke, 2015)
2.12 Instructional videos should provide learners with practice to consolidate and enhance learning and allow learners to judge their own learning.(Van der Meij and Van der Meij, 2013)
2.13 The manner of submission for assignments/ assessment is clear. (Sims, Dobbs, & Hand, 2002)
2.14 Course activities contain clear instructions and due dates. (Jahnke, 2015)
2.15 Instructions match the learning activity. (Quality Matters, 2018)
3.1 It is easy for students to get access to the course/ materials for reference even after they are done with the course. (Dringus & Cohen, 2005)
3.2 Users should not have to wonder whether different words, situations, or actions mean the same thing. (Nielsen, 1994)
3.3 Buttons and selections are of sufficient viewable size. (Dringus & Cohen, 2005)
3.4 The system allows the learner to leave whenever desired, but easily return to the closest logical point in the system. (Nielsen, 1994)
3.5 It is easy to upload files. (Dringus & Cohen, 2005)
3.6 Files and media are easy to view and download. (Dringus & Cohen, 2005)
3.7 Users are able to edit messages in discussion posts. (Dringus & Cohen, 2005)
3.8 Give learners controls (pause, go back, go forward, skip...) to allow them access the video at their own pace. (Boyle, 1997)
4.1 Materials and contents have consistent layouts and designs. (Chao, Saj, and Tessier, 2006)
4.2 Users should not have to wonder whether different words, situations, or actions mean the same thing. (Nielsen, 1994)
4.3 In other words, information is presented with a structured layout by… (Stone et al., 2005)
4.4 Links are easily recognizable. (Dringus & Cohen, 2005)
4.5 No extraneous or irrelevant information, visual noise, or unnecessary styles are present. (Moore et al., 2014)
4.6 Links and icons stand out from their background. (OLC Checklist, 2018)
4.7 Underlines are only used to indicate working links to relevant sections. (Nielsen, 2004)
4.8 Design, font, text size and heading styles look consistent from one part of the course to another. (Benson et al., 2002)
4.9 Content is segmented/ chunked. (Van der Meij and Van der Meij, 2013)
4.10 If something is important for the user, it should be placed in a prominent position. (Stone et al., 2005)
4.11 The hierarchy of content in a page or document is clearly indicated through heading styles and/or sizes. (Unpublished)
4.12 White space or negative space is used around content to help increase comprehension and reduce eye fatigue that occurs with large blocks of text. (Dringus & Cohen, 2005)
4.13 Proper and consistent use of graphics, font and size: simpler fonts are chosen over more ornate fonts, and the number of font families is limited to one or two. (Benson et al., 2002)
4.14 Use highlighting to signal screen objects or locations. (Van der Meij and Van der Meij, 2013)
4.15 Proper use of color contrast: Text is clearly distinguishable from the background. Black text on a white or light background is the most readable. (Quality Matters, 2018)
4.16 If the content of the next page(s) is the continuation of a previous page, all pages should share the same page title. (Unpublished)
4.17 Flashing and blinking text are avoided. (OLC Checklist, 2018)
4.18 Tables, if used are accompanied by a title and summary description. (OLC Checklist, 2018)
4.19 Editing and proofreading errors (spelling, grammar, punctuation, word choice, syntax) are minimal. (OLC Checklist, 2018)
4.20 Texts and/or figures in images should be easily seen. (Xavier University, 2018)
4.21 Font sizes are of proper size. (Dringus & Cohen, 2005)
4.22 Blocks of information are broken up into incremental chunks to support memory.(Xavier University, 2018)
4.23 Page title and page content should match. (Unpublished)
4.24 The design and presentation of information is consistent: layout, color, text size, text style, font etc. (Presentation of pieces of information) (Stone et al., 2005)
4.25 Things go together are grouped together using the same shape or color. (Stone et al., 2005)
4.26 Sizes of videos and images should be consistently the same and appropriate. (Dringus & Cohen, 2005)
4.27 Use display of the whole screen. (Van der Meij and Van der Meij, 2013)
4.28 Acronyms or abbreviations are only used if the full form is earlier mentioned. (Unpublished)
5.1 If the course includes links to external resources, the links are kept up-do-date. (Benson et al., 2002)
5.2 Resources are provided in a manner that replicates as closely as possible their availability and use in the real world. E.g Resources should be listed on the main menu. (Benson et al., 2002)
5.3 The messaging system to the instructor and other classmates is accessible. (Dringus & Cohen, 2005)
5.4 Frequently used technology tools are easily accessed. (Quality Matters, 2018)
5.5 The syllabus should be listed on the main navigation menu for quick access to it. (Xavier University, 2018)
5.6 Language use is consistent throughout the course. (Chao, Saj, and Tessier, 2006)
5.7 Sections, pages, links, files, and icons are labeled with common, brief, easy-to-understand, self-describing, and meaningful names/ terminologies. (Terms/Labels) (Moore et al., 2014)
5.8 Files are well organized according to sequence. (Dringus & Cohen, 2005)
5.9 The headings in the breadcrumbs match the headings listed in the main menu and on the page. (Dringus & Cohen, 2005)
5.10 Menu titles are brief yet long enough to communicate the intended purpose. (Dringus & Cohen, 2005)
5.11 Draw attention to the interconnection of user actions and system reactions. (Van der Meij and Van der Meij, 2013)
5.12 The course provides access to all the resources necessary to support effective learning. (Benson et al., 2002)
5.13 Things that go together should be grouped together. For example, all information related to destination should be in the destination column. (Stone et al., 2005)
5.14 The consequences of plagiarism, cheating, and failure to properly cite copyrighted materials are emphasized. (OLC checklist, 2018)
5.15 The course models the academic integrity expected of learners by providing both source references and permissions for use of instructional materials. (Quality Matters, 2018)
5.16 Course instructions articulate or link to an explanation of how the institutions academic support services and resources can help students succeed in the course and how students can access the services. (Sims, Dobbs, & Hand, 2002)
5.17 The course provides learners with information on protecting their data and privacy. Reviewers look for links to privacy policies and measures taken to product student data. (Quality Matters, 2018)
6.1 Navigation instructions should cover all alternative ways to navigate. (Unpublished)
6.2 Course design enables learners to easily locate where they are within the course. (OLC checklist, 2018)
6.3 Pages and sections mentioned in the instructions or throughout the course should be linked to in order to provide shortcuts for efficient navigation. (Douglas, 2017)
6.4 Related materials and resources are joined through hyperlink. (Moore et al., 2014)
6.5 Hyperlinked metadata is employed to enable learners to efficiently access related resources and tools. (Moore et al., 2014)
6.6 Interface provides adequate Back/Next button functionality to navigate to the respective screen. (Dringus & Cohen, 2005)
6.7 The interface of the system speaks for itself so that learners do not have to extensively consult instructions, a manual or other documentation to navigate in the course. (Xavier University, 2018)
6.8 All links within the course, external and internal, are working properly; there are no broken links. (Xavier University, 2018)
7.1 The system keeps the learner informed about what is happening, through appropriate feedback within reasonable time. E.g. The user is provided with information that the system is operating correctly or the status of download is communicated clearly etc. (Benson et al., 2002)
7.2 Technology interactions/ learning products adhere to widely recognized standards for technology/software interactions. (Benson et al., 2002)
7.3 The system is designed so that the learner recognizes when and where he/she has made a mistake. (Error Prevention)(Nielsen, 1994)
7.4 Technology works consistently and reliably. (Jahnke, 2015)
7.5 The intended functionality for each button and/or option is clear. (Dringus & Cohen, 2005)
7.6 Technology is compatible to all devices. (Satar, 2007).
7.7 Course identifies large files to help learners consider download times and alternative (smaller) files are provided where appropriate. (Quality matters, 2018)
7.8 The system or learning material should provide the student with encouraging and immediate feedback. (Nokelainen, 2006)
7.9 Online resources open in new windows. (Chao, Saj, and Tessier, 2006)
7.10 The course includes a map or table of contents that allows the learner to see what has been seen and not seen. (Benson et al., 2002)
8.1 Images and graphics contain alt text or descriptive captions. (OLC Checklist, 2018)
8.2 If not, the novel interactions are appropriate for the content and target audience. (Benson et al., 2002)
8.3 Course provides user accessibility statements for all technologies and/or materials required in the course. (Quality Matters, 2018)
8.4 Course instructions articulate or link to the institution’s accessibility policies and services. (Quality Matters, 2018)
8.5 The course provides guidelines and/or Q&A for disabled students to seek for technology help and/or academic help. (Quality Matters, 2018)
8.6 Course includes links to privacy policies for technology tools. (Quality Matters, 2018)
8.7 For accessibility, provide a means for the learner to access the text of the narration. (Van der Meij and Van der Meij, 2013)
9.1 Use a spoken human voice over the narration. (Van der Meij and Van der Meij, 2013)
9.2 Tablet drawing tutorials (e.g. Khan-style table drawing tutorials) are more engaging than Powerpoint slide presentation with voice-over. (Guo et al., 2014)
9.3 Videos should display the instructor’s talking head at opportune times. (Guo et al., 2014)
9.4 Give videos meaningful titles and organize them so learners can easily find relevant material. (Van der Meij and Van der Meij, 2013)
9.5 Actions/ images on screen are in sync with narration. (Van der Meij and Van der Meij, 2013)
9.6 Keep videos short. However, video length should be governed by the nature and complexity of the content: 1-3 minutes: introductions, brief overviews; 2-5 minutes: a topic, overview, more substantial learning; 5-15 minutes: complex topics, sections, modules; 15+ minutes: full modules, learning programs, documentaries. (Obsidian learning, n/d)
9.7 Videos are produced with a more personal feel e.g. filmed in an informal setting. (Guo et al., 2014)
9.8 Videos should instructors’ high enthusiasm e.g. instructors speak fairly fast with high enthusiasm. (Guo et al., 2014)
9.9 When providing tutorials, use a first person rather than a third-person perspective. (Van der Meij and Van der Meij, 2013)
9.10 Short videos of less than 10 minutes are used rather than long. (Mayer, 2002)
9.11 Course enables students to learn better with the inclusion of conversational style lessons in the content. (Mayer, 2002)
9.12 Course narration in multimedia lessons is spoken in a friendly human voice rather than a machine voice to enable better learning for the students. (Mayer, 2002)
9.13 People learn better when corresponding words and pictures are presented near rather than far from each other on the page or screen.(Mayer, 2002)
9.14 As for videos if used, animation and narration are presented concurrently rather than consecutively for better learning. (Mayer, 2002)
9.15 Powerpoint or lessons of the similar kind is presented in user-paced segments rather than as a continuous unit. (Mayer, 2002)
9.16 Provide both spoken narration and related imagery/action. (Van der Meij and Van der Meij, 2013)
10.1 Associate content with emotion (positive or negative) to increase motivation and retention. (Van der Meij and Van der Meij, 2013)
10.2 Content can be organized using hierarchical classification. (Moore et al., 2014)
10.3 Content can be organized using faceted classification. (Moore et al., 2014)
10.4 Content can be grouped by main categories and organized based on four key facets: action (post, read, submit, view), time, topic, and type (article, discussion board, assignment, grade, announcement, grade, discussion board etc). (Moore et al., 2014)
10.5 The learning materials should contain diverse assignments and provide enough time for the learner to finish in a thoughtful and meaningful way. (Dringus & Cohen, 2005)
10.6 Introduce new concepts by showing their use in context. In other words, knowledge is presented at the point when the user needs that information to perform the task. (Van Merrienboer, Kirshner & Kester, 2003)
10.7 Information and instructions are provided regarding how the tools support the learning objectives or competences. (Quality Matters, 2018)
10.8 Information and instructions are provided regarding how the tools required/ recommended to use in the course support the learning objectives or competences. (Quality Matters, 2018)
10.9 The instructional materials contribute to the achievement of the stated course and module/unit learning objectives. (Quality Matters, 2018)
10.10 Course materials and resources can be grouped by main categories and organized based on a hierarchy of classes, divisions, and subdivisions. For example, course material is grouped by main categories such as required readings, assignments, discussion activity, and resources. (Moore et al., 2014)
10.11 Technical skills required for participation in course learning activities scaffold in a timely manner (orientation, practice, and application - where appropriate). (Quality Matters, 2018)
10.12 For blended courses, instructions make clear which materials are to be used in the face-to-face classroom and which are specific to the online portion of the course. (OLC Checklist, 2018)
10.13 The distinction between required and optional materials is explained clearly. (Quality Matters, 2018)
10.14 Materials consistently indicate when activities or assessments take place onsite versus online. (Stein & Graham, 2014)
10.15 The instructor posts guidelines that assist the learner in identifying relevant materials. (Quality Matters, 2018)
11.1 Course content is “chunked” into small and manageable pieces of content. (Moore et al., 2014)
11.2 One topic or idea is introduced at one time. (Moore et al., 2014)
11.3 When presenting one topic/ idea, follow the “rule of seven” guideline: presenting a maximum of seven pieces of content at a time. (Clement, 1985)
11.4 Instructional materials are provided in a way/ order to be useful to the learner in completing learning activities. (OLC Checklist, 2018)
11.5 The manner of presenting new knowledge and skills is varied, including text, lists, organizational activities, reflective quizzes, readings, images, graphs, charts, etc. (Xavier University, 2018)
11.6 Content/ information is presented logically and intuitively. (OLC Checklist, 2018)
11.7 Content elements are presented in a logical sequence. (Dringus & Cohen, 2005)
12.1 There is no extraneous processing in using materials, resources, and multimedia. (Moore et al., 2014)
12.2 The instructional materials represent up-to-date theory and practice in the discipline. (Quality Matters, 2018)
12.3 A variety of instructional materials are used in the course like textbooks, publications, instructor-created resources, websites, multimedia. (Quality Matters, 2018)
12.4 A variety of tools are used in the course and support the course learning objectives or competencies. Examples of tools are discussion boards, chat rooms, grade book, social media, games, whiteboard, wikis, blogs, virtual classrooms, web conferencing, etc. (Quality Matters, 2018; Jahnke, 2015)
12.5 Interactive video is more preferred than non-interactive video. (Zhang et al., 2006)
12.6 The materials provide meaningful interactions for the user, rather than simply presenting long sections of text. (Benson et al., 2002)
12.7 The visual text rather than audio/ video will be helpful for learners to understand, and retrieve information. (Mayer, 2002)
12.8 Various forms of media are included for remediation and/or enrichment. (Benson et al., 2002)
12.9 The course engages the learner in content-specific tasks to complete and problems to solve that take advantage of the state-of-the-art of online learning capabilities. (Benson et al., 2002)
12.10 Users must be able to interact with videos like they often do in their daily lives like watching in full view or playing backward or forward. (Schade, 2014).
12.11 The interactivity with technology has meaningful learning purposes. (Reeves et al., 2002)
12.12 Images if used should be relevant to learning content and enhance the knowledge acquisition. (Lenzner et al., 2012).
12.13 Media included has a strong connection to the objectives and design of the course. (Benson et al., 2002)
13.1 Course information specifies how successful completion of the course will be recognized. (Jahnke, 2015)
13.2 Activities and assessments are adequate and reasonable for the course duration. (Xavier University, 2018)
13.3 The types of assessments selected measure the stated learning objectives and are consistent with the course activities and resources.(Quality matters, 2018)
13.4 The course provides opportunities for self-assessment that advances learner achievement. (Benson et al., 2002)
13.5 Ongoing assessments are conducted to verify the learner’s readiness for the next lesson. (Jahnke, 2015)
13.6 The assessment instruments selected are sequenced or in order relevant to what they have learned, varied and appropriate to the student work being assessed. (Quality matters, 2018)
13.7 Timeframe, time limit, and format of assessments are appropriate to the learned content. (Jahnke, 2015)
13.8 Wherever appropriate, higher order assessments (e.g. analysis, synthesis, and evaluation) are provided rather than lower order assessments (e.g., recall and recognition). (Benson et al., 2002)
13.9 Teacher gives active, specific, and consistent feedback and feedforward to student learning progress. (Jahnke, 2015)
13.10 Teacher has set up specific steps for students to ask for feedback, e.g. after initial draft of student products, students get feedback from teacher before students can continue. (Jahnke, 2015)
13.11 Assessment provides sufficient feedback to the learner. (Jahnke, 2015)
13.12 Feedback from a variety of sources (from teachers, and other peers) corrects, clarifies, amplifies, and extends learning. (Jahnke, 2015)
13.13 The grading rubric is consistent with the graded activity and assignment. (Quality Matters, 2018)
13.14 Grading criteria are outlined in the course syllabus and within the assignment or activity itself. (Quality matters, 2018)
13.15 Specific and descriptive criteria are provided for the evaluation of students’ work, participation, and mastery and are tied to the course grading policy. (Quality matters, 2018)
13.16 Criteria and procedures for peer review and evaluation are clear. (Quality matters, 2018)
13.17 The relationship between graded elements and the final grade is clear. (Jahnke, 2015)
14.1 Syllabus contains information regarding the course summary or the main parts of the course. (Xavier University, 2018)
14.2 Syllabus contains information regarding instructor presence and response time on assignments on the syllabus. (Xavier University, 2018)
14.3 Syllabus provides titles of assignments and points relevant. (Fink, 2012)
14.4 Syllabus communicates student expectations for discussion participation. (OLC Checklist, 2018)
14.5 Syllabus contains information regarding ‘Drop/withdraw/return’ policy. (Das, 2012)
14.6 Course overview and/or introduction mentions if any prerequisite knowledge in the discipline and/or any required competencies are required for the successful completion of the course. (Quality Matters, 2018)
14.7 Consequences of insufficient class participation are clearly stated and fair. (Chao, Saj, and Tessier, 2006)
15.1 The course/module learning objectives describe outcomes that are measurable and consistent with the course-level objectives. (Quality Matters, 2018)
15.2 All learning objectives are stated clearly, written from the students’ perspective, and prominently located in the course. (Jahnke, 2015)
15.3 The learning objectives are appropriately designed for the level of the course. (Quality Matters, 2018)
15.4 Students know where to find the goals/ objectives and students understand the goal. (Jahnke, 2015)
15.5 The objectives/ goals of the course and each module are present so that learners know what objectives/ goals they can achieve. (Reeves, 1994)
15.6 Learners are introduced to the purpose and structure of the course. (Quality Matters, 2018)
15.7 Objectives are developed based on Bloom's taxonomy. (Bloom’s taxonomy, 1956)
16.1 Home page includes a brief course description. (Xavier University, 2018)
16.2 Clear course navigation instructions are provided on the Home page. (Xavier University, 2018)
16.3 How to ‘get started’ is present and stands out in the Home page. (Xavier University, 2018)
16.4 Instructions for navigation match what actually happens in the environment. (Xavier University, 2018)
16.5 All help and documentation is written clearly and succinctly. (Benson et al., 2002)
16.6 The course provides help and documentation that is readily accessible to the user when necessary. (Benson et al., 2002)
16.7 User offered with sufficient FAQ, system, and human support to obtain necessary help. (Quality Matters, 2018)
16.8 Instructor contact information for course questions is provided. (OLC Checklist, 2018)
16.9 Students are introduced to the content organization, important documents, course policies, expectations, and assignment submission procedures at the beginning of the course. (Moore et al., 2014)
16.10 Requisite skills for using technology tools (websites, software, and hardware) are clearly stated and supported with resources. (Quality Matters, 2018)
16.11 Course overview can be included in a course syllabus in which students should be directed to the syllabus at the beginning of the course. (Quality Matters, 2018)