Description

View the full paper version with tables and images here!

2nd Workshop on Challenges and Opportunities of Learning Analytics Adoption in Higher Education: Prerequisites for Smart and Engaging Dashboards

L.W. van Meeuwen1, R. Conijn1, E. Ventura-Medina1, M. Specht2, B. Rienties3, O. Viberg4, R. Kaliisa5, I. Jivet6
 

1 Eindhoven University of Technology, The Netherlands
2 Delft University of Technology, The Netherlands
3 The Open University, The United Kingdom
4 Royal Institute of Technology, Sweden

 5 University of Oslo, Norway

6 University of Hagen, Germany

ABSTRACT: Despite the potential of learning analytics (LA) to enhance student learning in higher education, the adoption of learning analytics is still lagging. In this workshop, participants will share evidence and experiences with implementations of LA applications in higher education. The workshop aims to lower the threshold for a wider audience to engage with study data, thereby scaling LA. The focus is on increasing stakeholders’ engagement and data availability, with an emphasis on the methods and conditions for making data available to LA stakeholders. A key area of interest is the use of Learning Analytics Dashboards (LADs), aimed, for example, at improving student self-directed and self-regulated learning by actively engaging stakeholders to use the data. The workshop will provide insights into applications of dashboards, the commonalities and differences between them, as well as the potential and the use of AI to empower LADs. It will also focus on determining the theoretical principles that are not yet implemented in practice, and what is needed to accomplish more evidence-based LAD design. Accordingly, the workshop will provide a platform for a thorough discussion on the mapping between theory and practice in LA adoption.

Keywords: LA Adoption at Scale, Learning Analytics Dashboards, LAD Design, LA and AI, Self-Regulated Learing, Self-Directed Learning.

1. Introduction

Learning Analytics (LA) can enhance learning and has the potential to increase educational quality in higher education (e.g., Viberg & Gronlund, 2023). Several higher education (HE) institutions have ambitions to exploit opportunities of LA for improving the quality of education (e.g., The Open University, 2015; Lopez-Arteaga et al., 2023). Yet adoption is still lagging (Hernandez-de-Menendez et al., 2022). In our first workshop on “Challenges and Opportunities of Learning Analytics Adoption in HE” (Van Meeuwen, et al., 2024) this topic was successfully approached from the perspective of ‘stakeholder engagement’ and ‘obtaining data’. The (preliminary) analysis of last year's workshop suggests that a major opportunity for scaling LA lies in lowering the threshold for a wider audience to engage with study data in teaching and learning.

Therefore, our second workshop will focus specifically on the methods and conditions for making data available and actionable to LA stakeholders in HE. The objective is to find ways to actively engage stakeholders in HE to use these data, with the ultimate goal of improving student learning, including self-directed and self-regulated learning. In addition, we aim to  find examples where AI has been applied. To this end, in this workshop we focus on displaying data within learning support systems, such as learning analytics dashboards with the particular aim to make these data engaging and actionable.

Workshop aim

Sharing knowledge and practical insights about evidence-informed interactive learning analytics dashboard design to enable student learning.

1.1 Engaging with Data

A common approach to providing students and educators with insights into learners, their learning processes, and their contexts is through the use of LA Dashboards (LADs). LADs could be implemented as stand-alone dashboards or integrated within an Intelligent Tutoring Systems or Learning Management Systems. Since 2020, research interest in LADs has significantly increased, with a notable trend of leveraging LADs to unlock the potential of LA in fostering student autonomy in learning (cf. Masiello et al., 2024). A series of systematic reviews reveal that LADs’ long-term use and impact is still limited, potentially due to the shortcomings in both the design and evaluation of LADs (Jivet et al., 2018; Matcha et al., 2020; Kaliisa et al., 2024). This underlines the idea that many bumps still should be overcome towards scalable LA applications beneficial for learning (cf., Alzahrani, 2023).

In particular, the effects found of a LAD on academic performance are limited, with small to negligible effect sizes (Kaliisa et al., 2024). With some studies even showing negative effects of dashboard use (Lonn et al., 2015). These results are partly due to under-powered studies, showing the need to scale up the evaluation of LADs. In addition, it has been argued that the limited impact might be caused by the limited grounding of the LAD design in learning theories (e.g., Matcha et al., 2020). As stated by Masiello et al. (2024): “there is a need for a clear connection between the design of LADs and what educational science asserts works for learning” (p 9.).

Recently, more studies ground their LAD design into learning theories, of which self-regulated learning theory is most common (Heikkinen et al., 2022). Heikkinen et al. (2022) reported on 27 studies which used LADs to support sel-regulated learning. Out of these, most studies focus on the performance phase as compared to monitoring and reflection. However, the main focus on performance might improve extrinsic motivation rather than intrinsic motivation, and might not always be actionable.

Therefore, it is necessary to focus on additional strategies, such as adding specific prompts, smart filtering of the data to be presented, letting users interact with the data, or providing targeted strategy instructions − potentially with the use of AI − to make the data actionable and useful over time (Jivet et al., 2021). In addition, (generative) AI can play an increasing role in enhancing the explainability, functionality and data aggregation in LADs (Ouyang & Zhang, 2024), resulting in potential further personalization of the dashboard and creating adaptive interventions. This workshop therefore focuses on: (1) what is needed for learning analytics dashboards to be impactful and engaging over time; (2) grounding of learning analytics dashboard design in theory.

2 Workshop format and submissions

To the participants, the workshop first yields insights in applications of dashboards in HE, and what they have in common. The combination of participants with a LA researchers and practitioners background will yield a thorough discussion on the overlap between theory and practice and where practice violates with theory.

Second, the activities will focus on determining the theoretical principles that are not yet implemented in practice, finding out why, and gaining insights in what is needed to accomplish more evidence-based LAD design. This comes with the exchange of knowledge on specific and generic functionalities for tooling and (technical) bottlenecks for scaling up.

The workshop design will allow for half a day meeting and comprises a combination of an Interactive Workshop and a Mini-track Symposium. The design (see Table 1) includes discussions, group discussions, presentations, and voluntary contributions. We ask participants to consider a contribution detailing (case-)studies where the LAD influences students learning, specifying the LAD design process and stakeholder engagement (such as participatory design and institutional collaboration). Contributions of 10-minute demos may be submitted in the form of an abstract of up to 300 words. Based on the abstract, the workshop organization will carefully review the submissions and select a compelling array of diverse contributions that fit the workshop structure, workshop schedule, and enrich the workshop. Further interaction before and after the workshop will be supported by a website.

2.1 Attract Participants and Community Building

The organizing committee anticipates two main groups of participants. First, scholars who contribute to the LAD domain. Second, participants who want to share more practical experiences about embedding dashboard functionalities in education and curriculum design (e.g., education designers, teaching staff, (program) managers). The organizers will use their professional networks as well as the university alliance networks of their institutions to approach both target groups. The organizers receive examples for LinkedIn posts and information to share on relevant social media and mailing lists to recruit broadly. The participants of the workshop of last year will receive a personal invitation to participate in this year’s workshop. An extended website will become available as preparation, getting to know people with LAD-design interests and sharing the findings and relevant documents prior and after the workshop meeting. We are aiming for 25 – 35 participants.

Table 1 Program Design Outline

---

REFERENCES

Alzahrani, A. S., Tsai, Y. S., Iqbal, S., Marcos, P. M. M., Scheffel, M., Drachsler, H., ... & Gašević, D. (2023). Untangling connections between challenges in the adoption of learning analytics in higher education. Education and Information Technologies, 28(4), 4563-4595.

Heikkinen, S., Saqr, M., Malmberg, J., & Tedre, M. (2023). Supporting self-regulated learning with learning analytics interventions–a systematic literature review. Education and Information Technologies, 28(3), 3059-3088.https://doi.org/10.1007/S10639-022-11281-4

Hernández-de-Menéndez, M., Morales-Menendez, R., Escobar, C. A., & Ramírez Mendoza, R. A. (2022). Learning analytics: state of the art. International Journal on Interactive Design and Manufacturing (IJIDeM), 16(3), 1209-1230. https://doi.org/10.1007/s12008-022-00930-0

Jivet, I., Scheffel, M., Specht, M., & Drachsler, H. (2018, March). License to evaluate: Preparing learning analytics dashboards for educational practice. In Proceedings of the 8th international conference on learning analytics and knowledge (pp. 31-40). https://doi.org/10.1145/3170358.3170421

Jivet, I., Wong, J., Scheffel, M., Valle Torre, M., Specht, M., & Drachsler, H. (2021, April). Quantum of choice: How learners’ feedback monitoring decisions, goals and self-regulated learning skills are related. In LAK21: 11th international learning analytics and knowledge conference (pp. 416-427). https://doi.org/10.1145/3448139.3448179

Kaliisa, R., Misiejuk, K., López-Pernas, S., Khalil, M., & Saqr, M. (2024, March). Have Learning Analytics Dashboards Lived Up to the Hype? A Systematic Review of Impact on Students' Achievement, Motivation, Participation and Attitude. In Proceedings of the 14th Learning Analytics and Knowledge Conference (pp. 295-304). https://doi.org/10.1145/3636555.3636884

Lonn, S., Aguilar, S. J., & Teasley, S. D. (2015). Investigating student motivation in the context of a learning analytics intervention during a summer bridge program. Computers in Human Behavior, 47, 90-97. https://doi.org/10.1016/J.CHB.2014.07.013

Lopez-Arteaga, I., Koenraad, P., Rijk, K. C.H.A.M., (2023). TU/e Vision on Education. Eindhoven: University of Technology.

Masiello, I., Mohseni, Z., Palma, F., Nordmark, S., Augustsson, H., & Rundquist, R. (2024). A Current Overview of the Use of Learning Analytics Dashboards. Education Sciences, 14(1), 82. https://doi.org/10.3390/educsci14010082

Matcha, W., Gašević, D., & Pardo, A. (2019). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE transactions on learning technologies, 13(2), 226-245. https://doi.org/10.1109/TLT.2019.2916802

Ouyang, F., & Zhang, L. (2024). AI-driven learning analytics applications and tools in computer-supported collaborative learning: A systematic review. Educational Research Review, 44, 100616.

The Open University (2015). Policy on ethical use of student data for learning analytics. The Open University. September 2014 (2015), 1–11.

Van Meeuwen, L. W., Rienties, B., Drachsler, H., Viberg, O., Conijn, R., Vonk, C., Brijan, J. W., & Specht, M. (2024). Challenges and Opportunities of Learning Analytics Adoption in Higher Education Institutes: A European Perspective. In Companion Proceedings of The Fourteenth International Conference on Learning Analytics & Knowledge. Kyoto, Japan.

Viberg, O., & Grönlund, Å. (Eds.). (2023). Practicable Learning Analytics. Springer Nature.