Follow-up

In response to a recognised problem in higher education, being the limited use of scalable learning analytics (LA) (e.g. Hernandez-de-Menendez et al., 2022, Viberg et al., 2018 ), this study has designed and implemented a transition course for mathematics. This transition course aims to support first year university students in their transition from secondary to tertiary education with a focus on providing real time feedback to students and teachers through the use of LA (Kula et al., 2023). Feedback was provided with the data that was gathered throughout the course via students’ answers and attempts to questions and data from the interactive videos. Students were presented with an individual overview of their effort using a star based visualisation per sub-topic of the transition course indicating their level of understanding to be adequate for the university mathematics. Moreover, the teacher is presented with a whole class heatmap visualisation of the mathematical subjects which is based on the student attempts and their current result (adapted from Lee et al., 2016). The bridging course was implemented in the Civil Engineering bachelor program, and will be adapted to many bachelor programs at the University of Twente. We have received interest from various national and international universities, both in this course and its use of LA. This use of LA, to give students and teachers the feedback based on question attempts and results, is replicable to different courses. We would like to scale up the transition course as well as its use of LA both horizontally to a large audience of students and teachers national and international dimensions, and also vertically to various disciplines to adapt the use of LA. References: Hernández-de-Menéndez, M., Morales-Menendez, R., Escobar, C. A., & Ramírez Mendoza, R. A. (2022). Learning analytics: state of the art. International Journal on Interactive Design and Manufacturing (IJIDeM), 16, 1209–1230. Kula, F. , Horstman, E. M., Lanting, L. S., & Ten Klooster, L. R. (2023). Exploring Strategies To Promote Engagement And Active Learning Through Digital Course Design In Engineering Mathematics. Paper presented at 51st Annual Conference of the European Society for Engineering Education, SEFI 2023, Dublin, Ireland. Lee, J. E., Recker, M., Bowers, A. J., & Yuan, M. (2016, June). Hierarchical Cluster Analysis Heatmaps and Pattern Analysis: An Approach for Visualizing Learning Management System Interaction Data. In EDM (pp. 603-604). Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89(December 2018), 98-110. 

A data culture in Higher Education Institutions characterizes the tradition, people, narratives, and symbols around data and datafication (Raffaghelli, & Sangrà, 2023). The data culture of centralized Information Systems (IS) departments is commonly aimed at creating an all-inclusive data center that can integrate all sources of organizational data. This data culture cultivates data savvy users to act upon dashboards and use various information systems to promote various organizational goals. Assuming that Learning Analytics (LA) is just another information system, it should be owned by the IS department. Alternatively, the educational technology culture is seeking to seamlessly embed LA into current learning and teaching practices and to evaluate learning outcomes. This culture tightly links learning design as a form of documentation of pedagogical intent to making sense of diverse sets of analytic data (Lockyer et al., 2013). Therefore, ensuring the effective adoption and dissemination of LA can only be achieved if the ownership of LA is placed in the hands of the educational technology department. The leadership of the Open University of Israel debated who should own the LA initiative. The IS department suggested using the organizational Tableau platform while the center for educational technology suggested improving the open source LA plugin in MOODLE, its organizational Virtual Learning Environment (VLE). The short presentation will delineate the debate that led to choosing the embedded LA solution within the VLE, while creating a fine balance between the two cultures. We will show how the data culture considerations were a major factor of the successful adoption of LA practices into all the university's courses. Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439-1459. Raffaghelli, J. E., & Sangrà, A. (2023). Data Cultures in Higher Education: Acknowledging Complexity. In Data Cultures in Higher Education: Emergent Practices and the Challenge Ahead (pp. 1-39). Cham: Springer International Publishing.‏

The main challenge that we encountered at our Dutch university in the large-scale adoption of Learning Analytics (LA) is that the data-informed working environment was not mature enough to facilitate LA. For example, a policy describing the educational goals, legal requirements or ethical values for LA was missing (Gasevic et al., 2019). Further, the technical infrastructure was unsuitable for the more advanced analytics often used in LA, such as AI or real-time analytics (Webber & Zheng, 2020). We tackled these challenges by combining a top-down and bottom-up approach in a roadmap for LA (Perez-Sanagustin et al., 2022). From the top-down, a policy was created, based on a national LA and AI reference framework, literature (Tsai & Gasevic, 2017), and conversations with stakeholders. From the bottom-up, projects are initiated at the staff level within faculties, thereby ensuring stakeholder involvement. All LA initiatives need to follow the roadmap, which ensures that they adhere to the LA policy. Additionally, a new technical infrastructure, the LA data platform, was built to enable advanced analytics. The roadmap and the LA data platform are first steps towards the large-scale adoption of LA, but we are not there yet. We identified a larger overarching challenge: the value of data is not recognized on all levels in our university. This is reflected in, for example, the absence of a data governance program which is essential for data-informed working, such as LA. A mindset accommodating a data-informed working environment requires a cultural change, but this proves difficult to achieve (Dama International, 2017; Webber & Zheng, 2020). In this workshop, we will elaborate on the development of the roadmap and our experiences so far. Your suggestions on how to facilitate the cultural changes that are needed to make LA a fixture in our university would be highly appreciated. References. Dama International. (2017). DAMA-DMBOK Data management body of knowledge (2nd ed.). Technics Publications LLC. Gasevic, D., Tsai, Y.-S., Dawson, S., & Pardo, A. (2019). How do we start? An approach to learning analytics adoption in higher education. The International Journal of Information and Learning Technology, 36(4), 342–353. https://doi.org/10.1108/IJILT-02-2019-0024 Perez-Sanagustin, M., Hilliger, I., Maldonado-Mahauad, J., & Perez-Alvarez, R. (2022). Building Institutional Capacity for Learning Analytics: Top-Down & Bottom-Up Initiatives. IEEE Revista Iberoamericana de Tecnologias Del Aprendizaje, 17(3), 281–289. https://doi.org/10.1109/RITA.2022.3191413 Tsai, Y.-S., & Gasevic, D. (2017). Learning analytics in higher education --- challenges and policies: A review of eight learning analytics policies. Proceedings of the Seventh International Learning Analytics & Knowledge Conference, 233–242. https://doi.org/10.1145/3027385.3027400 Webber, K. L., & Zheng, H. (2020). Big data on campus. Data analytics and decision making in higher education. John Hopkins University Press.

Despite the growing demand across higher education institutions to use learning analytics within the practice of teaching, learning and assessing, learning analytics adoption literature highlights a significant lag in the adoption of learning analytics among higher education sector academics. This survey study aims to understand the barriers faced by educators, when using learning analytics in their teaching practice. The survey study targets higher education sector academics, who use technology-enabled learning instances in teaching. The survey questionnaire incorporates indicators from a conceptual model constructed using popular technology adoption models, to assess the factors influencing the learning analytics usage of educators in their teaching practice. The survey study has currently captured 120 responses representing an experienced sample with an average of over 11-20 years using educational technology (EdTech). Findings indicated the lack of available support, confidence and competency, in utilising and interpreting learning analytics to inform effective teaching decisions resulting improved learning outcomes. This study contributes by identifying barriers to learning analytics adoption in higher education, highlighting the importance of proper training and support to enable educators to effectively interpret and apply learning analytics in teaching. Addressing these challenges can promote integration of learning analytics in teaching practices to enhance overall learning experience. 

In response to the growing need for student support and retention, Thomas More university of applied sciences, home to over 20,000 students and at the time very new to data-driven approaches, has embarked on a new data journey: the adoption of a learning analytics (LA) model to predict potential drop-outs among first-year students. This model aims to assist study coaches finding ‘at-risk’ students to help them navigating their academic journey. A few essential steps needed to be undertaken before implementing this model for an optimal adoption of LA into the organization. 1. Policy Framework Design In collaboration with key stakeholders including the data protection officer, the director of student facilities, data analysts and students, we have meticulously designed a comprehensive policy framework. This framework outlines the ethical use of data and defines the purpose of LA for this and future projects. The collaboration between the different stakeholders was crucial in this step to have a multidisciplinary view and have many colleagues on board from the start. 2. Student Approval To ensure transparency and trust, students’ approval was sought through the student council. This process involved providing students with a detailed explanation of the model's purpose. It also proved crucial to emphasize what the model is not for (e.g. teachers do not get access to the data, the model predictions will never affect their grade) and that they have the option to opt-out of the tool. 3. Training for Study Coaches Recognizing that the study coaches lack experience in working with data, training sessions were organized, facilitated by two study coaches with a good knowledge about data and the LA model. The sessions focused on demystifying the model, explaining its functionalities, and providing practical insights into interpreting and utilizing the data effectively. 

The integration of Learning Analytics (LA) within higher education is pivotal for enhancing educational outcomes. A strategic approach is vital, including nurturing digital competencies among educators, especially in data literacy (Clark, Liu, & Isaías, 2020). The adoption of LA can be accelerated by advancing staff abilities and employing varied analytical strategies (Ifenthaler, 2020). Our study tackles the barriers to LA adoption through a user-friendly method that boosts educators’ data literacy, thereby facilitating the adoption of LA. We implemented a completion tracking system that monitors students’ academic progress both autonomously and via self-reporting. This system empowered educators to identify and assist students at risk of falling behind. The project involved close collaboration with faculty for curriculum development and comprehensive training on the tool's usage, emphasizing its simplicity and low maintenance. After three cycles of implementation, student evaluations expressed a strong preference for its adoption across all courses. This enthusiasm was largely attributed to educators’ effective use of the system to initiate timely interventions. The tool not only bolstered educators’ ability to utilize data for actionable insights but also provided students with a personal dashboard, thereby enhancing their engagement. The successful deployment of our system in a study program presents a scalable model for addressing the challenges of data collection and stakeholder engagement in LA adoption. It is consistent with research advocating for clear educational objectives, effective risk management, and thorough training (Nenadić, Krajnović, & Jašić, 2012). This approach not only promises to enhance student support and learning outcomes but also allows both educators and students to employ data for insights, thus resonating with the critical considerations of power dynamics, monitoring, and student identity in LA (Slade & Prinsloo, 2013).