Beyond Bloom’s 2 Sigma Problem: Data-Driven Approaches for Improving Student Success

Abstract

The teaching-learning approach proposed by Benjamin Bloom in his 2-sigma problem (Bloom, 1984), although testing exam performance, suggests a rigid, systemised education with rigorous testing that suites quantitative subjects in which achievement is predefined with a universally agreed fixed answer. In this essay I intend to discuss scenarios where this may not be the case through exploring alternative approaches involving data-driven platforms and the impact they may have on pedagogy; this includes gamification, constructivism, flipped-mastery and peer-peer learning. I critically evaluate Brown’s (2020) work on Seeing students at Scale and also offer a posthumanist perspective to Dijck’s et al (2018) work on the influence of tech companies in education.

Introduction

The industrial model of education has lead to standardised testing, age-based grouping of learners and mass instructions of all students at the same time (Robinson, 2016).

Although this may have evolved to more differentiated models of classroom instruction and activities, the issue of students accumulating gaps in their learning or yet to master the pre-requisites before approaching more difficult areas of the curriculum remains due to the linear progression of learners. Providing regular paper-based assessment for learning and further differentiated instruction and activities by a sole teacher in person for an average class size of 30 is impractical (Glymour, 2010). With the advent of the internet, MOOCS and data-driven platforms, assessment for learning and content ‘delivery’ has been digitised in both quantitative and non-quantitative subjects. This has revoked discussions around Bloom’s 2-sigma problem, who demonstrated students perform better by one standard deviation when a mastery-based learning approach and two standard deviations when mastery learning is combined in a one-one setting (Bloom, 1984) compared to conventional teaching of 1:30 student-teacher ratio. In conventional teaching, there is little scope for differentiated instruction and according to studies, teachers often cater for the top third of students (Bloom 1984). By mastery-learning Bloom (1984) suggests ‘formative tests are given for feedback followed by corrective procedures and parallel formative tests to determine the extent to which the students have mastered the subject matter’. One-one tutoring allows for further differentiated instruction based on the student’s needs and level.

Essa and Laster (2017) have argued that mastery-learning can help close the achievement gap and it is the only pedagogical alternative to the lecture-and-test paradigm dominating institutionalised education. An experiment in 1969 at Princton involving seven black students illustrates this point well. The students failed the the first semester but achieved grades A’s and B’s in the second semester after a master-based learning approach was adopted (Glymour, 2010). Lectures were replaced with Q&A sessions and ‘extemporaneous mini-lectures’ on difficult topics, with regular tests and re-tests at the students choosing. At the end of the term, Glymour (2010) concluded it was not practical to teach this way and was ‘ready for the hospital by the end of term’. He suggested, however, that the strategies used could be carried out by computers. Considered by some to be the holy grail of EdTech, the unresolved problem has been to see if teaching-learning conditions can be created to enable the majority of students under group-instruction to achieve the same level of achievement that can be achieved under mastery-based learning and good tutoring.

Gamification and Constructivism

Data-driven platforms not only provide a multitude of parallel tests or activities, but through adaptive learning the level of challenge can be modified for a particular task to the students’ level of ability. Learning analytics can be used to determine gaps, strengths and weaknesses. Gamification is increasingly being used by platforms such as LinkedIn, Khan Academy and Code Academy. In an attempt to solve Bloom’s 2-sigma problem, Betts (2013), created an online environment based on the principles of gamification to facilitate peer-peer learning without the need for a teacher to be present. Betts (2013), found the following:

1-Increased participation

2-Predicted grades based on experience-points

3-Highlight ‘at risk’ students who struggle in a game

The teacher would actively intervene with the ‘at risk students’ to work with them in other environments and use alternative approaches. After evaluating the level of participation using a cognitive presence model (Kanuka, Rourke and Laflamme, 2007), Betts found 78% of comments and contributions did not demonstrate constructive critical thinking. Changing the experience-points systems to reward competence and not completion, effort and quality of contribution, Betts (2013) was able to replicate the results of critical thinking levels when compared to in-class teaching (Kanuka, Rourke and Laflamme, 2007). An example of where a teacherless gamified environment worked successfully is 42¹ – a free coding university in Paris where some 3,000 students of age 18-30 with no prior experience of coding learn in an online learning environment through project-based peer learning and gathering experience points, which unlock levels as they progress through their learning map. So far 42 has sent each of its students to a top job in the tech sector (Beard, 2018).

Minecraft Digital Artefact:

Please see Prezi

Platformatisation and the Impact on Pedagogy

The Minecraft gaming environment can foster more ‘collaborative and participatory classroom experience’ and provide an environment to foster a ‘constructionist approach to learning’ in which players are making both within and without the game on platforms such as YouTube (Niemeyer and Gerber, 2015; Schifter and Cipollone, 2015). What made Minecraft so compelling was the fact that it did not come with instructions. Users had to self-learn or peer-learn through YouTube or synchronously in a Minecraft world. Although Microsoft had acquired the Minecraft and created the Education Edition with online teaching and student resources on Microsoft OneNote (Microsoft, 2020), it still relied on a teacher-lead approach and is very much lacking in offering a self-paced gamified environment in where students can go on to learn Python on their own. Prodigy Learning (2021) however, recently released comprehensive Coding in Minecraft programme, which not only provided self-contained Minecraft worlds that can be explored individually or in groups, but also allowed for digital certification, leading to its industry-standard Microsoft Technology Associate MTA Python exam. The level of support available to the student at various stages of coding, although time-saving, may, however, reduce some of the challenge inherent in the original design of the game. The information of user uptake amongst classes is useful from a managerial perspective but could potentially be a concern as it could be used for surveillance or progress checks of teachers by senior management.

With the recent Ransomware attack on education establishments, there is no doubt educational data is amongst the most valuable currencies in the ecosystem; the most recent one being Harris Federation, revealing their top managers are paid £123k more than their counterparts in other trusts (Harris Federation, 2021; TES, 2021). It can be argued that tech companies such as Microsoft, who purchased Minecraft for $2.5bn (BBC, 2014), are shaping the learning experience of millions of learners, which ‘uproots or bypasses the values that are fundamental to publicly funded education: Building, a knowledge-based curriculum, autonomy for teachers, collective affordability, and education as a vehicle for socioeconomic equality (Van Dijck, Poell and de Waal, 2018). In addition to data-driven platform-based tools developed by the Big Tech companies which offer ‘content production and distribution, student performance tracking, class communication and administrative organisation’ (Van Dijck, Poell and de Waal, 2018) there is also digital certification, which Prodigy Learning (2021) for example has provided as sole Microsoft partners for the UK. The Microsoft Office Specialist exams have been used as a means of gamification through national and international competitions (Prodigy Learning, 2019).

On the one hand, it can be argued digital certification comes with a ‘collective affordability’ limitation and is shaping the learning experience and curriculum of a certain sector of schools (van Dijck, Poell and de Waal, 2018). This may not have to be completely at odds with the ‘values of education’, as argued by Van Djick et al (2018). When considering the question around educational theories supported by outcomes-based data-driven approaches of mastery learning being reasserted in authoritative ways, it should be kept in mind that the current model of education is shaped heavily by the industrial age and following the COVID 19 pandemic, has lead many educators to suggest this may be ample time to re-think the whole structure of institutionalised education and adopt an educational paradigm suited for the fourth revolution (van Dijck, et al 2018). Also, from a posthumanism perspective, Djick et al (2018) has not addressed the question of the ‘digital tools’ shaping the ‘subjects’, as social media platforms such as Twitter have done; what may appear as an inherent limitation of characters on a Tweet, has lead to creative uses within education where students can be challenged to write essays using Twitter (Stommel, 2015). In the case of Minecraft, one of the unexpected outcomes has been how it helps children with autism who would find traditional face-face teaching challenging (Rutkin, 2016; Wolf et al., 2016). While the Big Five Big² tech companies are without a doubt shaping the educational experience of millions through their investment in the education sector, and Silicon Valley may develop schooling programmes such as AltSchool, would they have the necessary buy-in when compared to educational programmes such as Pearson’s Online Academy?

Flipped Mastery and Seeing Students at Scale

Another variation of the standard group-instruction model is that of flipped-mastery. Similar to Glymour’s (2010) experiment at Princeton in 1969, but now using technology, (Sparks and Bergmann, 2019) suggest how self-paced video content followed by practice & learn tasks (any activities that helps the students better understand: labs, activities, games etc) are used before a mastery-check and the next topic is learned. Failing a mastery check would involve new practice & learn tasks. Furthermore, the lecture time is now freed up for more quality one-one intervention based teaching and feedback.

The teacher is continually assessing mastery with every interaction with the student. Not only does the teacher assess mastery of the content and skills but also the student’s mindset, their grit and their agency. The teacher can then tailor their intervention specifically to each student. This includes enrichment and extension activities for the students who master the content quickly. (Griffiths, 2019)

The mastery check need not be tied to right-wrong quantitative answers through a digital means, but the data can also be created by the teacher based on written, verbal and other means and take into consideration qualitative measures. Where data is delivered algorithmically to an instructor about students in a flipped lecture scenario, to create a Learning Analytics Dashboard, Brown (2020) has explored case studies to show how this has lead to ‘frustration extended from the lack of clarity about how the algorithm worked, what information was being foregrounded, and what actionable insights could be drawn from the data.’ The case studies Brown (2020) presents are that of an online system imposed on instructors rather than a set of digital tools they have chosen, which only accepts multiple-choice questions with little scope for alternative forms of input e.g. diagrams. With the use of clickers for the case study, the technology seems outdated when compared to applications such as Wooclap or Socrative that require students to engage via apps on their smartphone phone or tablet. Where the use of digital tools is pedagogy based, a case study of flipped lecture approach has found 47/56 students preferring the method to conventional lectures and also scoring 10 percent higher than average than the other questions (UCL, 2014). Although Brown (2020) mentions active learning strategies such as Peer Instruction activity, there is little discussion of how the polling tool could be used to measure the level of critical thinking in discussion between students during such an activity. Finland’s most popular teacher, Pekka Peura for example, would pose a question to identify misconceptions; once the students have responded to the online poling question using their smartphones, without revealing the answer he would display their responses on a bar chart, before asking them to discuss the responses in pairs before they would beam in their answers for a second time. It was found that the number of correct answers had significantly increased, thereby demonstrating the impact of peer-peer learning (Beard, 2018). Incidentally, Peura developed some of his unconventional teaching approaches by studying how Google creates successful teams. He saw his role more as a skill and attitude builder than a content deliverer; students assign themselves their own grades and had access to all the curriculum resources including tests and solutions at the beginning of the course.

The datafication of student engagement discussed by Brown (2020), as he suggests, seems to be a better measure of engagement with the learning management system, rather than meaningful engagement with the course material and activities. Such data could however be used for surveillance, behaviour management or monitoring of student participation (Brown, 2020). There is also surprisingly few data collection misconceptions or comprehension of activities outside class time, given all the lectures in Brown’s (2020) case study were flipped – how do the synchronous and asynchronous leverage and inform each other? The issue instructors raised on more preparation time needed for their lectures when planning to use the clickers may partly be down to the digital platform or the fact that at higher education instructors design their syllabus and may therefore have less scope for pre-made content. Learning by Questions, for example, is a platform designed for synchronous data-based intervention with pre-made questions for primary and secondary levels, allowing teachers to make intervention-based teaching approaches. The levels of questions progress from understanding, fluency and reasoning to problem-solving. Question 13 in Fig. 3 is red for the number of incorrect attempts made on average; the yellow boxes show questions answered correctly after having attempted it multiple times, indicated by the number in the box (LendED, 2021). The instructor can easily click on a question and start teaching from it.

Figure 4 – Data dashboard from LBQ LendED (2021)

Summary

The use of digital tools, data-driven platforms, and video content creation has lead to the evolution of new pedagogies which leverage synchronous and asynchronous teaching and learning. Constructivism, peer-peer learning, gamification, differentiated instruction and adaptive learning are some examples of how technology can be leveraged by a single teacher of a large class size and hence bring us closer to solving Bloom’s 2-sigma problem. This has allowed for new forms of ‘corrective measures’ and ‘testing’ than originally used by Bloom (1984), allowing more scope for dialogic feedback and qualitative measures. A pedagogy informed approach is vital to the use of digital tools if staff frustration is to be avoided. While profit-based big tech organisations are informing and shaping the teaching and learning culture for millions of learners through their platforms, the situation may not be a binary one; several innovative uses e.g. Minecraft Education Edition or Peura using Google’s principles of successful teamwork to influence group dynamics in his classroom can also be seen from a posthumanist perspective of the digital re-shaping an educational culture which has its roots in the industrial age.

¹– https://www.42.us.org/

²– Facebook, Alphabet-Google, Apple, Amazon and Microsoft

References

BBC (2014) ‘Microsoft pays $2.5bn for Minecraft maker Mojang’, BBC News, 15 September. Available at: https://www.bbc.com/news/technology-29204518 (Accessed: 24 April 2021).

Beard, A. (2018) Natural Born Learners: Our Incredible Capacity to Learn and How We Can Harness It. London: Weidenfeld & Nicholson.

Bloom, B. S. (1984) ‘The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring’, Educational Researcher, 13(6), pp. 4–16. doi: 10.2307/1175554.

Brown, M. (2020) ‘Seeing students at scale: how faculty in large lecture courses act upon learning analytics dashboard data’, Teaching in Higher Education, 25(4), pp. 384–400. doi: 10.1080/13562517.2019.1698540.

van Dijck, J., Poell, T. and de Waal, M. (2018) The Platform Society. Oxford University Press. doi: 10.1093/oso/9780190889760.001.0001.

Freire, P. (2017) Pedagogy of the Oppressed. Erscheinungsort nicht ermittelbar: Penguin Classics.

Fullan, M. and Quinn, J. (2020) Reimagining education: From remote to hybrid learning | Microsoft EDU. Available at: https://educationblog.microsoft.com/en-us/2020/06/reimagining-education-from-remote-to-hybrid-learning/ (Accessed: 23 April 2021).

Glymour, C. (2010) Galileo in Pittsburgh. Cambridge, MA: Harvard University Press.

Gokalp, M. (2017) ‘COMPARE THE EFFECT OF MASTERY LEARNING AND CONSTRUCTIVIST APPROACHES TO THE ACADEMIC ACHIEVEMENTS OF TEACHERS’, 3(1), p. 6.

Griffiths, S. (2019) ‘How Flipped Learning Makes Mastery Learning Possible’. Available at: https://flr.flglobal.org/how-flipped-learning-makes-mastery-learning-possible/ (Accessed: 23 April 2021).

Guskey, T. R. and Gates, S. L. (1986) ‘Synthesis of Research on the Effects of Mastery Learning in Elementary and Secondary Classrooms’, p. 10.

Harris Federation (2021) Ransomware Attack. Available at: https://www.harrisfederation.org.uk/50/harris-highlights/post/92/ransomware-attack (Accessed: 23 April 2021).

Kanuka, H., Rourke, L. and Laflamme, E. (2007) ‘The influence of instructional methods on the quality of online discussion’, British journal of educational technology, 38(2), pp. 260–271. doi: 10.1111/j.1467-8535.2006.00620.x.

LendED (2021) Learning by Questions, BESA LendED. Available at: https://www.lended.org.uk/product/learning-by-questions/ (Accessed: 24 April 2021).

Microsoft (2020) ‘Basic Moves & the Agent’, Minecraft: Education Edition. Available at: https://minecraftedu.westus.cloudapp.azure.com/lessons/basic-moves-the-agent (Accessed: 23 April 2021).

Niemeyer, D. J. and Gerber, H. R. (2015) ‘Maker culture and Minecraft: implications for the future of learning’, Educational Media International, 52(3), pp. 216–226. doi: 10.1080/09523987.2015.1075103.

Prodigy Learning (2019) ‘Prodigy Learning – UK Champions in World Finals’, Prodigy Learning, 1 August. Available at: https://prodigylearning.com/uk-champions-in-world-finals/ (Accessed: 24 April 2021).

Robinson, K. (2016) Creative schools. UK: Penguin Books.

Rutkin, A. (2016) How <em>Minecraft</em> is helping children with autism make new friends, New Scientist. Available at: https://www.newscientist.com/article/mg23030713-100-how-is-helping-children-with-autism-make-new-friends/ (Accessed: 5 December 2020).

Schifter, C. C. and Cipollone, M. (2015) ‘Constructivism vs Constructionism: Implications for Minecraft and Classroom Implementation’, in Isaías, P. et al. (eds) E-Learning Systems, Environments and Approaches: Theory and Implementation. Cham: Springer International Publishing, pp. 213–227. doi: 10.1007/978-3-319-05825-2_15.

Sparks, G. and Bergmann, J. (2019) ‘Flipped Mastery: Bloom’s Dream Comes True’. Available at: https://flr.flglobal.org/bloomsmastery/ (Accessed: 23 April 2021).

Stommel, J. (2015) Teaching with Twitter, Teaching in Higher Ed. Available at: https://teachinginhighered.com/podcast/teaching-with-twitter/ (Accessed: 6 December 2020).

TES (2021) Harris spends £750K on the wages of its top two earners, Tes. Available at: https://www.tes.com/news/harris-spends-ps750k-wages-its-top-two-earners (Accessed: 23 April 2021).

UCL (2014) What I wish I knew before I flipped my lecture, Teaching & Learning. Available at: https://www.ucl.ac.uk/teaching-learning/case-studies/2014/aug/what-i-wish-i-knew-i-flipped-my-lecture (Accessed: 5 December 2020).

Wolf, C. et al. (2016) ‘“Will I always be not social?”: Re-Conceptualizing Sociality in the Context of a Minecraft Community for Autism’, in. doi: 10.1145/2858036.2858038.