Module P44410 Creating Frameworks for Learning

A critical analysis of on-line formative assessment activities utilised in higher/ professional education.

This essay will present a critical analysis of online formative assessment activities utilised in higher/ professional education. It commences with a brief account of the background of information and communication technologies within higher education and e-learning; providing a rationale for the chosen area of focus.

Following this the benefits and limitations of the approaches in online formative assessments are presented and critically analysed. Themes explored include…. purpose of assessment; why we might want to introduce ICT-based assessment; flexible and personalised learning; challenges in developing e-assessments; efficiency gains; e-assessment stage; formative e-assessment; supporting learners and mastering ICT.

Definitions adopted for the terms applied in this paper (CAA, CBA, e-assessment, e-Learning and ICT) are presented in appendix 1 .

Main Body

According to the Joint Information Systems Committee (JISC) (2007) Higher Education Institutions (HEI’s) are faced with increased student numbers and reduced student-to-staff ratios the combination of which have focused the attentions and efforts within HEI’s on ways in which e-assessment can be developed to support student learning. Gipps (2005) reports that the role of information and communication technologies (ICT) in higher education is well established, she identifies that universities were the first to develop networked computer systems; and that e-mail has become the main mode of communication across campuses and has extended into the wider community.

Gipps (2005) goes on to report that the use of web-based resources for teaching is now common-place and, increasingly teaching on campus is supported by learning management systems such as Blackboard™, or via websites (Internet and Intranet). Where universities use a learning management system (LMS) or virtual learning environment (VLE), they often utilise these environments to deliver online assessments (Gipps, 2005). This use of ICT was also observed by Sharpe et al, (2006) who stated that computer-aided assessment is a dominant feature in the e-learning & blended learning agenda for some HEIs.

Within my own university student learning is supported by the Blackboard™ VLE, the implementation and utilisation of this in teaching, learning and assessment activities is driven by the university e-learning strategy (Oxford Brookes University (OBU 2002; OBU 2005) and the school e-learning strategy (OBU, 2004) together these shape both our current and future developments in e-learning.

The strategic plan for the university indicates that the university will… “develop innovative approaches to learning, teaching and assessment particularly with regard to e-learning and more flexible and convenient opportunities for learning on-campus, at home and in the workplace” (Lidgey 2002: 1.2).

With regard to learning, teaching and assessment; the strategic plan for the university indicates that the university will… “develop innovative approaches to learning, teaching and assessment particularly with regard to e-learning and more flexible and convenient opportunities for learning on-campus, at home and in the workplace” (Lidgey 2002: 1.2). The e-learning strategy (OBU, 2002) recognised the potential advantages of computer aided assessment (CAA); both formative and summative assessment, indicating that this will facilitate faster performance feedback of information to the student. The strategy also identifies that the introduction of CAA provides the opportunity to “broaden and refine assessment methods and strategies” (OBU 2002: 11). In the second stage of the implementation specific outcomes were set for developments in e-learning these include further opportunities for students to engage with their studies through blended and flexible distributed learning (FDL) and where e-learning is used for communication, assessment, collaboration and/or content delivery (OBU, 2005).

The emphasis on increasing the development of e-learning and the implementation of online assessment within the organisation, within which I work, has influenced my decision to explore online formative assessment activities. This paper will begin to identify the advantages and disadvantages of using online formative assessments to develop student knowledge and skills in professional education.

Initially it is important to consider the purpose of assessment and then to decide whether e-assessment is appropriate. In setting an assessment it is necessary to consider whether the assessment measures what the student has learned and also supports and encourages the student to develop a deeper understanding of the subject area. Prosser & Trigwell (1999) identify that most academics have anxieties about testing student learning and whether the assessments they set encourage or support deep or surface learning. Ramsden indicates that surface approaches to learning are“uniformly disastrous” (1992:45); he cautions that a student may ‘imitate’ learning and may not recall what they have learned after the point of assessment. Dismissing these concerns, JISC (2007) emphasise that there are opportunities to embrace new pedagogic approaches which challenge the perceptions that e-assessment is only suited to testing surface knowledge. It is important to ensure that the e-assessment meets the purpose of the learning activity.

Innovative examples of e-assessment for learning identified by JISC include; “virtual world scenarios for professional training, and web-based tools, such as e-portfolios, blogs and wikis, to develop skills in reflection and self-assessment. Online tasks involving peer- and self-assessment, which can capture evidence of achievement in collaborative activities, and increasing use of the assessment tools within VLEs, indicate the diversity of approaches that has been established in this sector” (JISC 2007:11).

It is also important that we consider why we might want to introduce ICT-based assessment in place of a more traditionally delivered assessment. Historically marking student work has placed a significant demand on the tutors’ time, another factor in considering e-assessment is the speed and reliability of marking student work. Pollock et al, (2000) point out that this provides and another reason why e-assessment has grown in popularity is for its apparent efficiency in an increasingly demanding higher education environment.

This is endorsed by Gipps indicates that there are at least three significant reasons to implement this, namely “to avoid disjunction between teaching and assessment modes with e-based learning; to save staff time in marking, and to enable formative feedback to students” (2005: 173-174); further advantages in utilising CAA, reported by JISC include: “Online formative assessment activities, for example, multiple-choice questions and quizzes are available to learners any time, anywhere; Creation of online multiple-choice quizzes by learners for other learners motivates and engages learners; Online formative assessment reduces practitioners’ workload and increases learners’ responsibility for their own learning; Online assessment can offer instant feedback and opportunities to reinforce learning; Online quizzes testing conceptual understanding increase learners’ confidence and therefore participation in face-to-face sessions” (2004:36). In a follow up report JISC emphasise that “e-Assessment can play a significant part in a more flexible and personalised environment for learning” (2007:10). Further advantages to utilising online assessment are presented in appendix 2 .

There are however several reported disadvantages to online assessment (Gladwin 2005; SWAP no date), namely the construction of good objective tests requires skill and practice and so is initially time-consuming and potentially very costly; large bank of questions required preventing banding and hence loss of discrimination; students and academic staff require adequate ICT skills. There is the potential for hardware and software failure at key times, the HEI needs to ensure there is adequate provision of system maintenance to maintain the integrity of the system, some institutions invest in separate servers for formative and summative assessments which bring further costs. There are limitations on the knowledge and skills that can be assessed, text based assessments are mainly limited to MCQs or short answers. Further limitations to utilising online assessment are presented inappendix 3.

The emphasis on flexible and personalised learning has lead to the need for more teaching and learning materials to be available to the students when they want to engage with the learning (OBU, 2005), Paciello (2001) cited in (Seale 2006: 89) describes a move towards true personalization where web design ensures accessibility for every user by adapting to the users preference. Loewenberger & Bull (2003) report that the flexibility of online formative assessments means that students have the opportunity to study at their own pace; whilst Sharpe et al (2006: 14) identifies ‘flexibility of provision’ as being one of the five attributes that are part of the blended learning agenda for some HEIs. Personalisation also brings is own challenges, students with Special Educational Needs often require the environment to be personalised to meet their individual needs e.g. colour of screen and text for a sudden with dyslexia or a screen reader to read the text for a sudden with a visual impairment. HEIs must comply with the Special Educational Needs and Disability Act (SENDA, 2001) in the preparation of teaching and assessment materials. There is guidance available to assist the tutor developing general teaching materials however few guidelines exist to assist the development of online teaching and assessment materials (Wiles, 2002).

International standards are being set through the Web Accessibility Initiative (WAI) have produced guidance on web accessibility however these guidelines focus generally on web accessibility and do not specifically focus on accessibility of e-assessment. TechDis, an advisor service funded by JISC, have developed a substantial number of documents which are informative and are an asset when developing e-learning and e-assessment materials.

The challenges of implementing robust systems of CAA or computer-based assessment (CBA) are many, and these mainly revolve around allocation of resources and adequate investment. JISC (2007) reported that to a large extent innovative work in CAA & CBA in HEIs has occurred where systems and tools development has been supported through project funding. They report that this is often limited to a few departments, and is seldom a university wide initiative. This is rather limiting as projects are usually time limited with defined funding, which when the project ends may result in the survival of the initiative reliant upon an ‘enthusiastic few’. Conole & Warburton cite “Lack of resources, individual inertia and risk propensity are the key barriers for individual academics, while proper resourcing and cultural factors outweigh technical barriers at institutional level” (2005: 17). Networks of national projects help to overcome this potential isolation by working together and developing shared expertise which can be disseminated and implemented across a wider area (JISC, no date).

As previously indicated; “effective assessment needs to reflect the content of the programme and be fair, valid and reliable” (Wakeford 1999: 59). Effective e-assessment for formative purposes can be set up relatively simply by using the assessment tool within the institution’s VLE (JISC, 2007). Several authors predicted an increase in use of ‘simple’ computer-assisted formative assessment and that their use will continue to develop, as will the use of multi-media materials for assessment tasks (Zakrzewski & Bull 1998; Lee & Weerakoon 2001; McKenna 2001; Gipps 2005; OBU 2005).

With the growing development in e-assessment McKenna (2001) identifies that there is also an increase in attention being paid to staff development in writing questions and tests. Paxton (2000) identifies that we may set out to develop assessments which test higher-order skills and knowledge, unfortunately in practice e-assessments tend to focus on basic facts.

Bull & McKenna (2003) challenge us to consider whether we want to encourage every course tutor/lecturer to become an expert in question design, and whether we can afford it, given that many modules have small numbers of students. They also remind us that designing high-quality multiple-choice and other ‘objective’ questions is a skill that requires investment in staff training and need to have its quality assured.

Both Paxton (2000) and Gipps (2005) recognise the merits of the online ‘objective’ test they caution that have a limited match to ‘real’ tasks. Gipps (1994) cited in Gipps (2005: 174), stated that “validity is an issue because of disjunction between mode of teaching and mode of assessment”, this is endorsed by Salmon (2002) and Sharpe et al, (2006) who state that it is important to ensure that e-assessment strategies are constructively aligned with the learning objectives of the online learning activities.

Zakrzewski & Bull (1998) remind us that writing and production of objective items, with the need for trialling and statistical analysis to produce item data means that training, time and support are needed for staff to take this on; they report that this is a requirement that is often underestimated and subsequently reduces efficiency claims. Some six years later, Sim et al, (2004) continued to emphasise the need for focussed staff development suggesting that this may alleviate the aforementioned issues. Gipps identifies that the “true costs involved mean that this approach is really only feasible for large cohorts and/or items that can be reused; and this raises technical issues around item banking and estimates of item difficulty” (2005: 174).

Where the resources are unavailable locally to develop ‘robust’ and ‘good quality’ e-assessments, it is possible to implement online formative assessments through ready to use packages, e.g. SCORM (Sharable Content Object Reference Model) packages these are available to download and use within a VLE; and commercial packages. The commercial market is developing rapidly to meet this demand with an increasing number of educational packages’ including CD-ROM materials which include formative assessments to test student learning. However it is important to consider the country of origin as the assessments may be socially and culturally bound, and spelling and terminology may differ between countries e.g. UK & US. It is also necessary to ensure you have appropriate copyright and Intellectual Property Rights (IPR) permission for use before creating e-learning and e-assessment materials (HEFCE, 2006).

JISC (2007) emphasise that scaling up the use of e-assessment requires financial commitment and senior management support and a sustainable programme of technical and pedagogic support for academic staff is still needed. As in their earlier report, JISC (2007), reassert that well designed computer based formative assessment reduces practitioners’ workload and increases learners’ responsibility for their own learning.

With regards to the perceived efficiency gains in implementing CAA, e.g. reducing practitioners’ workloads and potential cost savings, Bull (1999) found that efficiency gains in terms of time saving had been elusive. In a follow up study Loewenberger & Bull (2003) identified that although CAA and CBA may initially appear to be resource heavy in terms of time and cost, where e-assessment resources are developed and reused or shared, either school or institution wide or between institutions, the resource impact reduces. Although they conceded that it was difficult to accurately cost this at modular level. More positively subsequent reports identify that there is a growing confidence in the “capability of e-assessments to deliver a return on investment, and a wider acceptance of the benefits of change amongst staff” (JISC 2007:14).

Having considered the value of e-assessment in its many guises, it is necessary now to consider the e-assessment stage; an exploration of the benefit of formative assessment and e-assessment will now follow.

Firstly it is necessary to consider what we are trying to achieve through setting an assessment and what the benefits are for the students and ourselves as tutors. Brown identifies three purposes for assessment; "to give a licence to proceed to the next stage or to graduation; to classify the performance of students in rank order; to improve their learning" (2001:6).

Gladwin (2005) and JISC (2007) identify four stages of assessment, indicating that the formative assessment approach lends itself well to developing student learning. This is endorsed by Biggs (1999) who states that results of formative assessments are used to give feedback to students, which may act to both improve student learning and improve teaching. Similarly, East states that the “essence of formative assessment is that undertaking the assessment constitutes a learning experience in its own right” (2005: online), feedback presented to the student following formative assessment should help them to develop in their learning; ergo the aim of formative assessment is to improve learning. Biggs (1999) presents formative assessment as being inseparable from teaching; therefore if e-assessment is to be adopted then it is surely best placed within a well designed e-learning programme. According to East (2005) it is acceptable for the results of formative assessments to be part of the final grade. However with regard to summative assessment, both Biggs (1999) and East (2005) infer that it usually occurs at the end of a period of learning in order to generate a grade that reflects the student's performance and has little value on enhancing the students learning experience. Biggs (1999) goes on to state that summative assessment is a negative activity, which causes high anxiety amongst students and where student error is no longer instructive. Conversely East (2005) argues that all forms of assessment have some formative element, and identifies that the modular degree course is in essence a formative learning experience culminating in the attainment of the final award.

With regard to automated formative feedback, Gipps (2005) questions its value; she argues that the provision of feedback of scores and marks alone is not truly formative, she indicates that for a student to enhance their learning they require feedback which includes comments about the good features of the work/performance, and what needs to be done to improve; beliefs which echo those of Biggs (1999) and East (2005). Holmes & Gardner are equally sceptical about the value of online formative assessment stating “formative assessment… is a form of assessment in which ‘old’ learning and teaching contexts still win hands down” (2006: 155). Conversely Morris (2007) reports that automated formative assessment, utilising MCQ & EMQ, are useful for the following reasons. They provide a rapid, reliable assessment of ‘core’ factual knowledge; allow a high degree of flexibility in terms of different question types; questions can be designed which require calculation; depending on the software package utilized assessments can be very flexible in terms of marking protocols; positive or negative marking for difficult parts, bad mistakes etc and formative questions allow student self-testing.

“When interactive online tests are available for formative assessment, there is evidence that learners retake tests voluntarily and that availability of these any time, anywhere, can help to establish more regular patterns of study” (JISC 2007:16)

“When interactive online tests are available for formative assessment, there is evidence that learners retake tests voluntarily and that availability of these any time, anywhere, can help to establish more regular patterns of study” (JISC 2007:16).

However Morris (2007) emphasises the need for caution in designing e-assessments and emphasises the need for ‘good quality assessment design’, he reminds us that utilising automated formative assessments makes it imperative to provide students with immediate feedback of the correct answers, he indicates that this prevents students from remembering and learning incorrect answers.

JISC (2007) indicated that there is a growing body of evidence which indicates that well-designed and well-deployed formative assessments can cultivate more effective learning for a wider diversity of learners. Several studies identified the greater the number of occasion’s students engaged with formative assessment the greater the apparent increase in average marks (Catley 2004; Morris & Walker 2006; Nichol & Macfarlane-Dick 2006). This is obviously of positive note for HEI’s and the current climate of performance indicators and league tables.

In respect to students self testing and self assessment, Holmes & Gardner (2006) develop into advocates of formative e-assessment when self and peer assessment is enabled, they see the e-learning environment as ideally suited to peer-to-peer discussion where comment and feedback can be made by other students within a supportive learning environment. Simpson (2002) identifies that self-assessment questions and answers (SAQs) are a valuable way of engaging students in self assessment to provide some sense of progress on their course, he emphasises this of particular value to students on distance education packages, arguably these are equally valuable to students on blended learning courses. However he identifies that ‘passive feedback’ may also be harmful, identifying those students who have not completed the SAQs may feel guilty and as a result they may feel less competent. Secondary feedback from the tutor is valuable in providing support the student to work through these feelings and explore the possible barriers to engagement in the e-learning/ e-assessment. Supporting the student in the e-learning/ e-assessment activity is likely to increase access and motivation and for the student who has opted out of the learning activity it has the potential beneficial of motivating the student to reengage in the learning activity (Salmon, 2002).

Bull (1999) identified that computer-assisted assessment (CAA) was mostly used for (large) classes where the size of the cohort lent it self well to this form of assessment, it is therefore important that students are prepared for their e-learning experience in good time. This is endorsed by Simpson (2002) and Salmon (2003) who emphasise that supporting students is a fundamental part of the role of the tutor in e-learning. Preparation of the student for online formative assessment are important aspects of support and is often crucial to a successful assessment outcome; OBU advocate the need for students to be "provided with appropriate support and training for CAA so that their performance during an examination is not diminished by fear of, or inexperience with, technology" (OBU 2002:11). This is endorsed by Davies et al., who identified that students developed skills in using ICT at different rates and to different levels and even with an improvement in ICT skills during their modules; they observed that,“using technology for assessment was not as intuitive as using pen and paper” (2005: 847); from their observations it is possible to surmise that ICT can be used effectively to enhance the learning experience, and through increased exposure in use the students become more experienced and confident in its use. Learning outcomes specifically directed towards developing transferrable skills in ICT are invaluable in today’s curriculum, helping students to develop essential transferrable skills (OBU, 2005).

Madden et al, emphasise, when exploring the apparent ‘new generation gap’ that “Reading and writing … permeated society over hundreds of years, so systems could adapt gradually. ICT has had a much more sudden impact” (2005:3). Increasingly within higher education programmes, adult learners must master ICT if they are successfully to complete any course for which online learning forms a significant component (Hara & Kling, 1999). This continues to be true for adult learners almost ten years on.

Conclusion

Exploration of the benefits and limitations to implementing e-assessment has raised more challenges than it has answered. It is evident that successful implementation of online formative assessment is reliant on a complex ‘joined up’ approach, between the Higher Education Institution and it’s students’ and workforce. At the HEI level it is important that the strategies support and encourage the development and implementation of e-learning and e-assessment at all levels.

At an individual level the paper has shown that preparation is vital, that staff developing and moderating e-assessments must be adequately prepared for this role, they need to develop the necessary skills to ‘master’ this role, it is essential that they are supported to achieve this at all levels. For the student, it is similarly important to prepare them for e-assessment, so they are familiar with the assessment procedure and not disadvantaged by lack of skill in the assessment environment. Regarding the implementation of online formative assessment, it is clearly apparent that there are significant benefits to all parties in implementing formative e-assessments. Evidence presented has shown how through repeated experience in completing formative assessments students have developed deeper knowledge in the subject areas and increased their overall grades and greater skill in the use of ICT. Benefits for the tutor may appear limited at first, with substantial initial investment of resources, especially individual time and effort, however following the development and implementation of the e-assessment the development time reduced and potential time savings ensued, which may lead to significant resource savings for the HEI.

The use of automated marking and automated feedback proved beneficial for both students and tutors alike, for students being given feedback for answers supported their learning and reduced the likelihood of their remembering incorrect answers. For tutors the benefits became apparent when marking time reduced, usually following the initial development period and resources could be reused.

On a final note it is important to approach the development of e-assessment with caution; it is crucial that the assessment is fit for purpose and assesses the outcomes of the learning activity; it is essential that sufficient resources are available to develop, test and deliver a ‘robust’ e-assessment. It is also important to remember that tutors do not need to become ‘experts’ in developing e-assessments, there are established assessment packages available and frequently there are pockets of expertise which can be drawn upon within the HEI’s and the wider national and international ICT community.

References

Biggs J (1999) Teaching for quality learning at university. Buckingham: The Society for Research into Higher Education & Open University Press. Brown G (2001) Assessment: A Guide for Lecturers. Available at: http://www.swap.ac.uk/docs/ltsn/assess/03Lecturers.pdf(accessed 04/01/08)

Bull J (1999) Computer-Assisted Assessment: Impact on Higher Education Institutions. Available at:http://www.ifets.info/journals/2_3/joanna_bull.pdf(accessed 07/01/08)

Bull J & McKenna C (2003) Blue Print for Computer-assisted Assessment (2^ndedition). Abingdon: Routledge.

Catley P (2004) One lecturer's experience of blending e-learning with traditional teaching or how to improve retention and progression by engaging students [online]. Brookes eJournal of Learning and Teaching. 1(2), Available at:http://www.brookes.ac.uk/publications/bejlt/volume1issue2/academic/catley05_1.html!(accessed 03.01.08)

Conole G & Warburton B (2005) A review of computer-assisted assessment. ALT-J Research in Learning Technology. 13(1), 17-32.

Davies A, Ramsay J, Lindfield H & Couperthwaite J (2005) A blended approach to learning: added value and lesson learnt from students' use of computer-based materials for neurological analysis. British Journal of Educational Technology. 36 (5), 839-849.

East R (2005) Formative vs. summative assessment. Available at: http://www.ukcle.ac.uk/directions/resources/assessment/formative.html(accessed 01/01/08) Gipps CV (2005) What is the role for ICT-based assessment in universities? Studies in Higher Education, 30(2), 171 – 180

Gladwin R (2005) Getting started with Computer Assisted Assessment. Available at:https://www.heacademy.ac.uk/assets/ps/documents/primers/primers/ps0084_getting_atarted_with_computer_assissted_assessment_mar_2005.pdf(accessed 01/01/08)

Hara N & Kling R (1999). Students' Frustrations with a Web-Based Distance Education Course. Available at:http://www.firstmonday.org/issues/issue4_12/hara/index.html(accessed 04/01/08)

HEFCE (Higher Education Funding Council for England) (2006) Intellectual property rights in e-learning programmes. Bristol: HEFCE.

Holmes B & Gardner J (2006) E-learning: Concepts and practice. London: Sage Publications.

JISC (Joint Information Systems Committee) (no date) What we do. Available at: http://www.jisc.ac.uk/whatwedo.aspx(accessed 04/01/08)

JISC (2004) Effective Practice with e-Learning: A good practice guide in designing for learning.http://www.jisc.ac.uk/media/documents/publications/effectivepracticeelearning.pdf (accessed 04/01/08)

JISC (2007) Effective Practice with e-Assessment: An overview of technologies, policies and practice in further and higher education. Available at: http://www.jisc.ac.uk/media/documents/themes/elearning/effpraceassess.pdf (accessed 01/01/08)

Lee G & Weerakoon P (2001) The role of computer-aided assessment in health professional education: a comparison of student performance in computer-based and paper-and-pen multiple-choice tests. Medical Teacher, 23(2), 152-157.

Lidgey J (2002) Learning, Teaching and Assessment Strategy 2002/2005. Available at:http://www.brookes.ac.uk/brookes/LTAStrategy2002.doc(accessed 04/01/08)

Loewenberger P & Bull J (2003) Cost-effectiveness analysis of computer-based assessment. ALT-J Research in Learning Technology. 11(2), 23-45

Madden AD, Nunes JMB, McPherson M, Ford N, Miller D & Rico M (2005) A New Generation Gap? Some thoughts on the consequences of increasingly early ICT first contact. Available at: http://eprints.whiterose.ac.uk/1114/1/consequences_of_increasingly_early_ICT_first_contact.pdf (accessed 04/01/08)

McKenna, C (2001)Introducing computers into the assessment process: what is the impact upon academic practice? Paper presented at the Higher Education Close Up Conference 2, LancasterUniversity, 16-18 July 2001. Available at:http://www.leeds.ac.uk/educol/documents/00001805.htm(accessed 01/01/08)

Morris J (2007) Assessment of preclinical medical students’ ‘core’ knowledge: use of IT, MCQ, EMQ._Available at:http://www.medev.ac.uk/workshop_resources/99/99_On_line_assessment_2007.ppt (accessed 01/01/08)

Morris L & Walker DJ (2006) CAA sparks chemical reaction: Integrating CAA into a learning and teaching strategy_. In:_ QAA Scotland.(2006) Flexible Delivery: An evaluation of the use of the virtual learning environment in higher education across Scotland. Mansfield: The Quality Assurance Agency for Higher Education. 25-31.

OxfordBrookesUniversity(2002) e-Learning Strategic Plan 2002 to 2005. Available at: http://www.brookes.ac.uk/virtual/strategy/files/brookes_e-learning_strategy.pdf(accessed 04/01/08)

Oxford Brookes University(2004) School of Health and Social Care: E-learning Strategy 2004-2006. Oxford: Oxford Brookes University.

OxfordBrookesUniversity(2005) e-Learning Strategy 2005 - 2008: Embedding e-learning. Available at:http://www.brookes.ac.uk/virtual/strategy/files/Appendix%202-5%20e-L%20Strategy%202005-08%20post%20consultation%20vers51.pdf (accessed 01/01/08)

Paciello MG (2000) Web accessibility for people with disabilities. Kansas: CMP Books. Cited in: Seale JK (2006) E-learning and disability in higher education: accessibility research & practice. London: Routledge.

Pass-It (no date) ICT-based assessment. Available at: http://www.pass-it.org.uk/ICT_based_assessment.asp(accessed 04/01/08)

Paxton M (2000) A linguistic perspective on multiple choice questioning, Assessment and Evaluation in Higher Education, 25(2), 109-119.

Pollock MJ, Whittington CD & Doughty GF (2000) Evaluating the Costs and Benefits of changing to CAA. Available at:https://dspace.lboro.ac.uk:8443/dspace/bitstream/2134/1782/1/pollockm00.pdf(accessed 04/01/08)

Prosser M & Trigwell K (1999) Understanding Learning and Teaching, on Deep and Surface Learning. Buckingham: Society for Research into Higher Education & Open University Press.

Ramsden P (1992) Learning to teach in higher education. London: Routledge-Falmer. Salmon G (2002) E-tivities: The key to active online learning. London: Routledge.

Salmon G (2003) E-moderating: The key to teaching and learning online, (Second edition). Abingdon: Routledge-Falmer.

Sharpe R, Benfield G, Roberts G & Francis R (2006) The undergraduate experience of blended e-learning: a review of UKliterature and practice. Available at: http://www.heacademy.ac.uk/assets/York/documents/ourwork/research/Sharpe_Benfield_Roberts_Francis.pdf (accessed 09/01/08)

Simpson O (2002) Supporting students in online, open and distance learning (2^nd edition). London: Kogan Page.

Special Educational Needs and Disability Act 2001 (c.10) Available at: http://www.opsi.gov.uk/ACTS/acts2001/ukpga_20010010_en_1(accessed 03/01/08)

Sim G, Holifield P & Brown M (2004) Implementation of computer assisted assessment. ALT-J Research in Learning Technology. 12(3), 215-229

SWAP (Social Policy & Social Work subject centre of the Higher Education Academy) (no date) Initial guide to computer aided assessment. Available at:http://www.swap.ac.uk/elearning/elcaaguide.asp(accessed 06/01/08)

Wakeford R (1999) Principles of assessment. In: Fry H, Ketteridge S & Marshall S. A handbook for teaching and learning inhigher education. London: Kogan Page, 58-69.

Wiles K (2002) Accessibility and computer based assessment: a whole new se of issues? In: Phipps L, Sutherland A & Seale J. Access all areas: disability, technology and learning. York & Oxford: TechDis & ALT. 1-66.

Zakrzewski S & Bull J (1998) The mass implementation and evaluation of computer-based assessments. Assessment and Evaluation in Higher Education, 23(2), 141-152.

Bibliography

Bone, A (2004a) The impact of formative assessment on student learning. Available at:http://www.ukcle.ac.uk/directory/research/projects/bone.html(accessed 01/01/08)

Bone, A (2004b) Nature of formal formative assessment. Available at:http://www.ukcle.ac.uk/research/projects/formativeassessment.html(accessed 01/01/08)

Bone, A (2004c) Key findings and some recommendations. Available at:http://www.ukcle.ac.uk/research/projects/recommendations.html(accessed 01/01/08)

Buzzetto-More NA & Alade AJ (2006) Best Practices in e-Assessment. Available at: http://jite.org/documents/Vol5/v5p251-269Buzzetto152.pdf(accessed 01/01/08)

Computer Assisted Assessment - Web Resources -http://ltsnpsy.york.ac.uk/LTSNPsych/Specialist/MacAndrew/MacAndrew_Website/html/computer_assisted_assessment.html(accessed 01/01/08)

Gipps, CV. (2005) What is the role for ICT-based assessment in universities? Studies in Higher Education, 30(2), 171--180

Higher Education Academy (no date) eLearning.Available at: http://www.heacademy.ac.uk/ourwork/learning/elearning (accessed 01/01/08)

Hooper H & Cuculescu M (2004) Can computer aided formative assessment impact on student performance?_Available at:http://www.bioscience.heacademy.ac.uk/resources/projects/hooper.aspx (accessed 01/01/08)

Macdonald J (2004) Developing Competent E-learners: The Role of Assessment. Assessment and Evaluation in Higher Education, 29(2), 215-226.

Nicol DJ (2006) Increasing success in first year courses: assessment re-design, self-regulation and learning technologies, paper presented at ASCILITE Conference, Sydney, 3-6 December, 2006. Available at:http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p41.pdf(accessed 01/01/08)

Nicol DJ & Macfarlane-Dick, D (2006) Formative assessment and self-regulated learning: a model and seven principles of good feedback practice.Studies in Higher Education, 31(2), 199--218

Qualifications and Curriculum Authority (2007) e-Assessment - Guide to effective practice. Available at: http://www.efutures.org/docs/guide.pdf (accessed 09/01/08)

Public Health Resource Unit (2007) Critical Appraisal Skills Programme. Available at: http://www.phru.nhs.uk/Pages/PHD/CASP.htm(accessed 01/01/08)

Ridgway J, McCusker S & Pead D (2004) Literature Review of E-assessment. Available at:http://www.futurelab.org.uk/resources/documents/lit_reviews/Assessment_Review.pdf (accessed 09/01/08)

Sharpe R & Benfield G (2005) The student experience of e-learning in higher education: a review of the literature. Brookes e-Journal of Learning and Teaching, Available at: www.brookes.ac.uk/publications/bejlt/volume1issue3/academic/sharpe_benfield.html (accessed 09/01/08)

Shephard K, Warburton B, Maier P & Warren A (2006) Development and evaluation of computer-assisted assessment in higher education in relation to BS7988, Assessment & Evaluation in Higher Education, 31(5), 583 - 595

Stephens D, Bull J & Wade W (1998) Computer-assisted assessment: Suggested guidelines for an institutional strategy. Assessment and Evaluation in Higher Education, 23(3), 283-294.

TechDis (no date) E-assessment. Available at: http://www.techdis.ac.uk/resources/sites/staffpacks/Staff%20Packs/E-Assessment/index.xml (accessed 01/01/08)

University of Bristol (no date) Computer Assisted Assessment. Available at: http://www.ltss.bris.ac.uk/resources/caa/resources_caa.html(accessed 01/01/08).

Appendix 1: Definitions

e-Assessment is the end-to-end electronic assessment processes where ICT is used for the presentation of assessment activity, and the recording of responses. This includes the end-to-end assessment process from the perspective of learners, tutors, learning establishments, awarding bodies and regulators, and the general public (JISC 2007: 43)

Computer-assisted assessment (CAA) to refer to practice that relies in part on computers – for example, use of online discussion forums for peer-assessment, audience response systems in group work, completion and submission of work electronically, or storage of work in an e-portfolio (JISC 2007: 43).

Cmputer-based assessment (CBA) is used in this publication to refer to assessments delivered and marked by computer (JISC 2007: 43).

e-Learning: The process of learning which is supported by the use of ICT. Also used loosely to describe the actual content delivered on screen, and the more general use of ICT to contribute to learning processes. (JISC 2007: 43)

ICT-based-assessment – is the assessment of knowledge, understanding and skills using technologies such as the Internet, intranet and CD-ROM (Pass-IT, no date)

Appendix 2: Benefits of computer-based assessment systems include

• A wide range of topics can be tested very quickly;

• Can monitor the progress of students through more frequent assessments;

• Graphics and multimedia can widen the scope of questions;

• Computerised marking of tests saves time;

• Large groups can be assessed quickly;

• Diagnostic reports and analyses can be generated;

• Results can be automatically entered into administration systems;

• Students can monitor their own progress through self-assessment;

• Students acquire information technology (IT) skills;

• Formative assessments can be used to aid learning;

• Students can be provided with clues and marked accordingly;

• Adaptive testing can be used to match the test to the student’s ability;

• Feedback is available to students during and after a test.

(Gladwin 2005:1) - To broaden the range of knowledge assessed;

• To place the emphasis and time spent on assessment design rather than marking;

• To increase consistency and objectivity of marking and minimise human error

• To decrease staff marking loads;

• To aid administrative speed and efficiency;

• To expand optional tools not available for use with paper-based exams, for example multimedia;

• To facilitate an increase in the frequency of testing, which may assist with student motivation;

• To increase the amount of feedback both learners (scores, hints, praise, guidance) and lecturers receive (question statistics, scores, reports);

• Quality of questions can be monitored easily using statistical analysis produced by course management software;

• Potential for sharing questions with other team members;

• Provides opportunities to practise skills learned;

• May help to identify individual student and cohort difficulties and weakness.

(SWAP no date: online)

Appendix 3: Limitations of computer-based assessment systems include

• Construction of good objective tests requires skill and practice and so is initially time-consuming;

• Testing of higher order skills is difficult;

• Possible to reinforce misconceptions;

• Implementation of a CAA system can be costly and time-consuming;

• Good system maintenance is required to avoid downtime during examinations;

• Difficult to reproduce freedom of paper examination - e.g. scanning questions to choose which to answer;

• Students require adequate IT skills and experience of the assessment type;

• Assessors and invigilators need training in assessment design, IT skills and examinations management;

• A high level of organisation is required across all parties involved in assessment;

• Overuse may promote surface learning;

• Large bank of questions required to prevent banding and hence loss of discrimination.

(Gladwin 2005:2) - Hardware and software failure possibilities;

• Secure servers are required to ensure learners cannot go elsewhere on the Internet to gain the answers;

• Security issues need to be addressed to ensure the identity of each student is verifiable;

• Learners need to demonstrate a minimum level of IT skills prior to taking the assessment;

• The development of rigorous and appropriate questions requires training and staff time ;

• High level of co-ordination is necessary between department, computer support, invigilators and administrative staff ;

• There are limitations on the knowledge and skills that can be assessed;

• Most assessment development software does not support textual responses beyond a sentence. Higher order skills are required to develop appropriate questions and alternative answers for 'soft knowledge'.

(SWAP no date: online)

Appendix 4: Stages of e-assessment

Some form of e-assessment may be used at each of the stages at which a learner’s attainment and progress come under review:

1. Diagnostic – assessment of a learner’s knowledge and skills at the outset of a course.

2. Formative – assessment that provides developmental feedback to a learner on his or her current understanding and skills. Formative assessment can also be described as ‘assessment for learning’ since an assessment that is entered into voluntarily, and on which no final qualification depends, can prompt learners to adjust their own performance.

3. Summative – the final assessment of a learner’s achievement, usually leading to a formal qualification or certification of a skill. Summative assessment is also referred to as assessment of learning. (JISC 2007: 6)

4. Self assessment – to test ones own understanding (Gladwin 2005:1) this could also be administered within a formative or summative assessment framework.

© J Pawlyn September 2007