One of the most powerful functions of the learning management system (LMS) is assessment, because it allows teachers to quickly and easily scale up to massive levels on a variety of dimensions. Perhaps the first that comes to mind is the number of students it can reach, but this seems pointless if you only teach a class of, say, 25. Certainly, this is the case for most teachers, so they often don't feel that the effort to move their curriculum into the LMS is worth it.

There are, however, a few areas where scale can even help teachers in this particular situation. Specifically, it is quite easy to create large banks of items from a simple pedagogical start, and then implement them into a random draw system that collects detailed information on student responses.

Before we start, it is important to put this effort into the proper perspective. To cut to the chase: large banks of auto-scored items are not a magic solution to learning. Assessment items are very easy to create, but determining whether or not they are effective is a considerable undertaking. For this reason, teachers should be extremely cautious about making assumptions regarding the results of such items. At the same time, students should be realistic about the potential rewards of studying for and taking such assessments. In short, a good score does not necessarily imply that a particular goal has been attained. Without robust data that strongly correlates a score with multiple measures of concept attainment, any "assessment" is, in fact, only an *attempt* at assessment. It may be true that a teacher's intuition is often a very strong indicator, but experienced teachers know that test scores, good or bad, usually do not tell the whole story of any one student's achievement.

Now that we have that point out on the table, let's start with how to create the items. I'll give two examples, both related to language learning, but hopefully the method can be expanded to other fields. The paradigm I use for language assessment is the idea that learning moves to mastery by progressing from recognition to comprehension to production. Assessing production automatically is quite difficult, but assessing recognition is quite straightforward. 

By far the most common item type in assessments is multiple choice, but for most test-takers, this is usually just a process of elimination, not driven by a pure understanding.  Effective items are possible, but they are quite difficult to create without a significant investment in data analysis. Recognition of correctness, on the other hand, is much simpler: Is this OK or not? For grammar, is this sentence grammatical? For vocabulary, does this word fit in this context? 

My process for creating grammaticality test items is very straightforward. I start with an example from a text that students have seen that includes a grammatical construction that they should know or that I have explained. I create another sentence that means the same thing with a similar but different grammatical construction, and then two that are either grammatically incorrect or have a different meaning. In this way, from one correct example, I have two sentences that are "OK" and two that are "Not OK," for a total of 4 items. Thus, for any one particular concept, I get 4 items. It is possible to argue that learners might be able to make use of this pattern by simply recognizing pattens that are ungrammatical or differ from the original, but it is also possible to argue that any student who has gained that level of recognition has, in fact, mastered the grammaticality judgment of that construction. 

As an example, I will use a few sentences from https://openstax.org/books/introduction-political-science/pages/7-5-how-do-governments-bring-about-civil-rights-change:

• The emergence of social movements demanding political change and the extension of civil rights may help minority groups secure those rights, but it is the work and responsibility of political institutions to fully address past discrimination.

Creating three more sentences to add to the first one:

• The emergence of social movements demanding political change and the extension of civil rights may help minority groups secure those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=OK.}

• It is the work and responsibility of political institutions are responsible for fully addressing past discrimination, and it is their job to accomplish that goal, even though the emergence of social movements demanding political change and the extension of civil rights may help minority groups secure those rights. {=OK.}

• The emergence of social movements demanding political change and the extension of civil rights may help minority groups secure those rights, but this is the work and responsibility of political institutions to fully address past discrimination. {=Not OK.}

• Social movements demanding political change emerging and civil rights extending may help minority groups secure those rights, but the work and responsibility of political institutions is to fully address past discrimination. {=Not OK.}

Vocabulary is somewhat more difficult, and my understanding of the challenge has evolved. I started by creating fill-in-the-blank items with the target word, which allowed me to create OK/NotOK items by substituting other words for the target word, similar to what Quizlet does. 

• The emergence of social movements demanding political change and the extension of civil rights may help minority groups [secure] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=OK.}

• The emergence of social movements demanding political change and the extension of civil rights may help minority groups [safe] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=Not OK.}

• The emergence of social movements demanding political change and the extension of civil rights may help minority groups [have] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=Not OK.}

• The emergence of social movements demanding political change and the extension of civil rights may help minority groups [got] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=Not OK.}

One annoying limitation of this model is that, while it is easy to find words that do not have the same meaning, it is often very challenging to find synonyms that actually conveys the exact same meaning in context. For this reason, I would make three incorrect items for each correct sentence, trying to use words that look similar or are at least more likely to be more unfamiliar than the correct word. Another limitation is that some words still "fit" in the sentence, even though they are not the original word. Realistically, it is very difficult to expect students to deem these sentences "Not OK" unless they have memorized what word was in the original sentence.

Recently, I took a step back to think about what I am really trying to assess with these vocabulary items, and I realized that my item structure tests the *sentences* and not the *words*. So rather than substituting different words into the same sentences, I needed to substitute different sentences for each word. In other words, I needed incorrect usages of the target words / phrases. This paradigm also allowed me to easily create another correct example so that I could balance two OK sentences with two Not OK sentences, giving the ideal 50% possibility. 

As I started to put this idea into practice, especially when extending it to student work, I realized that there are actually many ways to make an incorrect sentence with a word. Some are trivially easy to recognize, especially if they make use of broader grammatical principles rather than specific usage characteristics of that word. Ideally, the perfect incorrect distractor is something that a learner would mistakenly see as reasonable. This type of item is not impossible to create, but it does require a bit more thought (and experience) than just picking out three wrong words from a list. 

(Sentences from other pages in the above textbook, and a citation is provided below).

• The emergence of social movements demanding political change and the extension of civil rights may help minority groups [secure] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=OK.}

• “The State shall [secure] that the operation of the legal system promotes justice, on a basis of equal opportunity, and shall, in particular, provide free legal aid, by suitable legislation or schemes or in any other way, to ensure that opportunities for securing justice are not denied to any citizen by reason of economic or other disabilities.” {=OK.}

• Some examples of civil liberties mentioned in the Bill of Rights include “the freedom of speech, or of the press” (1st Amendment); “the right of the people to be [secure] in their persons, houses, papers, and effects” (4th Amendment); and the right against “excessive bail” and “cruel and unusual punishment” (8th Amendment). {=Not OK.}

• The Malawi “model” proved hugely successful in improving the civil rights of prisoners who did not have the means for [secure] good representation and in helping the accused know their rights. {=Not OK.}

It struck me, however, that this particular problem is unique to testing comprehension. Grammaticality judgments are a great way to test speaking and writing skills, because teachers want to give learners practice in precisely that type of recognition, so that they can recognize and fix problems in their own production. Following that train of thought, even the sentence (rather than word) substitution is not an ideal test, but it is somewhat more pedagogically valid. Once again, it is tempting to fall back on translation, but I feel like the disadvantages of encouraging the idea of a one-to-one lexical relationship between two languages are much larger than that of an awkward item structure.

Hopefully, the way that the sentences above are juxtaposed gives away my secret. There is a pattern where one part of the sentence is changed. This similarity makes it easy to copy/paste the original sentence four times and then just modify three versions. By the same token, it is very easy to create one row of a spreadsheet and then just change one small element of it. More specifically, split the sentence at the point where the variation happens, copy the portions that do not change down for three more rows, and then manually add the part that varies for those three rows.  The spreadsheet also makes it very easy to put this block of sentences into a format that can be imported into the learning management system. I use Moodle, so the GIFT format is an easy choice for me:

The emergence of social movements demanding political change and the extension of civil rights may help minority groups secure those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=OK. ~Not OK.}

It is the work and responsibility of political institutions are responsible for fully addressing past discrimination, and it is their job to accomplish that goal, even though the emergence of social movements demanding political change and the extension of civil rights may help minority groups secure those rights. {=OK. ~Not OK.}

The emergence of social movements demanding political change and the extension of civil rights may help minority groups secure those rights, but this is the work and responsibility of political institutions to fully address past discrimination. {~OK. =Not OK.}

Social movements demanding political change emerging and civil rights extending may help minority groups secure those rights, but the work and responsibility of political institutions is to fully address past discrimination. {~OK. =Not OK.}

<---

The emergence of social movements demanding political change and the extension of civil rights may help minority groups [secure] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=OK. ~Not OK.}

The emergence of social movements demanding political change and the extension of civil rights may help minority groups [safe] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {~OK. =Not OK.}

The emergence of social movements demanding political change and the extension of civil rights may help minority groups [have] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {~OK. =Not OK.}

The emergence of social movements demanding political change and the extension of civil rights may help minority groups [got] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {~OK. =Not OK.}

<---

The emergence of social movements demanding political change and the extension of civil rights may help minority groups [secure] those rights, but it is the work and responsibility of political institutions to fully address past discrimination. {=OK. ~Not OK.}

“The State shall [secure] that the operation of the legal system promotes justice, on a basis of equal opportunity, and shall, in particular, provide free legal aid, by suitable legislation or schemes or in any other way, to ensure that opportunities for securing justice are not denied to any citizen by reason of economic or other disabilities.” {=OK. ~Not OK.}

Some examples of civil liberties mentioned in the Bill of Rights include “the freedom of speech, or of the press” (1st Amendment); “the right of the people to be [secure] in their persons, houses, papers, and effects” (4th Amendment); and the right against “excessive bail” and “cruel and unusual punishment” (8th Amendment). {~OK. =Not OK.}

The Malawi “model” proved hugely successful in improving the civil rights of prisoners who did not have the means for [secure] good representation and in helping the accused know their rights. {~OK. =Not OK.}

To create the uploadable file, I just copy one specific column of my multiple-row spreadsheets into a .txt file. Once these files are in the system, those questions can be specified to for use in random-draw quizzes. I like to set a 10-item quiz with a 180 second time limit, and then allow unlimited attempts. This method creates four times the number of original sentences, so with twenty originals, you get 80 items, which provides not only good practice, it challenges students with an opportunity to get a higher and higher score as a good incentive to continue interacting with the content. Moodle provides rich data on these items which can be aggregated and analyzed to determine which are better and which are worse. 

It is important to take a step back and point out just how powerful this method is. Where teachers all have lists of good examples from which students gain an understanding of a specific point, those sentences (or whatever) can now become not only powerful assessment tools, those .txt files can be imported into any Moodle system. With modification, it is likely that they can be imported into *any* learning management system. Just as a previous generation found mimeographs and then xerographic copying to be a powerful way to bring content from course to course or from institution to institution, question banks give teachers mobility like never before, with the added benefit of powerful data analysis. 

More on GIFT format

One small note on the GIFT format method introduced above: There are many other options for item creation, even in Moodle, even in GIFT format. For a peek into that world, one place to start would be the Moodledocs page on GIFT. Near the end of the page, there is a section titled "Tools that create or process GIFTs," and many of the links lead to other very interesting areas.

References and examples

Examples are from https://openstax.org/books/introduction-political-science/pages/1-introduction

Access for free at https://openstax.org/books/introduction-political-science/pages/1-introduction

Excel example: Example-QuizItemBank-GIFT.xlsx

How-to video: https://youtu.be/GB-aatVS-uM