Data and Access

Surveys are usually administered on paper or electronically.   They are formed of a series of predetermined questions, often free-response or multiple choice.  Some surveys have branching logic, (e.g. if participant answer X, then show question A; if they answer Y, show question B), but many surveys do not.

Surveys are a great choice ...

• to bring prevalence information to rich qualitative data.

• for screening potential interview participants.

• to replicate other studies or compare across groups.

Surveys are a poor choice ...

• when you have few participants

• if you don't validate them

• as a sole source of information in a project.

Other considerations about surveys

Some people use a survey to screen participants for some other research participation.  For example, you might survey all students in a class to identify students with particularly interesting ideas for follow-up interviews (or just interest and availability for interviews), or you might conduct followup surveys of all your interview participants one year later, to see how their ideas have developed.   

Other people use surveys as a primary data source about student ideas.  For example, you might administer a concept inventory to all students before and after instruction, so that you can see how their ideas are changing.  Alternately, you might want to look for differences in student beliefs about science as a function of student demographics or institutional characteristics.   

• Where can I get them?

  • If you know which one you want, write to the author

  • If you want to know what's available, do a literature review

• Can I make my own?

  • This is very common if the purpose of your survey is very local, or if you're using the survey information primarily to screen participants for some other research participation.   You should check that participants are responding to your questions in ways that you understand, but you don't need to go through an extensive validation process. 

  • Alternately, some researchers want to develop a new concept inventory or generalizable research survey, for use beyond their particular local context.   Validating a new survey about student ideas can be a really extensive process without much payoff at the end, especially because new survey developers often overestimate the desire of other faculty to use their developed surveys.   While building a new research survey is an appealing idea to many new researchers, it is rarely actually a good choice.   It's often better to start with qualitative analysis of written / verbal information, and use your initial research to motivate other extensions or generalizations of your work.

• If you use surveys, you might need to use statistics to understand your results quantitatively.

  • Some people are really excited about this.  Some people are intimidated.

  • If your research design might require the use of statistics, then you need to conduct a power analysis (or similar quantitative estimation) to guess at how many participants you will need in order to see the differences you expect.  If your power analysis suggests that you will not be able to detect differences with the group of participants you have, then surveys are unlikely to be a good use of your time (or the time of your participants).

  • (of course, if you're using a survey to collect screening or scheduling information about participants, statistics might be excessive or nonsensical).

Interviews are characterized by real-time human interactions, usually between an interviewer (who asks specific questions from an interview protocol) and an interviewee.  Some interviews have more structure to the questions, while others flow more like a conversation.  There's a specific primer on interviews which covers why you should use them, the major kinds of interviews, how many you need, and some other common considerations. 

Interviews are a great choice ...

• to probe how individuals/groups think about a concept or procedure

• to probe how people reflect on a situation or experience

• for validating potential survey questions

Interviews are a poor choice ...

• when you aren't quite sure what you're looking for

• if you need to see what happens in classrooms

• if there's a big power differential between you and your participants

Other considerations about interviews:

• There are lots of different kinds of interviews.

  • Think-aloud, problem solving, focus group....

• Practice your protocol!

  • You should plan to try it out on 2-3 people, then revise the protocol, then try it out again.

  • Your IRB will want to see the final protocol after revision, before data.

• How will I know how many people to interview?

  • Exploratory interviews to see what kinds of ideas are available: 3-5

  • Saturation to find all the ideas: emergent process, but usually 10-15

  • Population: all the people in your population (unusual, but possible)

• Generally, you want to overestimate how many you'll need

Classroom artifacts include anything the students are doing as part of their work in the class, for example copies of their homework or exams, or pictures of their whiteboards.  It also includes artifacts that instructors generate, like slides or problem sets, as part of their work in the class.  It doesn't include anything that researchers specifically ask students to do for research purposes only. 

Classroom artifacts are a great choice...

• because they are cheap and easy to collect

• when you are curious about what happens in real classrooms

• because they usually permit reanalysis for multiple projects

• as an additional source of data for a larger project

• when you want to explore new ideas

Classroom artifacts are a poor choice...

• when you need to find out why students do something

• if you want to test a new idea or strategy

Other considerations about artifacts:

• IRB for these can be tricky to navigate

  • IRBs are very different in their interpretations of the regulations

  • Post-fact permission for classroom artifacts is ethically fraught.

• You will see awesome things ...

  • ... and then not be able to follow-up with those students

• Artifacts are common in archival data

  • Meta-data is very important in understanding artifacts.

  • Always put in more meta-data than you think you need.

Classroom observations are a great choice for when you want to see what real students and teachers are really doing in real classrooms.  Broadly speaking, collecting observational classroom data means recording what happens in a classroom.  Video is the gold standard for recording, but it's also possible (and sometimes desirable) to collect only audio data, or only field notes from observers.  In online or computer-based contexts, classroom observations usually take the form of screen recordings or zoom recordings; things like their written work are more often considered artifacts. 

Classroom observations are a great choice...

• because the data are rich and multifaceted.

• when you need to see what "really happens"

• if you're interested in how people interact with each other or with equipment

• early in a research project, to help you generate new ideas

Classroom observations are a poor choice...

• when you want to be able to probe participants' ideas in the moment

• if you cannot also collect classroom artifacts to understand what they're doing

• for research questions that depend on controlled conditions or individuals acting alone.

Other considerations with classroom observations

• Some observations are highly unstructured and emergent

  • This is more common in the early stages of a research project, to help you generate new ideas and questions

  • Video data is commonly used to support repeated viewings of the same interaction to gain insight.

• Some observations are highly structured and deterministic

  • There are observation protocols which can be used in live classrooms or with classroom video.

  • They are more common in situations where video recordings are difficult to obtain (e.g. with children or in the EU), or where large quantities of classroom time need to be analyzed to find patterns. 

• Archival videos of classroom observations are super common, but the ethical considerations of how to analyze the video are complicated. 

Reflections include writing that researchers generate to systematically document or understand a setting or the people in it.  They can include your generative writing or reflective memoing.  The central feature of this data type is that you, the researcher, generate this data.   

In contrast, analyzing reflections written by your research participants (either as part of a free-response survey or as assignments for their classes) is another data type.  

Reflections are a great choice...

• If you want to describe a curriculum or the implementation

• If you have a dual role as an instructor and as a researcher, and your research is about your own growth in teaching.

• as one of several data streams, to help contextualize them or tie them together.

Reflections are a poor choice...

• as a sole stream of data 

Archival data are research data which already exist.  This is not the same thing as classroom records which are sitting around in the back of the file cabinet!  For a data set to be archival research data, it must have been collected as research data and (still) covered under a relevant ethics approval.  

Archival data are a great choice...

• when you want to explore ideas and possibilities

• when you need to write a paper quickly

• if your research question is about development over a long time

• if you can't access new data

• if you want to make arguments about baseline vs. change

Archival data are a poor choice...

• when your research questions are very rigid or specific

• when they are very poor quality or missing crucial metadata

• if the IRB which covered it has expired

Other considerations with archival data:

• Can I use old exams / homeworks / etc as data?

  • This is an IRB question.

  • If they aren't already research data, then probably not.

• You get what you get.

  • sometimes, the data are not well-aligned to your research questions

  • you cannot ask follow-up questions of the same participants

• Is it ok to write new papers if parts of the data have already been analyzed?

  • Emphatically: yes. 

  • Articulate difference & cite other papers.

• Who has some archival data that I can work with?

  • Lots of people!  Start by emailing the authors of recent studies that you find interesting.

• The older the data are, the harder it is to remember what's going on.

  • Meta-data is enormously important when building a catalog of archival data or maintaining a data library.

  • You're going to lose data because files get corrupted or whatever, but it turns out that's not the worst.  The worst is when you still have some data files and you have no idea what's in them or what the students are doing or when the data were taken or what the answers correspond to.   

Some research is great for small sample sizes

• Very rich data / many observations of the same people

• case studies / exemplars

• claims about existence

Some research is not

• Impoverished data (e.g. surveys, homework responses)

• claims about prevalence 

My papers include a study of 50k students over 20 years, and a close analysis of 37s of video in one group where only two people talk.  Sample size matters less than matching data with research question.

Emily thinks that teaching her students explicitly how to craft scientific arguments will help them understand popular press articles in science.  She wants to develop an in-class activity about crafting scientific arguments, and a homework set about reading and understanding popular press articles.  She teaches two sections of 30 students every fall.

• What kinds of data could Emily collect?  Be specific, e.g.: 

  • not just "surveys": from whom? how frequently? on which topics?

  • not just "registrar data": what pieces of information? 

  • not just "classroom artifacts": which ones? on which topics? 

Oftentimes, in considering this scenario, people have an implicit research question in mind for Emily to pursue.   What research question is Emily pursuing?  For this research question, is she likely to be able to collect enough data?  If not, it's time to develop a different research question that matches the amount of data she can collect. 

Emily's project could be a great example of a design-based research project.

Arthur's department teaches an introductory class with 8 different instructors and 10 different GTAs.  The department is contemplating making big changes to the format and coverage of the class, and has tasked Arthur with collecting baseline data so that they will know if the changes are working.

• What kinds of data could Arthur collect?  Be specific, e.g.: 

  • not just "surveys": from whom? how frequently? on which topics?

  • not just "registrar data": what pieces of information? 

  • not just "classroom artifacts": which ones? on which topics? 

• Some baseline data will come from the upcoming year, before the department makes changes to their teaching.  Other baseline data concerns semesters which have already passed.  What kinds of retrospective data might be available ethically?  

• What do you think Arthur's department (or Arthur), means by "if the changes are working"?  Working for whom, and in what ways?  Think about what kinds of research questions Arthur could pursue. and how those are supported (or not) by each of the data types he could collect. 

Arthur's project could be a great example of a "getting bigger" iterative design.