Final Project

Overview

The purpose of the final project is to provide you with the opportunity to demonstrate and improve your capabilities as a researcher. To accomplish this, you will either independently or in a group (your choice) propose a research question that can be answered with one of the many datasets made available to you. After deciding on a research question you will write a proposal explaining your intended course of action to answer your research question. After receiving feedback and adjusting your course of action accordingly you will begin your research.

Later in the term you will present a brief progress report to the class. The purpose of this progress report is to both give you practice presenting in front of the class, which you will have to do again upon your project’s completion, and to give you feedback on the current state of your work, and perhaps inform you of other directions or specific constraints to consider. At the end of the term you will complete your research, create a final presentation, and write a report, which you will submit to a venue on artificial intelligence in education. The quality and completeness of this presentation and report, in addition to the thoughtfulness of your discussion board questions, will determine your final grade.

Deliverables

• Project Proposal (Due Sept 29th before class)

  • 1-2 pages answering the following:

    • Who is on your team?

    • What data will you use?

    • What is your research question?

    • What novelty does your research provide?

    • What steps will you take to answer your research question?

    • How will you gather your results?

    • What metrics do you intend to use to report your results?

    • Where will you be submitting your paper?

• Progress Presentation (5 minutes; due Oct 24th before class)

  • Everyone will present their progress presentation during class.

  • At minimum your presentation should summarize the following:

    • What is your research question?

    • What is the value of your research?

    • What similar research has been done in the past?

    • What methods are you using to address your research question?

    • What data are you using?

      • If you haven’t prepared your data yet, ask the class for advice.

      • If you have prepared your data, tell the class how you did so.

    • What information do you plan on getting for your results?

    • What metrics will you use to evaluate your results?

      • If you have preliminary results, share them with the class.

    • If you have any questions, ask them during your presentation so the class can give you feedback. This is the progress report, you're not expected to have anything complete, just to be working. Often it is important to try something and show it, especially if it’s not complete, so that people can see what you’re doing and give advice before you’re done and while you have time to improve your research.

• Final Presentation (15 minutes; due the day of your presentation before class)

  • Everyone will present their final presentation during class.

  • At minimum your presentation should summarize the following:

    • What is your research question?

    • What is the value of your research?

    • What similar research has been done in the past?

    • What methods are you using to address your research question?

    • What data are you using and how did you prepare it?

    • What were the results of your research?

    • What were the limitations of your research?

    • What conclusions should we make about your research?

• Research Paper (Due Dec 12th before class)

  • Must be in the proper format specified by the chosen venue

  • Each research paper should have sections similar to the following, but not necessarily exactly these sections:

    • Abstract

    • Introduction

    • Background / Previous Work

    • Data Preparation / Methodology

    • Results / Discussion

    • Limitations / Future Work

    • Conclusion

Research Question Guidelines

• You should aim for your research question to be answerable using the data made available from the ASSISTments platform. The individuals helping you succeed in this class have the most experience with this data and will therefore be able to help you the most.

  • Good:

    • What features of hints are most likely to improve learning? (ASSISTments Student Support Dataset)

    • Do patterns in students' success mastering previous skills predict trends in how much they will struggle with future skills. (2019-2020 School Year Dataset)

  • Bad:

    • Can smartphone accelerometer data be used to predict users' age?

    • Can ad personalization be improved by a novel clustering algorithm?

• A good research question has a clear, actionable goal in mind. This goal can involve looking for a trend or pattern in data, comparing existing methodologies, or developing a new model or process, but there must be a clear next step or application of the results of your work. Think “What will I have to show for my work?” and “How will this work be useful to other people?”.

  • Good:

    • Are there common patterns of curriculum content assigned to students using ASSISTments and how do these patterns effect students’ learning?

    • How does a novel theory-based approach to latent feature extraction compare to common methods when predicting students’ homework completion?

  • Bad:

    • What is the distribution of content used in ASSISTments?

    • Which support vector machine kernel is best for predicting students’ homework completion?

• A good research question must provide something novel to the learning science community. The novelty can be from a new process, model, or analysis, but doing the same thing someone else did with nothing to add is not good enough. To ensure you’re doing something novel, search on google scholar for anything related to your work.

  • Good:

    • How do different teachers compare to each other in the TeacherASSIST data when using partial pooling instead of fixed effects?

    • How do the various methods of knowledge tracing compare to each other using the new and much larger dataset on knowledge components included in the 2019-2020 school year dataset?

    • Neil recognizes that for you to recognize novelty is super hard for a young student.

  • Bad (only because they are no longer novel):

    • What were the individual treatment effects for each problem during the ASSISTments Maine study?

    • Can bandit algorithms be used to improve learning gains during education experiments?

• A good research question will provide value to the learning science / user modeling community. These questions sometimes necessitate the creation of an artificial intelligence, statistics, or machine learning based model (neural net, decision tree, logistic regression, hierarchical clustering, Bayes net, Q-learning, ANOVA, etc.). The model doesn’t have to be complex, but a question that can be answered without some kind of model fitting or that doesn't help the community is unlikely to be a good research question.

  • Good:

    • How does teachers’ experience correlate with the difficulty for students to master a variety of skills?

    • Which features and models are the most effective for predicting problem difficulty?

  • Bad:

    • How many students used ASSISTments each month over the last two years?

    • What curriculum is used the most in ASSISTments?

• A good research question has clear metrics for evaluating success. If you are looking for trends, the correlation coefficients could be your metric, if you are designing a new model, AUC, accuracy, R-squared, or training time could be your metric. You need a way for the quality of your method or model or results to be evaluated and understood by a reviewer.

  • There aren’t really examples of specifically good or bad questions for this one, the same question could be both good or bad depending on how you choose to evaluate your findings. For example, if your question was “What is the best type of tutoring?” and you simply concluded in your paper that it was scaffolding, without any metrics, that would be bad, but if you showed students’ average and standard deviation of their next problem correctness after receiving each type of tutoring, then now you can use a t-test and have a metric to evaluate the quality of scaffolding.

Presentation Guidelines

There are many different guidelines on how to make a good presentation. Many of them contradict each other. This guide will explain methods of presenting that have tended to work well in the past, and you can take what you want from it.

• You are the only person who knows about your research. You have to explain everything. Make sure to structure your presentation so that the high level concepts are explained at the beginning, and then later you can get into the weeds of how you implemented your specific algorithm and the implications of the values of your metrics. For example, if your goal is to explain how an algorithm works, it would be salient to first explain why the algorithm is needed, and in what circumstances it will be used. Providing that information helps the audience contextualize the algorithm and gives them a better chance to understand it.

• The purpose of a presentation is to convey information to your audience. YOU are the best tool for that, not the screen. The presentation should be structured such that speaking to the audience is the primary way that information is conveyed. The slides should be used to help illustrate things like examples of things you talked about, difficult to process concepts, multi-dimensional data, and results tables.

• A good slide has one purpose. Don’t overwhelm the audience with dense slides, one or two images or a table per slide is plenty. Don’t have slides with paragraphs or lots of bullet points, that’s stuff you say out loud. There’s no penalty for having lots of slides, so focus on one point that you want to convey per slide and let the presentation be as many slides as it needs to be.

Research Paper Guidelines

The best way to learn how to write a research paper is to write a research paper. The second best way is to read other peoples’ research papers. Lucky for you that’s what you have to do twice a week for this class, so pay attention and copy the formats you see.

Keep in mind, these research papers are 7-10 pages, one or two column, single spaced, tiny font papers. It’s A LOT to write. If you find yourself without enough content to fill a full paper, consider writing a short paper or poster paper, both of which are often available tracks for submissions at conferences. It’s better to have a good short paper than a bad long paper, because the former will be accepted and the latter rejected. Each conference will have templates you can use and length limits depending on the track.