Resources‎ > ‎

Running Experiments Online Using Amazon Mechanical Turk


This website is no longer updated, the version at www.josephjaywilliams.com/mturk (or via short URL tiny.cc/mechanicalturk) is kept up to date.



Video overview of the benefits of Mechanical Turk to run experiments

Scientific publications on Mechanical Turk

Paolacci, G., & Chandler, J. (2014). Inside the Turk Understanding Mechanical Turk as a Participant Pool. Current Directions in Psychological Science, 23(3), 184-188.

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk A New Source of Inexpensive, Yet High-Quality, Data?Perspectives on Psychological Science6(1), 3-5.

Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on amazon mechanical turk. Judgment and Decision Making5(5), 411-419.

AHN, D., & YO, N. (2011). Social science for pennies. Science

Article in Behavior Research Methods

Online resources on Mechanical Turk

A go-to site aimed at the research community, that aggregates resources as well as having regular blogs.
    Sample blog posts: Emailing workers using Python, Screening participants using Qualtrics, The Fundamental AMT Articles.

PsiTurk - a platform that augments Mechanical Turk and makes it easier to conduct experiments, by preventing repeat participation.

How to set up Mechanical Turk in 5-10 minutes?


What to do if you need help right now?


A go-to site aimed at the research community, that aggregates resources as well as having regular blogs.
    Sample blog posts: Emailing workers using Python, Screening participants using Qualtrics, The Fundamental AMT Articles.

Getting started using Mturk


These are some useful links for how Amazon Mechanical Turk can be used for asking people to participate in studies. 

Our old lab guidelines for running Mturk experiments


(1) Use a script in the HIT to pipe MturkIDs into Qualtrics survey URL
(2) Match participants' MturkIDs to a list kept in an embedded data field in Qualtrics survey flow


How to post 

The search bar in mturk, and what an HIT with an autogrant qualification might look like.

Using a pre-screening image survey


This idea is based on the notion that workers will remember an image better than an mturk requester or type of HIT. You'll set up a survey with an image that will redirect people to another survey. This is imperfect, but it has the added bonus of being able to redirect to any survey you would like. This method relies on appending an embedded data value to the survey url -- for more information on this, see http://www.qualtrics.com/university/researchsuite/append-information-to-the-url

Creating the image survey


To do this, first create a new survey in Qualtrics. This survey only needs one block, and does not need to have an actual question. The block should have an image (it doesn't matter what the image is, so long as it is distinctive and recognizable) and some text saying something along the lines of "If you recognize this image, please do not take this survey." 

Next, go into the survey flow.  The first thing you will want to do is add an embedded data block right at the very top with a single embedded data field. You can name the embedded data field anything you'd like, but it will be the variable that controls what survey people are sent to after clicking through your image survey (see screenshot below).

Next, for every value you want the embedded data field to have (with each value representing a separate survey you want to redirect to), create a conditional branch where the condition specifies what to do if the embedded data field is equal to that value (again, see screenshot). 

 

Under each branch, add an end of survey element, and hit the customize button. Select the "redirect to a URL" option, and paste the URL to the survey you want people to take into the box. See the screenshot below. Do this for every individual survey you want to use this particular image.


Using Bonuses

You can't pay someone directly, and so you can create a HIT with zero reward, and then bonus somebody who you want to compensate.

Intermediate level information


Amazon's site for requesters that gives an overview of Mturk
Amazon Mechanical Turk User Interface Guide for Requesters (step by step instructions & images on doing things)




Block Worker from taking same HIT multiple times (link)

Experimental Turk also has an aggregation of Academic Articles on Mturk

Useful tips from IT World article (including follow-up studies)

Specific Goals

How to Contact Mturk Workers, Do follow up experiment



The Developer Forum Search on this topic of how to contact Mturk Workers (and any others) was useful

Emailing workers using Python, where the MturkIDs can just be pasted in in a list

How to link Mturk with Qualtrics (Not recommended by Dave)

http://reactiontimes.wordpress.com/2012/02/08/connecting-mturk-to-qualtrics-to-scriptingrt/



Other resources


This website has a collection of programming tools built up around Mturk, and the owner is looking to open source it: Headlampresearch.com link



Other uses of Mturk
These links may give ideas for other ways that Mturk can be used: With a bit of creativity, it probably offers valuable solutions for subcontracting out other work that is involved in doing research, developing experiments, running a lab, finding resources, providing & assessing online education, & providing & assessing web-based resources for habit and health behavior change.

Using Amazon Mechanical Turk for Data Projects, journalism uses by ProPublica