#4 How to perform multi-stage and complex experiments

MTurk is not optimized for following up with a subject after a few months. It does allow you to assign predicates with each user who performs a task, and later offer HITs only to users who have (or who don't have) predicates of your choice. That's a bit complicated, though, and leads to much lower opt-in rates than directly contacting the desired subjects. Here's what you can do:

• Post a HIT where you asks turkers to perform some task, which allows you to collect demographics or other information for future subject calls. The task might appear independent of your study, and primarily serve to classify potential subjects so that you later can determine whom to ask to participate.

• Use the "send an email message" feature to ask subjects to participate in a follow-up study. By using MTurk as an anonymous proxy, they will receive an email with your e-mail address without disclosing their contact information to you. You can also send an email by clicking on "Worker ID" in the list of completed tasks.

• Include a URL in your email. This URL could be personalized for each subject, so once you know a given URL was accessed (and task completed), you would know whom to pay. Alternatively, you could ask the subject to enter a "payment code" on the site. You need to give them this piece of information in the email, and it has to be unique so that you can pay them after the task is performed. I like using the temporary user pseudonym as an identifier. This is a tag that is specific to this user and to this HIT. You can put together the email by cutting and pasting text and worker IDs which you can obtain from the MTurk site.

• Pay a "bonus" payment once a follow-up task has been completed. How? Look up the pseudonym you received from the list of completed HITs from the task described in step 1 above, click on the worker ID, and press "Give bonus payment". Regular payments that are not part of a multi-phase study are paid using the normal MTurk payment interface, which is straightforward and allows payment of several subjects at the same time. [To simplify your life and increase efficiency, you can write a script that handles subject interaction and payments. However, the script has to be able to parse error messages from Amazon which aren't uncommon.]

For constructing and deploying more complicated surveys, MTurk offers a programmable tool. But it isn't easy to do, and doesn't offer an easy visualization of results. Instead, I set up a survey on SurveyMonkey and link to that in my recruiting message.

It's also possible to ask your subjects to visit a URL of your suggestion, perform a task there, and report back an observation to another site including MTurk. Or, you can ask subjects to input their observations on the visited site.

For example: you can ask subjects to visit 3 websites and rank them.  One of them is yours or your client's, the others belongs to competitors — which one is nicest, and why? Similarly, you may have constructed a site that dispenses user advice – such as the SecurityCartoon one I run – and want to know whether typical Internet users understand a particular piece of advice. You can ask them to read the advice (include a URL for them to visit); then ask them to tell you, in their words, what it means. Or ask them to judge five fictional situations and tell you which ones are safe. If they understood the advice, that should impact their selection. Or ask them simply whether they learnt anything new, and whether they would suggest to friends of theirs concerned with the associated issue to read the advice.

Notifying workers using boto

1)     If you don't have python, download version 2.7 of python (should be already installed on macs).

2)     Then go to boto's github repository and download the latest version (if you feel comfortable with tarballs, feel free to use those; otherwise download as zip). Unzip the file anywhere (though somewhere easy to find with command line is best).

3)     open a terminal window (windows: Command prompt), and navigate to the directory where boto is unzipped.

4)     Type "python setup.py install" (without quotes).

5)     Start python (mac: type “python” (without quotes) into a terminal window. Windows: recommend using IDLE (the python gui which should have come with your download). You can use the command prompt, but kind of a pain.)