Thanks for doing our HITs! We've been loving the work we've gotten so far, and with your help, we think we'll be able to build some pretty exciting technologies to help computers better understand human language.
Can I have more example?
Here are some sentences from our pilot study that we liked.
Will you reject all of my work?
No. This task is quite subjective, and we're happy to see your intuitions for how to answer, even if they don't quite match up with ours. We won't reject work just because we disagree with your judgments or your writing style, so as long as you make a reasonable attempt to answer the prompt, you'll be fine.
Do you have any preferences for how we construct sentences?
You are free to construct sentence in any way you see fit. However, please avoid systematic and repetitive answers. We want your writing to be varied, so don't stick to a single recipe to answer the questions. We can't use highly repetitive data.
For the 'definitely correct' prompt, can I just copy the source caption?
No. You can write something similar (see the examples), but we don't want copy-and-paste answers.
For the 'definitely incorrect' prompt, can I write something completely unrelated?
Yes, but we'd rather you don't. For example, if the caption is "Two kids play in a park.", "A car driving down a dirt road." would be an acceptable answer for definitely incorrect, since it can't describe the same event. However, please try to keep your responses at least somewhat related to the original prompt, and don't do this for every example.
Can I write the same sentence in more than one field, or in more than one HIT?
No. If you accidentally reuse a sentence a couple of times in the course of doing many HITs, that's fine, but try to make every sentence original.
Some of the prompts are really long. Do my responses have to be that long?
Long responses are especially useful for us, but your responses don't have to be as long as the prompts. Though, if the prompt gives you lots of information, please avoid extremely short sentences like "dog runs."
Is there any limit to how many of these HITs I can do?
Nope. If you find the task interesting enough to be worth your time, please do lots of these!
When do you approve HITs?
We'll try to approve as early as we can, but we can't promise to look at the data directly any more often than once a day.
(For HITs with bonus) When do you send bonuses?
Since there is another validation phase, we can't send bonuses right away, but we promise to send them as soon as possible. We expect to send bonuses at maximum one week after all the HITs are completed.
Would you prefer that we use words in the original sentence or synonyms for those words in our own sentences?
Synonyms and repeated words are both ok. A mixture of both would be ideal, but we will accept either.
Are we able to expand the prompt based on our personal knowledge? For example, if we know the prompt relates to 9-11, and 9-11 happened in the morning, are we able to include information like that in our sentences even if it is not given in the original sentence?
We'd prefer if you didn't. It'd be best for us if you write sentences based only on the prompt and beliefs or assumptions about what generally happens in the world. Even if you happen to recognize a person or event, please don't use any outside knowledge you have about that person or event to write the sentence.
Do I have to write complete sentences?
If the prompt is a complete sentence, please write a complete sentence. However, if the prompt is just a short fragment, you don't have to. So if the prompt is "A dog", you can just write something like "A mammal" if you don't want to say more.
When should I fill out the 'problems' field?
You should fill this out if you can't complete the HIT, and not otherwise. This should be either if the prompt is total nonsense (you can't even guess what it's describing), or if HIT interface is broken (no prompt, for example). If there is a typo in the prompt, but you think you know what it means anyway, please don't report it. Never put anything in this field if there isn't a problem.
I accidentally submitted a HIT before I finished what I was writing. Will you reject it?
It happens. If you do this one or two times, and you're submitting other HITs in the same day (or the same batch), we will accept all of them. Don't email us or report a problem unless either you wound up doing this more than twice or there is some problem with the HIT interface that is making it hard to submit complete HITs.
Who are you?
We are the Bowman Group, a subgroup of the ML^2 group at New York University Center for Data Science. We are also affiliated with the NYU Departments of Computer Science and Linguistics.
I have more questions!
Email us through MTurk!