Amazon’s Mechanical Turk (MTurk) platform has become popular among researchers to gather all sorts of survey data from participants. The platform has good and not so good aspects but one thing I didn’t like for sure so far: Even though there is a way to upload a CSV to accept/reject submissions in batches, I had difficulties doing the same for paying bonuses. Here is a way to get around that.
As mentioned above, there is already a way to upload a CSV file to MTurk in order to accept and reject submissions. So when you got your say 20 submissions, you can create a CSV file that looks like this and hit the “Upload” button:
HITId,AssignmentId,WorkerId,Approve,Reject WB15LX4VZCSMYZG5AE8P8O92JFUCW5,B9EXUQM2U268WERQWPE6ESBEZGSIMR,9N5ETZFIN8Y16H,X,"" WB15LX4VZCSMYZG5AE8P8O92JFUCW5,OAHG122NPCADB4YIQOYB5ERUWFUVXS,F5N4Y9MNZ098M8,"",X
And you can even do the same in another file to assign Qualifications (e.g., to help establish a pool of participants to exclusively invite to or exclude from future studies).
So far so good. But what if you would like to pay workers a bonus based on their performance? The only way I found on the platform to pay those extra amounts was to open each individual worker’s profile page, click on “Bonus Payment”, enter a reason and assignment ID and click on OK. For every single one… That’s not only inefficient and exhausting but can easily lead to error.
After some research, I stumbled upon Amazon AWS, which MTurk is technically a part of. They offer “Developers” a command-line tool with which you can perform all sorts of Amazon services related things, including all sorts of MTurk-related things, such as accepting/rejecting submissions, … and paying bonuses!
The most difficult part is setting the tool up on your machine and link your AWS account with your MTurk Requester account. Here is how to do that:
Once you’re done, you can use that tool in many ways. For example, you could just open your Terminal and enter something like:
aws mturk help
To assign a Qualification to a worker, you can type:
aws mturk associate-qualification-with-worker --worker-id --qualification-type-id 3OUT9PHWFXXUXL5GBE --no-send-notification --integer-value 50
To approve a submission:
aws mturk approve-assignment --assignment-id B9EXUQM2U268WERQWPE6ESBEZGSIMR --requester-feedback "Thank you!"
And to send a bonus:
aws mturk send-bonus --worker-id 9N5ETZFIN8Y16H --bonus-amount 1.50--assignment-id B9EXUQM2U268WERQWPE6ESBEZGSIMR --reason "Very good performance!"
As a command-line tool, this can generally be called from other applications as well, such as an R script that reads in each participant’s responses, calculates their score and then automatically assigns a Qualification (based on their score), approves their submission and sends a bonus (between $0.00 and $2.00 based on their score):
system2("aws", shQuote(c("mturk", "associate-qualification-with-worker", "--worker-id", workerId, "--qualification-type-id", qualificationId, "--no-send-notification", "--integer-value", sprintf("%d", score))))
system2("aws", shQuote(c("mturk", "approve-assignment", "--assignment-id", assignmentId, "--requester-feedback", sprintf("Thank you for your participation! You achieved a score of %d%%.", score))))
system2("aws", shQuote(c("mturk", "send-bonus", "--worker-id", workerId, "--bonus-amount", sprintf("%.2f", score/100*2.00), "--assignment-id", assignmentId, "--reason", sprintf("Performance bonus"))))
Note that in the example above, you will need to set the qualificationId (same for all submissions) as well as workerId, assignmentId and score (value between 0 and 100) based on each individual submission. You should make sure to save workerId, assignmentId (and maybe hitId) somewhere in the dataset submitted by the participant.
As accepting/rejecting, qualification assignment, and bonus payment are three completely separate calls, you can also just pick the one(s) you need. For example, if you’re not paying bonuses, you could also just use the call for accepting submissions and/or assigning a qualification. This also already helpful as well because it spares you doing that by hand on the MTurk website or generating CSVs to upload there.
Using these few lines in a loop that iterates over the submissions will automate all of the mentioned steps and ensure that your research funding and time are spent right. 🙂
