"Conducting Behavioral Studies on Crowdsourcing Platforms"

Dr. Philip Chapkovsky held a scientific seminar on the experience of behavioral research on the Toloka crowdsourcing platform

Description:

COVID-19 pandemic put a severe strain on the conventional offline human subjects studies, forcing behavioral labs across the world to transition to online interactive experiments. The ‘traditional’ platforms used for this purpose such as Amazon Mechanical Turk or Prolific became flooded with researchers' requests which resulted in deteriorating data quality. Moreover, these platforms have limited geographical coverage (the US for MTurk, the UK for Prolific) which also limits the generality of findings. All these factors created the demand for alternatives, and in this talk, we report our experience on running behavioral studies on Toloka crowdsourcing platform. We demonstrate how to run both asynchronous studies without real-time interactions between participants and synchronous, interactive studies where the decisions of a participant’s partner are delivered to them in real-time. We discuss several major challenges researchers face with conducting interactive studies online such as controlling for attention and drop-offs.