Pay for Performance, Satisfaction and Retention in Longitudinal Crowdsourced Research

Overview

Journal PLoS One

Specialties General Medicine
Science

Date 2021 Jan 20

PMID 33471835

Citations 8

Authors

Elena M Auer

Tara S Behrend

Andrew B Collmus

Richard N Landers

Ahleah F Miles

Affiliations

Soon will be listed here.

Abstract

In the social and cognitive sciences, crowdsourcing provides up to half of all research participants. Despite this popularity, researchers typically do not conceptualize participants accurately, as gig-economy worker-participants. Applying theories of employee motivation and the psychological contract between employees and employers, we hypothesized that pay and pay raises would drive worker-participant satisfaction, performance, and retention in a longitudinal study. In an experiment hiring 359 Amazon Mechanical Turk Workers, we found that initial pay, relative increase of pay over time, and overall pay did not have substantial influence on subsequent performance. However, pay significantly predicted participants' perceived choice, justice perceptions, and attrition. Given this, we conclude that worker-participants are particularly vulnerable to exploitation, having relatively low power to negotiate pay. Results of this study suggest that researchers wishing to crowdsource research participants using MTurk might not face practical dangers such as decreased performance as a result of lower pay, but they must recognize an ethical obligation to treat Workers fairly.

Citing Articles

Work Environment and Depressive Symptoms of Webtoon Writers.

Lee J, Min J, Lee Y, Park M, Kim H Saf Health Work. 2024; 15(2):172-180.

PMID: 39035797 PMC: 11255920. DOI: 10.1016/j.shaw.2024.03.003.

Effects of pay rate and instructions on attrition in crowdsourcing research.

Ritchey C, Jimenez-Gomez C, Podlesnik C PLoS One. 2023; 18(10):e0292372.

PMID: 37792848 PMC: 10550147. DOI: 10.1371/journal.pone.0292372.

Effects of reduced nicotine content cigarette advertising with warning labels and social media features on product perceptions among young adults.

Johnson A, Mercincavage M, Tan A, Villanti A, Delnevo C, Strasser A J Behav Med. 2023; 46(6):948-959.

PMID: 37605036 PMC: 10591832. DOI: 10.1007/s10865-023-00441-7.

Assessing Attention to Tobacco Warnings With a Heatmapping Task.

Johnson A, Mercincavage M, Souprountchouk V, DeAtley T, Mays D, Strasser A Am J Prev Med. 2023; 65(5):809-817.

PMID: 37257762 PMC: 10592484. DOI: 10.1016/j.amepre.2023.05.020.

Using Reddit as a recruitment strategy for addiction science research.

Goodwin S, Dwyer M, Caliva S, Burrows C, Raiff B J Subst Use Addict Treat. 2023; 148:209011.

PMID: 36924845 PMC: 11366419. DOI: 10.1016/j.josat.2023.209011.

References

Lucas R, Clark A, Georgellis Y, Diener E . Reexamining adaptation and the set point model of happiness: reactions to changes in marital status. J Pers Soc Psychol. 2003; 84(3):527-39. DOI: 10.1037//0022-3514.84.3.527. View

WATSON D, Clark L, Tellegen A . Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol. 1988; 54(6):1063-70. DOI: 10.1037//0022-3514.54.6.1063. View

Stewart N, Chandler J, Paolacci G . Crowdsourcing Samples in Cognitive Science. Trends Cogn Sci. 2017; 21(10):736-748. DOI: 10.1016/j.tics.2017.06.007. View

Meade A, Craig S . Identifying careless responses in survey data. Psychol Methods. 2012; 17(3):437-55. DOI: 10.1037/a0028085. View

Dabos G, Rousseau D . Mutuality and reciprocity in the psychological contracts of employees and employers. J Appl Psychol. 2004; 89(1):52-72. DOI: 10.1037/0021-9010.89.1.52. View

Litman L, Robinson J, Rosenzweig C . The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behav Res Methods. 2014; 47(2):519-28. DOI: 10.3758/s13428-014-0483-x. View

. The Belmont Report. Ethical principles and guidelines for the protection of human subjects of research. J Am Coll Dent. 2015; 81(3):4-13. View

Donnellan M, Oswald F, Baird B, Lucas R . The mini-IPIP scales: tiny-yet-effective measures of the Big Five factors of personality. Psychol Assess. 2006; 18(2):192-203. DOI: 10.1037/1040-3590.18.2.192. View

Tankersley D, Stowe C, Huettel S . Altruism is associated with an increased neural response to agency. Nat Neurosci. 2007; 10(2):150-1. DOI: 10.1038/nn1833. View

10.

DeShon R, Gillespie J . A motivated action theory account of goal orientation. J Appl Psychol. 2005; 90(6):1096-127. DOI: 10.1037/0021-9010.90.6.1096. View

11.

Bruine de Bruin W, Parker A, Fischhoff B . Individual differences in adult decision-making competence. J Pers Soc Psychol. 2007; 92(5):938-956. DOI: 10.1037/0022-3514.92.5.938. View

12.

HELSON H . Adaptation-level as frame of reference for prediction of psychophysical data. Am J Psychol. 2010; 60(1):1-29. View

13.

Latham G, Pinder C . Work motivation theory and research at the dawn of the twenty-first century. Annu Rev Psychol. 2005; 56:485-516. DOI: 10.1146/annurev.psych.55.090902.142105. View

14.

Devine E, Waters M, Putnam M, Surprise C, OMalley K, Richambault C . Concealment and fabrication by experienced research subjects. Clin Trials. 2013; 10(6):935-48. DOI: 10.1177/1740774513492917. View

15.

Behrend T, Sharek D, Meade A, Wiebe E . The viability of crowdsourcing for survey research. Behav Res Methods. 2011; 43(3):800-13. DOI: 10.3758/s13428-011-0081-0. View

16.

Colquitt J . On the dimensionality of organizational justice: a construct validation of a measure. J Appl Psychol. 2001; 86(3):386-400. DOI: 10.1037/0021-9010.86.3.386. View

17.

Buhrmester M, Kwang T, Gosling S . Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?. Perspect Psychol Sci. 2015; 6(1):3-5. DOI: 10.1177/1745691610393980. View