Realistic Precision and Accuracy of Online Experiment Platforms, Web Browsers, and Devices

Overview

Journal Behav Res Methods

Publisher Springer

Specialty Social Sciences

Date 2020 Nov 3

PMID 33140376

Citations 99

Authors

Alexander Anwyl-Irvine

Edwin S Dalmaijer

Nick Hodges

Jo K Evershed

Affiliations

Soon will be listed here.

Abstract

Due to increasing ease of use and ability to quickly collect large samples, online behavioural research is currently booming. With this popularity, it is important that researchers are aware of who online participants are, and what devices and software they use to access experiments. While it is somewhat obvious that these factors can impact data quality, the magnitude of the problem remains unclear. To understand how these characteristics impact experiment presentation and data quality, we performed a battery of automated tests on a number of realistic set-ups. We investigated how different web-building platforms (Gorilla v.20190828, jsPsych v6.0.5, Lab.js v19.1.0, and psychoJS/PsychoPy3 v3.1.5), browsers (Chrome, Edge, Firefox, and Safari), and operating systems (macOS and Windows 10) impact display time across 30 different frame durations for each software combination. We then employed a robot actuator in realistic set-ups to measure response recording across the aforementioned platforms, and between different keyboard types (desktop and integrated laptop). Finally, we analysed data from over 200,000 participants on their demographics, technology, and software to provide context to our findings. We found that modern web platforms provide reasonable accuracy and precision for display duration and manual response time, and that no single platform stands out as the best in all features and conditions. In addition, our online participant analysis shows what equipment they are likely to use.

Citing Articles

Psychological, psychiatric, and behavioral sciences measurement scales: best practice guidelines for their development and validation.

Stefana A, Damiani S, Granziol U, Provenzani U, Solmi M, Youngstrom E Front Psychol. 2025; 15:1494261.

PMID: 39916786 PMC: 11798685. DOI: 10.3389/fpsyg.2024.1494261.

Usability and optimization of online apps in user's context.

Iqbal M, Shinan K, Shahid Rafique S, Alourani A, Ashraf M, Rahim N PeerJ Comput Sci. 2025; 10:e2561.

PMID: 39896397 PMC: 11784876. DOI: 10.7717/peerj-cs.2561.

Words before pictures: the role of language in biasing visual attention.

Calignano G, Lorenzoni A, Semeraro G, Navarrete E Front Psychol. 2025; 15:1439397.

PMID: 39744025 PMC: 11688633. DOI: 10.3389/fpsyg.2024.1439397.

Individual differences in online research: Comparing lab-based and online administration of a psycholinguistic battery of linguistic and domain-general skills.

McConnell K, Hintz F, Meyer A Behav Res Methods. 2024; 57(1):22.

PMID: 39702511 PMC: 11659378. DOI: 10.3758/s13428-024-02533-x.

Input device matters for measures of behaviour in online experiments.

Warburton M, Campagnoli C, Mon-Williams M, Mushtaq F, Morehead J Psychol Res. 2024; 89(1):29.

PMID: 39607544 PMC: 11604694. DOI: 10.1007/s00426-024-02065-1.

References

Barnhoorn J, Haasnoot E, Bocanegra B, van Steenbergen H . QRTEngine: An easy solution for running online reaction time experiments using Qualtrics. Behav Res Methods. 2014; 47(4):918-929. PMC: 4636512. DOI: 10.3758/s13428-014-0530-7. View

Bridges D, Pitiot A, MacAskill M, Peirce J . The timing mega-study: comparing a range of experiment generators, both lab-based and online. PeerJ. 2020; 8:e9414. PMC: 7512138. DOI: 10.7717/peerj.9414. View

Bohannon J . PSYCHOLOGY. Mechanical Turk upends social sciences. Science. 2016; 352(6291):1263-4. DOI: 10.1126/science.352.6291.1263. View

Button K, Ioannidis J, Mokrysz C, Nosek B, Flint J, Robinson E . Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013; 14(5):365-76. DOI: 10.1038/nrn3475. View

Reips U, Stieger S . Scientific LogAnalyzer: a web-based tool for analyses of server log files in psychological research. Behav Res Methods Instrum Comput. 2004; 36(2):304-11. DOI: 10.3758/bf03195576. View

Hecht H, Oesker M, Kaiser A, Civelek H, Stecker T . A perception experiment with time-critical graphics animation on the World-Wide Web. Behav Res Methods Instrum Comput. 1999; 31(3):439-45. DOI: 10.3758/bf03200724. View

Damian M . Does variability in human performance outweigh imprecision in response devices such as computer keyboards?. Behav Res Methods. 2010; 42(1):205-11. DOI: 10.3758/BRM.42.1.205. View

de Leeuw J . jsPsych: a JavaScript library for creating behavioral experiments in a Web browser. Behav Res Methods. 2014; 47(1):1-12. DOI: 10.3758/s13428-014-0458-y. View

Reips U . The Web Experimental Psychology Lab: five years of data collection on the Internet. Behav Res Methods Instrum Comput. 2001; 33(2):201-11. DOI: 10.3758/bf03195366. View

Furmanski C, Engel S . Perceptual learning in object recognition: object specificity and size invariance. Vision Res. 2000; 40(5):473-84. DOI: 10.1016/s0042-6989(99)00134-0. View

Garaizar P, Reips U . Best practices: Two Web-browser-based methods for stimulus presentation in behavioral experiments with high-resolution timing requirements. Behav Res Methods. 2018; 51(3):1441-1453. DOI: 10.3758/s13428-018-1126-4. View

Garaizar P, Vadillo M, Lopez-de-Ipina D, Matute H . Measuring software timing errors in the presentation of visual stimuli in cognitive neuroscience experiments. PLoS One. 2014; 9(1):e85108. PMC: 3883681. DOI: 10.1371/journal.pone.0085108. View

Mathot S, Schreij D, Theeuwes J . OpenSesame: an open-source, graphical experiment builder for the social sciences. Behav Res Methods. 2011; 44(2):314-24. PMC: 3356517. DOI: 10.3758/s13428-011-0168-7. View

10.

McGraw K, Tew M, Williams J . The integrity of Web-delivered experiments: can you trust the data?. Psychol Sci. 2001; 11(6):502-6. DOI: 10.1111/1467-9280.00296. View

11.

Neath I, Earle A, Hallett D, Surprenant A . Response time accuracy in Apple Macintosh computers. Behav Res Methods. 2011; 43(2):353-62. DOI: 10.3758/s13428-011-0069-9. View

12.

Plant R . A reminder on millisecond timing accuracy and potential replication failure in computer-based psychology experiments: An open letter. Behav Res Methods. 2015; 48(1):408-11. DOI: 10.3758/s13428-015-0577-0. View

12.

Pronk T, Wiers R, Molenkamp B, Murre J . Mental chronometry in the pocket? Timing accuracy of web applications on touchscreen and keyboard devices. Behav Res Methods. 2019; 52(3):1371-1382. PMC: 7280355. DOI: 10.3758/s13428-019-01321-2. View

13.

Reimers S, Stewart N . Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behav Res Methods. 2014; 47(2):309-27. PMC: 4427652. DOI: 10.3758/s13428-014-0471-1. View

14.

Reimers S, Stewart N . Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments. Behav Res Methods. 2016; 48(3):897-908. PMC: 5003904. DOI: 10.3758/s13428-016-0758-5. View

15.

Tripathy S, Cavanagh P . The extent of crowding in peripheral vision does not scale with target size. Vision Res. 2002; 42(20):2357-69. DOI: 10.1016/s0042-6989(02)00197-9. View