A Scale to Assess the Methodological Quality of Studies Assessing Usability of Electronic Health Products and Services: Delphi Study Followed by Validity and Reliability Testing

Overview

Journal J Med Internet Res

Publisher JMIR Publications

Specialty Medical Informatics

Date 2019 Nov 16

PMID 31730036

Citations 8

Authors

Anabela G Silva

Patricia Simoes

Rita Santos

Alexandra Queiros

Nelson P Rocha

Mario Rodrigues

Affiliations

Soon will be listed here.

Abstract

Background: The usability of electronic health (eHealth) and mobile health apps is of paramount importance as it impacts the quality of care. Methodological quality assessment is a common practice in the field of health for different designs and types of studies. However, we were unable to find a scale to assess the methodological quality of studies on the usability of eHealth products or services.

Objective: This study aimed to develop a scale to assess the methodological quality of studies assessing usability of mobile apps and to perform a preliminary analysis of of the scale's feasibility, reliability, and construct validity on studies assessing usability of mobile apps, measuring aspects of physical activity.

Methods: A 3-round Delphi panel was used to generate a pool of items considered important when assessing the quality of studies on the usability of mobile apps. These items were used to write the scale and the guide to assist its use. The scale was then used to assess the quality of studies on usability of mobile apps for physical activity, and it assessed in terms of feasibility, interrater reliability, and construct validity.

Results: A total of 25 experts participated in the Delphi panel, and a 15-item scale was developed. This scale was shown to be feasible (time of application mean 13.10 [SD 2.59] min), reliable (intraclass correlation coefficient=0.81; 95% CI 0.55-0.93), and able to discriminate between low- and high-quality studies (high quality: mean 9.22 [SD 0.36]; low quality: mean 6.86 [SD 0.80]; P=.01).

Conclusions: The scale that was developed can be used both to assess the methodological quality of usability studies and to inform its planning.

Citing Articles

Software evaluation on infant development to support teaching and professional training.

Melo W, Braga H, Cardoso M, Melo E, Monteiro F Rev Lat Am Enfermagem. 2024; 32:e4284.

PMID: 39166627 PMC: 11335069. DOI: 10.1590/1518-8345.7248.4284.

Acceptance and Privacy Perceptions Toward Video-based Active and Assisted Living Technologies: Scoping Review.

Mujirishvili T, Maidhof C, Florez-Revuelta F, Ziefle M, Richart-Martinez M, Cabrero-Garcia J J Med Internet Res. 2023; 25:e45297.

PMID: 37126390 PMC: 10186188. DOI: 10.2196/45297.

A roadmap for the development and evaluation of the eHealthResp online course.

Estrela M, Silva T, Almeida A, Regueira C, Zapata-Cachafeiro M, Figueiras A Digit Health. 2022; 8:20552076221089088.

PMID: 35360007 PMC: 8961349. DOI: 10.1177/20552076221089088.

Methodological Quality of User-Centered Usability Evaluation of Ambient Assisted Living Solutions: A Systematic Literature Review.

Bastardo R, Martins A, Pavao J, Goncalves Silva A, Rocha N Int J Environ Res Public Health. 2021; 18(21).

PMID: 34770022 PMC: 8582689. DOI: 10.3390/ijerph182111507.

A Provider-Facing eHealth Tool for Transitioning Youth With Special Health Care Needs From Pediatric to Adult Care: Mixed Methods, User-Engaged Usability Study.

McMaughan D, Lin S, Ozmetin J, Beverly J, Brog J, Naiser E JMIR Form Res. 2021; 5(5):e22915.

PMID: 34032579 PMC: 8188313. DOI: 10.2196/22915.

References

Brink Y, Louw Q . Clinical instruments: reliability and validity critical appraisal. J Eval Clin Pract. 2011; 18(6):1126-32. DOI: 10.1111/j.1365-2753.2011.01707.x. View

Sannino G, Forastiere M, De Pietro G . A Wellness Mobile Application for Smart Health: Pilot Study Design and Results. Sensors (Basel). 2017; 17(3). PMC: 5375897. DOI: 10.3390/s17030611. View

Choo S, Kim J, Jung S, Kim S, Kim J, Han J . Development of a Weight Loss Mobile App Linked With an Accelerometer for Use in the Clinic: Usability, Acceptability, and Early Testing of its Impact on the Patient-Doctor Relationship. JMIR Mhealth Uhealth. 2016; 4(1):e24. PMC: 4832121. DOI: 10.2196/mhealth.4546. View

Soto-Perez-de-Celis E, Kim H, Rojo-Castillo M, Sun C, Chavarri-Guerra Y, Navarrete-Reyes A . A pilot study of an accelerometer-equipped smartphone to monitor older adults with cancer receiving chemotherapy in Mexico. J Geriatr Oncol. 2017; 9(2):145-151. DOI: 10.1016/j.jgo.2017.09.008. View

Spook J, Paulussen T, Kok G, van Empelen P . Monitoring dietary intake and physical activity electronically: feasibility, usability, and ecological validity of a mobile-based Ecological Momentary Assessment tool. J Med Internet Res. 2013; 15(9):e214. PMC: 3785990. DOI: 10.2196/jmir.2617. View

Naslund J, Aschbrenner K, Bartels S . Wearable Devices and Smartphones for Activity Tracking Among People with Serious Mental Illness. Ment Health Phys Act. 2016; 10:10-17. PMC: 4845759. DOI: 10.1016/j.mhpa.2016.02.001. View

Klaassen B, van Beijnum B, Hermens H . Usability in telemedicine systems-A literature survey. Int J Med Inform. 2016; 93:57-69. DOI: 10.1016/j.ijmedinf.2016.06.004. View

Vorrink S, Kort H, Troosters T, Lammers J . A Mobile Phone App to Stimulate Daily Physical Activity in Patients with Chronic Obstructive Pulmonary Disease: Development, Feasibility, and Pilot Studies. JMIR Mhealth Uhealth. 2016; 4(1):e11. PMC: 4748139. DOI: 10.2196/mhealth.4741. View

Ellsworth M, Dziadzko M, OHoro J, Farrell A, Zhang J, Herasevich V . An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc. 2016; 24(1):218-226. PMC: 7654077. DOI: 10.1093/jamia/ocw046. View

10.

Lewis S, Warlow C . How to spot bias and other potential problems in randomised controlled trials. J Neurol Neurosurg Psychiatry. 2004; 75(2):181-7. PMC: 1738894. DOI: 10.1136/jnnp.2003.025833. View

11.

Sousa V, Lopez K . Towards Usable E-Health. A Systematic Review of Usability Questionnaires. Appl Clin Inform. 2017; 8(2):470-490. PMC: 6241759. DOI: 10.4338/ACI-2016-10-R-0170. View

12.

Whiting P, Rutjes A, Reitsma J, Bossuyt P, Kleijnen J . The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003; 3:25. PMC: 305345. DOI: 10.1186/1471-2288-3-25. View

13.

Williamson P, Altman D, Blazeby J, Clarke M, Devane D, Gargon E . Developing core outcome sets for clinical trials: issues to consider. Trials. 2012; 13:132. PMC: 3472231. DOI: 10.1186/1745-6215-13-132. View

14.

Guyatt G, Oxman A, Kunz R, Atkins D, Brozek J, Vist G . GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol. 2011; 64(4):395-400. DOI: 10.1016/j.jclinepi.2010.09.012. View

15.

Moja L, Telaro E, DAmico R, Moschetti I, Coe L, Liberati A . Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ. 2005; 330(7499):1053. PMC: 557223. DOI: 10.1136/bmj.38414.515938.8F. View

16.

Hong Y, Goldberg D, Vollmer Dahlke D, Ory M, Cargill J, Coughlin R . Testing Usability and Acceptability of a Web Application to Promote Physical Activity (iCanFit) Among Older Adults. JMIR Hum Factors. 2016; 1(1):e2. PMC: 4797672. DOI: 10.2196/humanfactors.3787. View

17.

Cruz Zapata B, Fernandez-Aleman J, Idri A, Toval A . Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst. 2015; 39(2):1. DOI: 10.1007/s10916-014-0182-2. View

18.

Stang A . Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010; 25(9):603-5. DOI: 10.1007/s10654-010-9491-z. View

19.

Jadad A, Moore R, Carroll D, Jenkinson C, Reynolds D, Gavaghan D . Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996; 17(1):1-12. DOI: 10.1016/0197-2456(95)00134-4. View

20.

Al Ayubi S, Parmanto B, Branch R, Ding D . A Persuasive and Social mHealth Application for Physical Activity: A Usability and Feasibility Study. JMIR Mhealth Uhealth. 2014; 2(2):e25. PMC: 4114463. DOI: 10.2196/mhealth.2902. View