Overcoming Challenges to Evidence-Based Policy Development in a Large, Integrated Delivery System

Overview

Journal Health Serv Res

Specialty Health Services

Date 2018 Jun 5

PMID 29862494

Citations 19

Authors

Austin B Frakt

Julia C Prentice

Steven D Pizer

A Rani Elwy

Melissa M Garrido

Amy M Kilbourne

David Atkins

Affiliations

Soon will be listed here.

Abstract

Objective: To describe a new Veterans Health Administration (VHA) program to foster the learning health system paradigm by rigorously evaluating health care initiatives and to report key lessons learned in designing those evaluations.

Principal Findings: The VHA's Quality Enhancement Research Initiative and its Health Services Research and Development Service are cooperating on several large, randomized program evaluations aimed at improving the care veterans receive and the efficiency with which it is delivered. The evaluations we describe involve collaborative design, outcomes assessment, and implementation science through partnerships between VHA operations and researchers. We review key factors to assess before committing to an evaluation. In addition to traditional design issues (such as ensuring adequate power and availability of data), these include others that are easily overlooked: the stability of intervention financing, means of controlling and commitment to adhering to randomized roll-out, degree of buy-in from key implementation staff, and feasibility of managing multiple veto points for interventions that span several programs, among others.

Conclusions: Successful program implementation and rigorous evaluation require resources, specialized expertise, and careful planning. If the learning health system model is to be sustained, organizations will need dedicated programs to prioritize resources and continuously adapt evaluation designs.

Citing Articles

The foundational science of learning health systems.

Kilbourne A, Borsky A, OBrien R, Braganza M, Garrido M Health Serv Res. 2024; 59(6):e14374.

PMID: 39165034 PMC: 11622276. DOI: 10.1111/1475-6773.14374.

Telehealth Research and Innovation for Veterans with Cancer: the THRIVE Center.

Zullig L, Makarov D, Becker D, Dardashti N, Guzman I, Kelley M J Natl Cancer Inst Monogr. 2024; 2024(64):70-75.

PMID: 38924789 PMC: 11207852. DOI: 10.1093/jncimonographs/lgae019.

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022.

Ashcraft L, Goodrich D, Hero J, Phares A, Bachrach R, Quinn D Implement Sci. 2024; 19(1):43.

PMID: 38915102 PMC: 11194895. DOI: 10.1186/s13012-024-01369-5.

Diffusion of excellence: evaluating a system to identify, replicate, and spread promising innovative practices across the Veterans health administration.

Jackson G, Fix G, White B, Cutrona S, Reardon C, Damschroder L Front Health Serv. 2024; 3:1223277.

PMID: 38420338 PMC: 10900518. DOI: 10.3389/frhs.2023.1223277.

Adaptive Designs in Implementation Science and Practice: Their Promise and the Need for Greater Understanding and Improved Communication.

Kilbourne A, Chinman M, Rogal S, Almirall D Annu Rev Public Health. 2023; 45(1):69-88.

PMID: 37931183 PMC: 11070446. DOI: 10.1146/annurev-publhealth-060222-014438.

References

Manning W, NEWHOUSE J, Duan N, Keeler E, Leibowitz A, Marquis M . Health insurance and the demand for medical care: evidence from a randomized experiment. Am Econ Rev. 1987; 77(3):251-77. View

Mor V, Volandes A, Gutman R, Gatsonis C, Mitchell S . PRagmatic trial Of Video Education in Nursing homes: The design and rationale for a pragmatic cluster randomized trial in the nursing home setting. Clin Trials. 2017; 14(2):140-151. PMC: 5376219. DOI: 10.1177/1740774516685298. View

Kaye H . Gradual rebalancing of Medicaid long-term services and supports saves money and serves more people, statistical model shows. Health Aff (Millwood). 2012; 31(6):1195-203. DOI: 10.1377/hlthaff.2011.1237. View

Done N, Oh D, Weinstock M, Whited J, Jackson G, King H . study: protocol for a stepped-wedge cluster randomised trial to compare access to care for a mobile app versus a workstation-based store-and-forward teledermatology process. BMJ Open. 2018; 8(12):e022218. PMC: 6303588. DOI: 10.1136/bmjopen-2018-022218. View

Bothwell L, Greene J, Podolsky S, Jones D . Assessing the Gold Standard--Lessons from the History of RCTs. N Engl J Med. 2016; 374(22):2175-81. DOI: 10.1056/NEJMms1604593. View

Johnson K, Neta G, Dember L, Coronado G, Suls J, Chambers D . Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016; 17:32. PMC: 4715340. DOI: 10.1186/s13063-016-1158-y. View

Atkins D, Kilbourne A, Shulkin D . Moving From Discovery to System-Wide Change: The Role of Research in a Learning Health Care System: Experience from Three Decades of Health Systems Research in the Veterans Health Administration. Annu Rev Public Health. 2017; 38:467-487. DOI: 10.1146/annurev-publhealth-031816-044255. View

Simon G, Beck A, Rossom R, Richards J, Kirlin B, King D . Population-based outreach versus care as usual to prevent suicide attempt: study protocol for a randomized controlled trial. Trials. 2016; 17(1):452. PMC: 5025595. DOI: 10.1186/s13063-016-1566-z. View

Etheredge L . A rapid-learning health system. Health Aff (Millwood). 2007; 26(2):w107-18. DOI: 10.1377/hlthaff.26.2.w107. View

10.

Kilbourne A, Elwy A, Sales A, Atkins D . Accelerating Research Impact in a Learning Health Care System: VA's Quality Enhancement Research Initiative in the Choice Act Era. Med Care. 2016; 55 Suppl 7 Suppl 1:S4-S12. PMC: 5472006. DOI: 10.1097/MLR.0000000000000683. View

11.

Ivers N, Halperin I, Barnsley J, Grimshaw J, Shah B, Tu K . Allocation techniques for balance at baseline in cluster randomized trials: a methodological review. Trials. 2012; 13:120. PMC: 3503622. DOI: 10.1186/1745-6215-13-120. View

12.

Oliva E, Bowe T, Tavakoli S, Martins S, Lewis E, Paik M . Development and applications of the Veterans Health Administration's Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide. Psychol Serv. 2017; 14(1):34-49. DOI: 10.1037/ser0000099. View

13.

Newhouse J, Normand S . Health Policy Trials. N Engl J Med. 2017; 376(22):2160-2167. DOI: 10.1056/NEJMra1602774. View

14.

Garrido M, Allman R, Pizer S, Rudolph J, Thomas K, Sperber N . Innovation in a Learning Health Care System: Veteran-Directed Home- and Community-Based Services. J Am Geriatr Soc. 2017; 65(11):2446-2451. PMC: 6317362. DOI: 10.1111/jgs.15053. View

15.

Baicker K, Taubman S, Allen H, Bernstein M, Gruber J, Newhouse J . The Oregon experiment--effects of Medicaid on clinical outcomes. N Engl J Med. 2013; 368(18):1713-22. PMC: 3701298. DOI: 10.1056/NEJMsa1212321. View

16.

Choudhry N . Randomized, Controlled Trials in Health Insurance Systems. N Engl J Med. 2017; 377(10):957-964. DOI: 10.1056/NEJMra1510058. View

17.

Mdege N, Man M, Taylor nee Brown C, Torgerson D . Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. J Clin Epidemiol. 2011; 64(9):936-48. DOI: 10.1016/j.jclinepi.2010.12.003. View

18.

Kilbourne A, Abraham K, Goodrich D, Bowersox N, Almirall D, Lai Z . Cluster randomized adaptive implementation trial comparing a standard versus enhanced implementation intervention to improve uptake of an effective re-engagement program for patients with serious mental illness. Implement Sci. 2013; 8:136. PMC: 3874628. DOI: 10.1186/1748-5908-8-136. View

19.

Milner A, Carter G, Pirkis J, Robinson J, Spittal M . Letters, green cards, telephone calls and postcards: systematic and meta-analytic review of brief contact interventions for reducing self-harm, suicide attempts and suicide. Br J Psychiatry. 2015; 206(3):184-90. DOI: 10.1192/bjp.bp.114.147819. View

20.

McCarthy J, Bossarte R, Katz I, Thompson C, Kemp J, Hannemann C . Predictive Modeling and Concentration of the Risk of Suicide: Implications for Preventive Interventions in the US Department of Veterans Affairs. Am J Public Health. 2015; 105(9):1935-42. PMC: 4539821. DOI: 10.2105/AJPH.2015.302737. View