TY - JOUR
T1 - What Makes a Quality Health App-Developing a Global Research-Based Health App Quality Assessment Framework for CEN-ISO/TS 82304-2
T2 - Delphi Study
AU - Hoogendoorn, Petra
AU - Versluis, Anke
AU - van Kampen, Sanne
AU - McCay, Charles
AU - Leahy, Matt
AU - Bijlsma, Marlou
AU - Bonacina, Stefano
AU - Bonten, Tobias
AU - Bonthuis, Marie-José
AU - Butterlin, Anouk
AU - Cobbaert, Koen
AU - Duijnhoven, Thea
AU - Hallensleben, Cynthia
AU - Harrison, Stuart
AU - Hastenteufel, Mark
AU - Holappa, Terhi
AU - Kokx, Ben
AU - Morlion, Birgit
AU - Pauli, Norbert
AU - Ploeg, Frank
AU - Salmon, Mark
AU - Schnoor, Kyma
AU - Sharp, Mary
AU - Sottile, Pier Angelo
AU - Värri, Alpo
AU - Williams, Patricia
AU - Heidenreich, Georg
AU - Oughtibridge, Nicholas
AU - Stegwee, Robert
AU - Chavannes, Niels H
N1 - Funding Information:
CEN-ISO/TS 82304-2:2021 Health software – Part 2: Health and wellness apps – Quality and reliability was funded in part by the European Commission under contract number SA/CEN/GROW/EFTA/000/2018-03. We would like to thank all the experts and stakeholders for participating in the studies, Noortje Wesdorp from the Research Support Center for her support in the review and analysis of the Delphi study and follow-up survey, interns Marianne van Rheenen and Jodie Assen for their contribution in the analysis of the follow-up survey, and intern Giulia Ensing for interviewing health app manufacturers. Sadly, MB passed away unexpectedly before this paper was published. We cherish her contribution to this work and the person she was.
Publisher Copyright:
© JMIR Formative Research 2023.
PY - 2023/1/23
Y1 - 2023/1/23
N2 - BACKGROUND: The lack of an international standard for assessing and communicating health app quality and the lack of consensus about what makes a high-quality health app negatively affect the uptake of such apps. At the request of the European Commission, the international Standard Development Organizations (SDOs), European Committee for Standardization, International Organization for Standardization, and International Electrotechnical Commission have joined forces to develop a technical specification (TS) for assessing the quality and reliability of health and wellness apps.OBJECTIVE: This study aimed to create a useful, globally applicable, trustworthy, and usable framework to assess health app quality.METHODS: A 2-round Delphi technique with 83 experts from 6 continents (predominantly Europe) participating in one (n=42, 51%) or both (n=41, 49%) rounds was used to achieve consensus on a framework for assessing health app quality. Aims included identifying the maximum 100 requirement questions for the uptake of apps that do or do not qualify as medical devices. The draft assessment framework was built on 26 existing frameworks, the principles of stringent legislation, and input from 20 core experts. A follow-up survey with 28 respondents informed a scoring mechanism for the questions. After subsequent alignment with related standards, the quality assessment framework was tested and fine-tuned with manufacturers of 11 COVID-19 symptom apps. National mirror committees from the 52 countries that participated in the SDO technical committees were invited to comment on 4 working drafts and subsequently vote on the TS.RESULTS: The final quality assessment framework includes 81 questions, 67 (83%) of which impact the scores of 4 overarching quality aspects. After testing with people with low health literacy, these aspects were phrased as "Healthy and safe," "Easy to use," "Secure data," and "Robust build." The scoring mechanism enables communication of the quality assessment results in a health app quality score and label, alongside a detailed report. Unstructured interviews with stakeholders revealed that evidence and third-party assessment are needed for health app uptake. The manufacturers considered the time needed to complete the assessment and gather evidence (2-4 days) acceptable. Publication of CEN-ISO/TS 82304-2:2021 Health software - Part 2: Health and wellness apps - Quality and reliability was approved in May 2021 in a nearly unanimous vote by 34 national SDOs, including 6 of the 10 most populous countries worldwide.CONCLUSIONS: A useful and usable international standard for health app quality assessment was developed. Its quality, approval rate, and early use provide proof of its potential to become the trusted, commonly used global framework. The framework will help manufacturers enhance and efficiently demonstrate the quality of health apps, consumers, and health care professionals to make informed decisions on health apps. It will also help insurers to make reimbursement decisions on health apps.
AB - BACKGROUND: The lack of an international standard for assessing and communicating health app quality and the lack of consensus about what makes a high-quality health app negatively affect the uptake of such apps. At the request of the European Commission, the international Standard Development Organizations (SDOs), European Committee for Standardization, International Organization for Standardization, and International Electrotechnical Commission have joined forces to develop a technical specification (TS) for assessing the quality and reliability of health and wellness apps.OBJECTIVE: This study aimed to create a useful, globally applicable, trustworthy, and usable framework to assess health app quality.METHODS: A 2-round Delphi technique with 83 experts from 6 continents (predominantly Europe) participating in one (n=42, 51%) or both (n=41, 49%) rounds was used to achieve consensus on a framework for assessing health app quality. Aims included identifying the maximum 100 requirement questions for the uptake of apps that do or do not qualify as medical devices. The draft assessment framework was built on 26 existing frameworks, the principles of stringent legislation, and input from 20 core experts. A follow-up survey with 28 respondents informed a scoring mechanism for the questions. After subsequent alignment with related standards, the quality assessment framework was tested and fine-tuned with manufacturers of 11 COVID-19 symptom apps. National mirror committees from the 52 countries that participated in the SDO technical committees were invited to comment on 4 working drafts and subsequently vote on the TS.RESULTS: The final quality assessment framework includes 81 questions, 67 (83%) of which impact the scores of 4 overarching quality aspects. After testing with people with low health literacy, these aspects were phrased as "Healthy and safe," "Easy to use," "Secure data," and "Robust build." The scoring mechanism enables communication of the quality assessment results in a health app quality score and label, alongside a detailed report. Unstructured interviews with stakeholders revealed that evidence and third-party assessment are needed for health app uptake. The manufacturers considered the time needed to complete the assessment and gather evidence (2-4 days) acceptable. Publication of CEN-ISO/TS 82304-2:2021 Health software - Part 2: Health and wellness apps - Quality and reliability was approved in May 2021 in a nearly unanimous vote by 34 national SDOs, including 6 of the 10 most populous countries worldwide.CONCLUSIONS: A useful and usable international standard for health app quality assessment was developed. Its quality, approval rate, and early use provide proof of its potential to become the trusted, commonly used global framework. The framework will help manufacturers enhance and efficiently demonstrate the quality of health apps, consumers, and health care professionals to make informed decisions on health apps. It will also help insurers to make reimbursement decisions on health apps.
U2 - 10.2196/43905
DO - 10.2196/43905
M3 - Article
C2 - 36538379
SN - 2561-326X
VL - 7
JO - JMIR Formative Research
JF - JMIR Formative Research
M1 - e43905
ER -