Randomized controlled trials reporting patient-reported outcomes with no significant differences between study groups are potentially susceptible to unjustified conclusions—a systematic review

Objectives: Ceiling effect may lead to misleading conclusions when using patient-reported outcome measure (PROM) scores as an outcome. The aim of this study was to investigate the potential source of ceiling effect–related errors in randomized controlled trials (RCTs) reporting no differences in PROM scores between study groups. Study Design and Setting: A systematic review of RCTs published in the top 10 orthopedic journals according to their impact factors was conducted, focusing on studies that reported no significant differences in outcomes between two study groups. All studies published during 2012–2022 that reported no differences in PROM outcomes and used parametric statistical approach were included. The aim was to investigate the potential source of ceiling effect–related errors—that is, when the ceiling effect suppresses the possible difference between the groups. The proportions of patients exceeding the PROM scales were simulated using the observed dispersion parameters based on the assumed normal distribution, and the differences in the proportions between the study groups were subsequently analyzed. Results: After an initial screening of 2343 studies, 190 studies were included. The central 95% theoretical distribution of the scores exceeded the PROM scales in 140 (74%) of these studies. In 33 (17%) studies, the simulated patient proportions exceeding the scales indicated potential differences between the compared groups. Conclusion: It is common to have a mismatch between the chosen PROM instrument and the population being studied increasing the risk of an unjustified “no difference” conclusion due to a ceiling effect. Thus, a considerable ceiling effect should be considered a potential source of error.