Skip to main content

Can a registered trial be reported as a one-group, pretest-posttest study with no explanation? A critique of Williams et al. (2021)

Williams et al. (2021) assessed the effects of a revised version of the Life Skills Training (LST) program in a juvenile justice diversionary setting. They concluded: “The results indicate that the intervention was effective for a variety of key outcomes” and that the program “can help overcome some of the shortcomings of current youth court programming in a way that is feasible and effective” (Williams et al., 2021, page 9).

There are a number of reasons why the interpretation of the results presented by Williams et al. (2021) in terms of program effectiveness, and their possible impact on youth in the juvenile justice system, are invalid. First, Williams et al. (2021) state “the study design did not include a control or comparison group” (page 10), yet it began as a randomized trial with a control group, as evident from a protocol registered in ClinicalTrials.gov (Williams, 2016). The registry shows the control group was removed from the protocol on January 3, 2019, five months after the study ended. It would be useful to know if the control group data had actually been collected and if the change in study design occurred before or after these data were analyzed. BMC requires registration of all trials and that registration number and date be included as the last line of the manuscript abstract (BMC, 2021). This information does not appear in Williams et al. (2021).

Second, the specific knowledge, attitudes and skills outcomes reported in Williams et al. (2021) are not those described in the registration. The registered primary outcomes were Substance Use Behavior (frequency of use of 13 drugs), Intentions, Attitudes and Perceived Norms, and the secondary outcomes Aggressive, Violent and Delinquent Behavior (frequency of these behaviors). Specific scales are named, such as the Reactive/Proactive Aggression Scale and Rochester Youth Development Self-Reported Delinquency Scale. School performance was also mentioned as an outcome in the “Detailed Description” section of the ClinicalTrials.gov registration. The registered follow-ups were pretest, posttest and 6-month; there is no mention of the latter in Williams et al. (2021). Trial registration is intended to limit selective outcome reporting in publications; this cannot be ruled out in the case of Williams et al. (2021). Concern about such bias has been raised regarding earlier evaluations of the LST program (Brown, 2001; Gorman, 2005a,2005b).

Without a control group one cannot even attribute the pretest-posttest differences in the variables reported in Williams et al. (2021) to participation in the modified version of the LST program. This weak design is further threatened by selection bias as parents of eligible children could decline their participation in the study, but the authors present no data on the number who refused and how they differed from those whose parents consented to their involvement. They do, in fact, acknowledge the inability of their one-group, pretest-posttest study design to rule out competing explanations of change. Yet they persist in claiming: “The findings indicate that an evidence-based prevention approach adapted for youth diversionary settings … can produce positive changes in psychosocial skills and protective factors known to prevent multiple risk behaviors among youth” (Williams et al., 2021, page 1). The registered study, with a control group, prespecified behavioral outcomes and six-month follow-up could have addressed the effectiveness of the LST program in the juvenile justice arena. The modified study reported in Williams et al. (2021) cannot and may be subject to selective outcome reporting bias.

Availability of data and materials

The data discussed are in the paper that is the focus of the critique and the ClinicalTrials.gov webpage.

References

  1. BMC. (2021). Editorial Policies: Trial Registration. Retrieved October 6, 2021, from https://www.biomedcentral.com/getpublished/editorial-policies#trial+registration. Accessed 6 Oct 2021.

  2. Brown, J. M. (2001). Youth, drugs and resilience education. Journal of Drug Education, 31(1), 83–122. https://doi.org/10.2190/365C-6F4J-7CX7-7JYH.

    Article  Google Scholar 

  3. Gorman, D. M. (2005a). Does measurement dependence explain the effects of the life skills training program on smoking outcomes? Preventive Medicine, 40(4), 479–487. https://doi.org/10.1016/j.ypmed.2004.07.005.

    Article  Google Scholar 

  4. Gorman, D. M. (2005b). Drug and violence prevention: Rediscovering the critical rational dimension of evaluation research. Journal of Experimental Criminology, 1(1), 39–62. https://doi.org/10.1007/s11292-004-6461-z.

    Article  Google Scholar 

  5. Williams, C. (2016). Testing a multilevel preventive intervention in youth courts. In ClinicalTrials.gov Retrieved October 6, 2021, from https://clinicaltrials.gov/ct2/show/record/NCT02764099. Accessed 6 Oct 2021.

  6. Williams, C., Griffin, K. W., Metrha, R. K., & Botvin, G. J. (2021). Testing an evidence-based drug abuse and violence preventive approach adapted for youth in juvenile justice diversionary settings. Health and Justice, 9(3), 1–12. https://doi.org/10.1186/s40352-021-00128-8.

    Article  Google Scholar 

Download references

Acknowledgements

None.

Funding

None.

Author information

Affiliations

Authors

Contributions

Dennis M. Gorman reviewed the paper by Williams et al. (2021) and the registry entry in ClinicalTrials.gov and wrote the critique. The author read and approved the final manuscript.

Corresponding author

Correspondence to Dennis M. Gorman.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

None.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Gorman, D.M. Can a registered trial be reported as a one-group, pretest-posttest study with no explanation? A critique of Williams et al. (2021). Health Justice 10, 2 (2022). https://doi.org/10.1186/s40352-021-00165-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40352-021-00165-3