Open Access Commentary

Adding effect sizes to a systematic review on interventions for promoting physical activity among European teenagers

Rik Crutzen

Author Affiliations

Maastricht University/CAPHRI, Maastricht, The Netherlands

International Journal of Behavioral Nutrition and Physical Activity 2010, 7:29 doi:10.1186/1479-5868-7-29


The electronic version of this article is the complete one and can be found online at: http://www.ijbnpa.org/content/7/1/29


Received:21 December 2009
Accepted:16 April 2010
Published:16 April 2010

© 2010 Crutzen; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

This commentary adds effect sizes to the recently published systematic review by De Meester and colleagues and provides a more detailed insight into the effectiveness of interventions to promote physical activity among European teenagers. The main findings based on this evidence were: (1) school-based interventions generally lead to short term improvement in physical activity levels, but there were large differences between interventions with regard to effect sizes; (2) a multi-component approach (including environmental components) generally resulted in larger effect sizes, thereby providing evidence for the assumption that a multi-component approach should produce synergistic results; and (3) if an intervention aimed to affect more health behaviours besides physical activity, then the intervention appeared to be less effective in favour of physical activity.

Commentary

The recently published systematic review by De Meester and colleagues [1] provides an excellent summary of the effectiveness of interventions to promote physical activity among European teenagers. The authors, however, based their conclusions merely on p-values while effect sizes are more appropriate to derive conclusions with regard to differences in the effectiveness of interventions. Therefore, in addition to the highly appreciated work done by De Meester and colleagues, I added effect sizes and provided an even more detailed insight into the effectiveness of interventions to promote physical activity among European teenagers.

Traditionally, null hypothesis-testing resulted in reporting of p-values as the key result of studies in behavioural sciences. Nowadays, however, reporting of effect sizes is deemed increasingly important [2]. Volker [3] argues on why to report effect sizes. In short, p-values are not particularly informative to determine whether a statistically significant effect is meaningful and substantive. They are merely "a conditional probability indicative of the probability of a result at least as extreme as the obtained difference assuming that the null hypothesis is true." [3] Therefore, effect sizes (i.e, Cohen's d) are reported in this commentary to gain a more detailed insight into the meaningfulness and substantiveness of the results of the studies included by De Meester and colleagues [1]. As correctly argued by De Meester and colleagues [1], performing a meta-analysis is problematic because of the heterogeneity of the outcome measures of the included studies. Nevertheless, effect sizes still provide the opportunity to quantify the intervention outcomes instead of having only a conditional probability. It needs to be stressed, however, that an effect size on its own is not equal to the public health impact of an intervention. According to the RE-AIM model, the public health impact of an intervention can only be evaluated by the assessment of five dimensions: Reach, Efficacy, Adoption, Implementation and Maintenance (hence the acronym RE-AIM) [4]. Thus, the public health impact of an intervention that has a small effect, but reaches a large group of people, can still be high. Furthermore, it is important, when looking at the potential of interventions, not to lose sight of the quality assessment of the studies (irrespective of the intervention outcomes).

Table 1 gives an overview of the effect sizes of intervention outcomes. Effect sizes were calculated by using the formulas described by Lipsey and Wilson [5]. As recommended by Morris [6], effect sizes were based on the pooled pre- and post-test standard deviation to obtain a more precise effect size estimate. Odds ratios were converted into Cohen's d (as described by Chinn [7]). Three studies did not report sufficient information to calculate effect sizes; one of the authors of these studies was able to provide the required information. In line with Cohen's classification [8,9], effect sizes were divided into five levels: trivial (Cohen's d ≤ .2), small (> .2), moderate (> .5), large (> .8), and very large (> 1.3).

Table 1. Effect sizes of intervention outcomes

The effects sizes in Table 1 clearly demonstrate the differences between effect sizes and p-values. For example, the p-value of the difference between the intervention group and the control group in the study by Christodoulos and colleagues [10], with regard to total moderate to vigorous physical activity, did not reach significance. The effect size for this outcome measure, however, was very large (d = 2.79). This demonstrates the potential of this intervention with regard to behaviour change, apart from the small sample size in the specific study. Another example, the study of Hill and colleagues [11], shows that although the conditions in this study did not differ significantly, the effect sizes indicate a clear difference between the impact of these interventions (ranging from 0.18 to 0.45).

Taking into account the effect sizes reported in Table 1, the conclusion of De Meester and colleagues [1] that school-based interventions generally lead to short term improvement in physical activity levels still holds. There were, however, large differences between interventions with regard to effect sizes. These differences should be taken into account when judging the potential of intervention solutions. Therefore, the evidence with regard to involvement of family is inconclusive and recommendations regarding family involvement should be interpreted with caution as they are premature. In contrast to De Meester and colleagues [1], the evidence provided by effect sizes is not inconclusive with regard to a multi-component approach. Interventions that included environmental components (as identified in the published review) generally resulted in larger effect sizes. This provides evidence for the assumption that a multi-component approach should produce synergistic results. With regard to interventions aimed at multiple behaviours, it can be concluded that these interventions resulted in smaller effects with regard to physical activity. There appeared to be no differences in effect sizes related to quality assessment of the studies (as assessed in the published review). Nevertheless, when homogeneous outcome measures are available for future studies, meta-analyses are needed to fully warrant strong conclusions with regard to potential moderators of effect sizes (e.g., methodological quality).

Conclusions

Based on the evidence identified by the review of De Meester and colleagues [1] and the effect sizes reported in this commentary, a detailed insight into the effectiveness of interventions to promote physical activity among European teenagers is provided. In summary, the main findings based on this evidence:

(1) School-based interventions generally lead to short term improvement in physical activity levels, but there were large differences between interventions with regard to effect sizes.

(2) A multi-component approach (including environmental components) generally resulted in larger effect sizes, thereby providing evidence for the assumption that a multi-component approach should produce synergistic results.

(3) If an intervention aimed to affect more health behaviours besides physical activity, then the intervention appeared to be less effective in favour of physical activity.

Competing interests

The author declares that they have no competing interests.

Authors' contributions

RC synthesized the data and calculated the effect sizes, interpreted the findings, and wrote the manuscript.

References

  1. De Meester F, Van Lenthe FJ, Spittaels H, Lien N, De Bourdeaudhuij I: Interventions promoting physical activity among European teenagers: a systematic review.

    International Journal of Behavioral Nutrition and Physical Activity 2009, 6:82. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  2. Kline R: Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: APA; 2004. OpenURL

  3. Volker MA: Reporting effect size estimates in school psychology research.

    Psychology in the Schools 2006, 43:653-672. Publisher Full Text OpenURL

  4. Glasgow RE, Vogt TM, Boles SM: Evaluating the public health impact of health promotion interventions: the RE-AIM framework.

    American Journal of Public Health 1999, 89:1322-1327. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  5. Lipsey MW, Wilson DB: Practical meta-analysis. Thousand Oaks, CA: Sage publications; 2001. OpenURL

  6. Morris SB: Estimating effect sizes from pretest-posttest control group designs.

    Organizational Research Methods 2008, 11:364-386. Publisher Full Text OpenURL

  7. Chinn S: A simple method for converting an odds ration to effect size for use in meta-analysis.

    Statistics in Medicine 2000, 19:3127-3131. PubMed Abstract | Publisher Full Text OpenURL

  8. Cohen J: Statistical power analysis for the behavioral sciences. 2nd edition. Hillsdale, NJ: Lawrence Erlbaum associates; 1988. OpenURL

  9. Cohen J: A power primer.

    Psychological Bulletin 1992, 112:155-159. PubMed Abstract | Publisher Full Text OpenURL

  10. Christodoulos AD, Douda HT, Polykratis M, Tokmakidis SP: Attitudes towards exercise and physical activity behaviours in Greek schoolchildren after a year long health education intervention.

    British Journal of Sports Medicine 2006, 40:367-371. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  11. Hill C, Abraham C, Wright DB: Can theory-based messages in combination with cognitive prompts promote exercise in classroom settings?

    Social Science & Medicine 2007, 65:1049-1058. OpenURL

  12. Haerens L, Deforche B, Vandelanotte C, Maes L, De Bourdeaudhuij I: Acceptability, feasibility and effectiveness of a computer-tailored physical activity intervention in adolescents.

    Patient Education and Counseling 2007, 66:303-310. PubMed Abstract | Publisher Full Text OpenURL

  13. Verstraete SJM, Cardon GM, De Clercq DLR, De Bourdeaudhuij IMM: Increasing children's physical activity levels during recess periods in elementary schools: the effects of providing game equipment.

    European Journal of Public Health 2006, 16:415-419. PubMed Abstract | Publisher Full Text OpenURL

  14. Tsorbatzoudis H: Evaluation of a school-based intervention programme to promote physical activity: an application of the theory of planned behavior.

    Perceptual and Motor Skills 2005, 101:787-802. PubMed Abstract | Publisher Full Text OpenURL

  15. Lubans D, Sylva K: Controlled evaluation of a physical activity intervention for senior school students: effects of the lifetime activity program.

    Journal of Sport & Exercise Psychology 2006, 28:252-268. OpenURL

  16. Murphy NM, Ni Dhuinn M, Browne PA, ÓRathaille MM: Physical activity for bone health in inactive teenage girls: is a supervised, teacher-led program or self-led program best?

    Journal of Adolescent Health 2006, 39:508-514. PubMed Abstract | Publisher Full Text OpenURL

  17. Lindberg LC, Ståhle A, Rydén L: Long-term influence of a health education programme on knowledge and health behaviour in children.

    European Journal of Cardiovascular Prevention and Rehabilitation 2006, 13:91-97. PubMed Abstract | Publisher Full Text OpenURL

  18. Chatzisarantis NLD, Hagger MS: Effects of a brief intervention based on the Theory of Planned Behavior on leisure-time physical activity participation.

    Journal of Sports & Exercise Psychology 2005, 27:470-487. OpenURL

  19. Digelidis N, Papaioannou A, Laparidis K, Christodoulidis T: A one-year intervention in 7th grade physical education classes aiming to change motivational climate and attitudes towards exercise.

    Psychology of Sport and Exercise 2003, 4:195-210. Publisher Full Text OpenURL

  20. Harrison M, Burns CF, McGuinness M, Heslin J, Murphy NM: Influence of a health education intervention on physical activity and screen time in primary school children: 'Switch Off-Get Active'.

    Journal of Science and Medicine in Sport 2006, 9:388-394. PubMed Abstract | Publisher Full Text OpenURL

  21. Haerens L, De Bourdeaudhuij I, Maes L, Cardon G, Deforche B: School-based randomized controlled trial of a physical activity intervention among adolescents.

    Journal of Adolescent Health 2007, 40:258-265. PubMed Abstract | Publisher Full Text OpenURL

  22. Simon C, Wagner A, Platat C, Arveiler D, Schweitzer B, Schlienger JL, Triby E: ICAPS: a multilevel program to improve physical activity in adolescents.

    Diabetes & Metabolism 2006, 32:41-49. OpenURL

  23. Jurg ME, Kremers SPJ, Candel MJJM, wal MF, De Meij JSB: A controlled trial of a school-based environmental intervention to improve physical activity in Dutch children: JUMP-in, kids in motion.

    Health Promotion International 2006, 21:320-330. PubMed Abstract | Publisher Full Text OpenURL

  24. Moon AM, Mullee MA, Rogers L, Thompson RL, Speller V, Roderick P: Helping schools to become health-promoting environments - an evaluation of the Wessex Healthy Schools Award.

    Health Promotion International 1999, 14:111-122. Publisher Full Text OpenURL

  25. Baxter AP, Milner PC, Hawkins S, Leaf M, Simpson C, Wilson KV, Owen T, Higginbottom G, Nicholl J, Cooper N: The impact of heart health promotion on coronary heart disease lifestyle risk factors in schoolchildren: lessons learnt from a community-based project.

    Public Health 1997, 111:231-237. PubMed Abstract | Publisher Full Text OpenURL

  26. Ortega-Sanchez R, Jimenez-Mena C, Cordoba-Garcia R, Muñoz-Lopez J, Garcia-Machado ML, Vilaseca-Canals J: The effect of office-based physician's advice on adolescent exercise behavior.

    2004., 38: PubMed Abstract | Publisher Full Text OpenURL

  27. Walker Z, Townsend J, Oakley L, Donovan C, Smith H, Hurst Z, Bell J, Marshall S: Health promotion for adolescents in primary care: randomised controlled trial.

    British Medical Journal 2002, 325:524-527. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  28. Kelleher CC, Fallon UB, McCarthy E, Dineen BD, O'Donnell M, Killian M, Hope A, Bluett D, Varley O, McDonagh G: Feasibility of a lifestyle cardiovascular health promotion programma for 8-15-years-olds in Irish general practice: results of the Galway Health Project.

    Health Promotion International 1999, 14:221-229. Publisher Full Text OpenURL

  29. Woods C, Mutrie N, Scott M: Physical activity intervention: a Transtheoretical Model-based intervention designed to help sedentary young adults become active.

    Health Education Research 2002, 17:451-460. PubMed Abstract | Publisher Full Text OpenURL