Citation
Cook, P., Dodge, K., Farkas, G., Fryer, R., Guryan, J., Ludwig, J. Mayer, S., Pollack, H., & Steinberg, L. (2014). The (surprising) efficacy of academic and behavioral intervention with disadvantaged youth: Results from a randomized experiment in Chicago. Cambridge, MA: National Bureau of Economic Research.
Highlights
- The study’s objective was to examine the impact of the Becoming a Man (BAM) program on youths’ academic outcomes, including grade point averages (GPAs), course failures, and achievement test scores. BAM provided mentoring and socialization activities to enhance cognitive, decision-making, and social skills.
- The study was a randomized controlled trial conducted in one high school on the south side of Chicago, Illinois, in 2012–2013. Eligible youth were randomly assigned to one of three conditions: (1) a treatment group receiving BAM only, in which individuals could participate in the BAM program; (2) a treatment group receiving BAM plus tutoring, in which individuals could participate in the BAM program and received daily tutoring in math; or (3) the control group, which could not participate in BAM but could access existing services at their school. The authors estimated regressions to measure the impact of the BAM program on academic achievement using administrative data from Chicago Public Schools.
- The study found that youth offered the opportunity to participate in BAM—either with or without supplemental tutoring—had significantly higher math achievement test scores than youth in the control group. In addition, youth in the BAM program group and the BAM plus tutoring program group had significantly higher math GPAs than members of the control group.
- The quality of causal evidence presented in this report is high for the GPA and course failures outcomes because it was based on a randomized controlled trial with low attrition. This means we are confident that the estimated effects are attributable to BAM and/or the tutoring sessions, and not to other factors. The quality of causal evidence is moderate for the math and reading achievement outcomes because these outcomes had high attrition but the authors included sufficient controls in their analysis. This means we are somewhat confident that the estimated effects are attributable to BAM and/or tutoring sessions, but other factors might also have contributed.
Intervention Examined
Becoming a Man
Features of the Intervention
The BAM program was aimed at youth at the highest risk of academic failure, based on a risk index factoring in number of course failures, unexcused absences, and age compared with classmates. The program consisted of up to 27 one-hour, weekly group mentoring sessions facilitated by Youth Guidance, a nonprofit organization in Chicago that developed and implemented the BAM program. Mentoring sessions were designed to help improve youths’ non-academic skills by promoting socialization and positive changes in behavior. Exercises included evaluating peers’ perspectives, foreseeing consequences, and interacting in groups.
For the BAM plus tutoring program, youth received the same weekly BAM mentoring sessions and daily one-hour math tutoring sessions during the school day. These sessions included two youth and one adult tutor.
Features of the Study
Study participants were male youth in 9th or 10th grades attending a high school on the south side of Chicago in the 2012–2013 school year. Students missing more than 60 percent of class days or failing more than 75 percent of their classes were not eligible for the study. Youth in the study came from highly disadvantaged backgrounds, with 99 percent eligible for free or reduced-price lunch.
Eligible youth were randomly assigned to one of three conditions: (1) a treatment group that could participate in BAM only; (2) a treatment group that could participate in BAM and received math tutoring; and (3) a control group that could not participate in the BAM program or tutoring, but could access existing services in the school or community.
The authors estimated the impact of the program by combining all students assigned to either treatment group and comparing them with the control group. The authors used administrative data from the Chicago Public Schools system to estimate regressions, controlling for demographic characteristics and pre-intervention academic achievement. The authors estimated the impact of the treatment groups on math and non-math GPAs, number of math and non-math course failures, and math and reading standardized test scores.
Findings
- The study found that students offered the opportunity to participate in BAM, either with or without the math tutoring, had significantly fewer non-math course failures. Specifically, the treatment groups experienced, on average, 1.5 fewer non-math course failures relative to the control group.
- Students in both treatment groups had math GPAs that were 0.43 points higher than the control group on a 1 to 4 scale. The study also found that the treatment groups’ national percentile rank in math achievement tests were on average 11.9 points higher than the control group. Both the math GPA and math achievement score findings were statistically significant.
- The study found no significant differences between the treatment groups and the control group for math courses failed, reading achievement test scores, and non-math GPAs.
Considerations for Interpreting the Findings
Although the study was a well-implemented randomized controlled trial, the math and reading achievement test score outcomes were missing for many students, which resulted in high attrition for those outcomes. Therefore, these outcomes cannot receive a high causal evidence rating. However, because the analysis of these results included sufficient controls for pre-program characteristics of the treatment and control groups, they receive a moderate causal evidence rating.
In addition, the authors combined the treatment group assigned to BAM and supplemental math tutoring sessions with the treatment group assigned only to BAM in their analysis. Although this did not affect the study’s causal evidence rating, it complicated interpretation of the results because it was not possible to disentangle the impacts of BAM itself and of the tutoring. It was possible that the math tutoring, and not BAM, was responsible for the observed effects, or vice versa.
Causal Evidence Rating
The quality of causal evidence presented in this report is high for the GPA and course failures outcomes because it was based on a randomized controlled trial with low attrition. This means we are confident that the estimated effects are attributable to BAM and/or tutoring sessions, and not to other factors. The quality of causal evidence is moderate for the math and reading achievement outcomes because these outcomes had high attrition, but the authors included sufficient controls in their analysis. This means we are somewhat confident that the estimated effects are attributable to BAM and/or tutoring sessions, but other factors might also have contributed.