Citation
Hirsch, B., Hedges, L., Stawicki, J., & Mekinda, M. (2011). After-school programs for high school students: An evaluation of after-school matters. Evanston, IL: Northwestern University.
Highlights
- The report’s objective was to evaluate the impact of the After School Matters (ASM) apprenticeship program on academic outcomes in high school and on the likelihood of being hired for a summer or permanent entry-level position in the labor market.
- For this evaluation, 535 students from 10 high schools in Chicago were randomly assigned to the treatment group, which could enroll in ASM, or the control group, which could not. Outcome data were derived from high school records from both semesters of the academic year of study participation and a mock job interview at the end of the spring semester of the same year.
- The study reported no statistically significant impacts of participation in ASM on high school grade point average (GPA), number of classes failed, school absences, or likelihood of being hired for a permanent or summer entry-level position.
- The quality of the causal evidence presented in this report is high for all academic outcomes. This means we are confident that estimated impacts for these outcomes would be solely attributable to ASM, and not other factors. For the likelihood of employment outcomes, the quality of evidence is moderate. This means we have confidence that the effects estimated in this study would be attributable at least in part to the ASM program, though other factors could also have contributed.
Intervention Examined
After School Matters
Features of the Intervention
ASM offered paid after-school apprenticeships in a variety of fields, including computer refurbishment, journalism, clothing design, storytelling, web design, claymation, dance and percussion, sports, songwriting, photography, painting and drawing, and the culinary arts. The program consisted of a 10-week apprenticeship during both the fall and spring semesters, so the intervention lasted 20 weeks over one academic year. Most apprenticeships took place within the high school building and met for three hours, three times a week (for a total of nine hours per week). Students who participated in at least 73 percent of the sessions were paid $900.
Two paid instructors led youth in each apprenticeship. The instructors were not typically teachers but made their living in the field of the apprenticeship. For the evaluation, strong instructors were selected by ASM program staff; all instructors in the evaluation had worked for at least a year on ASM before the evaluation.
Youth were eligible for the program if they had not completed the same apprenticeship in the previous year (youth who had participated in a different apprenticeship or ASM program previously were still eligible) and if the instructors judged them to be acceptable for the program.
Features of the Study
Students who expressed interest in the program, completed parental consent, and received instructor approval were randomly assigned to either the treatment or the control group. The treatment group was allowed to participate in the ASM program, whereas the control group was not. Data for the education and employment outcomes were collected for all students from high school records and a mock job interview. In the mock job interviews, two human resources professionals interviewed participants for one of two hypothetical summer jobs. The interviewers rated the likelihood they would recommend hiring the participant for either an entry-level summer or permanent position.
Findings
The ASM had no impact on high school GPA, school absences, number of failed courses, or likelihood of being hired for a permanent or summer entry-level position.
Considerations for Interpreting the Findings
Nearly 47 percent of students assigned to the treatment group dropped out of their apprenticeship before program completion. In addition, students in the control group were free to seek existing after-school programs in the community. Before the evaluation, it was assumed that few other programs were available for youth in the control group. However, 91 percent participated in some sort of organized after-school activity or paid work. Therefore, the comparison of the treatment and control groups measures the impact of ASM compared with the other available programming in which students in the control group participated.
The programs chosen by the authors to participate in this evaluation had strong experience in previously implementing ASM apprenticeships. Although chosen for their high quality, however, 3 of the 13 apprenticeship programs did not meet the authors’ expectations for quality.
Finally, the employment-related outcomes suffered from high attrition, so they are not eligible for a high causal evidence rating. However, the authors included sufficient control variables in their models to receive a moderate evidence rating.
Causal Evidence Rating
The quality of causal evidence for the academic outcomes presented in this report is high because it is based on a well-conducted randomized controlled trial. This means we are confident that estimated impacts would be solely attributable to ASM, and not to other factors. However, the quality of causal evidence for the employment outcomes presented in this report is moderate. This means we have confidence that the effects estimated in this study would be attributable at least in part to the ASM program, although other factors not accounted for in the study could also have contributed to the estimated effects.