Skip to main content

Reducing inequality summer by summer: Lessons from an evaluation of the Boston Summer Youth Employment Program (Modestino & Paulsen 2019)

Review Guidelines

Absence of conflict of interest.

Citation

Modestino, A. S., & Paulsen, R. J. (2019). Reducing inequality summer by summer: Lessons from an evaluation of the Boston Summer Youth Employment Program. Evaluation and Program Planning, 72, 40-53.

Highlights

  • The study's objective was to examine the impact of the Boston Summer Youth Employment Program (SYEP) on future work plans, academic aspirations, and job readiness skills. 

  • The study was a randomized control trial that used surveys to compare the outcomes of youth who participated in the Boston SYEP to a control group who applied to the program but were not enrolled.  

  • The study found that SYEP program participants were more likely than control youth to report having certain job readiness skills (such as having prepared a resume or cover letter or practiced interviewing). SYEP participants were also more likely than control youth to report educational aspirations such as plans to attend a two- or four-year college and less likely to report plans to work in the fall.  

  • The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention or include sufficient control variables. This means we are not confident that the estimated effects are attributable to the Boston SYEP; other factors are likely to have contributed. 

Intervention Examined

Boston Summer Youth Employment Program

Features of the Intervention

The Boston SYEP is designed to provide young adults (14- to 23-year-olds) with meaningful employment experiences that lead to new career and education opportunities. Boston SYEP participants receive job readiness training and are assigned a job where they work a maximum of 25 hours per week for a six-week period from July to mid-August.

Features of the Study

The study was a randomized control trial that assigned youth who applied to the Boston SYEP through Action for Boston Community Development (ABCD) to either participate in the program or a control group that did not receive employment or services. At the end of summer, youth completed a survey measuring job readiness, postsecondary aspirations, and financial capability. Treatment youth completed the survey in person when they picked up their final paycheck and control youth completed it via email. The authors used a statistical model to compare the outcomes of 663 treatment youth who participated in the program to 664 control youth who did not.

Findings

Education and Skills Gains  

  • The study found a positive statistically significant relationship between Boston SYEP participation and job readiness skills including having all key information to apply for a job, having prepared a resume, having prepared a cover letter, having developed answers to the usual interview questions, and having practiced interviewing skills with an adult.  

  • The study found a positive statistically significant relationship between Boston SYEP participation and both plans to attend a two-year college and plans attend a four-year college or university.  

  • The study found no statistically significant relationship between Boston SYEP participation and job readiness skills including having asked an adult to serve as a reference, having reviewed at least one job application form, having completed at least one online job application, and having searched for jobs online. 

  • The study found no statistically significant relationship between Boston SYEP participation and plans to enroll in an education or job training program after high school. 

Employment 

  • The study found a negative statistically significant relationship between Boston SYEP participation and plans to work the following fall.  

Considerations for Interpreting the Findings

The authors conducted a randomized control trial; however, the study had high levels of attrition. Additionally, although the authors controlled for some demographic characteristics of participants, they did not account for pre-intervention differences on measures related to the outcome domains. The review protocol requires that analyses of employment-related outcomes must include controls for previous employment or earnings from at least one year before program participation and that education-related outcomes should control for baseline education level and socioeconomic status. The authors did not include these control variables, so preexisting differences between the groups—and not Boston SYEP— could explain the observed differences in outcomes. Therefore, the study is not eligible for a high causal evidence rating, the highest rating available for experimental designs. 

Causal Evidence Rating

This study receives a low-evidence rating because the authors did not account for preexisting differences between the groups before program participation. This means we are not confident that the estimated effects are attributable to the Boston SYEP; other factors are likely to have contributed.

Reviewed by CLEAR

May 2022