Absence of conflict of interest.
Citation
Hendra, R., Greenberg, D. H., Hamilton, G., Oppenheim, A. Pennington, A. Schaberg, K., and Tessler, B. L. (2016). Encouraging evidence on a sector-focused advancement strategy. New York: MDRC. [Per Scholas]
Highlights
- The study’s objective was to examine the impact of the WorkAdvance sectoral training program at the Per Scholas site on employment, earnings, and education and training from 2011 to 2015. The authors investigated similar research questions with three other sites, the profiles of which are available here.
- The study used a randomized controlled trial design to compare the treatment group, which was able to access the WorkAdvance program at the Per Scholas site, and the control group, which was not eligible for WorkAdvance services but could access other services in the community. The authors collected data from two sources: a follow-up survey and unemployment insurance (UI) wage and employment data.
- The study found that the program had a statistically significant impact on quarterly earnings in the second and third years after random assignment (based on UI data) and average weekly earnings in the second year after random assignment (based on survey data). The study also found that the program did not have a significant impact on most employment outcomes. Using survey data, the authors found that members of the WorkAdvance group were significantly more likely to have obtained a degree or credential and to have completed a skills training program two years after random assignment.
- The quality of the causal evidence is moderate for employment and earnings outcomes based on UI data because those outcomes were based on a randomized controlled trial in which the authors did not demonstrate that they accounted for change in the probability of random assignment, but the authors did account for existing differences between the treatment and control groups. The quality of the causal evidence is moderate for education and training outcomes from the follow-up survey because those outcomes were based on a randomized controlled trial in which many people did not complete the follow-up survey, but the authors did account for existing differences between the treatment and control groups. This means we are somewhat confident that the estimated effects are attributable to the WorkAdvance program at the Per Scholas site, but other factors might also have contributed.
Intervention Examined
The WorkAdvance sectoral training program
Features of the Intervention
WorkAdvance was a sectoral training program that coupled a career readiness and occupational training program with job placement supports and post-employment job retention services. Per Scholas, a nonprofit in the Bronx, New York, implemented the 15-week program from 2011 to 2013. The intervention included 12 separate seven-hour training sessions geared toward obtaining occupational skills and industry certification in the information technology sector, pre-employment and career readiness services, job development and job placement services, and post-employment retention services. The program’s goal was to place participants in an entry-level information technology job.
To be eligible for the program, people had to be at least 18 years old, be legally allowed to work in the United States, have a family income of less than 200 percent of the federal poverty line, and either be unemployed or employed at a job making less than $15 per hour. In addition, people had to have math and reading scores at roughly a 9th-grade level or higher and have a high school diploma.
Features of the Study
The study used a randomized controlled trial design in which 700 people were assigned to either the treatment group (which was offered the WorkAdvance program) or a control group (which could not join the WorkAdvance program but could access other services available in the community). The sample was primarily male (87 percent) and between the ages of 18 and 24 (71 percent). In all, 45 percent of people in the sample were African American, and 36 percent were Latino.
The authors collected data from two sources: (1) a follow-up survey completed two years after random assignment by 552 members of the sample and (2) quarterly UI wage and employment data for nine quarters after random assignment from 690 members of the sample. The authors used a statistical model to compare the outcomes of treatment and control group members.
Findings
Employment
- Using UI data, the authors found that those assigned to the treatment group were 6 percentage points more likely to report ever being employed in the second year after random assignment than those assigned to the control group. There were no other significant impacts of the program on employment outcomes using UI data.
- Using follow-up survey data, the authors found no significant impacts of the program on employment outcomes.
Earnings and wages
- Using UI data, the authors found that those assigned to the treatment group earned $3,746 more in Year 2 and $1,289 more in Year 3 compared with those assigned to the control group. There were no significant impacts of the program on earnings in Year 1 using UI data.
- Using follow-up survey data, the study found that those assigned to the treatment group earned significantly more on average per week ($98) than the control group in Year 2.
Education and skills gains
- Using follow-up survey data, the study found that those assigned to the treatment group were significantly more likely to have obtained a degree or credential by Year 2 (36 percentage points) and to have completed a skills training program (37 percentage points), than those assigned to the control group.
Considerations for Interpreting the Findings
The outcomes using follow-up survey data came from a randomized controlled trial in which there were no follow-up data available for a large number of people in the treatment and control groups. The authors did account for potential existing differences between the groups, however, so these outcomes receive a moderate evidence rating.
The intervention included services and training for about 15 weeks, meaning services occurred in Quarters 1 and 2 following random assignment. Therefore, findings drawn from UI data from the second quarter after random assignment could underestimate the full treatment effect, as the full program was not complete for the full duration of that quarter.
A majority of control group members received some form of employment service from community-based or governmental organizations, and nearly 50 percent received skills training. If this training was similar in intensity and content to the WorkAdvance training, then there might not have been a clear difference between the treatment and control groups, making it more challenging to detect program impacts.
The study took place during the Great Recession, so there could have been fewer employment opportunities available to the sample.
Causal Evidence Rating
The quality of the causal evidence is moderate for employment and earnings outcomes based on UI data because those outcomes were based on a randomized controlled trial in which the authors did not demonstrate that they accounted for change in the probability of random assignment, but the authors did account for existing differences between the treatment and control groups. The quality of the causal evidence is moderate for education and training outcomes from the follow-up survey because those outcomes were based on a randomized controlled trial in which many people did not complete the follow-up survey, but the authors did account for existing differences between the treatment and control groups. This means we are somewhat confident that the estimated effects are attributable to the WorkAdvance program at the Per Scholas site, but other factors might also have contributed.