Absence of conflict of interest.
Citation
Hendra, R., Greenberg, D. H., Hamilton, G., Oppenheim, A. Pennington, A. Schaberg, K., and Tessler, B. L. (2016). Encouraging evidence on a sector-focused advancement strategy. New York: MDRC. [St. Nicks Alliance]
Highlights
- The study’s objective was to examine the impact of the WorkAdvance sectoral training program at the St. Nicks Alliance site on employment, earnings, education, and training from 2011 to 2015. The authors investigated similar research questions with three other sites, the profiles of which are available here.
- The study used a randomized controlled trial design to compare the treatment group, which was able to access the WorkAdvance program at the St. Nicks Alliance site, and the control group, which was not eligible for WorkAdvance services but could access other services in the community. The authors collected data from two sources: a follow-up survey and unemployment insurance (UI) wage and employment data.
- Using UI data, the authors found that those assigned to the treatment group were significantly more likely to be ever employed in Quarters 2 to 5 after random assignment than those in the control group. Using survey data, the authors also found that those in the treatment group were significantly more likely to complete education or training than the control group.
- The quality of the causal evidence is high for employment and earnings outcomes based on UI data because those outcomes were based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects for those outcomes are attributable to the WorkAdvance program at the St. Nicks Alliance site and not to other factors. For education and training outcomes from the follow-up survey, the quality of the causal evidence is moderate because those outcomes were based on a randomized controlled trial in which many people did not complete the follow-up survey, but the authors did account for existing differences between the treatment and control groups. This means we are somewhat confident that the estimated effects for those outcomes are attributable to the WorkAdvance program at the St. Nicks Alliance site, but other factors might also have contributed.
Intervention Examined
The WorkAdvance sectoral training program
Features of the Intervention
WorkAdvance was a sectoral training program that coupled a career readiness and occupational training program with job placement supports and post-employment job retention services. St. Nicks Alliance, a large social service agency in Brooklyn, New York, implemented the 12-week program from 2011 to 2013. The intervention included nine separate four-hour career readiness training sessions and up to 12 weeks of occupational training, including one-on-one preemployment coaching services two to three times a week. The targeted occupations for the training were environmental remediation technician, tanker or hazmat commercial driver, and pest control technician.
To be eligible for the program, people had to be at least 18 years old, be legally allowed to work in the United States, have a family income of less than 200 percent of the federal poverty line, and either be unemployed or employed at a job making less than $15 per hour. In addition, people had to have math and reading scores at the 8th-grade level or higher, have a driver’s license, pass a drug screen, and have not completed any other training at St. Nicks Alliance in the previous two years.
Features of the Study
The study used a randomized controlled trial design, in which 483 people were assigned to either the treatment group (which was offered the WorkAdvance program) or a control group (which could not join the WorkAdvance program but could access other services available in the community). The sample was primarily male (85 percent) and between the ages of 18 and 24 (63 percent). In all, 63 percent of people in the sample the sample were African American, and 23 percent were Latino.
The authors collected data from two sources: (1) a follow-up survey completed two years after random assignment by 384 members of the sample and (2) quarterly UI wage and employment data for nine quarters after random assignment from 479 members of the sample. The authors used a statistical model to compare the outcomes of treatment and control group members.
Findings
Employment
- Using UI data, the authors found that in Quarters 2 to 5 after random assignment, the treatment group was significantly more likely to be ever employed compared with the control group (78 percent and 70 percent, respectively). The authors found no other significant impacts of the program on employment outcomes, using either the UI data or survey data.
Earnings and wages
- The authors did not find any statistically significant effects on earnings from either the UI data or survey data.
Education and skills gains
- Based on the follow-up survey data, the authors found that 62 percent of the treatment group obtained a degree or credential by the second year after random assignment compared with 38 percent of the control group, which was a statistically significant difference. The treatment group was also significantly more likely to have completed a skills training program, obtained a credential, obtained a credential in the environmental remediation sector, and completed any education or training compared with the control group.
Considerations for Interpreting the Findings
The outcomes using follow-up survey data came from a randomized controlled trial in which there was no follow-up data available for a large number of people in the treatment and control groups. The authors did account for potential existing differences between the groups, however, so these outcomes receive a moderate evidence rating.
The St. Nicks Alliance site had to make considerable changes from its existing work to implement the WorkAdvance program. In particular, the site did not have strong connections with a wide network of employers, so it took a long time to establish those connections and relationships. A majority of control group members received some form of employment service from community-based or governmental organizations, and 45 percent received skills training. If this training was similar in intensity and content to the WorkAdvance training, then there might not have been a clear difference between the treatment and control groups, making it more challenging to detect program impacts.
The study took place during the Great Recession, so there could have been fewer employment opportunities available to the sample.
Causal Evidence Rating
The quality of the causal evidence is high for employment and earnings outcomes based on UI data because those outcomes were based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects for those outcomes are attributable to the WorkAdvance program at the St. Nicks Alliance site and not to other factors. For education and training outcomes from the follow-up survey, the quality of the causal evidence is moderate because those outcomes were based on a randomized controlled trial in which many people did not complete the follow-up survey, but the authors did account for existing differences between the treatment and control groups. This means we are somewhat confident that the estimated effects for those outcomes are attributable to the WorkAdvance program at the St. Nicks Alliance site, but other factors might also have contributed.