Citation
Hollenbeck, K., Huang, W.-J., and WE Upjohn Institute for Employment Research. (2008). Workforce program performance indicators for the commonwealth of Virginia. Upjohn Institute technical report no. 08-024, Kalamazoo, MI: W. E. Upjohn Institute for Employment Research. [FSET]
Highlights
- The study’s objective was to examine the impact of the Food Stamp Employment and Training Program (FSET) on employment and credential completion rates.
- The authors used administrative records to compare outcomes for low-income adults who took part in the FSET program to a nonexperimental matched group of adults who did not take part in the program.
- The study found that, compared to those who did not take part in the program, the employment rate was significantly lower. The FSET group also earned significantly fewer educational or training credentials.
- The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before program participation. This means we are not confident that the estimated effects are attributable to FSET; other factors are likely to have contributed.
Intervention Examined
The Food Stamp Employment and Training Program (FSET)
Features of the Intervention
The FSET program aimed to assist low-income adults find employment through job searches and skill-building.
Features of the Study
Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared those who exited the FSET program from July 2004 to June 2005 with those who did not take part in the program and used the Employment Services program. The authors used a nonexperimental statistical approach called propensity-score matching to create an Employment Services group that was similar in terms of demographics, education, and labor market history. The authors compared the two groups on employment two and four quarters after program exit, and on the percentage who had earned an education or training credential within one year of program exit. This analysis included 10,440 FSET participants in Virginia.
Findings
- Employment. The study found that FSET program participants were significantly less likely to be employed after program exit than the Employment Services group, with an employment rate that was 10.5 percentage points lower at both the second and fourth quarter after program exit.
- Education and/or training attainment and completion. The study found that those who took part in the FSET program were significantly less likely to earn an educational or training credential than those who did not take part in the program (0.3 percentage points lower).
Considerations for Interpreting the Findings
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the FSET and Employment Services participants were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in FSET services was six months, whereas that for Employment Services was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would be looking at FSET participants’ earnings about 12 months after they started receiving services and Employment Services participants’ earnings about 7 months after they started receiving services. If both programs were completely ineffective, and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the FSET participants earned more 6 months after their exit dates. However, this would not be attributable to receiving FSET services; it would be caused by the different elapsed time across the groups (12 months for FSET participants versus 7 months for Employment Services participants). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before program participation. This means we are not confident that the estimated effects are attributable to FSET; other factors are likely to have contributed.
Additional Sources
Hollenbeck, K. (2011). “Short-term net impact estimates and rates of return.” In Douglas J. Besharov and Phoebe H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 371-295). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.