Citation
Hollenbeck, K., & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [AEL]
Highlights
- The study’s objective was to examine the impact of the Adult Education and Literacy (AEL) program on participants’ employment and credential completion.
- The study used administrative records to compare outcomes of low-income adults who took part in the AEL program with outcomes of a nonexperimental matched group of adults who did not take part in the program.
- The study found that AEL participants had a lower employment rate but were more likely to have earned a GED compared with those of people who did not participate in the program.
- The quality of causal evidence presented in this report is low because the author did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to AEL; other factors are likely to have contributed.
- This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.
Intervention Examined
The Adult Education and Literacy (AEL) Program
Features of the Intervention
The AEL program is designed to help adults improve basic skills in areas such as reading, writing, and mathematics. The program also provides English as a Second Language courses to help non-native English speaking adults improve their communication skills.
Features of the Study
Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the AEL program with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors used a nonexperimental statistical approach called propensity score matching to compare AEL participants with similar nonparticipants. The authors compared the two groups on employment two and four quarters after program exit, and on the percentage of each group that had earned a GED within one year of program exit. The analysis included 3,716 AEL program participants in Virginia who exited the program from July 2004 to June 2005.
Findings
- Employment. The study found that AEL participants were significantly less likely to be employed than the comparison group, with an employment rate that was 11.5 percentage points lower in the second quarter and 9.6 percentage points lower in the fourth quarter after program exit.
- Education and/or training attainment and completion. The study found that AEL participants were significantly more likely than members of the comparison group to have earned a GED within one year of program exit (a difference of 70.5 percentage points).
Considerations for Interpreting the Findings
The authors presented unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the author.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the AEL participants and comparison group members (who participated in the employment services program only) were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in AEL services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe AEL participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective, and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the AEL participants earned more 6 months after their exit dates. However, this would not be attributable to receiving AEL services; it would be caused by the different elapsed time across the groups (12 months for AEL participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the author did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to AEL; other factors are likely to have contributed.
Additional Sources
Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.