Citation
Hollenbeck, K., & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [DBVI]
Highlights
- The study’s objective was to examine the impact of the Vocational Rehabilitation program administered by the Department for Blind and Vision Impaired (DBVI) on participants’ employment and credential completion.
- The study used administrative records to compare the outcomes of low-income adults who took part in the DBVI program with outcomes of a nonexperimental matched group of adults who did not take part in the program.
- The study found that DBVI participants had higher employment and credential completion rates compared with those of people who did not participate in the program.
- The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to DBVI; other factors are likely to have contributed.
- This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.
Intervention Examined
The Vocational Rehabilitation (VOC) Programs
Features of the Intervention
The Vocational Rehabilitation program administered by DBVI is designed to help blind and visually impaired adults find and keep employment. Program services include job training, job placement assistance, support for medical procedures, provision of adaptive equipment, and support for job site accommodations.
Features of the Study
Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the Vocational Rehabilitation program administered by DBVI with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors compared the two groups on employment two and four quarters after program exit, and on the percentage of each group that had earned an educational credential during the program or within one year of program exit. This analysis included 230 DBVI participants in Virginia who exited the program from July 2004 to June 2005.
Findings
- Employment. The study found that DBVI participants were significantly more likely to be employed than members of the comparison group, with an employment rate that was 18.3 percentage points higher in the second quarter and 25.4 percentage points higher in the fourth quarter after program exit.
- Education and/or training attainment and completion. The study found that DBVI participants were significantly more likely than members of the comparison group to earn an educational credential during the program or within one year of program exit (a difference of 4.1 percentage points).
Considerations for Interpreting the Findings
The authors presented unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the authors.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the DBVI participants and comparison group members were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in DBVI services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe DBVI participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the DBVI participants earned more 6 months after their exit dates. However, this would not be attributable to receiving DBVI services; it would be caused by the different elapsed time across the groups (12 months for DBVI participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to DBVI; other factors are likely to have contributed.
Additional Sources
Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.