Citation
Hollenbeck, K., & Huang, W. (2006). Net impact and benefit-cost estimates of the workforce development system in Washington State. Upjohn Institute technical report no. TR06-020. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research. [Dept. of Services for the Blind]
Highlights
- The study’s objective was to examine the impact of the Department of Services for the Blind (DSB) program on the employment rate and earnings of low-income adults who are blind.
- The authors used a nonexperimental method to compare the short-term (3 quarters after program exit) and long-term (9 to 12 quarters after program exit) employment and earnings between those who took part in the DSB program relative to those who were eligible but did not receive these services.
- The study found that, compared with those who did not receive these services, DSB program participants had significantly higher employment and earnings.
- The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to the DSB program; other factors are likely to have contributed.
- This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.
Intervention Examined
The Department of Services for the Blind (DSB) Program
Features of the Intervention
The DSB program provided vocational rehabilitation services to participants who are blind or visually impaired. These services could range from information, medical assessment, and referrals to vocational counseling, as well as occupational licenses, equipment, and technological aids that can increase the chances of employment.
Features of the Study
The authors used a nonexperimental approach of comparing DSB program participants with a group of people who registered at the Labor Exchange and were eligible for, but did not receive, services from the DSB. The authors compared the two groups on employment and quarterly earnings. Washington Workforce Intelligence System records and Unemployment Insurance (UI) records were collected for those who had exited the DSB program from July 2001 to June 2004 in Washington State. The study included a sample of 158 workers who received services through the DSB program and exited from July 2001 to June 2002 and 227 who exited from July 2003 to June 2004. The sample of participants was 40 years old on average, about 60 percent were male, and about 20 percent were of a racial or ethnic minority. The comparison group included 77 adults from July 2001 to June 2002 and 76 adults from July 2003 to June 2004.
Findings
Employment
- The study found that, compared with those who did not receive DSB support, the employment rate for those who received services significantly increased by 23.7 percentage points more in the 3rd quarter and by 20.3 percentage points more in the 9th to 12th quarter after exit. Compared with those who did not receive DSB services, the average quarterly hours worked for those who took part in the DSB program significantly increased by 78.4 more in the 9th to 12th quarter after program exit, but there was no significant difference in the 3rd quarter after program exit.
Earnings and wages
- The study found that average quarterly earnings significantly increased by $1,357 more in the 9th to 12th quarter after program exit for those who took part in the DSB program compared with those who did not receive DSB services, but there was no significant difference in the 3rd quarter after program exit. Compared with those who did not receive DSB services, the average hourly wages for those who took part in the DSB program significantly increased by $3.90 more in the 3rd quarter and $5.08 more in the 9th to 12th quarter after program exit
Considerations for Interpreting the Findings
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the DSB program participants and those who registered at the Labor Exchange but did not receive DSB services support were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in the DSB program was six months, whereas that for those not receiving DSB support was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe DSB program participants’ earnings about 12 months after they started receiving services and earnings of those who did not receive DSB support about 7 months after they started receiving services. If both programs were completely ineffective, and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the DSB program participants earned more 6 months after their exit dates. However, this would not be attributable to receiving DSB program services; it would be caused by the different elapsed time across the groups (12 months for DSB program participants versus 7 months for those not receiving DSB support). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because treatment is confounded with systematic differences in outcome measurement follow-up length for treatment and comparison group members. This means we are not confident that the estimated effects are attributable to the Department of Services for the Blind (DSB) program; other factors are likely to have contributed.
Additional Sources
Hollenbeck, Kevin. (2011). Short-term net impact estimates and rates of return. In Douglas J. Besharov & Phoebe H. Cottingham (Eds.), The Department of Blind Services program: Implementation experiences and evaluation findings, pp. 371-295. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.