Skip to main content

Return on investment analysis of a selected set of workforce system programs in Indiana. [Sub-baccalaureate Postsecondary Education] (Hollenbeck 2009)

Review Guidelines

Citation

Hollenbeck, K. (2009). Return on investment analysis of a selected set of workforce system programs in Indiana. Indianapolis, IN: Indiana Chamber of Commerce Foundation. [Sub-baccalaureate Postsecondary Education]

Highlights

  • The study objective was to examine the impact of the Sub-baccalaureate Postsecondary Education Program on the employment and earnings of adults with a high school education in Indiana.
  • The author used a nonexperimental method to compare the short-term (three quarters after program exit) and long-term (seven quarters after program exit) employment and earnings between those who took part in the Sub-baccalaureate Postsecondary Education Program and those who participated in the WorkOne program.
  • The study found that, compared with those who participated in the WorkOne program, Sub-baccalaureate Postsecondary Education Program participants had higher employment and earnings.
  • The quality of causal evidence presented in this report is low because the author did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to the Sub-baccalaureate Postsecondary Education Program; other factors are likely to have contributed.
  • This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.

Intervention Examined

Sub-baccalaureate Postsecondary Education Program

Features of the Intervention

The Sub-baccalaureate Postsecondary Education Program facilitated pre-baccalaureate education up to and including an Associate’s degree. The Indiana Commission on Higher Education oversaw the program, and courses were offered at both private and public institutions across the state.

Features of the Study

The author used a nonexperimental statistical approach called propensity-score matching to create a comparison group of people who participated in WorkOne and were similar to Sub-baccalaureate Postsecondary Education Program participants on characteristics such as age, educational attainment, employment history, and gender. The author then compared the two groups on employment and quarterly earnings after participation. The author collected Indiana Workforce Intelligence System records and Unemployment Insurance records for those who had exited the Sub-baccalaureate Postsecondary Education or WorkOne programs in fiscal year 2006. The study included 11,320 participants who received services through the Sub-baccalaureate Postsecondary Education Program. They were matched to an equal number of comparison cases drawn from the 292,616 participants who received services through the WorkOne program in Indiana during the same period.

Findings

  • Employment. The study found that, compared with those who participated in WorkOne, the employment rate for the Sub-baccalaureate Postsecondary Education group significantly increased by 17.9 percentage points in the third quarter after program exit and by 19.9 percentage points in the seventh quarter after program exit.
  • Earnings. The study found that average quarterly earnings significantly increased by $1,490 in the third quarter after program exit and by $1,547 in the seventh quarter after program exit for those who took part in the Sub-baccalaureate Postsecondary Education Program compared with members of the WorkOne group.

Considerations for Interpreting the Findings

Although the author accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the author’s decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the Sub-baccalaureate Postsecondary Education and WorkOne participants were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in Sub-baccalaureate Postsecondary Education services was six months, whereas that for WorkOne was one month. At the conclusion of participation, they exited the program.

If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe Sub-baccalaureate Postsecondary Education participants’ earnings about 12 months after they started receiving services and WorkOne participants’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the Sub-baccalaureate Postsecondary Education participants earned more 6 months after their exit dates. However, this would not be attributable to receiving Sub-baccalaureate Postsecondary Education services; it would be caused by the different elapsed time across the groups (12 months for Sub-baccalaureate Postsecondary Education participants versus 7 months for WorkOne participants). Therefore, studies defining the groups based on exit date rather than entry date cannot receive a moderate causal evidence rating.

Causal Evidence Rating

The quality of causal evidence presented in this report is low because the author did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to the Sub-baccalaureate Postsecondary Education Program; other factors are likely to have contributed.

Additional Sources

Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D. J. Besharov & P. H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.

Reviewed by CLEAR

May 2017