Citation
Raphael, S., & Stoll, M. A. (2006). Evaluating the effectiveness of the Massachusetts workforce development system using no-shows as a nonexperimental comparison group. Evaluation Review, 30(4), 379-429.
Highlights
- The study’s objective was to examine the impact of the Job Training Partnership Act (JTPA) workforce development program in Massachusetts on the earnings of disadvantaged adults.
- The study used a nonexperimental design to compare the earnings of program participants before and after the intervention, relative to the earnings of nonparticipants who were eligible and offered services but chose not to participate in a training program. The authors used administrative records from the Commonwealth Corporation of Massachusetts for their analysis.
- The study found that, although the participants in the Massachusetts JTPA program earned significantly less than nonparticipants during the year of the program, participants earned significantly more one year ($1,876) and two years ($2,285) after the program, compared with nonparticipants.
- The quality of causal evidence presented in this report is moderate because it was based on a well-implemented nonexperimental design. This means we are somewhat confident that the estimated effects are attributable to JTPA, but other factors might also have contributed.
Intervention Examined
The Massachusetts JTPA Workforce Development Program
Features of the Intervention
The JTPA workforce development program took place in the 1990s in Massachusetts and provided training for economically disadvantaged adults. Adults who received welfare benefits or had family incomes below the federal poverty level were eligible for the program. JTPA programs also targeted hard-to-serve populations, such as people without a high school degree, ex-offenders, homeless people, and people with disabilities. All those interested in the program first completed an assessment to determine program eligibility, and then eligible people received the opportunity to participate in training activities. The study did not provide details on the specific topics covered in the training activities.
Features of the Study
Using administrative data from the Commonwealth Corporation of Massachusetts, the authors compared the earnings of people who participated in the Massachusetts JTPA workforce development program with those who completed the assessment, were eligible for the program, and were offered services but did not participate in a training program. The analysis accounted for demographic characteristics such as gender, age, race, education level, and disability status, as well as math, reading, and basic skills levels. The analysis sample included 5,586 JTPA program participants and 4,025 comparison nonparticipants. The sample was predominantly female and economically disadvantaged.
Findings
- The study found that, during the year of the program, participants in the Massachusetts JTPA workforce development program earned $610 less than nonparticipants. This difference was statistically significant.
- JTPA workforce development program participants earned significantly more than nonparticipants in two follow-up periods: $1,876 more in the first year after the program and $2,285 in the second year after the program.
Considerations for Interpreting the Findings
The authors did not indicate that they accounted for the serial correlation of participants’ earnings over time—that is, how past earnings relate to future earnings. Failure to account for serial correlation can incorrectly inflate the chances of finding statistically significant impacts.
Causal Evidence Rating
The quality of causal evidence presented in this report is moderate because it was based on a well-implemented nonexperimental design. This means we are somewhat confident that the estimated effects are attributable to JTPA, but other factors might also have contributed.