Absence of conflict of interest.
Citation
Cosgrove, J. J., Cosgrove, M. S., & Bragg, D. D. (2016). Third party evaluation of MoManufacturingWINs: Implementation, outcomes, and impact. St. Louis, MO: Cosgrove & Associates and Bragg & Associates.
Highlights
- The study’s objective was to examine the impact of Missouri’s Manufacturing Workforce Innovation Networks(MMW) program on student education and employment outcomes.
- The study used a nonexperimental design to compare the outcomes of students who were in MMW to a comparison group of students in other manufacturing-technology related programs.
- The study found that MMW participation was significantly associated with higher retention and employment rates.
- The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention and they did not include sufficient controls. This means we are not confident that the estimated effects are attributable to MMW; other factors are likely to have contributed.
Intervention Examined
Missouri’s Manufacturing Workforce Innovation Networks (MMW)
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Eight Missouri colleges and the State's technical college formed the Missouri’s Manufacturing Workforce Innovation Networks (MMW) consortium in order to use TAACCCT funds to assist and bolster students in manufacturing related career pathways. The program targeted low-skilled, unemployed/underemployed, TAA-eligible students, veterans, and other students interested in the manufacturing industry. To better prepare students for the workforce and enhance educational and employment outcomes, MMW colleges developed a stackable certificate model that directly mapped to industry certifications. The colleges also combined coursework with basic academic skills, provided intensive academic and employment support, and developed multiple entrance and exit points into the Manufacturing Career Pathways program.
Features of the Study
The study used a nonexperimental design to compare the outcomes of students who participated in the MMW program to students who did not participate. The treatment group included 663 credit-seeking students who first enrolled in one of the consortium colleges' Manufacturing Career Pathways program in Fall 2013 and were assessed through 2016. The comparison group included 411 credit-seeking, first-time students who enrolled in a manufacturing-technology related program in a consortium college in Fall 2011 and were assessed through 2014. Outcomes included program retention rates and employment rates upon program completion. Using data from college databases, the Missouri Division of Workforce Development, direct follow-up engagement with students, and DOL-approved follow-up methods including employee paystubs and letters from employers, the authors used statistical models to examine differences in the outcomes between treatment and comparison groups.
Study Sites
- East Central College (ECC) in Union, Missouri
- Metropolitan Community College (MCC) in Kansas City, Missouri
- Mineral Area College (MAC) in Park Hills, Missouri
- North Central Missouri College (NCMC) in Trenton, Missouri
- Ozarks Technical Community College (OTC) in Springfield, Missouri
- St. Charles Community College (SCC) in Cottleville, Missouri St.
- Louis Community College (STLCC) in St. Louis, Missouri
- State Fair Community College (SFCC) in Sedalia, Missouri
- State Technical College of Missouri (STCM) in the Osage County, Missouri
Findings
Education and skills gain
- The study found that participation in the TAACCCT-enhanced MMW program was significantly associated with program completion as treatment group participants were 4.4 times more likely than comparison group participants to complete their program.
Employment
- The study found that participation in the TAACCCT-enhanced MMW program was significantly associated with employment rates as treatment group participants were 9.7 times more likely than comparison group participants to be employed upon program completion.
Considerations for Interpreting the Findings
The authors note that some of the observed outcomes could be attributed to the accelerated nature and timeframe of MMW programs versus the traditional manufacturing technology programs. Furthermore, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Lastly, although the authors controlled for several demographic variables in their regression analysis, they did not account for certain factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the MMW program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention and they did not include sufficient controls. This means we are not confident that the estimated effects are attributable to MMW; other factors are likely to have contributed.