Absence of conflict of interest.
Citation
Westat. (2016). Evaluation of the Illinois Network for Advanced Manufacturing: Final Report. Rockville, MD: Westat & GEM Software Development, Inc.
Highlights
- The study’s objective was to examine the impact of the Illinois Network for Advanced Manufacturing (INAM) program on education, earnings, and employment outcomes.
- The study used a nonexperimental design to compare the outcomes of students who were in the INAM program to a comparison group of students who took similar courses before the creation of INAM. Using data from college databases and the Illinois Department of Employment Security, the authors conducted statistical models to examine differences in outcomes between the groups.
- The study found that participation in the INAM program was significantly associated with improved rates of certificate or degree completion.
- The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the INAM program; other factors are likely to have contributed.
Intervention Examined
The Illinois Network for Advanced Manufacturing (INAM)
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories
The Illinois Network for Advanced Manufacturing (INAM) was developed using TAACCCT funds and consisted of 21 colleges throughout the state. INAM aimed to prepare participants for advanced manufacturing careers in six areas: certified production technicians, mechatronics, precision machining (CNC or computer numerical control), industrial maintenance, welding, and metalworking. The programs targeted TAA workers, veterans, incumbent workers, and the unemployed, as well as other students who applied. To achieve their goal, INAM: (1) developed student educational plans; (2) improved curricula to facilitate attainment of industry credentials; (3) obtained industry-relevant equipment; (4) developed online and technology-based learning; (5) recognized prior student academic achievements to accelerate program completion; (6) created internships and on-the-job training opportunities; (7) instituted career placement services; and (8) created articulation agreements with higher learning institutions.
Features of the Study
The nonexperimental study was conducted at the colleges that formed INAM, although several colleges were excluded due to the strategies used to recruit treatment and comparison group students. The treatment group for educational outcomes consisted of 2,645 students taking INAM courses who were recruited into the INAM program by education planners from summer 2013 through 2015. The planners intentionally excluded some students who did not fit the program intent, were not intending on taking further courses, had poor grades, or had already taken other INAM courses. For employment and wage outcomes, this sample was restricted to only students in one of four chosen colleges who completed a certificate. The comparison group consisted of students enrolled in similar courses before the creation on INAM, including those in INAM courses but not in the INAM program. Furthermore, the comparison group for employment outcomes was limited to 549 students from four colleges where wage data was available.
Outcomes included completion rates, incumbent workers with wage increases, incumbent workers with no wage increases, wage declines after enrollment, non-incumbent workers with increased wages, non-incumbent workers with zero wages before and after college entrance, predicted wages in the twelfth quarter, and employment. Using data from college databases and the Illinois Department of Employment Security, the authors conducted statistical models to examine differences in outcomes between treatment and comparison groups.
Findings
Education and skills gain
- The study found that participation in the INAM program was significantly associated with improved rates of completion; students in the treatment group were 1.4 times as likely to complete a certificate or degree than students in the comparison group.
Earnings and wages
- The study found no significant relationship between participation in the INAM program and predicted wages in the twelfth quarter.
Employment
- The study found no significant relationship between participation in the INAM program and employment.
Considerations for Interpreting the Findings
When creating the treatment and comparison groups, education planners selectively approached students in INAM courses with information on enrolling in the program. Potentially inconsistent selection of students, as well as student self-selection into the INAM program, represents a confound. Additionally, the authors used students from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Lastly, the authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. The authors did not provide demographic information about the comparison group. These preexisting differences between the groups—and not the INAM program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the INAM program; other factors are likely to have contributed.