Absence of conflict of interest.
Citation
Feldman, J., Staklis, S., Hong, Y., & Elrahman, J. (2016). Evaluation report of the Amplifying Montana's Advanced Manufacturing and Innovation Industry (AMAMII) project: Final report. Berkeley, CA: RTI International.
Highlights
- The study’s objective was to examine the impact of the Amplifying Montana’s Advanced Manufacturing and Innovation and Industry (AMAMII) program on education outcomes.
- The authors used a nonexperimental design to compare the outcomes of students participating in AMAMII to a matched comparison group using institutional research data.
- The study found a significant negative relationship between program participation and credit accumulation, with AMAMII students accumulating fewer credits than comparison students.
- The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analyses. This means we are not confident that the estimated effects are attributable to the AMAMII program; other factors are likely to have contributed.
Intervention Examined
The Amplifying Montana’s Advanced Manufacturing and Innovation and Industry (AMAMII) Program
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories. In 2012, Flathead Valley Community College (FVCC) was awarded a TAACCT grant to support the implementation and evaluation of its Amplifying Montana’s Advanced Manufacturing and Innovation and Industry (AMAMII) program. The AMAMII program offers certificate programs and curriculum in advanced manufacturing (developed using the input from local employers and business partners) to ensure students (especially low-skilled) obtain the required skills and certifications for high-demand, local jobs in manufacturing. The program utilizes Workforce Navigators (WFNs) who are dedicated to recruiting eligible program participants, advising, and coaching enrolled students on academic and career needs, and providing job development and placement assistance when needed. A key component of the program is that it includes a revamped foundational math sequence that is self-paced so students can work independently; these courses involve heavy use of computer-based exercises. Moreover, a dedicated math lab was established and staffed for students enrolled in the AMAMII program as the program requires students to spend time completing assignments in the math lab every week.
Features of the Study
The study was a nonexperimental design conducted at Flathead Valley Community College (FVCC) in Kalispell, Montana. The authors matched AMAMII program participants to similar nonparticipants using propensity scores developed from several demographic and pre-program enrollment characteristics. The treatment group was comprised of students enrolled in the AMAMII program’s remedial math sequence starting in fall 2013, when the program was fully implemented. The comparison group included students who were enrolled in foundational math courses prior to fall 2013. To ensure each enrolled student had data for two full years after program enrollment and to avoid treatment crossover, the authors excluded students enrolled between the winter 2012 and summer 2013 terms, as well as those enrolled after the winter 2015 term. The analysis sample included 1,944 students who enrolled in FVCC from fall 2009 to spring 2016, with 611 students in the AMAMII treatment group and 1,333 students in the matched comparison group. Using the college’s institutional research data, the authors conducted statistical models to examine differences in education outcomes between the groups. Education outcomes included foundational math course completion, repetition of one or more foundational math courses, taking a college-level math course, passing a college-level math course, two-year credit accumulation, and degree completion two years after program enrollment.
Findings
Education and skills gain
- The study found a significant negative relationship between program participation and credit accumulation with AMAMII students, accumulating fewer credits over the two-year period compared to students in the comparison group.
- The study did not find statistically significant relationships between AMAMII program participation and the other educational outcomes.
Considerations for Interpreting the Findings
Although the authors accounted for baseline characteristics and outcomes, they did not control for race/ethnicity in their analytic model which is required by the review protocol. The preexisting differences between the groups on this variable—and not AMAMII—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analyses. This means we are not confident that the estimated effects are attributable to the AMAMII program; other factors are likely to have contributed.