Absence of conflict of interest.
Citation
Highlights
- The study's objective was to conduct a formative evaluation of the Advanced Manufacturing Education (AME) Alliance to determine the extent to which the project was implemented as intended on education and training needs in Minnesota.
- The study authors conducted an implementation evaluation using both quantitative and qualitative data from a variety of sources that included surveys, interviews, and extant data from program staff, participants, and other stakeholders (e.g., partners/employers), as well as project records.
- The study found that all five key strategies outlined by the AME Alliance project were largely implemented across the Consortium of colleges.
- The AME Alliance implementation study had a comprehensive study design with multiple data sources and clear research questions. However, the study suffered from low survey response rates and unrealistic projected outcomes. It also lacked detailed information about the sample or the selection criteria for focus groups and interviews.
- The embedded impact study was reviewed by CLEAR in May of 2020.
Intervention Examined
The Advanced Manufacturing Education (AME) Alliance
Features of the Intervention
- Type of organization: Community and Technical Colleges
- Location/setting: Multi-site in Minnesota
- Population served and scale: Adults; Low-skilled, trade impacted and other dislocated workers; Long-term unemployed veterans; 2,163 participants served
- Industry focus: Manufacturing
- Intervention activities: Career Pathways; Student support services; Technology
- Organizational partnerships: Employers; Community Stakeholders
- Cost: Not included
- Fidelity Assessment: Included
The AME Alliance was established in September 2012 when three community colleges in Minnesota formed a strategic partnership with Minnesota's 360 Center for Manufacturing Excellence to serve the manufacturing industry's education and training needs in the state of Minnesota. The project was funded by a four-year grant from the Round 2 Trade Adjustment Assistance Community College and Career Training (TAACCCT) program sponsored by the United States Department of Labor (USDOL). The intervention aimed to serve trade-impacted and other dislocated workers, as well as long-term unemployed, veterans, and other adults who were seeking employment in the manufacturing industry. The primary goal of the AME Alliance sought to provide education and training so individuals could gain the necessary skills and knowledge to be successful in the manufacturing field. The program focused on the areas of automation/robotics, plastic technology, precision manufacturing, prototyping, and reverse engineering. In order to address the challenge of program completion and securing employment, the AME Alliance used a career pathways framework and implemented five key strategies. These strategies included: 1) a technology-enriched environment, 2) a hybridized and modularized curriculum, 3) comprehensive student support services, 4) marketing activities and program outreach, and 5) employer and workforce partner collaborations. The total number of participants served was 2,163.
The study used a logic model of the career pathways framework to conceptualize the summative and formative evaluation designs. The logic model described the connections between required resources, strategies to support student success, expected outputs as a result of the strategies, the mediators, and the main outcomes of interest. Resources included project staff, data coordinators, instructors, curriculum developers, and industry and workforce partners. The outputs were the participants and implemented strategies that adhered to the project work plan. The intermediate outcomes included academic integration, advising effectiveness, academic support, social-emotional support, and career integration. Finally, the study outcome measures consisted of program completion rates, further education enrollment rates, grade point average (GPA), employment rates, and percentage of participants receiving a wage increase.
Features of the Study
The AME Alliance project was implemented across three community college sites in Minnesota: Central Lakes College (CLC), Pine Technical and Community College (PTCC), and Saint Cloud Technical and Community College (SCTCC). A mixed-methods approach was used to collect qualitative and quantitative data through several data sources. Data collection included surveys, interviews, and extant data from program staff, participants, and other stakeholders (e.g., partners/employers), as well as project records. Extant data included meeting notes and minutes, quarterly reports, student support service logs, participant data (e.g., demographic information, educational records, employment history, wage data), and project records. All data was reviewed for data entry errors and improbable responses prior to data analysis. Data analysis included descriptive statistics such as frequencies, percentages and means for students, and partner surveys. Qualitative data were analyzed using computer software to identify prevalent themes and emerging issues.
Fidelity was assessed using an assessment tool developed by the project evaluators that examined three aspects of project implementation: adherence, quality, and participant responsiveness. The adherence indicator focused on the structural aspect of implementation, and the quality and participant responsiveness indicators focused on the procedural aspects of implementation. All three indicators were rated on a five-point scale from zero to four. The five program strategies were rated for each indicator. All individual ratings were added together to obtain a sum score and produce the project's implementation status.
Findings
Intervention activities/services
- The study found that each participant had an average of 3.75 meetings with their advisor and the advisors completed 7.66 contacts or outreach activities with each participant. Individualized learning plans were also completed for 95 percent of the grant participants enrolled in the first three years.
- The study found that collaboration with industry partners was essential to the success of the grant activities. The AME program had 18 partners when the project was first funded, but by the end of the grant period the partnerships grew to 154. The partnerships contributed to curriculum development, participant recruitment, leveraging resources, and sustainability planning.
- The study found that students who participated in the focus groups thought the most valuable part of the program was having high quality instructors and being able to complete coursework in a technologically advanced learning environment. Project staff noted that the AME Alliance gave their institutions the opportunity to greatly enhance their curricula in the manufacturing field.
Fidelity
- The study found that all five key strategies outlined by the AME Alliance project were largely implemented across the consortium of colleges. Additionally, the study found that the grant program developed a total of 23 certificate programs, 25 diploma programs, and 16 degree programs across the four program tracks.
- The study found that by the end of the grant Year 3, the AME Alliance program was at the 97th percentile for adherence to full implementation.
- The study found that the AME Alliance strategies were implemented with high quality and that participants were overall satisfied with the project implementation, especially with hybridized and modularized curricula and wraparound student support services.
Considerations for Interpreting the Findings
The strengths of the study design included a mixed-methods approach with multiple data sources, clear research questions, and guidelines on how to use the data to support the research questions. Study findings were also well-categorized and easily interpretable and data collection quality assurance procedures were utilized. The main weaknesses for the study as noted by the author included low survey response rates from participants and partners and unrealistic projected outcomes. Additionally, the study failed to provide detailed information about the sample or the selection criteria for focus groups and interviews. Fidelity was assessed by the study authors and not by the CLEAR team.