Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.
Citation
Horwood, T., Usher, K.., McKinney, M., & Passa, K. (2017). Maine is IT! program evaluation final report. Fairfax, VA: ICF.
Highlights
- The study’s objective was to evaluate the effects of the Maine is IT! program on education outcomes.
- The authors used a nonexperimental design to compare the outcomes of students enrolled in the Maine is IT! program to a matched comparison group of students enrolled at the same college and at the same point of entry in business administration programs of study. Using student tracking data, the authors conducted statistical models to examine differences between the groups.
- The study found that compared to non-participating students, students enrolled in the Maine is IT! program were significantly more likely to complete credit hours, be retained in their program of study, and earn a degree.
- The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Maine is IT! program; other factors are likely to have contributed.
Intervention Examined
Maine is IT! Program
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
A seven college consortium located in the state of Maine was awarded TAACCCT grant funds to develop the Maine is IT! program. The program was designed to address employment needs in the information technology (IT) field by improving access to IT training opportunities. The Maine is IT! program was comprised of 12 degree programs, 9 certificate programs, and 4 advanced certificate programs. Maine is IT! implemented several evidence-based strategies: 1) creating and enhancing industry recognized certifications, 2) increasing the number and type of stackable credentials, 3) building bridges between non-credit and credit courses, 4) standardizing and expanding credit for prior learning, and 5) introducing innovative approaches such as online and technology enabled competency-based learning strategies and accelerated time to completion strategies.
Features of the Study
The nonexperimental study took place across seven community colleges in Maine. The authors matched students enrolled in the Maine is IT! program to similar non-participating students using demographic and enrollment characteristics. The treatment group consisted of 812 students that were enrolled in the Maine is IT! program between Fall 2014 and Spring 2017 and received grant funded services. The comparison group consisted of 812 students who were enrolled in Business Administration programs of study at the same colleges, during the same entry point between Fall 2014-Spring 2017, but did not receive grant funded services. Using student tracking and longitudinal administrative data, the authors conducted statistical models to compare the groups on program retention, credit hour completion, and degree/credential attainment.
Study Sites
- Central Maine Community College (CMCC) in Auburn, Maine
- Eastern Maine Community College (EMCC) in Bangor, Maine
- Kennebec Valley Community College (KVCC) in Fairfield, Maine
- Northern Maine Community College (NMCC) in Presque Isle, Maine
- Southern Maine Community College (SMCC) in South Portland, Maine
- Washington County Community College (WCCC) in Calais, Maine
- York County Community College (YCCC) in Wells, Maine
Findings
Education and skills gain
- The study found that students in the Maine is IT! program were significantly more likely than comparison group students to be retained in their program of study (1.34 times more likely).
- The study also found that program participation was significantly related to higher odds of credit hour completion, where treatment participants were two times more likely to complete credit hours than participants in the comparison group.
- The study found that participation in the Maine is IT! program was significantly associated with an increased likelihood of degree attainment, with higher proportions of treatment students earning a degree relative to the comparison group (18.5% vs. 13.5%).
- However, the study did not find a significant relationship between Maine is IT! participation and credential attainment.
Considerations for Interpreting the Findings
Although the authors created a matched group of non-participating students to compare to the Maine is IT! students, they did not account for other factors that could have affected the differences between the treatment and comparison groups, such as a pre-intervention measure of financial disadvantage. These preexisting differences between the groups—and not the Maine is IT! program—could explain the observed differences in outcomes. Additionally, the authors used a comparison group composed of a concurrent cohort of students that attended the same colleges, but were enrolled in different programs of study. The program varying characteristics of the comparison groups creates a confound. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Maine is IT! program; other factors are likely to have contributed.