Absence of conflict of interest.
Citation
Hong, Y., Boyette, J., & Staklis, S. (2017). RevUp: Empowering Montana's workforce student impact analysis study report. Berkeley, CA: RTI International.
Highlights
- The study’s objective was to examine the impact of Montana’s RevUp program on education outcomes.
- The authors used a nonexperimental design to compare education outcomes of RevUp participants to a matched comparison group.
- The study found that RevUp participation was significantly associated with lower college credit accumulation, higher RevUp course credit accumulation, lower program persistence, and higher dropout.
- The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to the RevUp program; other factors are likely to have contributed.
Intervention Examined
The RevUp Program
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2013, a consortium of 13 public, community and tribal colleges in Montana were awarded a TAACCCT grant to develop and implement the RevUp program. The RevUp program provided short-term credential programs and online training in energy, manufacturing, and other related fields so adult students could obtain the credentials, skills, and degrees that align with Montana’s workforce needs. The program also provided support services to students.
Features of the Study
The nonexperimental study was conducted at 10 of the colleges in the consortium. The authors matched RevUp participants to similar nonparticipants using propensity scores developed from demographic and academic information. Across the 10 colleges, study participants included 1,192 students. The treatment group included 771 students who were enrolled in the advanced manufacturing program between the fall term of 2014 and the summer/fall term of 2016. The comparison group was a historical cohort composed of 421 students who were enrolled in the advanced manufacturing program prior to the spring term of 2014. Using Montana’s statewide postsecondary data system, the authors conducted statistical models to examine differences in outcomes between the groups. Outcomes included degree/certificate completion, program persistence, and cumulative credits one year after RevUp program enrollment.
Study Sites
- Bitterroot College in Hamilton, Montana
- City College in Billings, Montana
- Dawson Community College in Glendive, Montana
- Flathead Valley Community College in Kalispell, Montana
- Gallatin College in Bozeman, Montana
- Great Falls College - Montana State University in Great Falls, Montana
- Helena College in Helena, Montana • Highlands College in Butte, Montana
- Missoula College in Missoula, Montana
- Montana State University – Northern in Havre, Montana
Findings
Education and skills gain
- The study did not find a significant relationship between participation in the RevUp program and degree/certificate completion.
- However, the study found that RevUp participants were significantly less likely to remain enrolled in the program by the end of the first year and were significantly more likely to drop out of the program without completing a degree or certificate relative to the comparison group.
- The study also found that RevUp participation was significantly associated with a lower number of accumulated college credits but a higher number of accumulated RevUp course credits one year after program enrollment.
Considerations for Interpreting the Findings
Despite using matching to ensure that students in the treatment and comparison groups were similar in terms of baseline demographic characteristics, the composition of the treatment and comparison groups significantly varied by gender. However, the authors did not control for this variable in the analyses. These preexisting differences between the groups—and not the RevUp program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to the RevUp program; other factors are likely to have contributed.