Absence of conflict of interest.
Citation
Yamada, H., Bohannon, A. X., Grunow, A., & Thorn, C. A. (2018). Assessing the effectiveness of Quantway®: A multilevel model with propensity score matching. Community College Review, 46(3), 257-287. doi: 10.1177/0091552118771754
Highlights
- The study’s objective was to examine the impact of the Quantway® 1 math program on community college students’ developmental math completion rates.
- The authors used a nonexperimental design to compare the outcomes of students who were in the Quantway® 1 program to a matched comparison group.
- When compared to non-participating students, the study found that Quantway® 1 program participation was associated with a significantly higher likelihood of successfully completing the developmental math course.
- The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Quantway® 1 program; other factors are likely to have contributed.
Intervention Examined
Quantway®
Features of the Intervention
The Quantway® program was developed as part of the Carnegie Foundation for the Advancement of Teaching to improve mathematics sequencing. Quantway® is intended for non-STEM students placed two levels below college math and focuses on quantitative literacy. The program includes an accelerated, single term developmental math course (Quantway® 1) as well as a college level math course (Quantway® 2). Quantway® 1 is organized around everyday themes (including citizenship, health care, and financial literacy), and utilizes real world examples in the curriculum as a way to engage students. The program also provides two levels of student supports. One is "productive persistence" which is designed to help students persist through complex coursework grounded in social psychology. The second is language and literacy supports in understanding math concepts and vocabulary. The program also includes a faculty development program with online resources, mentorships, and workshops as well as the ability for faculty to participate in collaborative learning community.
The program was first implemented in Spring 2012 in eight community colleges across three states (California, Connecticut, Florida, Minnesota, New York, Texas, and Washington). By 2016, the program was implemented in 14 colleges across eight states (Georgia, Minnesota, New Jersey, New York, Ohio, Washington, West Virginia, and Wisconsin).
Features of the Study
The study used a nonexperimental design to compare the outcomes of students who were in the Quantway® 1 program to those who were not. The authors received student demographic, course enrollment, and course performance data from each institution. The authors used a hierarchical linear model framework when creating propensity scores to match students on 37 characteristics. The sample included data from 10 colleges for students who enrolled in the course from Spring 2012 to Fall 2014, resulting in a sample of 4,243 Quantway® 1 students. The comparison group included 83,887 students at the same institutions during the timeframe who did not participate in the program. The sample size was reduced due to the availability of data for the full sample, and the final analytic sample was 3,992 Quantway® 1 students and 12,448 comparison students.
Study Sites
- Georgia
- Minnesota
- New Jersey
- New York
- Ohio
- Washington
- West Virginia
- Wisconsin
Findings
Education and skills gain
- The study found a significant relationship between program participation and course completion, where Quantway® 1 students were twice as likely to successfully complete the developmental math course than students in the comparison group.
Considerations for Interpreting the Findings
The authors created a matched group of non-participating students to compare to Quantway® 1 students. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not Quantway® 1—could explain the observed differences in outcomes.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to Quantway® 1; other factors are likely to have contributed.