Citation
Bettinger, E., & Baker, R., (2011). The effects of student coaching in college: An evaluation of a randomized experiment in student mentoring. Stanford, CA: Stanford University School of Education.
Highlights
- The study’s objective was to evaluate the impact of InsideTrack student coaching on students’ persistence in college and degree completion.
- The study was a randomized controlled trial carried out with 17 different student cohorts from eight participating colleges, universities, and technical schools. For this review, CLEAR distinguished between sites with a 50/50 probability of assignment to coaching and sites with other probabilities of assignment. The authors assessed the program’s impact on retention rates using academic records for all students who participated in the study.
- The study found that on average, both in sites with a 50/50 probability of assignment and all sites combined, students participating in InsideTrack demonstrated increased persistence compared with control group students receiving only the regular suite of career or academic support services.
- The quality of causal evidence presented in this report for sites with a 50/50 probability of assignment is high because it was based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects for these sites are attributable to InsideTrack and not to other factors. The quality of causal evidence presented in this report for all sites combined is low because randomization was compromised and the analysis did not include sufficient controls. This means we are not confident that the estimated effects for the full sample are attributable to InsideTrack. Other factors are likely to have contributed.
Intervention Examined
InsideTrack Student Coaching
Features of the Intervention
InsideTrack is an independent provider of voluntary student coaching services for institutions of higher education. InsideTrack matches students to potential coaches, who work to help students prioritize their studies, plan for academic success, and identify and overcome barriers to academic success. A coach typically worked with a student for two semesters, and spent significant time assessing students’ lives outside of school in such areas as personal time commitments, primary caregiving responsibilities, and financial obligations. In addition to regular contact, coaches sometimes had access to course information and students’ performance in their specific courses. This information was used in an algorithm that directed coaches to specific issues to address. The ultimate goal of InsideTrack was to improve college retention and completion.
Each participating institution established its own eligibility criteria for study inclusion independent of the service provider, IndsideTrack. Different institutions restricted eligibility in different ways, some focusing on students in later years of study, some on new entrants, and some on full- or part-time students. Older and nontraditional students were more heavily represented in the study sample than in the institutions’ general student population, with only about 25 percent of participants younger than 23.
Features of the Study
Across eight universities, 13,555 students were randomly assigned to the treatment or control groups and considered in the reported analysis. The authors conducted regression analyses controlling for gender, age, high school grade point average, American College Test scores, Scholastic Assessment Test scores, residential location, Pell grant status, merit scholarship status, remediation in math or English, and school type.
Findings
- The study found that, on average among students in sites with a 50/50 probability of assignment, InsideTrack significantly increased persistence in college after 6, 12, and 18 months by 4, 5, and 7 percentage points, respectively.
- There was no statistically significant effect on degree completion.
Considerations for Interpreting the Findings
Although the study was designed and implemented as a randomized controlled trial, the authors reassigned some students in sites with non-50/50 probabilities of assignment, jeopardizing the study design. Even though the authors reassigned very few students and did so to improve balance on observable characteristics between the treatment and control groups, CLEAR considers studies with any post-random assignment manipulation of the research groups as nonexperimental designs. Because the authors neither established baseline equivalence nor controlled for all relevant background characteristics in their analysis, these sites receive a low evidence rating.
Across sites, the study reported only intent-to-treat findings, which can be interpreted as the impact of being offered the treatment. Participation was optional, so assignment to coaching did not guarantee participation. As a result, the program’s impact on students who actually participated might be higher than the study’s estimates.
Causal Evidence Rating
The quality of causal evidence presented in this report for sites with a 50/50 probability of assignment is high because it was based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects for these sites are attributable to InsideTrack and not to other factors. The quality of causal evidence presented in this report for all sites combined is low because randomization was compromised and the analysis did not include sufficient controls. This means we are not confident that the estimated effects for the full sample are attributable to InsideTrack. Other factors are likely to have contributed.