Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.
Citation
Rodríguez-Planas, N. (2012). Longer-term impacts of mentoring, educational services, and learning incentives: Evidence from a randomized trial in the United States. American Economic Journal: Applied Economics, 4(4):121–139.
Highlights
- The report’s objective was to evaluate the long-term impacts of the Quantum Opportunity Program (QOP)—which included case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives—on high school completion and postsecondary education.
- About 1,100 9th-grade students from 11 high schools were randomly assigned to the treatment group, which could enroll in QOP, or the control group, which could not participate in QOP activities. The study team administered a survey to treatment and control group members at three follow-up periods to collect information on postsecondary educational attainment and labor market outcomes.
- The report found few statistically significant impacts on the educational and employment-related outcomes examined.
- The quality of the causal evidence presented in this report for the analysis of the third follow-up period is high because it is based on a well-conducted randomized controlled trial. This means we are confident that any impacts observed would be attributable to the QOP, not to other factors. However, the quality of causal evidence for the analyses of the other two follow-up periods is low.
Intervention Examined
Quantum Opportunities Program (QOP)
Features of the Intervention
QOP was an intensive program composed of case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives. Services were provided year-round for up to five years to enrollees who had not yet graduated from high school. After high school graduation, participants received some continued mentoring and assistance applying to postsecondary education or training. According to the program model, case managers were to serve 15 to 25 students, and annual participation goals were 750 hours per enrollee who had not yet graduated from high school. For this evaluation, QOP was operated by seven community-based organizations, each affiliated with one to three high schools (11 high schools in total).
Eligibility requirements for students included being in 9th grade for the first time during the 1995–1996 academic year (except at one site, for which the relevant academic school year was 1996–1997) and being in the bottom two-thirds of the GPA distribution for their school in 8th grade. In addition, the students could not be so physically or learning disabled that, according to the school, the program was inappropriate for them.
Features of the Study
Students who met eligibility requirements were randomly selected to be invited into the study. Those who consented (about 1,100) were randomly assigned to either the treatment or control group. The treatment group was allowed to participate in QOP activities, whereas the control group was not. Participants in both groups were surveyed about one year after their scheduled graduation from high school—when they were entering their late teens—to collect information on their educational and labor market outcomes. High school transcripts for both groups were also collected.
Study Sites
- Cleveland, Ohio
- Fort Worth, Texas
- Houston, Texas
- Memphis, Tennessee
- Washington, D.C.
- Philadelphia, Pennsylvania
- Yakima, Washington
Findings
- The report found few statistically significant impacts on the educational and employment-related outcomes examined.
- The exceptions were that, at the time of the third follow-up, QOP students were 7 percentage points more likely to have completed two years of college or training (37 versus 30 percent) and 7 percentage points more likely to have ever enrolled in postsecondary education (63 versus 56 percent) than control group members.
Considerations for Interpreting the Findings
The analyses based on the third follow-up survey are based on a randomized controlled trial with low attrition and so received a high causal evidence rating. However, although based on the same trial, the study had high differential attrition across the study groups at the first and second follow-ups: the treatment group’s response rates to the surveys were substantially higher than the control group’s. This made the analyses based on these surveys ineligible to receive a high causal evidence rating. In addition, the analyses did not demonstrate equivalence of the two analytic groups on a measure of their financial disadvantage, which is required for studies reviewed in this topic area, nor did the analyses include a control for this characteristic. Therefore, the analyses of the first and second follow-ups could not receive a moderate causal evidence rating.
According to a related report (Schirm et al. 2006), the success of QOP implementation varied across the seven sites, and was not particularly well implemented in any sites. Two sites “deviated substantially” from the QOP model, whereas the other five “deviated moderately,” according to the study’s authors. Mentoring and developmental components were successfully implemented for the most part, but the education component was not effectively implemented and sites did not provide sufficient supportive services. Additionally, most enrollees participated in fewer activities than anticipated. These factors could help explain the lack of statistically significant findings.
Causal Evidence Rating
The quality of the causal evidence presented in this report for the analysis of the third follow-up survey is high because it was based on a well-conducted randomized controlled trial. This means we are confident that any impacts observed would be attributable to the QOP, not to other factors. However, the quality of causal evidence for the analyses of the other two surveys is low.