Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.
Citation
Schirm, A., Stuart, E., & McKie, A. (2006). The Quantum Opportunity Program demonstration: Final impacts. Washington, DC: Mathematica Policy Research.
Highlights
- The report’s objective was to evaluate the impact of the Quantum Opportunity Program (QOP)—which included case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives—on educational attainment and labor market participation nine years after program enrollment. Earlier reports presented impact findings over a shorter follow-up period.
- About 1,100 9th-grade students from 11 high schools were randomly assigned to the treatment group, which could enroll in QOP, or the control group, which could not participate in QOP activities. The study team administered a survey to treatment and control group members about nine years after program enrollment (six years after their scheduled graduation from high school) to collect information on education and labor market outcomes.
- The study reported no statistically significant impacts of participation in QOP on postsecondary educational attainment, likelihood of employment, or earnings for the full sample.
- The quality of the causal evidence presented in this report is high because it was a well-implemented randomized controlled trial. This means we are confident that estimated impacts would be solely attributable to QOP, and not other factors.
Intervention Examined
Quantum Opportunities Program (QOP)
Features of the Intervention
QOP was an intensive program composed of case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives. Services were provided year-round for up to five years to enrollees who had not yet graduated from high school. After high school graduation, participants received some continued mentoring and assistance applying to postsecondary education or training. According to the program model, case managers were to serve 15 to 25 students, and annual participation goals were 750 hours per enrollee who had not yet graduated from high school. For this evaluation, QOP was operated by seven community-based organizations, each affiliated with one to three high schools (11 high schools in total).
Eligibility requirements for students included being in 9th grade for the first time during the 1995–1996 academic year (except at one site, for which the relevant academic school year was 1996–1997) and being in the bottom two-thirds of the GPA distribution for their school in 8th grade. In addition, the students could not be so physically or learning disabled that, according to the school, the program was inappropriate for them.
Features of the Study
Students who met eligibility requirements were randomly selected to participate in the study. Those who consented (about 1,100) were randomly assigned to either the treatment or control group. The treatment group was allowed to participate in QOP activities, whereas the control group was not. Participants in both groups were surveyed approximately six years after their scheduled graduation from high school—when they entered their mid-20s—to collect information on their educational and labor market outcomes. High school transcripts for both groups were also collected.
Study Sites
- Cleveland, Ohio
- Fort Worth, Texas
- Houston, Texas
- Memphis, Tennessee
- Philadelphia, Pennsylvania
- Washington D.C.
- Yakima, Washington
Findings
QOP had no statistically significant impacts on postsecondary attainment, likelihood of employment, or earnings measured when study participants were entering their mid-20s.
Considerations for Interpreting the Findings
The success of QOP implementation varied across the seven sites and it was not particularly well implemented in any sites. Two sites “deviated substantially” from the QOP model and the other five “deviated moderately,” according to the study’s authors. Mentoring and developmental components were successfully implemented for the most part, but the education component was not effectively implemented and sites did not provide sufficient supportive services. Additionally, most enrollees participated in fewer activities than anticipated. These factors could help to explain the lack of statistically significant findings.
Causal Evidence Rating
The quality of causal evidence presented in this report is high because it is based on a well-conducted randomized controlled trial. This means we are confident that estimated impacts would be solely attributable to QOP, and not to other factors.