Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.
Citation
Schirm, A., Rodriguez-Planas, N., Maxfield, M., & Tuttle, C. (2003). The Quantum Opportunity Program Demonstration: Short-term impacts. Washington, DC: Mathematica Policy Research.
Highlights
- The report’s objective was to evaluate the impact of the Quantum Opportunity Program (QOP)—which includes case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives—on educational attainment and labor market participation approximately five years after program enrollment. Other studies examine longer-term impacts.
- About 1,100 9th-grade students from 11 high schools were randomly assigned to the treatment group, which could enroll in QOP, or the control group, which could not participate in QOP activities. The study team administered a survey to treatment and control group members about five years after program enrollment (one year after their scheduled graduation from high school) to collect information on high school and postsecondary educational attainment and labor market outcomes.
- QOP increased the likelihood that students earned a high school diploma one year after their scheduled graduation by 7 percentage points. No other employment or educational attainment outcomes were statistically significant.
- The quality of the causal evidence presented in this report is low. This means we do not have confidence that the estimated effects are attributable to QOP; other factors likely contributed.
Intervention Examined
Quantum Opportunities Program (QOP)
Features of the Intervention
QOP was an intensive program comprised of case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives. Services were provided year-round for up to five years to enrollees who had not yet graduated from high school. After high school graduation, participants received some continued mentoring and assistance applying to postsecondary education or training. According to the program model, case managers were to serve 15 to 25 students, and annual participation goals were 750 hours per enrollee who had not yet graduated from high school. For this evaluation, QOP was operated by seven community-based organizations, each affiliated with one to three high schools (11 high schools in total).
Eligibility requirements for students included being in 9th grade for the first time during the 1995-1996 academic year (except at one site, for which the relevant academic school year was 1996-1997) and being in the bottom two-thirds of the GPA distribution for their school in 8th grade. In addition, the students could not be so physically or learning disabled that, according to the school, the program was inappropriate for them.
Features of the Study
Students who met eligibility requirements were randomly selected to participate in the study. Those who consented (about 1,100) were randomly assigned to either the treatment or control group. The treatment group was allowed to participate in QOP activities, whereas the control group was not. Participants in both groups were surveyed about one year after their scheduled graduation from high school—when they were entering their late teens—to collect information on their educational and labor market outcomes. High school transcripts for both groups were also collected.
Study Sites
- Cleveland, Ohio
- Fort Worth, Texas
- Houston, Texas
- Memphis, Tennessee
- Philadelphia, Pennsylvania
- Washington D.C.
- Yakima, Washington
Findings
- QOP increased the likelihood that students earned a diploma or a general educational development (GED) diploma by 7 percentage points.
- No other employment or educational attainment outcomes were statistically significant using regression-adjusted impact estimates.
Considerations for Interpreting the Findings
Although it was based on a randomized controlled trial, the study had high differential attrition across the study groups: the treatment group’s response rate to the follow-up survey was substantially higher than was the control group’s. This made the study ineligible to receive a high causal evidence rating. In addition, the analysis did not demonstrate equivalence of the two groups on a measure of their financial disadvantage, which is required for studies reviewed in this topic area, nor did it include a control for this characteristic. Therefore, the study could not receive a moderate causal evidence rating.
Causal Evidence Rating
The quality of the causal evidence presented in this report is low. This means we do not have confidence that the estimated effects are attributable to QOP; other factors likely contributed.