Body,Title,"Study Type",Citation,AdditionalSources,"Original Publication Date","Original Publication URL","Reviewed by CLEAR","Topic Area","Subtopic Area","Study Evidence Rating","Causal Design",Conclusions,Body,"Features of Intervention","Features of the Study",Findings,"Considerations for Interpreting the Findings","Types of and Outcomes","Study Sites",ProfileFootnotes,"Conflict of Interest","Firm Characteristics","Geographic Setting","Study Type","Target Population","Topic Tags","Domain Effectiveness",Outcome,"Review Protocol"
"The Quantum Opportunity Program Demonstration: Initial post-intervention impacts (Schirm & Rodriguez-Planas 2004)","The Quantum Opportunity Program Demonstration: Initial post-intervention impacts (Schirm & Rodriguez-Planas 2004)","Study Type: Causal Impact Analysis","Schirm, A., & Rodriguez-Planas, N. (2004). The Quantum Opportunity Program Demonstration: Initial post-intervention impacts. Washington, DC: Mathematica Policy Research.",,2004,http://www.doleta.gov/reports/searcheta/occ/papers/June_04_QOP_report.pdf,"September 2014","Opportunities for Youth",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The report’s objective was to evaluate the impact of the Quantum Opportunity Program (QOP)—which includes case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives—on educational attainment and labor market participation approximately seven years after program enrollment. Other reports presented impact findings from different time periods.
About 1,100 9th-grade students from 11 high schools were randomly assigned to the treatment group, which could enroll in QOP, or to the control group, which could not participate in QOP activities. The study team administered a survey to treatment and control group members about seven years after program enrollment (three years after their scheduled graduation from high school) to collect information on postsecondary educational attainment and labor market outcomes.
Participation in QOP had no statistically significant impact on most measures of postsecondary attainment and employment. The exceptions were that participation increased enrollment in college or vocational/technical school, an apprenticeship, or the military by 10 percentage points. It decreased the likelihood of working at least 35 hours at a main job by 11 percentage points.
The quality of the causal evidence presented in this report is low. This means we do not have confidence that the estimated effects are attributable to QOP; other factors likely contributed.","QOP was an intensive program composed of case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives. Services were provided year-round for up to five years to enrollees who had not yet graduated from high school. After high school graduation, participants received some continued mentoring and assistance applying to postsecondary education or training. According to the program model, case managers were to serve 15 to 25 students, and annual participation goals were 750 hours per enrollee who had not yet graduated from high school. For this evaluation, QOP was operated by seven community-based organizations, each affiliated with one to three high schools (11 high schools in total).
Eligibility requirements for students included being in 9th grade for the first time during the 1995–1996 academic year (except at one site, for which the relevant academic school year was 1996–1997) and being in the bottom two-thirds of the grade point average distribution for their school in 8th grade. In addition, the students could not be so physically or learning disabled that, according to the school, the program was inappropriate for them.","Students who met eligibility requirements were randomly selected to participate in the study. Those who consented (about 1,100) were randomly assigned to either the treatment or control group. The treatment group was allowed to participate in QOP activities, whereas the control group was not. Participants in both groups were surveyed about three years after their scheduled graduation from high school—when they were entering their early 20s—to collect information on their educational and labor market outcomes. High school transcripts for both groups were also collected.","Participation in QOP had no impact on high school completion or most measures of postsecondary attainment and employment.
Participation in QOP increased enrollment in college, vocational/technical school, an apprenticeship, or the military by 10 percentage points, a statistically significant difference. Participation in QOP also increased the likelihood of ever attending a two- or four-year college by 8 percentage points.
QOP participation decreased the likelihood of working at least 35 hours at a main job by 11 percentage points, a statistically significant difference.","Although it was based on a randomized controlled trial, the study had high differential attrition across the study groups: the treatment group’s response rate to the follow-up survey was substantially higher than the control group’s. This made the study ineligible to receive a high causal evidence rating. In addition, the analysis did not demonstrate equivalence of the two groups on a measure of their financial disadvantage, which is required for studies reviewed in this topic area, nor did it include a control for this characteristic. Therefore, the study could not receive a moderate causal evidence rating.",,"Cleveland, Ohio
Fort Worth, Texas
Houston, Texas
Memphis, Tennessee
Philadelphia, Pennsylvania
Washington D.C.
Yakima, Washington",,"Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.",,"United States","Causal Impact Analysis",Youth,"Mentoring, Other training and education, Youth programs","Education and skills gains-Low-Favorable impacts, Employment-Low-Unfavorable impacts","Education and skills gains, Employment","Opportunities for Youth Review Protocol"
"The Quantum Opportunity Program Demonstration: Short-term impacts (Schirm et al. 2003)","The Quantum Opportunity Program Demonstration: Short-term impacts (Schirm et al. 2003)","Study Type: Causal Impact Analysis","Schirm, A., Rodriguez-Planas, N., Maxfield, M., & Tuttle, C. (2003). The Quantum Opportunity Program Demonstration: Short-term impacts. Washington, DC: Mathematica Policy Research.",,2003,http://www.doleta.gov/reports/papers/QOP_short_term_impacts.pdf,"September 2014","Opportunities for Youth",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The report’s objective was to evaluate the impact of the Quantum Opportunity Program (QOP)—which includes case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives—on educational attainment and labor market participation approximately five years after program enrollment. Other studies examine longer-term impacts.
About 1,100 9th-grade students from 11 high schools were randomly assigned to the treatment group, which could enroll in QOP, or the control group, which could not participate in QOP activities. The study team administered a survey to treatment and control group members about five years after program enrollment (one year after their scheduled graduation from high school) to collect information on high school and postsecondary educational attainment and labor market outcomes.
QOP increased the likelihood that students earned a high school diploma one year after their scheduled graduation by 7 percentage points. No other employment or educational attainment outcomes were statistically significant.
The quality of the causal evidence presented in this report is low. This means we do not have confidence that the estimated effects are attributable to QOP; other factors likely contributed.","QOP was an intensive program comprised of case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives. Services were provided year-round for up to five years to enrollees who had not yet graduated from high school. After high school graduation, participants received some continued mentoring and assistance applying to postsecondary education or training. According to the program model, case managers were to serve 15 to 25 students, and annual participation goals were 750 hours per enrollee who had not yet graduated from high school. For this evaluation, QOP was operated by seven community-based organizations, each affiliated with one to three high schools (11 high schools in total).
Eligibility requirements for students included being in 9th grade for the first time during the 1995-1996 academic year (except at one site, for which the relevant academic school year was 1996-1997) and being in the bottom two-thirds of the GPA distribution for their school in 8th grade. In addition, the students could not be so physically or learning disabled that, according to the school, the program was inappropriate for them.","Students who met eligibility requirements were randomly selected to participate in the study. Those who consented (about 1,100) were randomly assigned to either the treatment or control group. The treatment group was allowed to participate in QOP activities, whereas the control group was not. Participants in both groups were surveyed about one year after their scheduled graduation from high school—when they were entering their late teens—to collect information on their educational and labor market outcomes. High school transcripts for both groups were also collected.","QOP increased the likelihood that students earned a diploma or a general educational development (GED) diploma by 7 percentage points.
No other employment or educational attainment outcomes were statistically significant using regression-adjusted impact estimates.","Although it was based on a randomized controlled trial, the study had high differential attrition across the study groups: the treatment group’s response rate to the follow-up survey was substantially higher than was the control group’s. This made the study ineligible to receive a high causal evidence rating. In addition, the analysis did not demonstrate equivalence of the two groups on a measure of their financial disadvantage, which is required for studies reviewed in this topic area, nor did it include a control for this characteristic. Therefore, the study could not receive a moderate causal evidence rating.",,"Cleveland, Ohio 
Fort Worth, Texas 
Houston, Texas 
Memphis, Tennessee 
Philadelphia, Pennsylvania
Washington D.C.
Yakima, Washington",,"Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.",,"United States","Causal Impact Analysis",Youth,"Mentoring, Other training and education, Youth programs","Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Education and skills gains, Employment","Opportunities for Youth Review Protocol"
"A promising start: Initial impacts of Year Up on low-income young adults' careers (Roder & Elliot, 2011)","A promising start: Initial impacts of Year Up on low-income young adults' careers (Roder & Elliot, 2011)","Study Type: Causal Impact Analysis","Roder, A., & Elliott, M. (2011). A promising start: Initial impacts of Year Up on low-income young adults' careers. Economic Mobility Corporation.",,2011,,"April 2014","Opportunities for Youth",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"This report presents preliminary findings for the impacts of the Year Up program, which provides technical skills training and internships to low-income adults ages 18 to 24.
The authors randomly assigned eligible applicants to either be invited to participate in the program immediately (treatment group) or gain entry after 10 months (control group). They compared the outcomes of treatment and control group members for approximately two years after random assignment.
The authors reported many statistically significant and positive impacts of the program in the second year after random assignment. These included impacts on annual earnings and hourly wages.
The quality of causal evidence is low because the randomized controlled trial had high attrition and the study did not show that the groups being compared were equivalent prior to the program. This means we are not confident that the estimated effects are attributable to Year Up. Other factors are likely to have contributed.","Year Up is a nonprofit organization headquartered in Boston, Massachusetts. It was founded by a former software entrepreneur with the goal of providing a year of training and work experience to urban young adults. Participants in the Year Up program must be low-income and ages 18 to 24. A chief feature of the program model is six months of technical skills training in information technology and investment operations fields followed by a six-month internship with a top company in the region. The program also includes classes in business writing, communication, and professional skills training. Participants receive a weekly stipend and the support of social workers and mentors throughout the program.","In summer 2007, eligible Year Up applicants were randomly assigned to either be invited to participate in the program immediately (treatment group) or be placed on a waiting list to gain entry to the program after 10 months (control group). In total, 135 applicants were assigned to the treatment group and 60 to the control group. A follow-up survey was administered to both groups to capture post-program earnings and employment outcomes; the last survey was administered 24 to 30 months after random assignment.","During the first year after random assignment, the Year Up group’s earnings lagged those of control group members because most members of the Year Up group were participating in program-related activities. However, by two years after random assignment, Year Up participants were earning more than control group members by a statistically significant margin ($15,082 versus $11,621).
During the second year after random assignment, Year Up and control group members had similar levels of employment, but the Year Up group held higher-paying jobs, on average, than the control group ($12.58 per hour versus $10.32 per hour). 
The two groups were equally likely to be attending college two years after random assignment.","Although this study was based on a randomized controlled trial, which can provide the highest possible causal evidence if well-implemented, the trial suffered from high differential attrition across the study groups. In other words, outcomes were available for a larger proportion of Year Up members than control group members. This could bias the estimated effects of the program if characteristics of the control group are systematically related to responding to the follow-up survey. Therefore, this study cannot receive a high causal evidence rating.
In addition, the study did not demonstrate that the groups being compared were equivalent at the time of program application, nor was it clear from the study whether the authors included controls for baseline characteristics in their analysis approach. Attempts to reach the authors to clarify these issues were unsuccessful. Therefore, this study cannot receive a moderate causal evidence rating in the absence of additional information from the authors.",,"Year Up sites in three cities participated in the evaluation: 
Boston, Massachusetts 
New York City
Providence, Rhode Island",,,"Finance and insurance","United States, Urban","Causal Impact Analysis","Youth, Low income","Mentoring, Work based and other occupational training, Youth programs","Earnings and wages-Low-Mixed impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Opportunities for Youth Review Protocol"
"Youth Opportunity Grant Initiative: Impact and synthesis report (Jackson et al. 2007)","Youth Opportunity Grant Initiative: Impact and synthesis report (Jackson et al. 2007)","Study Type: Causal Impact Analysis","Jackson, R., Dixon, R., McCoy, A., Pistorino, C., Zador, P., Thomas, C., … Bruno, L. (2007). Youth Opportunity Grant Initiative: Impact and synthesis report. Houston, TX: Decision Information Resources, Inc.",,2007,http://wdr.doleta.gov/research/FullText_Documents/YO%20Impact%20and%20Synthesis…,"October 2013","Opportunities for Youth",,"Low Causal Evidence",Nonexperimental,,"This report’s objective was to examine the effectiveness of the Youth Opportunity (YO) Grant Initiative, which sought to improve job prospects and education outcomes for youth in high-poverty areas by providing them with a mix of 15 youth development activities. Companion reports describe the initiative’s implementation.
The initiative’s effectiveness was evaluated using two separate comparison group designs.
In general, the study found positive and statistically significant relationships between the YO initiative and educational enrollment outcomes. However, there were few statistically significant effects on employment-related outcomes and educational attainment.
The quality of the causal evidence presented in this study is low. This means we are not confident that the estimated impacts were caused by the YO Grant Initiative, and not other factors.","The YO Grant Initiative operated from 2000 until 2005. Thirty-six communities (24 urban, 6 rural, 6 Native American) provided comprehensive services to more than 92,000 economically disadvantaged youth, ages 14 to 21. Services were available regardless of income, but the program’s goal was to target youth programming in areas with high levels of joblessness and poverty, especially among out-of-school youth, and to serve as a foundation for community-wide efforts to mobilize resources for supporting youth to enter the economic mainstream.
The 36 grantees, selected by a competitive application process, provided services to both in-school and out-of-school youth in the community. The program’s core activities included job readiness training, internships or subsidized employment, short-term occupational skills training, short-term unsubsidized jobs, and GED preparation. Most youth participated in at least one core service as well as other services designed to support them, such as sports and recreation. On average, participants spent 563 hours participating in program activities at the study sites.","The study used two different nonexperimental designs to examine the effectiveness of YO. The first design was a census tract-level analysis using data from the American Community Survey (ACS) to form a comparison group. The second design was an individual-level analysis using data drawn from the Current Population Survey (CPS).
For the analysis using the ACS data, the authors first grouped 29 non-Native American YO sites and comparison census tracts into bins based on their similarity in terms of their population, percentage of home ownership, median contract rent, percentage of vacant housing units, percentage of whites in the population, percentage of rural population, and labor force participation rate. The authors then estimated YO’s impact by comparing the changes in employment and education outcomes from baseline (2001) to follow-up (2003–2004) for the YO and comparison communities within each bin. These were combined across bins to calculate an overall estimate.
For the analysis using the CPS data, the YO group consisted of all youth ages 16 to 21 who lived in the target areas of 23 of the 24 urban YO sites, because they were all eligible to participate in YO. The comparison group consisted of similarly aged youth in high-poverty (defined as poverty above 20 percent according to the 1990 Census) central-city neighborhoods. The authors estimated YO’s impact by comparing the changes in employment and education outcomes between April 2000 and March 2004 for youth in the YO and comparison communities.","The study examined 11 employment-related outcomes across the two designs. Of these, the study found a statistically significant positive relationship between living in a YO target area and the labor force participation rate and percentage of those employed working full time in the CPS analysis. There were no statistically significant differences between the groups on the other 9 employment-related outcomes examined across the two studies.
The study examined 5 educational attainment outcomes in the ACS analysis. Of these, the study found a statistically significant positive relationship between participation in the YO program and having completed 11th grade.
The authors examined 8 educational enrollment outcomes across the two designs. The authors found that participation in YO was associated with a statistically significant reduction in the percentage of youth not in school and an increase in the percentage of youth in secondary school (ACS analysis). They also found that living in a YO target area was associated with a statistically significant reduction in high school dropout spells (CPS analysis).
The authors also examined program impacts by gender, race/ethnicity, and nativity. Of note, the study found statistically significant increases in the labor force participation rates and mean hourly wages of women associated with living in a YO target area.","Although the ACS analysis attempted to match YO communities to other similar communities, differences in labor force participation rates remained and were not controlled for in the analysis. This could have been mitigated if there were similar trends in those rates before the study period, but this was not demonstrated. Also, although participants spent an average of more than 500 hours participating in YO programs, the total time spent varied widely among grantees and among individuals within grantees.
Similarly, the high-poverty communities selected to be part of the CPS analysis had very different racial compositions from the YO target areas. In addition, for this analysis there was no indication that YO target areas and comparison areas had similar levels or trends in employment and education-related outcomes at the start of the study period. These factors indicate that the comparison group does not provide a valid representation of what would have happened to the YO group in the absence of YO. Finally, the effects of YO may be diluted because youth who did not participate in YO but lived in YO treatment areas were included in the treatment group.",,"Albany, GA*
Baltimore, MD
Birmingham, AL
Boston, MA
Brawley/Calipatria, CA*
Brockton, MA
Buffalo, NY
Chicot/Desha County, AR*
Cleveland, OH
Denver, CO
Detroit, MI
Hartford, CT
Houston, TX
Kansas City, MO
Los Angeles, CA
Louisville, KY
Maui & Molokai, HI*
Memphis, TN
Milwaukee, WI
Monroe, LA*
Philadelphia, PA
Portland, OR
Robeson County, NC*
San Antonio, TX
San Diego, CA
San Francisco, CA
Seattle, WA
Tampa, FL
Tucson, AZ
* indicates rural sites",,,,"United States","Causal Impact Analysis","Youth, Low-skilled, Low income","Basic skills, Community college education and other classroom training, Job search assistance and supportive services, Other training and education, Work based and other occupational training, Youth programs","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Opportunities for Youth Review Protocol"
"The relative impact of a Career Academy on post-secondary work and education skills in urban, public high schools (Maxwell & Rubin 1997)","The relative impact of a Career Academy on post-secondary work and education skills in urban, public high schools (Maxwell & Rubin 1997)","Study Type: Causal Impact Analysis","Maxwell, N., & Rubin, V. (1997). The relative impact of a Career Academy on post-secondary work and education skills in urban, public high schools. Hayward, CA: The Human Investment Research and Education Center discussion paper.",,1997,,"June 2014","Career Academies",,"Low Causal Evidence","Quasi-Experimental Design (QED)",,"The study aimed to examine the academic and labor market outcomes of students who were enrolled in a Career Academy track in a large, West Coast, urban high school district, compared with peers enrolled in general, vocational, and academic track programs.
The analysis is based on a 1996 survey of students who were high school sophomores in 1990 to 1993.
Career Academy graduates reported higher graduation rates (8.7 percentage points) and higher rates of enrollment in a two- or four-year college (11.6 and 17.9 percentage points, respectively) than students in the general education track.
The quality of causal evidence presented in this study is low because the authors did not adequately control for the academic achievement of students before they enrolled in a Career Academy. This means we are not confident that the results estimated in the study are attributable to Career Academies; other factors are likely responsible.","Career Academies aim to make the high school experience more academically rigorous and relevant to subsequent careers. They achieve this by creating small learning communities; integrating academic and vocational curricula (typically focused around one occupation, profession, or industry); and offering work-based learning experiences, often through partnerships with local employers and post-secondary institutions. Career Academies work with the employers to ensure that the curriculum reflects their needs, and employers often provide summer work experience and mentoring to career academy students.
This study examined Career Academies in one large, West Coast, urban school district. The district first opened a Career Academy in 1985 and had 12 at the time of the study. The Career Academies varied in their career focus and were at different stages of implementation. Students self-selected into attending a Career Academy or continuing in an academic, vocational, or general track. The direct costs of the Career Academy were $1,000 more per student than the standard high school offering.","The study examined measures of educational achievement, such as on-time high school graduation and two- or four-year college enrollment, as well as various educational and labor market skills. The authors estimated the effects of high school program track—Career Academy, vocational education, or academic track—on these achievement and skills outcomes, controlling for students’ and schools’ characteristics. In addition, the authors attempted to control for students’ program track selection by estimating the effect of retrospective student-reported assessments of their own pre-program awareness and skills. This information was gathered through a post-graduation survey mailed to 10,102 students who attended school in the district as sophomores from 1990 to 1993. The authors also collected district records for these students. Students returned 1,223 surveys. The authors reported estimates for the full sample and for subgroups based on gender, ethnicity, and English language ability.","Career Academy graduates reported higher graduation rates (8.7 percentage points) and higher rates of enrollment in a two- or four-year college (11.6 and 17.9 percentage points, respectively) than students in the general education track.","The study authors attempted to control for students’ program selection by adjusting their results to account for known correlates between observable student characteristics and program selection. They also attempted to compensate for low survey response rates by corroborating their findings with a nationally representative National Educational Longitudinal Study sample. However, although the authors intended to control for standardized achievement test scores, these data were unavailable for almost half the study sample.
Without additional controls for the baseline nonequivalence of students from each of the three program tracks, and evidence of the representativeness of the survey respondent sample to the fielded sample, it is likely that the estimates of program effects are confounded by the initial student program selection and subsequent survey response bias.",,,,,,"Urban, United States","Causal Impact Analysis",Youth,"Youth programs, Community college education and other classroom training, Work based and other occupational training","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Career Academies Review Protocol"
"Step to College: Moving from the High School Career Academy through the 4 Year University (Maxwell 2001)","Step to College: Moving from the High School Career Academy through the 4 Year University (Maxwell 2001)","Study Type: Causal Impact Analysis","Maxwell, N. (2001). Step to College: Moving from the High School Career Academy through the 4 Year University. Evaluation Review, 25(6), 619–654.",,2001,,"July 2014","Career Academies",,"Low Causal Evidence","Quasi-Experimental Design (QED)",,"The study aimed to examine the effect of attending a Career Academy on students’ secondary and post-secondary academic outcomes. The authors compared the outcomes of students who had attended a Career Academy in a large, inner-city school district in California and had applied to a medium-sized state university in California, with those of peers who had graduated from a traditional public high school in the same district and had applied to the same university.
The analysis was based on university records, which provided information on student demographics, high school grade point average (GPA) and college GPA, enrollment status, and major of those students who actually attended the university.
The study found that students from Career Academies had higher GPAs in college and less need for remediation in English at the university level. Students from career academies still had a high need for remediation in math and low college graduation rates, but they graduated at a 4-percent-higher rate than students from traditional high schools.
The quality of causal evidence presented in this study is low, because the authors did not adequately control for the academic achievement of students before they enrolled in a career academy, nor for students’ pre-intervention financial disadvantage. This means that we are not confident that the results estimated in the study are attributable to Career Academies; other factors are likely to be responsible.","Career Academies aim to make the high school experience more academically rigorous and relevant to subsequent careers. They achieve this by creating small learning communities; integrating academic and vocational curricula (typically focused around one occupation, profession, or industry); and offering work-based learning experiences, often through partnerships with local employers and post-secondary institutions. Career Academies work with the employers to ensure that the curriculum reflects the employers’ needs, and employers often provide summer work experience and mentoring to Career Academy students.
This study examined Career Academies in one large urban school district on the West Coast. The district first opened a Career Academy in 1985, and by the time of the study, had 12 Career Academies. These varied in their career focus and were at different stages of implementation. Students self-selected into either attending a Career Academy or continuing in an academic, vocational, or general track. The direct costs of the Career Academy were $1,000 higher per student than the standard high school offering.","The study examined several measures of educational achievement, including high school GPA, route to college (including whether the student transferred from another college or university, and whether the student required remediation in math or English), entrance into college, college GPA, and type of exit from the university (graduated, dropped out, or was dismissed). The study had 1,402 participants. All participants graduated from high school between 1990 and 1997 and applied to the university between 1990 and 1998. One-quarter of the applicants to the university from the school district studied had graduated from Career Academies. The authors estimated the effects of high school program track—Career Academy or traditional high school—on these academic outcomes, controlling for student race and gender as well as for high school campus attended.","Students who attended Career Academies had higher college GPAs and less need for remediation in English at the university level than students from traditional high schools.
They had a high need for remediation in math and low college graduation rates, but they graduated at a 4-percent-higher rate than students from traditional high schools.","Students in the Career Academy were self-selected. Characteristics of the students and their families, such as previous academic achievement and degree of financial disadvantage, could shape both selection into the Career Academy and later outcomes, but the study did not control for these. In the absence of these controls, it is likely that the estimates of program effects are confounded by the initial student self-selection into the program.
The university and school district studied had built a strong relationship with one another, to help students successfully transfer from the districts’ high schools to the university. While this relationship minimized the institutional barriers for post-secondary admission for students, the findings from this university may not be relevant in other contexts, where the university does not have the same type of relationship with the school district.",,,,,,"Urban, United States","Causal Impact Analysis",Youth,"Youth programs, Community college education and other classroom training, Work based and other occupational training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Career Academies Review Protocol"
"Transformative graduate education programs: An analysis of impact on STEM and non-STEM Ph.D. completion (Kniola 2012)","Transformative graduate education programs: An analysis of impact on STEM and non-STEM Ph.D. completion (Kniola 2012)","Study Type: Causal Impact Analysis","Kniola, D., Chang, M., & Olsen, D. (2012). Transformative graduate education programs: An analysis of impact on STEM and non-STEM Ph.D. completion. Higher Education, 63(4), 473–495. doi:10.1007/s10734-011-9453-8",,2012,,"November 2014","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to determine whether universities participating in Transformative Graduate Education Programs (TGPs) conferred a greater number of science, technology, engineering, and mathematics (STEM) doctoral degrees, particularly to women and underrepresented candidates of color. The study also examined whether TGPs resulted in a greater number of non-STEM doctoral degrees, but that is not the focus of this review.
The authors analyzed data from the Integrated Postsecondary Education Data System to estimate the impact of a university’s participation in one of seven TGPs during the 2003–2005 period, controlling for institutional characteristics.
The study found that TGPs were associated with a higher number of doctoral degrees awarded in STEM fields to women across all institutions.
The quality of causal evidence presented in this report is low. This means we are not confident that the estimated effects are attributable to universities’ participation in TGPs. Other factors are likely to have contributed.","Despite growing graduate enrollments of women and underrepresented candidates of color, STEM doctoral conferral rates for these students remain sluggish. In response, prestigious foundations and institutional associations launched several grants to fund TGPs. Competitive TGP grants challenge universities to make doctoral programs more inclusive and supportive, offering more well-rounded preparation to meet the real-world demands of contemporary careers in research, teaching, and service.","The authors examined the effectiveness of 7 well-known TGPs sponsored by prominent institutions among 246 doctoral degree–granting universities in the United States from 2003–2005. Of those, 117 did not participate in a TGP and 129 participated in at least one TGP. The TGPs studied included (from largest to smallest):
National Science Foundation’s Integrative Graduate Education and Research Traineeship (IGERT) program (87 institutions)
Council of Graduate Schools’ PhD Completion Project (43 institutions)
Council of Graduate Schools and Association of American Colleges & Universities’ Preparing Future Faculty (43 institutions)
Carnegie Foundation for the Advancement of Teaching’s Initiative on the Doctorate (42 institutions)
National Science Foundation’s ADVANCE (24 institutions)
Woodrow Wilson National Fellowship Foundation’s Responsive PhD (20 institutions)
National Institutes of Health’s Graduate Partnerships program (10 institutions)
The authors employed hierarchical ordinary least squares regression models to predict the 2003–2005 average annual number of STEM doctoral degrees, STEM doctoral degrees awarded to women, and STEM doctoral degrees awarded to underrepresented students of color. They controlled for institution-level differences in institutional control, average annual student enrollment, level of research activity, and proportion of doctoral degrees conferred in STEM disciplines. The authors also compared the relative effectiveness of the four largest TGPs.","The study found that TGPs were associated with a higher number of doctoral degrees awarded in STEM fields to women across all institutions; however, this association did not hold when looking only at STEM-dominant institutions.
When looking at the four largest TGPs, only IGERT was associated with a higher number of doctoral degrees awarded to women in STEM fields.","Although the study controlled for several important institutional-level variables, the authors acknowledged the possibility that other institutional characteristics that were not controlled for in the model could encourage both the adoption of TGPs and the award of more doctoral degrees to women. For instance, institutional leadership and culture could drive both the desire to adopt a TGP and higher rates of doctoral degree completion by women. In addition, even though CLEAR does not require explicit demonstrations of baseline equivalence, the study indicated that the groups being compared were quite dissimilar, leading to doubts about the extent to which including controls in the model could reduce the potential for bias.",,,,,,"United States","Causal Impact Analysis","Other barriers, Female, STEM professional","Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"A Scholarship Model for Student Recruitment and Retention in STEM Disciplines (Yelamarthi & Mawsha 2010)","A Scholarship Model for Student Recruitment and Retention in STEM Disciplines (Yelamarthi & Mawsha 2010)","Study Type: Causal Impact Analysis","Yelamarthi, K., & Mawasha, R. (2010). A Scholarship Model for Student Recruitment and Retention in STEM Disciplines. Journal of STEM Education, 11(5 & 6), 25–32.",,2010,,"November 2014","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Comparison of Means",,"This paper evaluated the effects of the Computer Science, Engineering, and Mathematics Scholarship (CSEMS) program at Wright State University (WSU) on educational outcomes.
The study used academic records from the university to compare educational outcomes of CSEMS participants and nonparticipants of similar background at the undergraduate and graduate levels.
The study found that undergraduate program participants were more likely than traditional undergraduate students to secure a grade point average higher than 2.7.
The quality of causal evidence presented in this study is low. This means we are not confident that the estimated effects are attributable to the CSEMS program. Other factors are likely to have contributed.","The CSEMS program was initiated in 1999 in an effort to increase recruitment, retention, and academic excellence of underrepresented minority and female students studying and pursuing careers in STEM disciplines. The program includes scholarships, career orientation workshops, cooperative and internship programs, and academic and social support. Students must apply and be accepted to participate in the program. Selection criteria include both noncognitive and cognitive factors. At the time of the evaluation, the program had served 32 undergraduate and 7 graduate students.","The researchers used academic records from the university to compare the mean outcomes of CSEMS program participants with those of underrepresented minority students who did not participate in the CSEMS programs. The study sample included 39 program participants and 114 underrepresented minority students who did not participate in the CSEMS program but were pursuing degrees similar to those of CSEMS participants.","The study found 90 percent of CSEMS undergraduate program participants secured a grade point average higher than 2.7 versus 63 percent of traditional undergraduate students.
There was little difference in grade point averages between graduate students, all of whom secured a grade point average over 3.0.","The authors compared the mean outcomes of CSEMS program participants with the mean outcomes of nonparticipants. However, other than being of the same minority group and having similar majors, the authors did not provide other evidence to suggest that the two groups were similar before program participation. To provide such evidence, the authors could have demonstrated that the two groups did not differ on age, gender, and indicators of previous academic performance. This would give us more confidence that the two groups were similar before program participation, so that any observed differences in outcomes after program participation could be attributable to the program and not underlying differences between the groups.",,,,,,"United States","Causal Impact Analysis",Youth,"Science, Technology, Engineering, and Math (STEM) programs, Youth programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"Student enrollment in high school AP sciences and calculus: how does it correlate with STEM careers? (Robinson 2003)","Student enrollment in high school AP sciences and calculus: how does it correlate with STEM careers? (Robinson 2003)","Study Type: Causal Impact Analysis","Robinson, M. (2003). Student enrollment in high school AP sciences and calculus: how does it correlate with STEM careers? Bulletin of Science, Technology & Society, 23, 265–273.",,2003,,"December 2014","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Comparison of Means",,"This study’s objective was to examine differences in the intended choice of college science, technology, engineering, and mathematics (STEM) majors between females and males enrolled in high school advanced placement (AP) science and calculus courses.
The study used survey data from eight high schools to examine the percentage of AP students who chose a STEM field as their intended college major. The authors compared outcomes for minority and nonminority males and females.
The study found that among students taking AP science or calculus classes, males were generally more likely than females to plan to choose a college major in engineering but less likely to plan to choose a college major in medicine.
The quality of causal evidence presented in this study is low. This means we are not confident that the estimated effects are attributable to AP courses. Other factors are likely to have contributed.","The AP program was initiated in the 1955–1956 school year as a means for high school students to earn college credit for high achievement in specific subjects. In 2001, the AP program offered 35 courses in 19 subject areas, and more than 90 percent of colleges in the United States offered credit or placement for students with high AP scores. Students can take AP courses in grades 9 through 12, though the majority of AP students are in 11th or 12th grade.","The authors administered a survey to 10 high schools in one school district of an unnamed state in the western United States. Eight schools returned the survey, which contained information on school enrollment as well as individual student responses indicating their planned college majors. A total of 315 students who completed the survey were enrolled in at least one AP course during the 2002–2003 school year. The authors separated the sample into minority and nonminority students who were enrolled in AP science or calculus courses and compared the percentage of females who intended to major in a STEM field to the percentage of males who intended to major in a STEM field.","The study found that among nonminority students taking AP classes in science or calculus, males were more likely than females to plan to choose a college major in engineering, but females were more likely to plan to choose science, math, or medicine: 28 percent of males chose engineering as their planned college major versus 11.6 percent of females. However, 24.4 percent of females chose science or math as their planned major versus 19.2 percent of males, and 20.3 percent of females chose medicine versus 9.8 percent of males.
Among minority students taking AP science and calculus courses, males were more likely than females to plan to choose a college major in engineering, science, and math, but not medicine: 32.3 percent of males planned to choose engineering majors versus 8.3 percent of females; 19.1 percent of males planned to choose science or math versus 13.3 percent of females; but 21.6 percent of females planned to choose medicine versus 16.1 percent of males.","The authors compared females’ pursuit of STEM majors with that of males. These two groups are fundamentally different, so comparing them does not provide evidence of the effect of taking AP courses on females’ pursuit of STEM majors. To provide such evidence, the authors could have compared the pursuit of STEM majors among females taking AP science and calculus courses with that of females who did not have access to AP courses, controlling for relevant characteristics such as age, race/ethnicity, and previous grades in science and math courses.",,,,,,"United States","Causal Impact Analysis","Youth, Female, Male","Science, Technology, Engineering, and Math (STEM) programs, Youth programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"Supporting young women to enter engineering: Long-term effects of a middle school engineering outreach program for girls (Demetry et al. 2009)","Supporting young women to enter engineering: Long-term effects of a middle school engineering outreach program for girls (Demetry et al. 2009)","Study Type: Causal Impact Analysis","Demetry, C., Hubelbank, J., Blaisdell, S., Sontgerath, S., Nicholson, M.E., Rosenthal, E., & Quinn, P. (2009). Supporting young women to enter engineering: Long-term effects of a middle school engineering outreach program for girls. Journal of Women and Minorities in Science and Engineering 15, 119-142.",,2009,,"July 2015","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to assess the long-term impact of Camp Reach, a summer engineering enrichment program for middle school girls, on enrollment in STEM courses in high school and college.
Admission to Camp Reach was determined through a lottery of girls who applied to the program and met eligibility requirements. Six to seven years later, the authors administered a survey to 109 girls who had been admitted to the program and 107 girls who had not. Many girls in the control group attended other, similar programs.
The study found that members of the control group who had attended other summer science or engineering programs at Worcester Polytechnic Institute (WPI) were significantly more likely to have taken high school calculus than were members of the Camp Reach Partial group. There were no statistically significant differences on taking physics, computer science, other science and engineering courses, or planning to major in engineering in high school.
The quality of causal evidence presented in this report is low because the authors adjusted the research sample after random assignment and did not include adequate controls to ensure that the resulting groups were similar on all relevant attributes. This means we are not confident that the estimated effects were attributable to Camp Reach; other factors are likely to have contributed.","Camp Reach was a two-week summer engineering program for rising 7th-grade girls run through WPI. The program aimed to encourage girls’ interest in and pursuit of engineering and other STEM fields through design workshops and mentoring. It emphasized collaborative problem-solving through a collaborative public service engineering project, which showed how engineering can be a helping profession. The camp promoted continued engagement through alumnae newsletters and reunions.","Camp Reach made 30 places available to interested applicants each year; those applicants meeting minimum eligibility criteria were entered in a lottery through which camp organizers selected the summer’s participants, who formed the treatment group. The group of selected participants was then adjusted to reflect the demographic profile of Worcester, Massachusetts, where the program was held. Those who were not invited to participate formed the control group.
The authors contacted 252 girls who had applied to Camp Reach from 1997 to 2001 in 2004 through 2007, six to seven years after they had applied, to complete a survey. Of these, 176 responded. The authors compared the responses of the treatment and control group members on survey questions about their knowledge of engineering, the science and engineering courses they took in high school, and plans to major in science or engineering in college.
To estimate impacts, the authors organized the treatment and control groups into subgroups. First, they distinguished between girls in the treatment group who had attended the summer session of Camp Reach (Camp Reach Partial) and those who had participated in alumnae reunions and the initial summer session (Camp Reach Full). Because further program participation was nonexperimental, this review focuses on treatment survey respondents who attended only the Camp Reach summer session. In addition, the authors organized the control group into those who had or had not attended other science and engineering programs, because many survey respondents who were not selected for Camp Reach ultimately participated in similar programs offered by WPI.","The study found that members of the control group who had attended other summer science or engineering programs at WPI were significantly more likely to have taken high school calculus than were members of the Camp Reach Partial group. There were no statistically significant differences on taking physics, computer science, other science and engineering courses, or planning to major in engineering in high school.
There were no statistically significant differences between members of the Camp Reach Partial group and members of the control group who did not attend other science or engineering camps at WPI.","The authors acknowledged that the randomly selected attendee groups were modified after random selection to replicate Worcester’s demographic profile. This adjustment means that the assignment was no longer random, and therefore this study cannot receive a high causal evidence rating. The adjustment of the research sample after random assignment could have induced differences between the treatment and control groups on characteristics that also influenced the outcomes of interest. Because the techniques the authors used to estimate the program’s effect did not adequately control for these potential differences, they cannot distinguish the effects of Camp Reach from the effects of differences in relevant demographic characteristics.",,,,,,"United States","Causal Impact Analysis","Youth, Female","Science, Technology, Engineering, and Math (STEM) programs, Youth programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"Long-term effects of a middle school engineering outreach program for girls: A controlled study (Hubelbank et al. 2007)","Long-term effects of a middle school engineering outreach program for girls: A controlled study (Hubelbank et al. 2007)","Study Type: Causal Impact Analysis","Hubelbank, J., Demetry, C., Nicholson, S., Blaisdell, S., Quinn, P., Rosenthal, E., & Sontgerath, S. (2007). Long-term effects of a middle school engineering outreach program for girls: A controlled study. In Proceedings, American Society for Engineering Education Annual Conference & Exhibition.",,2007,https://www.wpi.edu/Images/CMS/News/1106_LONG_TERM_EFFECTS_OF_A_MIDDLE_SCHOOL_E…,"September 2015","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to assess the long-term impact of Camp Reach, a summer engineering enrichment program for middle school girls, on enrollment in STEM courses in high school and college.
Admission to Camp Reach was determined through a lottery of girls who applied to the program and met eligibility requirements. Six to seven years later, the authors administered a survey to 88 girls who had been admitted to the program and 41 girls who had not.
The study found that Camp Reach attendees were significantly more likely to have taken computer science and other science or engineering courses in high school than those who did not attend.
The quality of causal evidence presented in this report is low because the authors adjusted the research sample after random assignment and did not include adequate controls to ensure that the resulting groups were similar on all relevant attributes. This means we are not confident that the estimated effects are attributable to Camp Reach; other factors are likely to have contributed.","Camp Reach was a two-week summer engineering program operated by Worcester Polytechnic Institute (WPI) for rising 7th-grade girls. The program aimed to encourage girls’ interest in and pursuit of engineering and other STEM fields through design workshops and mentoring. It emphasized collaborative problem-solving through a group public service engineering project highlighting engineering as a helping profession. The camp promoted continued engagement through alumnae newsletters and reunions.","Camp Reach made 30 places available to interested applicants each year; those applicants meeting minimum eligibility criteria participated in a lottery through which camp organizers selected the summer’s participants, who formed the treatment group. The group of selected participants was then adjusted to reflect the demographic profile of Worcester, Massachusetts, where the program was held. Those who were not invited to participate formed the control group.
The authors contacted 178 girls who had applied to Camp Reach from 1997 to 2000 six to seven years after they had applied (that is, the authors contacted them in 2004 through 2006) to complete a survey. Of these, 129 responded. The authors compared the responses of the treatment and control group members to survey questions about their knowledge of engineering, the science and engineering courses they took in high school, and plans to major in science or engineering in college.","The study found that Camp Reach attendees were significantly more likely to have taken computer science and other science or engineering courses in high school than those who did not attend.","The authors acknowledged modifying the randomly selected attendee groups after random selection to replicate Worcester’s demographic profile. This adjustment means that the assignment was no longer random and, therefore, this study cannot receive a high causal evidence rating. The adjustment of the research sample after random assignment could have induced differences between the treatment and control groups on characteristics that also influenced the outcomes of interest. Because the techniques the authors used to estimate the program’s effect did not adequately control for these potential differences, the authors could not distinguish the effects of Camp Reach from the effects of differences in relevant demographic characteristics.",,,,,,"United States","Causal Impact Analysis","Youth, Female","Science, Technology, Engineering, and Math (STEM) programs, Youth programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"Coordinated studies programs: Their effect on student involvement at a community college (Tinto & Russo 1994)","Coordinated studies programs: Their effect on student involvement at a community college (Tinto & Russo 1994)","Study Type: Causal Impact Analysis","Tinto, V., & Russo, P. (1994). Coordinated studies programs: Their effect on student involvement at a community college. Community College Review, 22(2), 16-25.",,1994,,"October 2015","Community College",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of coordinated studies programs (CSPs) on students’ persistence in college.
The authors used administrative data to compare reenrollment rates between a treatment group of students enrolled in CSP classes and a comparison group of students enrolled in humanities and social science classes.
The study found that the spring reenrollment rate for CSP students was 84 percent, significantly higher than the 81 percent rate for comparison students. The fall reenrollment rate for CSP students was 67 percent, significantly higher than the 52 percent rate for comparison students.
The quality of causal evidence presented in this report is low because the authors did not control for relevant student characteristics in the analysis. This means we are not confident that the estimated effects are attributable to CSPs. Other factors are likely to have contributed.","A CSP consists of a set of linked courses organized around a central theme. Students enrolled as if they were taking separate courses in different disciplines and fields but attended the CSP as one course that met for a total of 11 to 18 hours per week in blocks of 4 to 6 hours over two to four days. CSPs were team-taught by two to four faculty members who were present and active in all class sessions.","The authors used administrative data to compare reenrollment rates between the treatment and comparison groups. The authors formed the treatment group (121 students) by selecting students in first-year CSP classes in the liberal arts and transfer division of Seattle Central Community College. They formed the comparison group (166 students) by selecting students in humanities and social science classes who program staff believed were a representative sample of first-year students who had not enrolled in the CSP. First-year students who completed two surveys on perceptions of the school environment took part in the study.","The study found that the spring reenrollment rate for CSP students was 84 percent, significantly higher than the 81 percent rate for comparison students.
The fall reenrollment rate for CSP students was 67 percent, significantly higher than the 52 percent rate for comparison students.","The authors did not include controls for student characteristics in their analysis of reenrollment rates, so estimated impacts could reflect pre-existing differences between the groups being compared, and not program impacts. For example, students who chose to enroll in the CSP might have had higher academic achievement and persistence than students who did not. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,,,"United States","Causal Impact Analysis",Adult,"Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Learning better together: The impact of learning communities on the persistence of low-income students (Engstrom & Tinto 2008)","Learning better together: The impact of learning communities on the persistence of low-income students (Engstrom & Tinto 2008)","Study Type: Causal Impact Analysis","Engstrom, C., & Tinto, V. (2008). Learning better together: The impact of learning communities on the persistence of low-income students. Opportunity Matters: A Journal of Research Informing Educational Opportunity Practice and Programs, 1(1), 5-21.",,2008,,"November 2015","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of learning communities on community college students’ persistence from freshman to sophomore year, an outcome in the progress toward degree completion domain.
The authors used a nonrandom approach to select treatment and comparison students to include in their analysis. To measure the relationship between participation in a learning community and persistence, the authors used survey and administrative data from 13 community colleges to estimate a logistic regression.
The study found that 62 percent of learning community students persisted from freshman to sophomore year, compared with 57 percent of comparison group students. This difference was statistically significant.
The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to learning communities. Other factors are likely to have contributed.","The characteristics of the learning communities implemented at the colleges included in this study varied. The authors selected institutions at which at-risk students were well represented and that had learning communities meeting several criteria: the learning community programs were well established and had institutional evidence of effectiveness for academically underprepared students, they situated basic skill development within a broader academic context rather than simply linking several courses, and they captured variations in how learning communities were being adapted.","The study used a nonexperimental design. The authors selected two groups of students at each participating college: a treatment group of students who participated in a learning community during their first year of college and a comparison group of students who did not. Comparison students were similar to learning community students in terms of attributes and academic preparation and were enrolled in courses that were similar in content to those that were part of the learning communities. To estimate the relationship between participation in a learning community and persistence, the authors estimated a logistic regression using the following control variables: highest education credential, mother’s education level, age, gender, English as native language, U.S. citizenship, ethnicity, and student engagement.","The study found that 62 percent of learning community students persisted from freshman to sophomore year, compared with 57 percent in the comparison group. This difference was statistically significant.","The authors reported that students in the learning communities were more likely to come from minority backgrounds, to be younger, and to be female than comparison group students. In addition, the authors did not include controls for state. Because participating colleges were located in several states throughout the United States, it is important to control for state to account for differences in state-level educational policies, among other factors, that could influence persistence. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"Thirteen two-year colleges in several states across the United States participated in the study:
Camden College Cerritos College
Community College of Baltimore County
DeAnza College
Grossmont College
Holyoke Community College
LaGuardia Community College
San Jose City College
Sandhills Community College
Santa Fe Community College
Seattle Central Community College
Shoreline Community College
Spokane Falls Community College",,,,"United States","Causal Impact Analysis","Adult, Other barriers, Low income","Community college education and other classroom training, Basic skills","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"More than robots: An evaluation of the FIRST robotics competition participant and institutional impacts (Melchior et al. 2005)","More than robots: An evaluation of the FIRST robotics competition participant and institutional impacts (Melchior et al. 2005)","Study Type: Causal Impact Analysis","Melchior, A., Cohen, F., Cutter, T., & Leavitt, T. (2005). More than robots: An evaluation of the FIRST robotics competition participant and institutional impacts. Waltham, MA: Brandeis University Center for Youth and Communities Heller School for Social Policy and Management.",,2005,http://www.techfire225.com/uploads/6/3/7/1/6371896/first_study.pdf,"January 2016","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of the FIRST Robotics Competition (FRC) on participants’ education and career outcomes. The FRC was a six-week competition in which teams of high school male and female students (with the help of mentors) used a set of standard parts to build a robot. Students showcased these robots by competing against teams from other high schools.
The authors used propensity-score matching to create treatment and comparison groups and conducted an analysis of retrospective survey data to measure education and career outcomes of FRC participants who graduated from the program from 1999 to 2003. They administered a researcher-designed survey to students in the treatment group in 2004 and collected data from a national survey—the Beginning Postsecondary Student Survey (BPSS)—to measure outcomes for the comparison group, who began college in 1995-1996.
The study found that a higher percentage of female FRC participants majored in engineering compared with females in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not demonstrate equivalency of the treatment and comparison groups on all required measures. This means we are not confident that the estimated effects are attributable to the FRC; other factors are likely to have contributed.","The FRC was a six-week competition in which teams of high school students (with the help of teachers at their school and community mentors) used a set of standard parts to build a robot. Students showcased these robots by competing against teams from other high schools in a variety of challenges at regional and national events. During the program, students were required to attend team meetings, raise funds, build and program the robot, develop a strategy for the competition, and present the robot to various audiences. Typically, teams divided these tasks among the team members so specific work varied by student. Schools across the country were able to form their own FRC team and students self-selected themselves into the teams. The goal of the program was to increase students’ STEM interests and abilities, while also developing teamwork, leadership, and project management skills.","The primary goal of the study was to retrospectively determine the impact of participation in the FRC on a variety of outcomes for all students, but particularly low-income and minority students. To do so, the authors first invited all of the high schools in three large metropolitan areas (New York City, Detroit/Pontiac metropolitan area, and San Jose/San Francisco Bay area) that had been running the FRC program for at least two years to participate in the study. Of the 30 schools invited to participate, 14 had the interest and necessary data to do so. These 14 schools identified 360 former program participants to form the treatment group, of which 173 ultimately filled out the study-administered survey (a 48 percent response rate) and were included in the analysis.
The researchers compared the study-administered survey data collected from the treatment group with the BPSS survey data to create a comparison group using propensity-score matching. Using 147 exact matched pairs—matching on participants’ demographics, parental education, economic status of the high school, and various measures of high school STEM engagement—the authors compared the two groups on a variety of education, career, and developmental outcomes by calculating the difference in average outcome between the treatment and comparison groups and determining the statistical significance.","The study found that a higher percentage of female FRC participants majored in engineering compared with the comparison group (32.6 versus 8.7 percent).
The authors also found a marginally statistically significant difference in the proportion of females majoring in STEM-related fields more generally (in addition to engineering) (41.3 percent of the treatment group compared with 21.7 percent of the comparison group).","This study has several issues that should be considered when interpreting the findings. First, the treatment and comparison groups might differ substantially on both observed and unobserved characteristics. The authors did not match students on age as required for this review, and students could self-select into the intervention, which could bias the results. Second, data for the treatment group came from study-administered surveys of participants who were several years removed from program participation, such that some survey results might be subject to recall bias. Third, the treatment sample might suffer from nonresponse bias because only 14 of 30 treatment schools participated and only 173 of 360 former students responded to the study-administered survey. Fourth, the treatment and comparison groups did not attend college at the same time, and the authors used different data sources to measure their outcomes, both of which could confound the results.",,,,,,"United States","Causal Impact Analysis","Youth, Female, Male","Science, Technology, Engineering, and Math (STEM) programs, Youth programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"Leading them to water: A study of the efficacy of a Mandatory Placement Project in first-year academic courses at a community college (Emmerson 2009)","Leading them to water: A study of the efficacy of a Mandatory Placement Project in first-year academic courses at a community college (Emmerson 2009)","Study Type: Causal Impact Analysis","Emmerson, J. (2009). Leading them to water: A study of the efficacy of a Mandatory Placement Project in first-year academic courses at a community college. Graduate Theses and Dissertations, paper 10661. Retrieved from http://lib.dr.iastate.edu/etd/10661",,2009,http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1685&context=etd,"December 2015","Community College",,"Low Causal Evidence","Regression Analysis",,"This study’s objective was to examine the effects of Des Moines Area Community College’s Mandatory Placement Project on credit completion and fall-to-spring semester persistence. The intervention included intensive counseling and developmental courses for first-time, full-time community college students whose late registration suggested a high risk of not persisting.
The study used administrative records to compare outcomes of a group of 270 students who were required to meet with an advisor and strongly encouraged to participate in one of six developmental courses with those of a comparison group of 174 students who received no counseling and were not advised to take developmental courses.
The study found that participation in one of the six developmental courses was significantly and positively associated with fall-to-spring semester persistence.
The quality of causal evidence presented in this report is low because the author did not adequately account for existing differences between the treatment and comparison groups. This means we are not confident that the estimated effects are attributable to the Mandatory Placement Project. Other factors are likely to have contributed.","Des Moines Area Community College began a pilot of the Mandatory Placement Project in fall 2006 to improve students’ grades, credit completion, and persistence. The Mandatory Placement Project was piloted among students registering during the two-week period before classes started; a 2004 study at the college had identified such students as being at a particularly high risk for low levels of academic success and persistence. Additional eligibility criteria included being a full-time student who had never enrolled at Des Moines Area Community College before. Mandatory Placement students were required to meet with an advisor who strongly advised that students enroll in at least one developmental course. Mandatory Placement students could choose among six developmental course options: (1) The College Experience, (2) Study Strategies, (3) Learning Communities, (4) College Preparatory Reading, (5) College Preparatory Writing, or (6) College Preparatory Math. In addition to taking developmental courses, students were required to meet with an advisor to modify their course schedules or reenroll for the following semester.","The study used administrative records to compare credit completion and fall-to-spring semester persistence of students in treatment and comparison groups at Des Moines Area Community College, Ankeny Campus. During the two-week period before courses began, 278 eligible students who registered made up the treatment group and were required to meet with an advisor and strongly encouraged to participate in one of six developmental courses. The comparison group consisted of 174 eligible students who registered during the first week of classes. These students received no counseling and were not advised to take developmental courses. The authors used statistical techniques to estimate the effect of the program.","The study found that participation in one of the six developmental courses was significantly and positively associated with fall-to-spring semester persistence.","The author did not control for the previous academic achievement of students in the treatment group and students in the comparison group. Moreover, students registering before courses begin (who were included in the treatment group) are likely to differ systematically from students registering later (who were included in the comparison group) in unobservable ways, such as motivation. Therefore, the estimated relationship between participating in a developmental course and persistence may reflect underlying differences between the students being compared and not the effect of participating in a developmental course. For this reason, the study is not eligible for a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Adult, Other barriers","Basic skills, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"The Engineering Admissions Partnership Program: A navigation strategy for community college students seeking a pathway into engineering (Laugerman et al. 2013)","The Engineering Admissions Partnership Program: A navigation strategy for community college students seeking a pathway into engineering (Laugerman et al. 2013)","Study Type: Causal Impact Analysis","Laugerman, M., Shelley, M., Mickelson, S., & Rover, D. (2013). The Engineering Admissions Partnership Program: A navigation strategy for community college students seeking a pathway into engineering. International Journal of Engineering Education, 29(5), 1260-1269.",,2013,,"January 2016","Community College",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of the Engineering Admissions Partnership Program (E-APP) on retention rates of community college students who transferred to Iowa State University (ISU) to pursue an engineering degree. The intervention consisted of coordinated academic advising, peer mentoring, campus visits, and online social and professional networking.
The study used student record data to examine outcomes for a treatment group of ISU students who participated in E-APP and a matched comparison group of ISU students who did not participate.
The study found that E-APP had positive impacts on first-year retention rates at ISU.
The quality of causal evidence presented in this report is low because the authors did not include sufficient controls in the analysis. This means we are not confident that the estimated effects are attributable to E-APP. Other factors are likely to have contributed.","E-APP was created in 2008 to improve transfer outcomes through connections to the university for community college students who transferred to ISU to pursue an engineering degree. E-APP fostered connections to the university through coordinated academic advising, peer mentoring, campus visits, and online social and professional networks. E-APP was a project initiative of Seeking Enrollment and Engagements through Connections (SEEC), a National Science Foundation-funded program that established a partnership between ISU and a multiple-location community college. E-APP was offered to all in-state community college transfer students at ISU and to students from the SEEC community college.","The study analyzed student record data from community college transfer students who enrolled in ISU’s College of Engineering from fall 2002 to fall 2008. The authors used average math scores on the American College Test (ACT) to match E-APP participants to a comparison group of community college transfer students who did not participate in E-APP. To measure the impacts of E-APP on retention rates at ISU for community college transfer students, the authors computed mean differences and conducted statistical tests of these differences. The authors compared retention rates of E-APP participants and all in-state community college transfer students, as well as of E-APP participants and nonparticipating community college transfer students from the SEEC community college.","The study found that, for all in-state community college students admitted to the College of Engineering at ISU, E-APP participants were 11 percent more likely to be retained at ISU. This difference was statistically significant. 
For SEEC community college students admitted to the College of Engineering at ISU, E-APP participants were 19 percent more likely to be retained in Engineering and 14 percent more likely to be retained at ISU. These differences were statistically significant.","The study used a matching method to form the comparison group, which is a way to control for existing differences between treatment and control groups that might bias impact estimates. However, the matching procedure matched only on ACT math scores. The authors did not include other relevant covariates in either the matching procedure or in the impact analysis. This lack of statistical controls raises concerns that factors other than participation in E-APP could account for the observed differences in retention rates between treatment and comparison students. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,,,"United States","Causal Impact Analysis",Adult,"Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Developing core skills in the major (Arcario et al. 2007)","Developing core skills in the major (Arcario et al. 2007)","Study Type: Causal Impact Analysis","Arcario, P., Clark, J., & Klages, M. (2007). Developing core skills in the major. In M. Smith & B. Williams (Eds.), Learning communities and student affairs: Partnering for powerful learning. Olympia, WA: Washington Center for Improving the Quality of Undergraduate Education, Evergreen State College.",,2007,,"January 2016","Community College",,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of participation in one of three first-year academies (Allied Health, Business/Technology, or Liberal Arts) on course failure rates and course attrition rates at LaGuardia Community College in New York City.
The study design was unclear: the authors might have used a pre-post design or a simple mean comparison. The authors did not specify the data source. 
The study found that failure rates in basic skills and discipline-area courses declined by 7.7 percent for academy students, and attrition in basic skills and discipline-area courses declined by 8 percent for academy students. The authors did not report information on the statistical significance of these findings.
The quality of causal evidence presented in this report is low because the authors did not establish that there were no observable differences between the treatment and comparison groups at baseline nor did they adjust for any differences. This means we are not confident that the estimated effects are attributable to first-year academies. Other factors are likely to have contributed.","New York City’s LaGuardia Community College offered three first-year academies to basic skills students (the college later expanded the program to all students). Students chose to participate in one of three academies based on their majors: the Allied Health Academy, the Business/Technology Academy, or the Liberal Arts Academy. The first-year academies’ goals were to (1) contextualize basic skills instruction; (2) include co-curricular experiences; (3) foster a sense of community; and (4) provide a framework for advisement, orientation, and professional development activities. Incoming academy students received orientation and advising and attended special registration sessions. During the school year, academy students took a new student seminar and a cooperative education course, received specialized advising, and attended events organized by their academy.","The empirical strategy was unclear: the authors might have used a pre-post design or a simple mean comparison. The authors did not specify the data source but it appears to be school administrative data from four semesters (2004–2006). The study did not provide information on the number of participants, eligibility criteria, or demographic characteristics. In addition, the authors did not discuss how they formed the treatment and comparison groups.","The study found that failure rates in basic skills and discipline-area courses declined by 7.7 percent for academy students. 
Attrition in basic skills and discipline-area courses declined by 8 percent for academy students. 
The authors did not report information on the statistical significance of these findings.","It is unclear if the results for the outcomes of interest involved a comparison group; they could be results from a pre-post contrast. If a comparison group was involved, the study was unclear whether the group was composed of LaGuardia Community College students who were not first-year academy students or all LaGuardia Community College students.",,,,,,"United States","Causal Impact Analysis","Adult, Other barriers","Basic skills, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Increasing retention of women engineering students (Sullivan & Davis 2007)","Increasing retention of women engineering students (Sullivan & Davis 2007)","Study Type: Causal Impact Analysis","Sullivan, K., & Davis, R. (2007). Increasing retention of women engineering students. American Society for Engineering Education. Proceedings of the 2007 American Society for Engineering Education Annual Conference and Meeting. Washington, DC: American Society for Engineering Education.",,2007,,"January 2016","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine how participating in a study-designed program called the Jornada Intervention affected undergraduate female engineering students’ attitudes toward and persistence in studying engineering. Program activities included receiving a handheld computer and an invitation to participate in meetings to share experiences using the computer and suggestions for improving the user’s experience.
The study used a comparison group design to compare outcomes for the treatment group with those of women who enrolled in engineering in adjacent years. Data sources included an attitudinal survey administered for the first analysis and administrative records to measure students’ graduation rates for the second analysis.
In the first analysis, the study found no significant differences in attitudes toward engineering between the treatment and comparison groups. For the second analysis, though results were not presented and methods were not described, the authors reported that they uncovered significant differences in persistence between the treatment and comparison groups. They noted that program participants were significantly more likely to graduate in engineering than both (1) women from the same year who did not participate in the program and (2) women who enrolled in engineering in adjacent years.
The quality of causal evidence presented in this report is low because it does not include control variables to adjust for differences between treatment and comparison groups. This means we are not confident that the estimated effects are attributable to the intervention; other factors are likely to have contributed.","The study took place at Santa Clara University and included all female freshman and sophomore students enrolled in engineering in fall 1999 who agreed to participate in the Jornada Intervention (65 percent of those asked). Students in the intervention or treatment group received a handheld computer called the Hewlett-Packard Jornada, either as a personal digital assistant device or a 3/4-sized laptop computer, and were invited to attend regular meetings with other treatment students to evaluate the computer and discuss how to improve its functioning. Participants also completed a workshop that focused on improving future generations of cell phone technology. The purpose of the intervention was to (1) “increas[e] their self-identification as ‘techies’” and (2) form a group of female students who would meet to discuss technology and technological issues.","The study included 54 participants in the treatment group and 17 students in the comparison group, comprising a single cohort, for the first analysis. Sample sizes were not provided for the second analysis, which included six comparison years.
For the first analysis, the authors administered a study-designed survey two years following implementation of the intervention using a scale constructed from attitudinal measures in the “Success in Engineering” instrument. Survey results were compared for the treatment and comparison groups and analyzed with a paired-samples t-test.
For the second analysis, the authors compared graduation rates for female students who entered the university during the year the intervention was implemented with those who enrolled in adjacent years, though they did not describe their methods for this analysis.","The first analysis found no significant differences between the two-year follow-up attitudinal measures of the treatment and comparison groups.
In the second analysis, the six-year graduation rate of the treatment group was 75.9 percent, compared with the 52.9 percent rate of the comparison group. The authors reported that these results were significant, though results were not presented and methods were not described.","There are several factors to consider when interpreting this study. The study used a quasi-experimental approach to evaluating the intervention; however no control variables were used to adjust for differences between groups. Further, the composition of the two groups included some systematic variation: the treatment group contained freshmen and sophomores, whereas the control group contained only freshmen. Sophomores had already persisted from the first to the second year of college, making them perhaps more likely to graduate and have positive attitudes toward engineering.
Further, selection bias likely influenced the study outcomes, as students opted in to the intervention. Expressing interest in attending group meetings to discuss technology (one component of the intervention) is likely correlated with an outcome of interest—holding positive attitudes toward engineering. Selection into the study might also be correlated with a number of other factors, including motivation, work and family obligations, and collaborative working style, that are in turn correlated with persistence outcomes.",,,,,,"United States","Causal Impact Analysis","Youth, Female","Science, Technology, Engineering, and Math (STEM) programs, Youth programs","Attitudes-Low-No impacts, Education and skills gains-Low-Favorable impacts","Attitudes, Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"A mixed-method approach to evaluating learning communities for underprepared community college students: The Integrated Studies Communities at Parkland College (Moore 2000)","A mixed-method approach to evaluating learning communities for underprepared community college students: The Integrated Studies Communities at Parkland College (Moore 2000)","Study Type: Causal Impact Analysis","Moore, L.H. (2000). A mixed-method approach to evaluating learning communities for underprepared community college students: The Integrated Studies Communities at Parkland College (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 9971142)",,2000,http://files.eric.ed.gov/fulltext/ED457936.pdf,"January 2016","Community College",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of the Integrated Studies Community (ISC) at Parkland College in Champaign, Illinois, on credit hours earned, course completion, and persistence during the 1998–1999 academic year.
The study used a matched comparison group design to compare ISC participants with nonparticipants. Data sources included administrative data from Parkland College and interview and survey data.
This study found that ISC students earned significantly more credit hours than comparison students.
The quality of causal evidence presented in this report is low because the study did not include sufficient controls in the analysis. This means we are not confident that the estimated effects are attributable to the ISC. Other factors are likely to have contributed.","The ISC at Parkland College was a coordinated studies learning community model that targeted underprepared students who lacked college-level skills in reading, writing, or mathematics and were considered to be academically at risk. The study had two parts: a pilot study in spring 1998 and the full implementation in fall 1998 and spring 1999. In the pilot study, the ISC model (or ISC II) integrated four three-credit hour classes: Critical Comprehension Skills (CCS) 098, English 098 or 099, Literature 120, and Orientation 101. Pilot study participants included all students enrolled in the spring 1998 ISC. In the full implementation study, the ISC II model continued to apply to the spring 1998 ISC participants. The third phase of ISC (or ISC III) consisted of four integrated courses: English 099, Math 094, Psychology 101, and CCS 099. Participants in the full implementation included students who enrolled in the second and third phases of ISC (ISC II and ISC III). In both ISC phases, study participants met regularly (three days each week) with their instructors.","The study used a matched comparison group design to compare ISC participants with nonparticipants. Students in the comparison group were enrolled in traditional, nonlearning community versions of the developmental reading courses that were part of the learning community. The study used multivariate regressions and t-tests of mean differences to estimate impacts on credit hours. The author did not conduct statistical tests for course completion or persistence outcomes.","Overall, ISC students earned 1.5 more credit hours than comparison students. Students in ISC III earned 3.5 more credit hours than those in the CCS 099 comparison group. These differences were statistically significant.","The study used a matched comparison group design to compare ISC participants with nonparticipants. However, for only one eligible outcome did the author conduct any statistical tests: credit hours earned. The author did not conduct statistical tests for the other two eligible outcomes: course completion and persistence. The author estimated impacts on credit hours, controlling for age, race, pre-intervention reading achievement, and gender. The regression model did not control for a measure of participant’s degree of financial disadvantage. This raises the concern that pre-existing differences between the groups, rather than ISC, could account for the observed differences in outcomes between treatment and comparison students.",,,,,,"United States","Causal Impact Analysis","Adult, Other barriers","Basic skills, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Reducing the gender achievement gap in college science: A classroom study of values affirmation (Miyake et al. 2010)","Reducing the gender achievement gap in college science: A classroom study of values affirmation (Miyake et al. 2010)","Study Type: Causal Impact Analysis","Miyake, A., Kost-Smith, L., Finkelstein, N., Pollock, S., Cohen, G., & Ito, T. (2010). Reducing the gender achievement gap in college science: A classroom study of values affirmation. Science, 330 (6008), 1234–1237.",,2010,,"January 2016","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to assess the effect of values affirmation on female students’ performance in an introductory college physics course.
Students in the course were randomly assigned to either a treatment group, in which they completed a writing exercise affirming their personal values, or a control group, in which they described values that might be important to others. The authors collected exam scores and course grades for students in both study groups.
The study found that, although male students’ exam scores remained higher than female students’ scores, the gender gap was significantly smaller for students in the values affirmation group than for students in the control group.
The quality of causal evidence presented in this report is low because it was a randomized controlled trial with high attrition that did not control for background characteristics likely to have affected the outcomes of interest. This means we are not confident that the estimated effects are attributable to values affirmation; other factors are likely to have contributed.","To test the hypothesis that women underperform in scientific fields relative to men in response to perceived gender stereotypes, the authors sought to boost students’ self-esteem through a values affirmation writing exercise. During weeks one and four of a 15-week introductory physics course, students in the values affirmation group selected from a list the values that were most important to them and then completed a 15-minute writing exercise in which they explained why the values were important to them.","Students in a 15-week introductory physics course at the University of Colorado-Boulder were randomly assigned to either the treatment group, in which they completed a values affirmation exercise, or to the control group, in which they completed a writing exercise describing values that might be important to others. The authors then examined men and women’s performance in the course within and across treatment groups using multiple regression analysis.","The study found that the gender gap in exam scores, both as an average across the course and for the final exam in isolation, was significantly smaller for students in the values affirmation group than for students in the control group.
Women in the values affirmation group were significantly more likely to earn Bs than women in the control group, who were more likely to earn Cs. The treatment had no impact on men’s grade distributions.
The authors determined that male and female students in the values affirmation group learned the same amount in the course, in which learning was measured as the difference between scores on a standardized physics test administered at the beginning and end of the term. In the control group, however, men learned significantly more than women.","Although the authors designed the study as a randomized controlled trial, the group of students included in the analysis included only those who followed the writing exercise instructions correctly, completed the online survey regarding gender stereotypes in physics, and had valid Scholastic Achievement Test/American College Test math data available. Including only a subset of those originally randomized in the analysis based on post-random assignment characteristics could introduce bias into the study results; therefore, the study is not eligible for a high causal evidence rating. In addition, the authors did not control for race or a pre-intervention measure of academic achievement, both of which are highly correlated with the academic outcomes of interest in this study. Because differences in these characteristics could also affect academic performance, the values affirmation exercise might not solely explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Youth, Female","Science, Technology, Engineering, and Math (STEM) programs, Youth programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"Impact of learning communities in developmental English on community college student retention and persistence (Barnes & Piland 2010)","Impact of learning communities in developmental English on community college student retention and persistence (Barnes & Piland 2010)","Study Type: Causal Impact Analysis","Barnes, R., & Piland, W. (2010). Impact of learning communities in developmental English on community college student retention and persistence. Journal of College Student Retention: Research, Theory, & Practice, 12(1), 7-24.",,2010,,"January 2016","Community College",,"Low Causal Evidence","Comparison of Means",,"This study’s objective was to evaluate the effectiveness of learning communities as an alternative pedagogical format for students enrolled in developmental reading and English composition courses at an urban community college.
The authors employed a quasi-experimental design to compare average retention and persistence rates of students enrolled in learning community developmental English courses with those of students enrolled in traditional developmental English courses in the same semester.
The study found that retention rates were higher for students who enrolled in learning community developmental courses compared with their counterparts who took the same courses but in the traditional format. Furthermore, for the four semesters studied, learning community students were more likely to enroll in courses the following semester than were students who were not enrolled in learning communities. 
The quality of causal evidence provided in this study is low because authors did not use sufficient controls in the analysis. This means we are not confident that the estimated effects are attributable to the learning communities. Other factors are likely to have contributed.","A learning community is an instructional model that provides students enrolled in linked courses opportunities to interact and engage with one another in a supportive learning environment. The learning community assessed in this study consisted of developmental English writing and reading courses at one and two levels below college-level English. Aside from the unique learning environment, students also received special services, such as in-course tutoring. In addition, course faculty received professional development and planning time to develop course themes.","The authors collected academic and administrative data on students enrolled in developmental English courses in a community college located in an urban area of southern California. The study evaluated students who enrolled in one of two developmental English course sequences: English 042 and 043 (two levels below college-level English) and English 051 and 056 (one level below college-level English) during any of the following four semesters: fall 2007, spring 2008, fall 2008, and spring 2009.
The intervention outcomes of interest in this study included (1) retention rates, which were equal to the proportion of students who completed the course relative to the total number of students enrolled after the drop deadline; and (2) persistence, defined as whether the student reenrolled the semester following completion of the developmental course sequence. The study used a quasi-experimental comparison group design, in which the authors compared retention and persistence rates of learning community and nonlearning community students to test for significant differences between groups.
The intervention sample included all 760 students enrolled in developmental English learning communities during the study period (fall 2007 to spring 2009). The comparison group consisted of a random sample of 760 students enrolled in the same developmental courses, but who took them in the traditional format that was not part of a learning community. For the persistence measure, however, only those students who successfully completed the developmental English course were evaluated, which reduced the overall sample size used for analysis by about half (61 percent of the original treatment group and 38 percent of the original comparison group).","The study found that retention rates were significantly higher for students enrolled in the learning community developmental courses English 043 and English 051 compared with their counterparts who took the same courses but in the traditional format. In addition, when courses were grouped by level (that is, English 042/043 and English 051/056) the differences between the treatment and comparison groups were statistically significant for both sets of courses.
Persistence rates in the grouped English 051/056 courses for fall 2007 to spring 2008 and for fall 2008 to spring 2009 were significantly higher for learning community students than for comparison students. In addition, persistence rates for fall 2008 to spring 2009 in English 051 were significantly higher for learning community students than for comparison students.","The authors did not account for self-selection into learning communities by including control variables in their analyses or by demonstrating that the treatment and comparison groups were similar on key characteristics before the intervention. In fact, the authors established that the treatment and comparison groups differed on observable characteristics (gender and ethnicity) in the analytic sample. Self-selection of treatment students into learning communities is a concern because it can bias estimates of the impacts of the intervention on outcomes. For example, more motivated students might be more likely to enroll in learning communities and to achieve higher persistence and retention rates. A lack of statistical controls for possible systematic differences between the treatment and comparison groups raises concerns that factors other than the intervention could account for the observed differences in retention and persistence rates. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,,,"United States, Urban","Causal Impact Analysis","Adult, Other barriers","Basic skills, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Success of joint programs between junior and senior colleges (Krudysz et al. 2012)","Success of joint programs between junior and senior colleges (Krudysz et al. 2012)","Study Type: Causal Impact Analysis","Krudysz, M., Walser, A., & Alting, A. (2012). Success of joint programs between junior and senior colleges. American Society for Engineering Education.",,2012,https://www.asee.org/public/conferences/8/papers/3558/download,"February 2016","Community College",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of the Joint/Dual (JD) program on retention and graduation rates for community college and undergraduate engineering students who were under-represented minorities and women. JD students received dual admission to a community college as well as a senior college, and transitioned into the upper division of the baccalaureate engineering program at the senior college upon successful completion of lower-division courses and degree requirements at the community college.
This study used data from junior and senior colleges within a major urban university system to compare outcomes of a treatment group of JD program participants and a comparison group of transfers into the urban university system from other, nonparticipating community colleges.
This study found that first- and second-year retention rates were higher for JD program participants than for transfers from other community colleges, but did not conduct statistical tests of these differences.
The quality of causal evidence presented in this report is low because the authors did not include controls for relevant student characteristics in the analysis. This means we are not confident that the estimated effects are attributable to the JD program. Other factors are likely to have contributed.","The JD program was designed to offer increased educational opportunities for under-represented minorities and women who intended to transfer from community colleges to senior colleges and pursue an engineering degree. JD students received dual admission to the community college as well as the senior college, and transitioned into the upper division of the baccalaureate engineering program at the senior college upon successful completion of lower-division courses and degree requirements at the community college. Program participants received special orientation and advising sessions, application fee waivers, and straightforward transfer credit evaluation. In addition, they were eligible for early registration and were able to register for courses at the senior college while still attending the junior college.","The study used a nonexperimental design to examine the impacts of the JD program on retention rates in junior and senior colleges in a major urban university system. The treatment group consisted of JD program participants who attended community and senior college at the same time for the first two years of undergraduate study, and who majored in civil, chemical, mechanical, or electrical engineering. The comparison group consisted of students who transferred to a senior college within the university system from other community colleges where no JD program was implemented. The study examined two sets of comparison transfer students: those with Associate in Science (AS) degrees and those without AS degrees.","This study found that first- and second-year retention rates were higher for JD program participants than for transfers from other community colleges, but did not conduct statistical tests of these differences.","The study did not include information on pre-intervention characteristics of treatment and comparison groups and did not include any controls in the analysis. This lack of statistical controls raises concerns that factors other than participation in the JD program could account for observed differences in retention rates between treatment and comparison students. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,,,"Urban, United States","Causal Impact Analysis","Adult, Female, Other","Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"The impact of curricular learning communities on furthering the engagement and persistence of academically underprepared students at community colleges (McIntosh 2012)","The impact of curricular learning communities on furthering the engagement and persistence of academically underprepared students at community colleges (McIntosh 2012)","Study Type: Causal Impact Analysis","McIntosh, J. (2012). The impact of curricular learning communities on furthering the engagement and persistence of academically underprepared students at community colleges (Doctoral dissertation). Retrieved from ProQuest Dissertation and Theses database. (UMI No. 3509985)",,2012,,"February 2016","Community College",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of basic skills curricular learning communities on the persistence of underprepared community college students at 13 community colleges in eight states. Curricular learning communities enroll cohorts of students in sets of two or three linked courses.
The study used student survey and administrative enrollment data in a quasi-experimental design in which the treatment group enrolled in the basic skills curricular learning communities and the comparison group did not enroll in the learning communities but took other, similar courses.
The study found that students enrolled in the basic skills curricular learning communities were 1.27 times more likely to be enrolled in a postsecondary educational institution in the year following program enrollment, compared with students not enrolled in learning communities.
The quality of causal evidence presented in this report is low because the author did not include sufficient controls in the analysis. This means we are not confident that the estimated effects are attributable to basic skills curricular learning communities. Other factors are likely to have contributed.","In the basic skills curricular learning communities, a cohort of students enrolled in the same set of two or three linked courses. Instructors for the linked courses worked together to create an interdisciplinary learning environment, making connections in content across the linked courses. These learning communities aimed to bolster connections between students and build a sense of community through students attending multiple classes with the same group of peers.
This study included 13 community colleges in eight states: 4 colleges in California; 3 in Washington; and 1 each in Florida, Maryland, Massachusetts, New Jersey, New York, and North Carolina. These community colleges offered basic skills curricular learning communities specifically designed for academically underprepared students, or students whose placement test scores or academic history indicated they might struggle to complete college-level courses. All schools in the sample had a history of successful learning communities and served at-risk students (first generation college, minority, or nontraditional students, and so on). At these schools, eligible students enrolled in basic skills and general education courses together as a cohort. The intervention lasted at least one semester or quarter, depending on the school.","This study used a quasi-experimental design. Colleges were selected through an application process, and all students at those colleges enrolled in basic skills curricular learning communities in fall 2003 were selected for participation in the study as the treatment group. Coordinators at the community colleges identified courses that covered similar content and had students with similar characteristics to the treatment group. All of the students enrolled in the similar courses in fall 2003 were assigned to the comparison group. The author used data from the National Student Clearinghouse for enrollment information and from a modified version of the Community College Survey of Student Engagement for background information and reported engagement with instructors and classmates. The author used student survey data from fall 2003 and administrative enrollment data from fall 2004. In total, 6,272 students made up the full sample, and 3,733 responded to the survey. The author estimated regression models comparing the outcomes of treatment and control group members, controlling for demographic and other characteristics.","The study found that students enrolled in the basic skills curricular learning communities were 1.27 times more likely to be enrolled in a postsecondary educational institution the following year, compared with students not enrolled in the learning communities. This difference was statistically significant.","The author used multivariate regressions to measure the relationship between participating in the curricular learning communities and students’ persistence. The regressions controlled for students’ demographic characteristics, the institution’s academic calendar system (semester or quarter), and pre-intervention socioeconomic status and academic achievement. However, the author did not control for the state in which the community college was located. This lack of statistical controls raises concerns that factors other than participation in basic skills curricular learning communities could account for the observed differences in persistence between treatment and comparison students. For example, differences in state education policies could lead to differences in students’ persistence, regardless of participation in curricular learning communities. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,,,"United States","Causal Impact Analysis","Adult, Other barriers","Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Does attending a STEM high school improve student performance? Evidence from New York City (Wiswall et al. 2014)","Does attending a STEM high school improve student performance? Evidence from New York City (Wiswall et al. 2014)","Study Type: Causal Impact Analysis","Wiswall, M., Stiefel, L., Schwartz, A., & Boccardo, J. (2014). Does attending a STEM high school improve student performance? Evidence from New York City. Economics of Education Review, 40, 93-105.",,2014,,"April 2016","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of New York City (NYC) STEM high schools on math and science test-taking behavior and scores.
The study used regression analyses to compare outcomes for students who attended a STEM-focused high school with those who attended a regular public high school. The study used administrative data from the NYC Department of Education.
The study found that young women attending STEM high schools in NYC were more likely to take various New York State Regents Examinations in math and science and to score higher on biology tests compared with young women who attended non-STEM public high schools.
The quality of causal evidence presented in this study is low because the treatment and comparison groups were not equivalent on key measures at baseline. This means we are not confident that the estimated effects are attributable to STEM high schools; other factors are likely to have contributed.","STEM high schools in NYC teach specific science and math programs to every student. Admission to these schools is based on a number of criteria, including student preference, school-specific admission criteria, and space availability. In addition to offering this specialized curriculum, STEM high schools employ different teachers than do the regular public high schools—on average, NYC STEM teachers have more experience teaching and are more likely to have earned a master’s degree than their counterparts at regular public high schools. Teachers at these schools also earn higher salaries than those in the traditional public schools. Because the study cannot disentangle the effects of the STEM programs themselves and the different teacher characteristics, this review considers STEM teachers to be a component of the treatment condition as implemented in this study.","The study used data from high schools in the NYC public school system, which included 30 STEM schools. There were 181 non-STEM schools in 2007 and 227 in 2008 included as comparisons. The authors used regression analyses to compare STEM with non-STEM schools on both the probability of taking New York State Regents Examinations in various subjects (Math A, Math B, Chemistry, Physics, and Biology) as well as the scores received on those tests. Students in New York must pass a series of five tests in order to earn a Regents high school diploma, including at least one math and one science exam, or can earn an Advanced diploma by passing additional exams (a non-Regents diploma can be earned by passing any one exam). The analyses controlled for middle school scores in math and reading, race and ethnicity, free lunch eligibility, English language use at home, limited English proficiency, and age. The authors ran regression models by gender to estimate gender gaps both within and across school types.","The study found that female students in NYC STEM schools were significantly more likely to take Math B, Chemistry, and Physics Regents exams and to earn higher scores on the Biology exam compared with female students in non-STEM schools.","Students in NYC chose whether to attend a STEM or non-STEM school. The authors were unable to control for unobservable differences between the two types of students, including potential selection factors such as parental support and student motivation, that might influence a student’s decision regarding which type of school to attend. Thus, estimated effects might reflect differences in outcomes that are due to these differences in student characteristics as well as the impact of the STEM school.
The study measured the effectiveness of STEM high schools in NYC. The results might not be externally valid in other cities, particularly if other STEM schools do not share the same characteristics as the NYC STEM schools, including offering higher teacher salaries and hiring more educated and experienced teachers.",,,,,,"United States","Causal Impact Analysis",Youth,"Science, Technology, Engineering, and Math (STEM) programs, Youth programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"Improving student outcomes via comprehensive supports: Three-year outcomes from CUNY’s Accelerated Study in Associate Programs (ASAP) (Kolenovic et al. 2013)","Improving student outcomes via comprehensive supports: Three-year outcomes from CUNY’s Accelerated Study in Associate Programs (ASAP) (Kolenovic et al. 2013)","Study Type: Causal Impact Analysis","Kolenovic, Z., Linderman, D., & Karp, M. (2013). Improving student outcomes via comprehensive supports: Three-year outcomes from CUNY’s Accelerated Study in Associate Programs (ASAP). Community College Review, 41(4), 271-291.",,2013,,"March 2016","Community College",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of the Accelerated Study in Associate Programs (ASAP) conducted by the City University of New York (CUNY) on retention, credit accrual, transfer, and degree attainment at six CUNY campuses.
The authors used administrative data from the CUNY Office of Institutional Research and Assessment to match students who participated in ASAP with similar students who enrolled the previous year and did not participate in ASAP and compared their outcomes.
The study found that ASAP students had higher one-year retention rates, were more likely to graduate within two or three years, and accrued more credits than comparison students.
The quality of causal evidence presented in this report is low because the analysis compared students in different cohorts, and we cannot rule out the possibility that differences in outcomes between the two groups were due to other changes taking place at the same time that only affected one of the cohorts. This means we are not confident that the estimated effects are attributable to ASAP; other factors likely contributed.","CUNY, in conjunction with the Center for Economic Opportunity, developed ASAP in 2007. The program aimed to support community college students seeking associate’s degrees by providing enhanced services. The program was implemented in all six CUNY community colleges and included the following components: (1) a requirement to attend college full time; (2) block-scheduled classes with other ASAP students for the first year of the program; (3) participation in an ASAP seminar that covers goal-setting, study skills, and academic planning for at least two semesters of the program; (4) free use of textbooks during the semester; (5) comprehensive advising from an ASAP adviser with a small caseload of 60 to 80 students; (6) tutoring; (7) career services; (8) a tuition waiver that covered any gap between a student’s financial aid and tuition and fees; and (9) free monthly MetroCards. ASAP encouraged participants to complete their developmental education early and graduate within three years. Therefore, ASAP included only students in approved majors. Approved majors were linked to New York’s economic needs and transfer opportunities within the CUNY system and had requirements that could reasonably be completed in three years.","Students in the study’s treatment group were New York City residents, had fewer than 12 college credits, were pursuing an associate’s degree in one of the approved majors for ASAP, were not enrolled in any other special CUNY support program, and had completed developmental education coursework. The authors used propensity-score matching procedures to select comparison group students who entered CUNY in 2006 and were similar to treatment students who entered in 2007. They estimated impacts by comparing outcomes of ASAP students to those of the matched comparison students. The authors matched ASAP and comparison students using the following variables: gender, race and ethnicity, age, college attended, exemption from taking CUNY assessment tests, continuing or transfer student status, dependency status for financial aid purposes, and Pell and New York State’s Tuition Assistance Program grant receipt as a proxy for income. Analyses used student-level data and followed students from initial enrollment into CUNY or ASAP through three academic years.","The study found that the one-year retention rate for ASAP students was 11.9 percent higher than that of the matched sample of comparison students, a statistically significant difference.
After three semesters, ASAP students had earned an average of 3.7 more credits than the matched comparison students. They also accumulated significantly more credits (an average of 6.6 more) than matched comparison students after three years of college enrollment. Both differences were statistically significant.
Two and three years after enrollment, 30.2 and 54.7 percent of ASAP students earned a degree, compared with 12.1 and 26.4 percent of matched comparison students, respectively. Both were statistically significant differences.","The treatment group and the comparison group were from different cohorts—the treatment group entered CUNY in 2007, and the comparison group entered CUNY in 2006. Thus, we cannot rule out the possibility that differences in outcomes between the treatment and comparison groups were due to other changes taking place at the same time that only affected one of the study groups. For example, suppose a policy change that was enacted in 2007 decreased income thresholds for financial aid. This change could have made it easier for ASAP students to remain at CUNY and/or earn more credits. In this case, any observed differences in outcomes between ASAP and comparison students could be due to the policy change rather than ASAP.",,,,,,"United States","Causal Impact Analysis",Adult,"Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Workforce program performance indicators for the Commonwealth of Virginia. Upjohn Institute technical report no. 08-024. [TANF/VIEW]  (Hollenbeck et al. 2008)","Workforce program performance indicators for the Commonwealth of Virginia. Upjohn Institute technical report no. 08-024. [TANF/VIEW]  (Hollenbeck et al. 2008)","Study Type: Causal Impact Analysis","Hollenbeck, K., Huang, W.-J., and WE Upjohn Institute for Employment Research (2008). Workforce program performance indicators for the Commonwealth of Virginia. Upjohn Institute technical report no. 08-024. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research. [TANF/VIEW]","Hollenbeck, K. (2011). “Short-term net impact estimates and rates of return.” In Douglas J. Besharov and Phoebe H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 371–295). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.",2008,http://research.upjohn.org/cgi/viewcontent.cgi?article=1027&context=up_technica…,"February 2017","Low-Income Adults",,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Virginia Initiative for Employment Not Welfare (TANF/VIEW) program on employment and credential completion rates.
The authors used administrative records to compare outcomes for low-income adults who took part in the TANF/VIEW program to a nonexperimental matched group of adults who did not take part in the program.
The study found that, compared with those who did not take part in the program, the employment rate was significantly lower for those who took part in the TANF/VIEW program, but the TANF/VIEW group earned significantly more educational or training credentials.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before program participation. This means we are not confident that the estimated effects are attributable to the TANF/VIEW program; other factors are likely to have contributed.","The TANF/VIEW program aimed to help adults who received TANF move into employment. The program provided services such as child care and transportation and could have included education and training support.","Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared those who exited the TANF/VIEW program from July 2004 to June 2005 with those who did not take part in the program and used the Employment Services program. The authors used a nonexperimental statistical approach called propensity-score matching to create an Employment Services group that was similar in terms of demographics, education, and labor market history. The authors compared the two groups on employment two and four quarters after program exit, and on the percentage who had earned an education or training credential within one year of program exit. This analysis included 6,373 participants of the TANF/VIEW in Virginia.","Employment. The study found that TANF/VIEW participants were significantly less likely to be employed after program exit than the Employment Services group, with an employment rate that was 2.1 percentage points lower at the second quarter and 2.0 percentage points lower at the fourth quarter after program exit.
Education and/or training attainment and completion. The study found that those who took part in the TANF/VIEW program were significantly more likely to complete an educational or training credentials than those who did not take part in the program (0.4 percentage points higher).","Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the TANF/VIEW and Employment Services participants were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in TANF/VIEW services was six months, whereas that for Employment Services was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would be looking at TANF/VIEW participants’ earnings about 12 months after they started receiving services and Employment Services participants’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the TANF/VIEW participants earned more 6 months after their exit dates. However, this would not be attributable to receiving TANF/VIEW services; it would be caused by the different elapsed time across the groups (12 months for TANF/VIEW participants versus 7 months for Employment Services participants). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Low income","Job search assistance and supportive services, Other employment and reemployment, Other training and education","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Education and skills gains, Employment","Low-Income Adults Review Protocol"
"Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [TAA] (Hollenbeck & Huang 2008)","Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [TAA] (Hollenbeck & Huang 2008)","Study Type: Causal Impact Analysis","Hollenbeck, K., & Huang, W.-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [TAA]","Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 371–295). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.",2008,http://research.upjohn.org/cgi/viewcontent.cgi?article=1027&context=up_technica…,"February 2017",,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Training Adjustment Assistance (TAA) program on employment and credential completion.
The study used administrative records to compare outcomes of low-income adults who took part in the TAA program with outcomes of a nonexperimental matched group of adults who did not take part in the program.
The study found that TAA participants had a lower employment rate but were more likely to have obtained a training certificate compared with those who did not participate in the program.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before program participation. This means we are not confident that the estimated effects are attributable to TAA; other factors are likely to have contributed.
This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.","Established under the Trade Act of 1974 and amended through the Trade Adjustment Assistance Reform Act of 2002, the TAA Program was intended to provide aid to adult workers (18 and older) within an industry or group whose employment had been lost or reduced as a result of foreign trade competition. The available services included employment and case management services, job search assistance and allowances, training, relocation allowances, and some cash income supports. Eligible employees must have lost their jobs or have been notified of unemployment risk because of import competition or shifts in foreign production.","Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the TAA program with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors used a nonexperimental statistical approach called propensity-score matching to compare TAA participants with similar nonparticipants. The authors compared the two groups on employment two and four quarters after program exit, and on the percentage of each group who had earned an educational credential during the program or within one year of program exit. This analysis included 2,254 TAA participants in Virginia who exited the program from July 2004 to June 2005.","Employment. The study found that TAA participants were significantly less likely to be employed than members of the comparison group, with an employment rate that was 6.2 percentage points lower in the second quarter and 5.6 percentage points lower in the fourth quarter after program exit.
Education and/or training attainment and completion. The study found that TAA participants were significantly more likely than members of the comparison group to earn an educational credential during the program or within one year of program exit (a difference of 67 percentage points).","The authors presented unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the authors.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the TAA participants and the comparison group members (who participated in the employment services program only) were on identical wage trajectories before receiving services from their respective programs, and that the average length of participation in TAA services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe TAA participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the TAA participants earned more 6 months after their exit dates. However, this would not be attributable to receiving TAA services; it would be caused by the different elapsed time across the groups (12 months for TAA participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date rather than entry date cannot receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Low-skilled, Low income, Dislocated or displaced worker","Job search assistance and supportive services, Other training and education","Education and skills gains-Low-Favorable impacts, Employment-Low-Unfavorable impacts","Education and skills gains, Employment","Review Protocol"
"Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [WIA-Adult] (Hollenbeck & Huang 2008)","Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [WIA-Adult] (Hollenbeck & Huang 2008)","Study Type: Causal Impact Analysis","Hollenbeck, K., & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [WIA-Adult]","Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.",2008,http://research.upjohn.org/cgi/viewcontent.cgi?article=1027&context=up_technica…,"March 2017","Job Search Assistance",,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Workforce Investment Act (WIA) Title I program for adults (including dislocated workers) on participants’ employment and credential completion rates.
The study used administrative records to compare the outcomes for low-income adults who took part in the WIA program with a nonexperimental-matched group of adults who did not take part in the program.
The study found that the WIA program participants had higher employment and credential completion rates than those of people who did not participate in the program.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to the WIA Title I program for adults (including dislocated workers); other factors are likely to have contributed.
This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.","The WIA adult program was authorized by Title I of the Workforce Investment Act of 1998 and was superseded by the Workforce Innovation and Opportunity Act (WIOA), effective in July 2015. WIA adult program services, which remain essentially the same under WIOA, were designed to provide quality employment and training services to eligible workers. Administered through Local Workforce Investment Areas, the adult program serves all people ages 18 and older through a set of core services, including job placement assistance, skills assessments, and provision of information on the labor market, among other services. In addition, those unable to obtain a job through core services alone can receive intensive services—which include counseling and specialized assessments—and vouchers for attending training. Some local areas provide supportive services such as child care, transportation, and work-related financial assistance to those who qualify.","Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the WIA program with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors compared the two groups on employment two and four quarters after program exit and on the percentage of each group that had earned an education credential during the program or within one year of program exit. This analysis included 6,373 WIA participants in Virginia who exited the program from July 2004 to June 2005.","Employment. The study found that the employment rate of WIA participants was significantly higher than that of the comparison group, with an employment rate that was 4.7 percentage points higher in the second quarter and 3.4 percentage points higher in the fourth quarter after program exit.
Education and/or training attainment and completion. The study found that WIA participants were significantly more likely than members of the comparison group to have earned an education credential during the program or within one year of program exit (54 percentage points higher).","The authors presented unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the authors.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the WIA participants and comparison group members were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in WIA services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe WIA participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the WIA participants earned more 6 months after their exit dates. However, this would not be attributable to receiving WIA services; it would be caused by the different elapsed time across the groups (12 months for WIA participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Unemployed, Low income","Adult and Dislocated Worker programs, Job search assistance and supportive services, Other employment and reemployment, Other training and education, Workforce Investment Act (WIA)/Workforce Innovation and Opportunity Act (WIOA)","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Job Search Assistance Review Protocol"
"Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [AEL] (Hollenbeck & Huang 2008)","Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [AEL] (Hollenbeck & Huang 2008)","Study Type: Causal Impact Analysis","Hollenbeck, K., & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [AEL]","Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.",2008,http://research.upjohn.org/cgi/viewcontent.cgi?article=1027&context=up_technica…,"May 2017",,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Adult Education and Literacy (AEL) program on participants’ employment and credential completion.
The study used administrative records to compare outcomes of low-income adults who took part in the AEL program with outcomes of a nonexperimental matched group of adults who did not take part in the program.
The study found that AEL participants had a lower employment rate but were more likely to have earned a GED compared with those of people who did not participate in the program.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to AEL; other factors are likely to have contributed.
This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.","The AEL program is designed to help adults improve basic skills in areas such as reading, writing, and mathematics. The program also provides English as a Second Language courses to help non-native English speaking adults improve their communication skills.","Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the AEL program with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors used a nonexperimental statistical approach called propensity score matching to compare AEL participants with similar nonparticipants. The authors compared the two groups on employment two and four quarters after program exit, and on the percentage of each group that had earned a GED within one year of program exit. The analysis included 3,716 AEL program participants in Virginia who exited the program from July 2004 to June 2005.","Employment. The study found that AEL participants were significantly less likely to be employed than the comparison group, with an employment rate that was 11.5 percentage points lower in the second quarter and 9.6 percentage points lower in the fourth quarter after program exit.
Education and/or training attainment and completion. The study found that AEL participants were significantly more likely than members of the comparison group to have earned a GED within one year of program exit (a difference of 70.5 percentage points).","The authors presented unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the author.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the AEL participants and comparison group members (who participated in the employment services program only) were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in AEL services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe AEL participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective, and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the AEL participants earned more 6 months after their exit dates. However, this would not be attributable to receiving AEL services; it would be caused by the different elapsed time across the groups (12 months for AEL participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Low-skilled, Low income","Other training and education","Education and skills gains-Low-Favorable impacts, Employment-Low-Unfavorable impacts","Education and skills gains, Employment","Review Protocol"
"Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [DRS] (Hollenbeck & Huang 2008)","Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [DRS] (Hollenbeck & Huang 2008)","Study Type: Causal Impact Analysis","Hollenbeck, K., & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [DRS]","Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.",2008,http://research.upjohn.org/cgi/viewcontent.cgi?article=1027&context=up_technica…,"May 2017",,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Vocational Rehabilitation program administered by Department of Rehabilitative Services (DRS) on participants’ employment and credential completion.
The study used administrative records to compare the outcomes of low-income adults who took part in the DRS program with outcomes of a nonexperimental matched group of adults who did not take part in the program.
The study found that DRS participants had higher employment and credential completion rates compared with those of people who did not participate in the program.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to DRS; other factors are likely to have contributed.
This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies","The Vocational Rehabilitation program administered by DRS is designed to help disabled veterans find and keep employment. Program services include job training, skills coaching, and job search assistance.","Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the Vocational Rehabilitation program administered by DRS with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors compared the two groups on employment two and four quarters after program exit, and on the percentage of each group that had earned an educational credential during the program or within one year of program exit. This analysis included 6,128 DRS participants in Virginia who exited the program from July 2004 to June 2005.","Employment. The study found that DRS participants were significantly more likely to be employed than members of the comparison group, with an employment rate that was 20.0 percentage points higher in the second quarter and 18.2 percentage points higher in the fourth quarter after program exit.
 Education and/or training attainment and completion. The study found that DRS participants were significantly more likely than members of the comparison group to earn an educational credential during the program or within one year of program exit (a difference of 6.2 percentage points).","The authors presented unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the authors.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the DRS participants and comparison group members were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in DRS services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe DRS participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the DRS participants earned more 6 months after their exit dates. However, this would not be attributable to receiving DRS services; it would be caused by the different elapsed time across the groups (12 months for DRS participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Disability, Low-skilled, Low income","Job search assistance and supportive services, Vocational rehabilitation","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Review Protocol"
"Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [DBVI] (Hollenbeck & Huang 2008)","Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [DBVI] (Hollenbeck & Huang 2008)","Study Type: Causal Impact Analysis","Hollenbeck, K., & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [DBVI]","Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.",2008,http://research.upjohn.org/cgi/viewcontent.cgi?article=1027&context=up_technica…,"April 2017",,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Vocational Rehabilitation program administered by the Department for Blind and Vision Impaired (DBVI) on participants’ employment and credential completion.
The study used administrative records to compare the outcomes of low-income adults who took part in the DBVI program with outcomes of a nonexperimental matched group of adults who did not take part in the program.
The study found that DBVI participants had higher employment and credential completion rates compared with those of people who did not participate in the program.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to DBVI; other factors are likely to have contributed.
This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.","The Vocational Rehabilitation program administered by DBVI is designed to help blind and visually impaired adults find and keep employment. Program services include job training, job placement assistance, support for medical procedures, provision of adaptive equipment, and support for job site accommodations.","Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the Vocational Rehabilitation program administered by DBVI with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors compared the two groups on employment two and four quarters after program exit, and on the percentage of each group that had earned an educational credential during the program or within one year of program exit. This analysis included 230 DBVI participants in Virginia who exited the program from July 2004 to June 2005.","Employment. The study found that DBVI participants were significantly more likely to be employed than members of the comparison group, with an employment rate that was 18.3 percentage points higher in the second quarter and 25.4 percentage points higher in the fourth quarter after program exit.
Education and/or training attainment and completion. The study found that DBVI participants were significantly more likely than members of the comparison group to earn an educational credential during the program or within one year of program exit (a difference of 4.1 percentage points).","The authors presented unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the authors.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the DBVI participants and comparison group members were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in DBVI services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe DBVI participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the DBVI participants earned more 6 months after their exit dates. However, this would not be attributable to receiving DBVI services; it would be caused by the different elapsed time across the groups (12 months for DBVI participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Disability, Low income","Supported employment or other employment supports, Vocational rehabilitation","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Review Protocol"
"Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [CTE] (Hollenbeck & Huang 2008)","Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). [CTE] (Hollenbeck & Huang 2008)","Study Type: Causal Impact Analysis","Hollenbeck, K., & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report No. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. [CTE]","Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.",2008,http://research.upjohn.org/cgi/viewcontent.cgi?article=1027&context=up_technica…,"April 2017",,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Carl Perkins Postsecondary Career and Technical Education (CTE) program on employment and credential completion.
The authors used administrative records to compare the outcomes of low-income adults who took part in the CTE program with the outcomes of a nonexperimental matched group of adults who did not take part in the program.
The study found that CTE participants had higher employment and credential completion rates compared with those of people who did not participate in the program.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups compared were similar before program participation. This means we are not confident that the estimated effects are attributable to CTE; other factors are likely to have contributed.
This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.","The postsecondary CTE program provides career-focused education though Virginia’s community college system. CTE students take courses and work towards credentials that will prepare them for specific occupational fields.","Using data from program administrative records, Unemployment Insurance records, the Wage Record Interstate System, and the community college system, the authors compared education and employment outcomes of those who took part in the Carl Perkins Postsecondary CTE program with outcomes of those who did not take part in the program (both groups participated in an employment services program). The authors compared the two groups on employment two and four quarters after program exit, and on the percentage of each group that had earned an educational credential during the program or within one year of program exit. This analysis included 30,058 CTE participants in Virginia who exited the program from July 2004 to June 2005.","Employment. The study found that CTE participants were significantly more likely to be employed than members of the comparison group, with an employment rate that was 1.0 percentage point higher in the second quarter and 1.9 percentage points higher in the fourth quarter after program exit.
 Education and/or training attainment and completion: The study found that CTE participants were significantly more likely than members of the comparison group to have earned an educational credential during the program or within one year of program exit (a difference of 86.3 percentage points).","The authors present unadjusted treatment effects in the study. This profile reports adjusted effects obtained directly from the author.
Although the authors accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the authors’ decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the CTE participants and comparison group members were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in CTE services was six months, whereas that for the employment services program was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe CTE participants’ earnings about 12 months after they started receiving services and comparison group members’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the CTE participants earned more 6 months after their exit dates. However, this would not be attributable to receiving CTE services; it would be caused by the different elapsed time across the groups (12 months for CTE participants versus 7 months for comparison group members). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Low income, Low-skilled","Other training and education, Job search assistance and supportive services","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Review Protocol"
"Cash transfer programme, productive activities and labour supply: Evidence from a randomised experiment in Kenya (Asfaw et al. 2014)","Cash transfer programme, productive activities and labour supply: Evidence from a randomised experiment in Kenya (Asfaw et al. 2014)","Study Type: Causal Impact Analysis","Asfaw, S., Davis, B., Dewbre, J., Handa, S., & Winters, P. (2014). Cash transfer programme, productive activities and labour supply: Evidence from a randomised experiment in Kenya. The Journal of Development Studies, 50(8), 1172-1196. doi: 10.1080/00220388.2014.919383",,2014,,"November 2018","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to assess the impact of a monthly cash transfer program on household food consumption, resources, and work, including child labor.
The study used survey data from a clustered randomized controlled trial. Households from seven districts in Kenya were randomly assigned to either the treatment or control group. Within those locations, 20 percent of the poorest households were recruited if they had one or more children 17 years old or younger who was an orphan or had a chronically ill parent or caregiver. 
The study found a statistically significant relationship between the program and a reduction in farm labor for children ages 10-15. This relationship was significant for boys but not for girls. There was also a statistically significant relationship between the program and an increase in school enrollment for children ages 12-18.
The quality of causal evidence presented in this report is low because it was based on a randomized controlled trial with high attrition and the authors did not demonstrate that groups were similar at baseline. This means we are not confident that the estimated effects are attributable to the Kenya Cash Transfer Programme for Orphans and Vulnerable Children; other factors are likely to have contributed.","The Kenya Cash Transfer Programme for Orphans and Vulnerable Children (CT-OVC) is the government’s primary poverty reduction program. It focuses on the poorest households with at least one vulnerable child defined as an orphan or child with a chronically ill parent or caregiver aged 17 or younger. The program aims to improve household economic outcomes and to improve the care of OVC through a monthly cash transfer (equivalent to $21 USD in 2007).
From 2007-2011, UNICEF contracted a randomized controlled trial to assess the impact of the CT-OVC program on household welfare indicators, including child labor. Seven Kenyan districts with high rates of poverty and HIV/AIDS were selected to implement the program. Since the program could not be implemented across all locations in the districts at the same time, four locations were selected within each district and were randomly assigned to either the treatment or control group. In the treatment locations, a committee of local citizens identified households to receive the monthly cash stipend based on poverty and the vulnerable child criteria described above, while households in the control group did not receive the stipend. Eligible households completed a socioeconomic questionnaire to confirm eligibility. The sample was narrowed through eligibility verification and availability of program funding per district, with preference first given to households that were child-led, and then to adult-led households, beginning with the oldest heads of household. In the control locations, the households identified by the local committee based on poverty and the vulnerable child criteria were included in the control group. The baseline sample included 2,294 households (1,542 treatment and 755 control). Treatment households received monthly cash transfers.","This study used survey data from the clustered randomized controlled trial to assess the effects of the CT-OVC program on child labor. Survey responses were examined at baseline and four years after the start of the program. The 2011 sample included 1,811 households (1,280 and 531 control). The authors used a regression model to compare farm labor participation rates between treatment and control group children ages 10-15. They also looked at the impact of the program on school enrollment.","Employment/Child labor
• The study found a statistically significant relationship between the CT-OVC program and a reduction in child labor. This relationship was significant for boys but not for girls.
Education (School participation/enrollment)
The study found a statistically significant relationship between the CT-OVC program and an increase in school enrollment for children ages 12-18. However, the authors did not provide the data associated with this finding.","The original study was a randomized controlled trial with high attrition from the sample. Because attrition affected the treatment and control groups’ comparability on background characteristics that could have affected the outcomes of interest, CLEAR reviews the study as a quasi-experimental design. While the study accounted for demographic differences between the treatment and control groups, the authors did not account for the child outcomes at baseline. As a result, preexisting differences between the groups—and not the program/intervention—could explain the observed differences in outcomes. In particular, there may have already been a lower rate of child labor in the treatment group prior to the study which could account for the significantly lower rate of child labor in the treatment group four years after the start of the program. Also, one of the Kenyan districts was not included in the assessment of the child labor outcomes and this may have created additional pre-existing differences between the two groups.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other barriers, Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Child education and work choices in the presence of a conditional cash transfer programme in rural Colombia (Attanasio et al. 2010)","Child education and work choices in the presence of a conditional cash transfer programme in rural Colombia (Attanasio et al. 2010)","Study Type: Causal Impact Analysis","Attanasio, O., Fitzsimons, E., Gomez, A., Gutierrez, M. I., Meghir, C., & Mesnard, A. (2010). Child education and work choices in the presence of a conditional cash transfer programme in rural Colombia. Economic Development and Cultural Change, 58(2), 181-210. doi:10.1086/648188.",,2010,http://openaccess.city.ac.uk/5381/1/Child_Education.pdf,"November 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The objective of this study was to examine the impact of the conditional cash transfer program, Familias en Acción, on children’s work and school participation in Columbia.
The study used a nonexperimental design to compare labor and schooling outcomes of children who received the conditional cash transfer with a comparison group of children who did not, based on data from a survey.
The study found that receipt of the cash transfer was significantly associated with a decrease in the likelihood of participating in domestic work for urban children aged 10 to 17, a decrease in the hours spent in income-generating work for urban children aged 14 to 17, and a decrease in the hours spent in domestic work for all but rural children aged 14 to 17. Receipt of the cash transfer was also significantly related to an increase in the hours spent in school for all children in urban and rural areas.
The quality of causal evidence presented in the study is low because the authors did not control for pre-intervention outcomes. This means we are not confident that the estimated effects are attributable to the Familias en Acción program; other factors are likely to have contributed.","The Familias en Acción program operates in Columbia and aims to alleviate poverty by providing financial subsidies for education, health, and nutrition. Eligible households receive a subsidy of $6.15 USD per child for primary school and $12.30 per child for secondary school. Children 7 to 17 years of age qualify for the program and subsidies are paid to the mother. The subsidy is conditional on attendance of 80 percent of classes. Mothers with children 0-6 are eligible to receive a monthly flat rate subsidy of $20.45 if they fulfill health care requirements including vaccinations, taking their children to growth and development check-ups, and attending courses on nutrition, hygiene, and contraception. In rural Columbia, the program has been operating since 2002.","Municipalities were eligible for the Familias en Acción program if they had a population of less than 100,000 individuals (not being a departmental capital), a bank, current welfare lists and other government documents, and basic health and education infrastructure. Households were eligible if they were in extreme poverty and had children under 18 years of age. Poverty was determined by a six-level indicator called SISBEN which is historically used to target households eligible for welfare programs (this program targeted SISBEN Level 1, extreme poverty). The study included a representative stratified sample of 122 municipalities (57 treatment and 65 comparison), stratified by region and infrastructure for education and health. Comparison municipalities satisfied eligibility requirements, except that they did not have a bank, and an effort was made to select areas that were similar in terms of population, area, and measures obtained from a quality of life index.
After the municipalities were chosen, approximately 100 households were randomly selected in each municipality for the treatment and comparison groups. In this study, the authors used a baseline survey conducted between June and October 2002 (n=11,500 households) and a follow-up survey conducted between July and November 2003 (n=10,742 households). In 26 of the 57 treatment municipalities, the baseline survey was conducted before the program started; however, knowledge of the program was widespread and registration had begun. In the other 31 areas, the baseline survey was conducted after the program started. Thus, the authors created early treatment (n=26 areas) and late treatment (n=31 areas) groups and only used the late treatment areas in analyses of work and schooling outcomes.
Child labor was measured by asking if the child participated in work (all work, income-generating, and domestic work) and the hours spent in work and school the day before the survey excluding Saturday and Sunday. The authors omitted children surveyed on Saturday and Sunday to avoid asking about weekend activities. Analyses were separated to determine program effects by location (urban or rural) and child age (10 to 13 or 14 to 17 years of age). The authors used statistical models with municipality, household, and individual controls to compare the outcomes of treatment and comparison group.","Employment/Child labor
The study found that receipt of the cash transfer was significantly associated with a decrease in the likelihood of participating in domestic work for urban children in both age groups. However, receipt of the cash transfer was not significantly associated with rates of domestic labor for rural children in either age group.
The study did not find a significant relationship between receipt of the cash transfer and rates of income-generating work (this includes work in the labor market and work in the family business).
The study found that receipt of the cash transfer was significantly associated with a decrease in the hours spent on income-generating work for urban children ages 14 to 17 years old. The program was not significantly related to hours spent on income-generating work for younger urban children, or rural children in either age group.
The study found that receipt of the cash transfer was significantly associated with a decrease in the hours spent on domestic work and the hours spent in total work for rural children ages 10 to 13 and for urban children in both age groups. The program was not significantly related hours spent on domestic work and the hours spent in total work for older rural children.
Education (School participation/enrollment)
• The study found that receipt of the cash transfer was significantly associated with an increase in the hours spent in school for all children in urban and rural areas.","The authors controlled for age, gender, and household education in the regression models but did not account for the outcomes at baseline, such as previous school attendance or child labor. These preexisting differences between the groups—and not the Familias en Acción program— could explain the observed differences in outcomes. Also, in 26 treatment municipalities, the baseline survey was conducted before the program started; however, knowledge of the program was widespread and registration had begun. The authors acknowledge that anticipation of the intervention was a problem in the late treatment group.",,,,"Absence of conflict of interest.",,"International, Rural","Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Are conditional cash transfers effective in urban areas? Evidence from Mexico (Behrman et al. 2012)","Are conditional cash transfers effective in urban areas? Evidence from Mexico (Behrman et al. 2012)","Study Type: Causal Impact Analysis","Behrman, J. R., Gallardo-Garcıa, J., Parker, S. W., Todd, P. E., & Velez-Grajales, V. (2012). Are conditional cash transfers effective in urban areas? Evidence from Mexico. Education Economics, 20(3), 233-259.",,2012,,"November 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of Mexico’s conditional cash transfer program (PROGRESA/Oportunidades) on school enrollment and employment in paid activities for children ages 6 to 18 in urban areas.
The study used a nonexperimental design to evaluate schooling and work outcomes from PROGRESA/Oportunidades program 1 and 2 years after implementation in urban areas.
The study found that PROGRESA/Oportunidades was significantly related to reduced labor rates for boys who were 12-14 in the first and second year, and girls ages 15 to 18 in the first year. The program was also significantly related to an increase in school enrollment for boys and girls.
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups were similar before program participation. This means we are not confident that the estimated effects are attributable to PROGRESA/Oportunidades; other factors are likely to have contributed.","The PROGRESA/Oportunidades program started in Mexican rural areas in 1997 and was modified and renamed Oportunidades in 2000. It has gradually expanded to urban areas. The program provides monthly conditional cash transfers to five million families. The cash transfer is conditional upon child school attendance and health clinic visits for family members. The amount of the cash transfer varies by gender and increases with the grade of the child. However, if a child repeats a grade twice, he or she will permanently lose the cash transfer.","In this study, authors used the Urban Evaluation Survey that was collected in 2002 prior to the announcement and start of the cash transfer, and follow-up data 1 year in 2003 and 2 years in 2004 post-implementation. The study utilizes a difference-in-differences propensity score matching design, which compares the changes in the outcomes of interest for the treatment group participating families with the outcomes for a matched set of program nonparticipants from the comparison group. The data included 2,972 households who were eligible, 2,556 households who were eligible and did not participate, and 3,607 households who were eligible but resided in areas that were not implementing the program.","Employment/Child labor 
The study found a statistically significant relationship between PROGRESA/Oportunidades and a decrease in child labor. Boys ages 12-14 in 2002 had decreased labor rates of about 8 percentage points in year 1 and 12-14 percentage points in year two. The program reduced child labor for girls ages 15 to 18 by 11 percentage points in year one.
Education (School participation/enrollment) 
The study found a statistically significant relationship between PROGRESA/Oportunidades and an increase in school enrollment for boys and girls ages 8 to 11, with a 2 to 3 percentage point increase in enrollment for the 1 and 2 year program impacts.","The authors created a matched group of non-participating households to compare to PROGRESA/Oportunidades participating-households. However, the authors did not account for the outcomes at baseline, such as previous school attendance or child labor. Preexisting differences between the groups—and not the program/intervention itself— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"International, Urban","Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Do conditional cash transfers for schooling generate lasting benefits? A five-year followup of PROGRESA/Oportunidades (Behrman et al. 2011)","Do conditional cash transfers for schooling generate lasting benefits? A five-year followup of PROGRESA/Oportunidades (Behrman et al. 2011)","Study Type: Causal Impact Analysis","Behrman, J. R., Parker, S. W., & Todd, P. E. (2011). Do conditional cash transfers for schooling generate lasting benefits? A five-year followup of PROGRESA/Oportunidades. Journal of Human Resources, 46(1), 93-122. doi:10.3368/jhr.46.1.93",,2011,https://www.mcgill.ca/isid/files/isid/behrman_et_al_2011.pdf,"November 2018","Child Labor",,"High Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of Mexico’s conditional cash transfer program (PROGRESA/Oportunidades) on long-term schooling and work for children ages 9 to 15 (during the time of the intervention).
Using survey data from a randomized controlled trial, the authors analyzed the impact of the PROGRESA/Oportunidades program on schooling and work 18-months after implementation. The authors also used a nonexperimental design to evaluate schooling and work outcomes five-and-a-half years post-implementation.
The study found that 18 months of program exposure significantly increased the school grade completion by 2.4 percent for boys and 2.7 percent for girls. The 18-month program exposure significantly decreased participation in work by 4.1 percent for boys but had no significant effect for girls. The study found a statistically significant relationship between the program and increased school grade completion five-and-a-half years post-implementation for all groups except for girls aged 17-19. It also found a statistically significant relationship between the program and a reduction in child labor for boys aged 15-16 in 2003 but did not find a significant relationship between the program for girls or for any other age groups of boys.
The quality of causal evidence presented in this report is high for short-term outcomes (18 months) because they are based on a well-implemented randomized controlled trial but low for longer-term outcomes (five-and-a-half years) because the authors did not account for the outcomes of schooling and work at baseline.","The PROGRESA program started in Mexican rural areas in 1997 and was modified and renamed Oportunidades in 2000. It has gradually expanded to urban areas. The program provides monthly conditional cash transfers to five million families. The cash transfer is conditional upon child school attendance and health clinic visits for family members. The amount of the cash transfer varies by gender and increases with the grade of the child. However, if a child repeats a grade twice, he or she will permanently lose the cash transfer.","The study used survey data from a randomized controlled trial. The original study randomly assigned 320 communities to the treatment group and 186 communities to the control group. After 18 months, the control group also began to receive the treatment (cash benefits).
In this study, authors used the 1997 baseline Survey of Household Socio-Economic Conditions and linked it with data from the 2003 rural Evaluation Survey of PROGRESA/Oportunidades. This linkage provided longitudinal data for children who were 9 to 15 in 1997 and 15 to 21 in 2003. The authors had complete data for 8,894 children in the treatment group and 5,591 children in the control group.
To test the longer-term impact of the PROGRESA/Oportunidades program on schooling and work, the authors used propensity score matching to create a comparable group of non-participating households from 2003 data. The non-participating households had similar household and community characteristics as the participating households. This new comparison group comprised households from 152 communities that had not been involved in the program in 2003.","Employment/Child labor
The study found that 18 months of program exposure significantly reduced participation in work by 4.1 percent for boys but had no significant effect for girls.
The study found a statistically significant relationship between the program and a reduction in child labor for boys aged 15-16 in 2003 but not for girls or any other age group of boys.
Education (School participation/enrollment)
The study found that 18 months of program exposure significantly increased the school grade completed by 2.4 percent for boys and 2.7 percent for girls.
The study found a statistically significant relationship between the program and increased school grade completion five-and-a-half years post-implementation for all groups with the exception of girls aged 17-19.","The authors created a matched group of non-participating households to compare to PROGRESA/Oportunidades participating-households. However, the authors did not account for the outcomes at baseline, such as previous school attendance or child labor. Preexisting differences between the groups—and not the program/intervention— could explain the observed differences in outcomes. Also, the education outcomes, but not the work outcomes, were the same for the original treatment and control groups at baseline.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor","Child labor-Mod/high-Favorable impacts, Child labor-Low-Favorable impacts, Education and skills gains-Mod/high-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"The impact of conditional cash transfer program on human capital formation in Brazil (Helfand & Souza 2010)","The impact of conditional cash transfer program on human capital formation in Brazil (Helfand & Souza 2010)","Study Type: Causal Impact Analysis","Helfand, S.M., & Souza, A.P. (2010). The impact of conditional cash transfer program on human capital formation in Brazil. University of California, Riverside, CA: Sao Paulo School of Economics.",,2010,,"December 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to analyze the impact of Brazil’s Bolsa Escola program on school enrollment, school progression, and child labor for children ages 6 to 15 in rural areas.
Using data from the 2001 and 2003 Brazilian Census, the authors used a structural model to examine the average program impact of the Bolsa Escola program.
The study found that the Bolsa Escola program was significantly associated with an increase in school enrollment and school progression. However, the study did not find a statistically significant relationship between program participation and child labor.
The quality of causal evidence presented in the study is low because the authors did not control for pre-intervention outcomes. This means we are not confident that the estimated effects are attributable to the Bolsa Escola program; other factors are likely to have contributed.","The Bolsa Escola Program was a conditional cash transfer program for poor families in urban areas of Brazil. The program gave cash grants to poor families with children between ages 7 and 14 conditional on school enrollment (15 Reais per month, equivalent to $6 USD). Households lose a month of benefits if their children miss too many days of school per month (the number of days is not specified in the article).","The authors used a non-experimental analysis. The study measured the impact of Brazil's Bolsa Escola Program on school attendance, school progression, and child labor in rural Brazil. The study used the 2001 and 2003 Brazilian Census to collect data on outcomes for children ages 6 to 15 who received the program. The Census has information about families who participated in a social program. The authors used the information on program participation to form a treatment group. The sample consisted of all families with at least two children, aged 6 to 15, who received the program. The authors used ages 6 to 15 to include baseline data for children who started the program at age 7 and endline data for children who were age 14 when starting the program. There was no comparison group. The study used a structural model to estimate program impacts. The model controlled for characteristics of the child and the household.","Employment/Child labor
The Bolsa Escola program was not significantly related to child labor.
Education (School participation/enrollment)
The study found a significant positive relationship between the program and school attendance. The program raised the probability of school attendance by 10 percentage points among children in a two-child family in rural areas, and by 0.5 percentage points in urban areas.
The Bolsa Escola program was significantly related to school progression. The program reduced the delay in school by 0.4 years among rural children in a two-child family and 0.115 years among urban children.","The study used data from the Brazilian Census to identify children who received the program. The Census question does not explicitly ask about participation in a named program. Rather, it asks if the child participates in a social program conditional on school enrollment. The authors assumed that the social program was Bolsa Escola.
Although the structural model controlled for the child’s age, gender, and household socioeconomic status, the authors did not account for child outcomes at baseline. The children might be enrolled in school prior to the intervention. Also, the families chose to participate in the program rather than being randomly assigned to the program. It is possible that unobservable factors, such as individual motivation to attend school, might have contributed to the estimated effects.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-No impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"The impact of cash transfers on child labor and school enrollment in Brazil (Cardoso & de Souza 2009)","The impact of cash transfers on child labor and school enrollment in Brazil (Cardoso & de Souza 2009)","Study Type: Causal Impact Analysis","Cardoso, E., & de Souza, A. F. (2009). The impact of cash transfers on child labor and school enrollment in Brazil. In P. F. Orazem, G. Sedlacek, & Z. Tzannatos (Eds.), Child labor and education in Latin America: An Economic perspective. Houndmills, U.K. and New York: Palgrave Macmillan.",,2009,https://ideas.repec.org/p/van/wpaper/0407.html,"December 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of two Brazilian income transfer programs (Minimum Income and Bolsa Escola) on child labor and school enrollment.
The authors used a nonexperimental design to compare the outcomes of children ages 10 to 15 that received the conditional cash transfers with those who had not, based on data from the 2000 Brazilian Census. Using several demographic characteristics, they created a matched comparison group of families who had not received the transfers but who had similar characteristics to the treatment group to assess the effectiveness of the cash transfer program.
The study found that the income transfer programs were significantly associated with an increase in school enrollment but not significantly associated with child labor.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the income transfer programs; other factors are likely to have contributed.","Brazil’s income transfer programs provide grants to children between the ages of 7 and 14 conditional on school attendance. The programs were implemented in 1995 in one district and expanded in 1996 across the country. By 1999, the authors estimate that there were 61 programs implemented across the country, under municipality jurisdiction. Additionally, the Minimum Income Program was developed by the Ministry of Education in 1998 with municipalities with lower per capita income, targeting children ages 7 to 14 in school. The program provided an average of 37 Reais per month for 504,000 families, assisting around one million children in 20 percent of Brazilian municipalities as of December 1999. In 2001, the federal government replaced the Minimum Income Program with Bolsa Escola Federal, which was implemented in 99.7 percent of Brazilian municipalities, and provided cash transfers to benefit five million children. The program paid 15 Reais ($6 USD) for each child in families with a per capita monthly income of below 90 Reais, conditional on school attendance.","The authors used a nonexperimental design (propensity score matching) to compare the outcomes of children ages 10-15 that received an income transfer with those who had not received it (waitlisted families), based on data from Brazil’s 2000 Census. The Census collected information on whether families received any income transfer or assistance from either the Minimum Income or Bolsa Escola programs, as well as unemployment insurance and handicap assistance. This study was focused on the Minimum Income and Bolsa Escola programs and dropped children from the sample if they had a handicapped member in their family, because the Census question included a handicapped aid transfer with the Bolsa Escola program, and the authors wanted to eliminate the effect of the handicapped transfer. Using several demographic characteristics (child, household, and community level) to assess the effectiveness of the cash transfer programs, the authors created a matched comparison group of families who reported that they had not received any income transfer to compare them to families in the treatment group who reported they had received an income transfer. The authors completed four analyses to include samples of: all families, poor children only, children with employed parents only, and poor children with employed parents only. The authors used a logit model separately for girls and boys to compare the schooling attendance and child labor outcomes of the treatment and control groups.","Employment/Child labor
The study did not find a significant relationship between the income transfer programs and child labor for boys or girls.
Education (School participation/enrollment)
The study found that participation in the income transfer programs was significantly associated with a 3-4 percentage point increase in school enrollment for boys and girls.","The authors created a matched group of non-participating eligible families to compare to income transfer participating families. However, the authors did not account for the outcomes at baseline, such as previous school attendance or child labor. Preexisting differences between the groups—and not the program/intervention— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-No impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Exploring the differential impact of public interventions on indigenous people: Lessons from Mexico's conditional cash transfer program (Lopez-Calva & Patrinos 2015)","Exploring the differential impact of public interventions on indigenous people: Lessons from Mexico's conditional cash transfer program (Lopez-Calva & Patrinos 2015)","Study Type: Causal Impact Analysis","Lopez-Calva, L. F., & Patrinos, H. A. (2015). Exploring the differential impact of public interventions on indigenous people: Lessons from Mexico's conditional cash transfer program. Journal of Human Development and Capabilities, 16(3), 452-467. doi:10.1080/19452829.2015.1072378",,2015,https://www.researchgate.net/publication/242464961_Exploring_the_Differential_I…,"December 2018","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The objective of the study was to examine the impact of Mexico’s conditional cash transfer program (PROGRESA/Oportunidades) on child labor and school attendance for children ages 8 to 17 across 3 indigenous, Spanish-speaking, and bilingual households.
Using survey data from a cluster randomized controlled trial, the authors analyzed the average program impact of the PROGRESA/Oportunidades program on indigenous households versus individuals who were bilingual or Spanish-speaking.
The study found that the PROGRESA/Oportunidades program was significantly related to a decrease in child labor and an increase in school attendance for indigenous speaking children.
The quality of causal evidence presented in this study is low because it was based on a randomized controlled trial with unknown attrition and the authors did not control for the required variables. As a result, we are not confident that the estimated effects are attributable to PROGRESA/Oportunidades; other factors are likely to have contributed.","Since 1997, the PROGRESA/Oportunidades program has provided monthly conditional cash transfers, which is approximately 20 percent of the average total household income. By 2012, PROGRESA/Oportunidades had served 5.8 million households in Mexico. The cash transfer is conditional upon child school attendance (85% of days) for children aged 8-17, and health clinic visits for household members, as well as the requirement for pregnant women and lactating women to take nutritional supplements and go to five pre-natal visits. The amount of the cash transfer varies by gender as girls are more likely to drop out of school at younger ages and increased with the grade of the child. However, if a child repeats a grade twice, he or she will permanently lose the cash transfer.","The study used survey data from a randomized controlled trial across rural communities in seven states in Mexico (320 in the treatment; 185 in the control). Poor households, based on a marginality index, were selected to participate in the program. In this study, authors used the 1997 baseline survey of Household Socioeconomic Characteristics (ENCASEH) as well as the Mexican national household survey, Encuesta de Evaluacion de los Hogares, in 2000, 1999, and 1997. The authors use multinomial logit, sequential probit, and difference- in-difference regressions, looking at differences between individuals who speak the indigenous language, those were bilingual, and those who were Spanish-speaking on the outcomes of child labor and school attendance. The difference-in-differences (DID) model explored the effect of PROGRESA/Oportunidades on the three language groups (indigenous, Spanish, and bilingual).","Employment/Child labor
The study showed that participation in PROGRESA/Oportunidades was significantly related to a decrease in the probability of working among children aged 8-17 in all language groups, with the largest reduction among indigenous speaking children.
Education (School participation/enrollment)
The study showed that participation in PROGRESA/Oportunidades was significantly related to an increase in the probability of school attendance among children aged 8-17 in all language groups, with the largest increase among indigenous speaking children.","The authors do not include attrition data for the randomized controlled trial. In cases of high or unknown attrition, a study can receive a moderate causal evidence rating if the analysis controls for possible differences in background characteristics of the analytic treatment and control groups. For the difference-in-differences model, the authors include a vector of household characteristics but not child gender.",,,,"Absence of Conflict of Interest.",,"International, Rural","Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Education and child labor: Experimental evidence from a Nicaraguan conditional cash transfer program (Maluccio 2009)","Education and child labor: Experimental evidence from a Nicaraguan conditional cash transfer program (Maluccio 2009)","Study Type: Causal Impact Analysis","Maluccio, J. A. (2009). Education and child labor: Experimental evidence from a Nicaraguan conditional cash transfer program. In P. F. Orazem, G. Sedlacek, & Z. Tzannatos (Eds.), Child labor and education in Latin America: An economic perspective (pp. 187-204). New York, NY: Palgrave Macmillan. doi.org/10.1057/9780230620100_12",,2009,,"December 2018","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of the Red de Proteccion Social (RPS) cash transfer program on school attendance and enrollment and on child labor for children aged 7-13 in rural Nicaragua.
Using panel survey data collected before and one year after program implementation, the author measured the impact of RPS using a randomized controlled trial (RCT).
The study found that participation in RPS was significantly related to a reduction in child labor and a significant increase in school enrollment and attendance.
The quality of the causal evidence presented in this report is low because, while it was based on an RCT, attrition was high and the authors did not demonstrate that the groups were similar at baseline. This means we are not confident that the estimated effects are attributable to the Red de Proteccion Social; other factors are likely to have contributed.","The Red de Proteccion Social (RPS) was a pilot government program to reduce both current and future poverty via cash transfers to households living in extreme poverty in rural Nicaragua. The transfers were conditional on school attendance and promotion to the next grade. The objectives of the program were to supplement the household income for up to three years to increase expenditures for food, increasing healthcare, and nutritional status of children under age 5, and reducing school dropout during the first four years of school (cash transfer equivalent to $112 USD annually in 2000).","Creation of the treatment and control groups occurred in three stages of geographic targeting. First, two departments (i.e., administrative or political subdivisions) were selected based on need and capacity to implement the program, including existing education and health facilities. Second, six municipalities within the departments were chosen based on poverty level and capacity to participate in the pilot program. Finally, 42 localities within the municipalities were selected using a marginality index based on 1995 National Population and Housing Census data. Localities were rank ordered by a marginality index then stratified into seven groups of six localities each. From each group, three localities were randomly selected into the treatment group, while the remaining three localities served as controls, resulting in 21 treatment localities and 21 control localities. All households in the treatment group received a $224 USD cash transfer annually; families with children ages 7 to 13 who had not completed the fourth grade received an additional $133 USD annually for a school attendance and school supplies transfer. The control group did not receive the cash transfers.
Data were obtained through a household level panel survey. At baseline, the survey was administered to a stratified random sample of 1,585 households, approximately 13 percent of the household population in the study area. A follow-up survey was also conducted to the original baseline households, re-interviewing 1,494 households.
The purpose of the analysis was to assess whether there were differences between the treatment and the control groups due to the RPS on child labor, and school attendance and enrollment for children 7 to 13 years of age.","Employment/Child labor
The study found a statistically significant relationship between the RPS program and a reduction in child labor. When compared to control group children, there was a significant reduction in the percentage of children working (9-percentage point difference) for RPS participating children ages 10 to 13.
Education (School participation/enrollment)
The study found a statistically significant relationship between the RPS program and an increase in school enrollment. When compared to control group children, there was a significant increase in the percentage of RPS participating children enrolled in school (22-percentage point difference).
The RPS was also significantly associated with an increase in school attendance, with an average 29-percentage point difference between the treatment and control group.","The original study was a randomized controlled trial with high attrition, which affected the treatment and control groups’ comparability on background characteristics that could have affected the causal outcomes of interest; CLEAR reviews the study as a regression analysis. The authors also did not report tests of baseline equivalence on the control variables. There is also no explanation as to how the localities were stratified into the seven groups before randomization into the treatment or control groups. Another consideration was the potential for measurable changes in localities due to the novelty of the program, or Hawthorne Effect, which may inflate the impact of the RPS on the treatment localities.",,,,"Absence of Conflict of Interest.",,"International, Rural","Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Effects of cash transfers on child labor and schooling in Kenya (Owoko 2014)","Effects of cash transfers on child labor and schooling in Kenya (Owoko 2014)","Study Type: Causal Impact Analysis","Owoko, S. A. (2014). Effects of cash transfers on child labor and schooling in Kenya (Unpublished master's thesis). Nairobi, Kenya: University of Nairobi.",,2014,,"December 2018","Child Labor",,"Low Causal Evidence",Pre/Post,,"The objective of the study was to assess the impact of Kenya’s Cash Transfer to the Orphaned and Vulnerable Children (CT-OVC) program on children’s participation in work and school.
The study used a post-test only design to assess differences in school enrollment and work participation between 5-17 year-old children in the CT-OVC program and those who were not in the program.
The study found that receipt of the cash transfer was significantly related to lower rates of child labor and higher rates of school enrollment.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to CT-OVC; other factors are likely to have contributed.","The Kenya’s Cash Transfer to the Orphaned and Vulnerable Children CT-OVC program was operated by the local government with assistance from the United Nations Children’s Fund (UNICEF). The program aimed to support the welfare of children considered orphaned or vulnerable; this included children who had one or more parent who died, or who had a parent with a chronic health condition. These children had become more common due to the HIV/AIDs epidemic, which led to the development of the program. The selection process for the CT-OVC program was designed to target households with an OVC, and was conducted with input from the local communities, government, and donors. Households participating in the program received a cash transfer. Per the authors, there were no conditions for receiving the cash transfer but those receiving the payment were told of their responsibilities to care of the OVC who lived in their household. The program was implemented in multiple regions of Kenya, but the study was limited to the Kwale District.","The study had a post-test only design that compared child labor and school participation outcomes between children in households who participated in the CT-OVC program and those in households that did not. The researchers divided the potential sample into groups based on the type of OVC: two parents decreased, one parent decreased, or having both parents alive, but with at least one who was chronically ill. The researchers randomly selected a sample for the treatment and comparison groups within those categories. For each of the CT-OVC and the comparison groups, 100 households were sent the survey (200 households total). The researchers conducted analyses with completed surveys received from 86 of households in the CT-OVC group, and 86 in the comparison group (172 total), after removing three CT-OVC surveys that were physically damaged and could not be used. The researchers used regression analyses to compare the percent of children ages 5-17 in each group who participated in child labor and school, controlling for differences in a number of characteristics such as: child gender and age, household head education level and occupation, household size, and the type of OVC (one or no parents, or parent with a chronic illness).","Employment/Child labor 
The study found that the cash transfers were significantly associated with lower rates of child labor (84% of children in the treatment group worked compared to 93% of children in the comparison group).
Education/School participation/Enrollment
The study found that the cash transfers were significantly associated with higher rates of school enrollment (98% of children in the treatment group were enrolled in school compared to 73% of children in the comparison group).","The authors did not account for preexisting differences between the groups before program participation, such as initial participation in child labor. The preexisting differences between the groups—and not the program—could explain the observed differences in outcomes.",,,,"Absence of Conflict of Interest.",,International,"Causal Impact Analysis","Other barriers, Other","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Cash transfers, conditions, school enrollment, and child work: Evidence from a randomized experiment in Ecuador (Schady & Araujo 2006)","Cash transfers, conditions, school enrollment, and child work: Evidence from a randomized experiment in Ecuador (Schady & Araujo 2006)","Study Type: Causal Impact Analysis","Schady, N. & Araujo, M. C. (2006). Cash transfers, conditions, school enrollment, and child work: Evidence from a randomized experiment in Ecuador. Policy Research Working Paper No. 3930. Washington, DC: The World Bank. Retrieved from https://openknowledge.worldbank.org/handle/10986/8452  ",,2006,https://openknowledge.worldbank.org/handle/10986/8452,"December 2018","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The purpose of the study was to assess the impact of the Bono de Desarrollo Humano (BDH) program on child labor and school enrollment for children 6-17 years old.
The study used a randomized controlled trial to compare the outcomes of children in households assigned to participate in the program to outcomes of those who did not participate.
The study found a significant relationship between being selected for the BDH program and whether children worked and hours per week children worked; both were significantly lower for those in the treatment group than those in the control group. The study also found that children selected into the BDH program had significantly higher rates of school enrollment and significantly lower rates of having dropped out since the program began.
The quality of causal evidence presented in this report is low because it is a randomized controlled trial with unknown attrition and the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Bono de Desarrollo Humano program; other factors are likely to have contributed.","The Bono de Desarrollo Humano (BDH) program was adapted from the Bono Solidario program, which was previously implemented. Like its predecessor, BDH was implemented in Ecuador to address a low rate of enrollment in secondary schooling. Although both programs aimed to serve poor households, the selection process for BDH was based on a wealth index to better select the target population. BDH was an unconditional cash transfer program, in which women received $15 monthly transfers for their household. While there were no conditions to receive the cash transfers, some of the local households thought that children were required to be enrolled in school because program administrators and television advertisements encouraged school enrollment when discussing the program. The program was also funded by the government, and included four provinces: Carchi, Imbabura, Cotopaxi, and Tungurahua. To be eligible to participate, households needed to have not previously received BDH or Bono Solidario transfers and be in the lowest 40 percent of wealth. The report did not specify whether having a child of a certain age was required to receive the transfers, but the analyses only included households with children ages 6-17.","The study used a randomized controlled trial design to test program impact. Parishes were randomly selected from the provinces, and then a total of 1,391 households were selected from those parishes to be randomized into the treatment and control group. The exact household sampling process was not described, however, the report said that households needed to be in the lowest 40 percent of wealth, that those close to the cut-off between the lowest and second lowest fifth of wealth were overrepresented, and that those selected had not previously received BDH or Bono Solidario transfers. Among the 1,391 households selected for the study, there was a total of 3,072 children. Of those households, half were randomly assigned to the treatment group and the others were randomly assigned to the control group. Households in the control group did not receive transfers or other benefits, but would potentially be able to receive the transfers after two years. However, some control households ended up receiving treatment benefits despite being in the control group. This may have occurred because some program administrators began giving out transfers before they had obtained the list of which households were selected, and did not want to discontinue giving out transfers to households who had already received them.
Two surveys were administered to households: a baseline survey given prior to start of the treatment that assessed child and household demographics and child labor and schooling outcomes (e.g., school enrollment, work participation), and a follow-up survey administered about a year and half afterwards that assessed child schooling and labor outcomes. The authors also used bank records to verify the receipt of transfers. The impact of the program was evaluated by assessing the difference in child labor and schooling outcomes between the treatment and control groups, controlling for baseline differences between the groups. The regression model included the relevant outcome variable (school enrollment or whether the child was working), along with household and child demographic characteristics, and the household’s canton (geographic area).","Employment/Child labor
The study found a significant relationship between BDH program participation and a reduction in the number of hours worked, the proportion of children working 1.5 years after the program began, and the proportion of children who started working between the baseline and 1.5 year follow-up survey. However, there was no significant relationship between program participation and the proportion of children who stopped working between the baseline and 1.5 year follow-up survey.
Education (School participation/enrollment)
The study found that program participation was significantly related to an increase in school enrollment 1.5 years after the program began and a decrease in school dropout. However, there was no a significant relationship between program participation and the proportion of children who enrolled in school between the baseline and follow-up survey.","While the study was a randomized controlled trial, the level of differential attrition was not reported. In addition, baseline equivalence could not be affirmed; while some outcome variables were controlled for in their respective analyses (e.g., enrolled in school, working); for some, that was not the case (e.g., hours worked). In addition, there was a high proportion of those in the control group who received the cash transfers and also a high proportion of those in the treatment group who did not receive the cash transfers, even though they were assigned to that group.",,,,"Absence of Conflict of Interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Protecting education for the poor in times of crisis: An evaluation of a scholarship programme in Indonesia (Sparrow 2007)","Protecting education for the poor in times of crisis: An evaluation of a scholarship programme in Indonesia (Sparrow 2007)","Study Type: Causal Impact Analysis","Sparrow, R. (2007). Protecting education for the poor in times of crisis: An evaluation of a scholarship programme in Indonesia. Oxford Bulletin of Economics and Statistics, 69, 99-122. doi: 10.1111/j.1468-0084.2006.00438.x",,2007,https://pdfs.semanticscholar.org/8f79/add16372936ea1e388e3a9c82e320435a87d.pdf,"December 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of Jaring Prengaman Sosial (JPS), a scholarship program in Indonesia, on child labor and school enrollment.
The author used data from an annual Indonesian national survey to estimate an instrumental variable (IV) regression model to examine if scholarship receipt affected school enrollment, and probit models to examine the effects on child labor and school attendance.
The study found that the receipt of a scholarship was significantly associated with a decrease in child labor and an increase in school attendance.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before receiving the JPS in the probit model and no evidence was provided on the strength of the instrument used in the IV model. This means we are not confident that the estimated effects are attributable to the Jaring Pengaman Sosial (JPS) scholarship program; other factors are likely to have contributed.","The Jaring Prengaman Sosial (JPS) scholarship started in the 1998-1999 academic year. Scholarships were targeted to children from poor households in grade 4 or higher. The scholarship amount increased with grade level: 10,000 rupiahs per month for enrollment in primary school; 20,000 rupiahs per month for enrollment in junior secondary school; and 25,000 rupiahs per month for enrollment in senior secondary school. The scholarship funds were distributed in three phases. During the first phase, the funds were distributed to districts based on poverty estimations from the 1996 Indonesian national economic survey (Susenas). During the second phase, district committees determined which schools would receive the funding based the economic conditions of the community that the school served. During the third and final phase, school committees assigned scholarships to children based on prosperity status, household composition (with preference given to single-parent and large households), and distance to school. The school committees were instructed to award at least half of the scholarships to girls.","The study used a nonexperimental design to compare the treatment group receiving the Jaring Prengaman Sosial (JPS) scholarship in 1999 to the comparison group that did not receive the JPS. The author used data from Indonesia’s 1998 and 1999 national economic survey, Susenas, for a sample of 113,187 children (ages 10 to 18). Using these data, the author estimated bivariate probit and IV regression models to analyze the relationship between JPS and both child labor and school enrollment. The JPS scholarships were implemented using decentralized targeting, which resulted in delivery of the program in some areas that should not have been targeted. The author capitalized on the mistargeting and used geographic separability (at the district level) to select the instrument for the instrumental variable analysis. The instrument chosen to identify the scholarships effect on outcomes was constructed from the actual poverty profile for the district in 1998 and the mistargeting due to selection rules (1996 poverty estimate and number of enrolled students in the district). In the probit models, the author controlled for age, household size, agricultural or non-agricultural income, head of household demographics, and schooling type (public or private).","Employment/Child labor
The receipt of a JPS scholarship was significantly associated with a decrease in the probability of child labor by 3.8 percentage points.
Education (School participation/enrollment)
The receipt of a JPS scholarship was significantly associated with an increase in the probability of school attendance in the previous week by 1.5 percentage points.
However, the receipt of a JPS scholarship was not significantly associated with school enrollment.","The author did not include a test of the instrument’s strength in his analysis of the IV model. Also, the author did not account for preexisting differences between the groups before program participation including baseline school attendance and child labor outcomes in the probit models. These preexisting differences—and not the JPS scholarship— could explain the observed differences in child labor and school attendance.",,,,"Absence of Conflict of Interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor, Tuition assistance","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomised controlled trial in Zimbabwe (Fenton et al. 2016)","Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomised controlled trial in Zimbabwe (Fenton et al. 2016)","Study Type: Causal Impact Analysis","Fenton, R., Nyamukapa, C., Gregson, S., Robertson, L., Mushati, P., Thomas, R., & Eaton, J. W. (2016) Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomized controlled trial in Zimbabwe. Psychology, Health & Medicine, 21(8), 909-917",,2016,https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5044770/,"December 2018","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the effects of conditional cash transfers (CCT) and unconditional cash transfers (UCT) on child labor and schooling outcomes. This summary focuses on the comparison between the CCT group and the control group.
The study was a randomized controlled trial in which households were randomly assigned to receive one of two cash transfers (conditional or unconditional) or to participate in the control group that did not receive a cash transfer. Using household survey data, the authors analyzed the impact of the treatment conditions on child labor and schooling outcomes a year after the intervention began.
The study found that receipt of a conditional cash transfer was significantly associated with a reduction in child work and increases in school attendance for the poorest households.
The quality of causal evidence presented in this report is low because it based on a randomized controlled trial with unknown attrition and the authors did not ensure that the groups being compared were similar before receiving the cash transfer. This means we are not confident that the estimated effects are attributable to the conditional cash transfer; other factors are likely to have contributed.","The intervention took place in Manicaland, Zimbabwe, an area where there is a high rate of HIV. Due to HIV, many children in that area have lost their parents. The program offered cash transfers to households to improve the children’s school attendance and child labor outcomes. There were two treatment conditions: a conditional cash transfer (CCT) and unconditional cash transfer (UCT). In both conditions, households received the equivalent of $18 USD every other month and $4 USD per child for a maximum of three children. However, to receive the stipend in the CCT condition, participants had to demonstrate that household children had attended the majority (80%) of school days in the last two months, had a vaccination record, applied for a birth certificate, and that the parents had participated in parenting classes. In both treatment conditions, households also received maize seed and fertilizer. To be eligible for the CCT or UCT program, children needed to live in households that were not in the wealthiest 20 percent, and whose household either had a child head of household, an orphan, a person with a disability, or was in the lowest 20 percent for wealth. The intervention was conducted by a non-governmental organization called the Diocese of Mutare Community Care Programme.","The study used a randomized controlled trial (RCT) in which households were assigned to either the CCT group, UCT group, or control group. Households were divided into 10 geographic sections and then further divided into three clusters. Within those clusters, households were randomly assigned to one of the two treatment groups or to the control group (1,319 assigned to the CCT group 1,525 to the UCT group, and 1,199 to the control group).
To assess the intervention, the study used a baseline and follow-up survey. The authors compared outcomes one year after the start of the intervention between each treatment and the control group, and between the two treatment groups and the control group using regression models. For school attendance, the authors compared the treatment groups and the control group with low attendance at baseline, to see what proportion of children had low attendance at follow-up. For child labor, the authors compared the hours of paid employment between each of the treatment groups and the control group, and between the two treatment groups.","Employment/Child labor
The study found that the receipt of the CCT was significantly associated with lower number of hours worked (an average of .31 fewer hours).
 Education (School participation/enrollment)
For the poorest households, the study found that the receipt of the CCT was significantly associated with higher rates of attendance.","The study was a randomized controlled trial with unknown attrition in each group. In cases of high or unknown attrition, a study can receive a moderate causal evidence rating if the analysis controls for possible differences in background characteristics of the treatment and control groups. However, the authors did not account for preexisting differences between the groups before program participation including child age, gender, and baseline school and child labor outcomes. These preexisting differences between the groups—and not the CCT program— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other barriers, Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomised controlled trial in Zimbabwe (Fenton et al. 2016)","Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomised controlled trial in Zimbabwe (Fenton et al. 2016)","Study Type: Causal Impact Analysis","Fenton, R, Nyamukapa, C., Gregson, S., Robertson, L., Mushati, P., Thomas, R., & Eaton, J.W. (2016). Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomised controlled trial in Zimbabwe. Psychology, Health & Medicine, 21(8), 909-917",,2016,https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5044770/,"December 2018","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the effects of conditional cash transfers (CCT) and unconditional cash transfers (UCT) on child labor and schooling outcomes. This summary focuses on the comparison between the UCT group and the control group.
The study was a randomized controlled trial in which households were randomly assigned to receive one of two cash transfers (conditional or unconditional) or to participate in the control group that did not receive a cash transfer. Using household survey data, the authors analyzed the impact of the treatment conditions on child labor and schooling outcomes a year after the intervention began.
The study found that receipt of an unconditional cash transfer was significantly associated with an increases in school attendance for the poorest households, but not significantly related to child labor outcomes.
The quality of causal evidence presented in this report is low because it based on a randomized controlled trial with unknown attrition and the authors did not ensure that the groups being compared were similar before receiving the cash transfers. This means we are not confident that the estimated effects are attributable to the unconditional cash transfers; other factors are likely to have contributed.","The intervention took place in Manicaland, Zimbabwe, an area where there is a high rate of HIV. Due to HIV, many children in that area have lost their parents. The program offered cash transfers to households to improve the children’s school attendance and child labor outcomes. There were two treatment conditions: a conditional cash transfer (CCT) and unconditional cash transfer (UCT). In both conditions, households received the equivalent of $18 USD every other month and $4 USD per child for a maximum of three children. However, to receive the stipend in the CCT condition, participants had to demonstrate that household children had attended the majority (80%) of school days in the last two months, had a vaccination record, applied for a birth certificate, and that the parents had participated in parenting classes. In both treatment conditions, households also received maize seed and fertilizer. To be eligible for the CCT or UCT program, children needed to live in households that were not in the wealthiest 20 percent, and whose household either had a child head of household, an orphan, a person with a disability, or was in the lowest 20 percent for wealth. The intervention was conducted by a non-governmental organization called the Diocese of Mutare Community Care Programme.","The study used a randomized controlled trial (RCT) in which households were assigned to either the CCT group, UCT group, or control group. Households were divided into 10 geographic sections and then further divided into three clusters. Within those clusters, households were randomly assigned to one of the two treatment groups or to the control group (1,319 assigned to the CCT group 1,525 to the UCT group, and 1,199 to the control group).
To assess the intervention, the study used a baseline and follow-up survey. The authors compared outcomes one year after the start of the intervention between each treatment and the control group, and between the two treatment groups and the control group using regression models. For school attendance, the authors compared the treatment groups and the control group with low attendance at baseline, to see what proportion of children had low attendance at follow-up. For child labor, the authors compared the hours of paid employment between each of the treatment groups and the control group, and between the two treatment groups.","Employment/Child labor
The study did not find a significant relationship between receipt of the UCT and the number of hours worked.
Education (School participation/enrollment) 
For the poorest households, the study found that the receipt of the UCT was significantly associated with higher rates of attendance.","The study was a randomized controlled trial with unknown attrition in each group. In cases of high or unknown attrition, a study can receive a moderate causal evidence rating if the analysis controls for possible differences in background characteristics of the treatment and control groups. However, the authors did not account for preexisting differences between the groups before program participation including child age, gender, and baseline school and child labor outcomes. These preexisting differences between the groups—and not the CCT program— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other barriers, Other, Low income","Child labor","Child labor-Low-No impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomised controlled trial in Zimbabwe (Fenton et al. 2016)","Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomised controlled trial in Zimbabwe (Fenton et al. 2016)","Study Type: Causal Impact Analysis","Fenton, R., Nyamukapa, C., Gregson, S., Robertson, L., Mushati, P., Thomas, R., & Eaton, J.W. (2016). Wealth differentials in the impact of conditional and unconditional cash transfers on education: Findings from a community-randomised controlled trial in Zimbabwe. Psychology, Health & Medicine, 21(8), 909-917.",,2016,https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5044770/,"December 2018","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the effects of conditional cash transfers (CCT) and unconditional cash transfers (UCT) on child labor and schooling outcomes. This summary focuses on the comparison between the CCT group and the UCT group.
The study was a randomized controlled trial in which households were randomly assigned to receive one of two cash transfers (conditional or unconditional) or to participate in the control group that did not receive a cash transfer. Using household survey data, the authors analyzed the impact of the treatment conditions on child labor and schooling outcomes a year after the intervention began.
The study found that the CCT group had significantly higher levels of school attendance than the UCT group for the least poor households. However, the study found no statistically significant difference in the number of hours worked between the two groups.
The quality of causal evidence presented in this report is low because it based on a randomized controlled trial with unknown attrition and the authors did not ensure that the groups being compared were similar before receiving the cash transfers. This means we are not confident that the estimated effects are attributable to the cash transfers; other factors are likely to have contributed.","The intervention took place in Manicaland, Zimbabwe, an area where there is a high rate of HIV. Due to HIV, many children in that area have lost their parents. The program offered cash transfers to households to improve the children’s school attendance and child labor outcomes. There were two treatment conditions: a conditional cash transfer (CCT) and unconditional cash transfer (UCT). In both conditions, households received the equivalent of $18 USD every other month and $4 USD per child for a maximum of three children. However, to receive the stipend in the CCT condition, participants had to demonstrate that household children had attended the majority (80%) of school days in the last two months, had a vaccination record, applied for a birth certificate, and that the parents had participated in parenting classes. In both treatment conditions, households also received maize seed and fertilizer. To be eligible for the CCT or UCT program, children needed to live in households that were not in the wealthiest 20 percent, and whose household either had a child head of household, an orphan, a person with a disability, or was in the lowest 20 percent for wealth. The intervention was conducted by a non-governmental organization called the Diocese of Mutare Community Care Programme.","The study used a randomized controlled trial (RCT) in which households were assigned to either the CCT group, UCT group, or control group. Households were divided into 10 geographic sections and then further divided into three clusters. Within those clusters, households were randomly assigned to one of the two treatment groups or to the control group (1,319 assigned to the CCT group 1,525 to the UCT group, and 1,199 to the control group).
To assess the intervention, the study used a baseline and follow-up survey. The authors compared outcomes one year after the start of the intervention between each treatment and the control group, and between the two treatment groups and the control group using regression models. For school attendance, the authors compared the treatment groups and the control group with low attendance at baseline, to see what proportion of children had low attendance at follow-up. For child labor, the authors compared the hours of paid employment between each of the treatment groups and the control group, and between the two treatment groups.","Employment/Child labor
 The study did not find a significant difference in the number of hours worked between the two groups.
Education (School participation/enrollment)
For the poorest households, the study did not find a significant difference in the attendance rates between the two groups.
For the least poor households, the study found a significantly higher proportion of those in the CCT group with a high level of school attendance than those in the UCT group.","The study was a randomized controlled trial with unknown attrition in each group. In cases of high or unknown attrition, a study can receive a moderate causal evidence rating if the analysis controls for possible differences in background characteristics of the treatment and control groups. However, the authors did not account for preexisting differences between the groups before program participation including child age, gender, and baseline school and child labor outcomes. These preexisting differences between the groups—and not the cash transfer programs— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other barriers, Other, Low income","Child labor","Child labor-Low-No impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Same bureaucracy, different outcomes in human capital? How indigenous and rural non-indigenous areas in Panama responded to the CCT (Arriaz & Rozo 2011)","Same bureaucracy, different outcomes in human capital? How indigenous and rural non-indigenous areas in Panama responded to the CCT (Arriaz & Rozo 2011)","Study Type: Causal Impact Analysis","Arriaz, I., & Rozo, S. (2011). Same bureaucracy, different outcomes in human capital? How indigenous and rural non-indigenous areas in Panama responded to the CCT. Inter-American Development Bank, Office of Evaluation & Oversight.",,2011,https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1847124,"December 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of Red de Oportunidades (RdO), a conditional cash transfer program in Panama, on child labor and school enrollment.
The authors used cross-sectional data to compare the treatment group to the comparison group on several outcomes, including child labor (if the child worked in the previous week) and school enrollment (if the child was enrolled in formal schooling).
In the analyses for rural areas, the study found that the receipt of the RdO was significantly associated with a decrease in child labor by 6.2 percentage points and an increase in school enrollment by 8.8 percentage points. In the analyses for the indigenous areas, the receipt of RdO was significantly associated with an increase in school enrollment by 6.3 percentage points, but not related to child labor.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Red de Oportunidades (RdO) program; other factors are likely to have contributed.","Red de Oportunidades (RdO) began in 2006 as a government effort to reduce poverty and increase human capital in Panama. The program targeted the poorest households in all areas of Panama. Household eligibility was estimated by examining socioeconomic variables to predict the likelihood that a household was impoverished. The thresholds used to determine inclusion varied by region and were as follows: for urban households the criteria was a probability of being poor at 35 percent or higher; for rural households, the criteria was a probability of being poor at 25 percent or higher; and for indigenous households, the criteria was a probability of being poor at 20 percent or higher. The cash transfer was delivered bimonthly to eligible households at a rate of $70 USD. When households did not comply with the conditions of the program, the bimonthly payment was reduced to $20 USD per month. In 2008, the bimonthly payment increased to $100 USD. Compliance was monitored by self-report and the conditions included: children being enrolled in school 80 percent of the time, children under 5 and pregnant women in the household receiving health care, and one adult in the household attending an information session on human development every 6 months.","Data were drawn from the 2008 Living Standards Measurement Survey. The authors used propensity score matching to compare the treatment group to the comparison group on several outcomes, including child labor (if the child worked in the previous week) and school enrollment (if the child was enrolled in formal schooling). The analysis was completed separately for (1) indigenous individuals residing in independent areas, and (2) non-indigenous individuals residing in rural areas. The majority of the population in both areas live in poverty.
The treatment and comparison groups were formed by their receipt of the conditional cash transfers (CCTs) via RdO. Those that received the CCTs were considered the treatment group. The comparison group included households that did not receive the CCTs. The groups were matched on household characteristics (e.g., household has adequate roofing; household has potable water). Importantly, the government used fridge ownership as a criterion to target RdO to households; this variable was not used in the propensity score matching.
Households with children aged 4- to 17-years of age were included in the study, the age group eligible for RdO. There were 1,203 households in the treatment condition, and 5,730 in the comparison condition. The analyses examining labor activities included a subsample of the households with children aged 10-17.","Employment/Child labor
In the analyses examining the data in rural areas, the study found a statistically significant relationship as the receipt of RdO decreased the likelihood of a child working by 6.2 percentage points.
In the analyses for the indigenous areas, the receipt of RdO was not significantly associated with child labor.
Education (School participation/enrollment)
In the analyses examining the data in rural areas, the study found a statistically significant relationship as the receipt of RdO increased the children's enrollment in formal schooling by 8.8 percentage points.
In the analyses for the indigenous areas, the receipt of RdO significantly increased enrollment in formal schooling by 6.3 percentage points.","The authors did not account for other factors that could have affected the difference between the treatment and comparison group, as they did not match on some of the required controls (age, gender, baseline child labor, baseline school enrollment), and they were not used as controls in their models.
The authors also discuss that there may be issues with supply of middle and high schools. The authors attempt to address the issue by reanalyzing the data where propensity score matching was conducted with region/area as a matching criterion. The authors report that the findings held; however, they note that the method they use does not address how geographic barriers impact school access. Moreover, the authors note that compliance was verified by self-report and that the self-report indices were verified by an annual stamp received from teachers.",,,,"Absence of conflict of interest.",,"International, Rural","Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"The impact of conditional cash transfer programs on household work decisions in Brazil (Ferro et al. 2010)","The impact of conditional cash transfer programs on household work decisions in Brazil (Ferro et al. 2010)","Study Type: Causal Impact Analysis","Ferro, A. R., Lúcia Kassouf, A., & Levison, D. (2010). The impact of conditional cash transfer programs on household work decisions in Brazil. In R. K. Akee, E. V. Edmonds, & K. Tatsiramos (Eds.), Child Labor and the Transition between School and Work (pp. 193-218). Bingley, UK: Emerald Publishing Limited.",,2010,http://www.anpec.org.br/encontro2009/inscricao.on/arquivos/000-dde4869521f17def…,"December 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The purpose of the study was to examine the impact of the Bolsa Escola program, which provided conditional cash transfers to families in Brazil, on child labor and school enrollment.
The authors used a nonexperimental matched comparison group design to compare the outcomes of children ages 6-15 that received the conditional cash transfers with those who had applied but who had not received it (waitlisted families), based on data from Brazil’s national household survey.
The study found that receipt of the Bolsa Escola benefit was significantly related to higher rates of school enrollment and lower rates of child labor.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Bolsa Escola program; other factors are likely to have contributed.","The Bolsa Escola program is a conditional cash transfer program in Brazil. The program was launched nationally by the Brazilian government in 2001 to focus on education. In 2001, Bolsa Escola was providing monthly stipends to 8.6 million children in 5 million families. The participants included families with children ages 6 to 15 and monthly incomes below half of Brazil's minimum salary. The program paid 15 Reais ($6 USD) for each child who attended 85 percent of school days for a maximum of 3 children per family. In 2004, the program was merged with four other cash transfer programs, and while the school requirements remained the same, the program was expanded to include prenatal care and vaccines for children ages 0 to 6.","The authors used a nonexperimental design to compare the outcomes of children ages 6-15 that received the conditional cash transfers with those who had applied but who had not received it (waitlisted families), based on data from Brazil’s national household survey, the Pesquisa Nacional pr Amostra de Domicilios (PNAD) – 2003. Using several demographic characteristics, they created a matched comparison group of families who had applied but had not yet received the benefit to families in the treatment group to assess the effectiveness of the cash transfer program. The sample included families with no missing data across all variables, resulting in 14,434 children (4,230 in the control group and 10,204 in the treatment group) and 8,202 parents who participated in the study. The authors used statistical models to compare the outcomes of treatment and control groups and ran separate models for children in urban and rural areas.","Employment/Child labor
The study found that receipt of the Bolsa Escola benefit was significantly related to a reduction in child labor, with decreases in labor rates by 2 percent in urban areas and 6 to 9 percent in rural areas.
Education (School participation/enrollment)
The study found that receipt of the Bolsa Escola benefit was significantly related to an increase in school enrollment, with increases in enrollment rates by 2 percent in urban areas and 4 percent in rural areas.","The authors created a matched group of non-participating eligible families to compare to Bolsa Escola participating families. However, the authors did not account for the outcomes at baseline, such as previous school attendance or child labor. Preexisting differences between the groups—and not the program/intervention— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Do conditional cash transfers reduce child labor?: Evidence from the Philippines (Galang 2016)","Do conditional cash transfers reduce child labor?: Evidence from the Philippines (Galang 2016)","Study Type: Causal Impact Analysis","Galang, I. M. (2016). Do conditional cash transfers reduce child labor?: Evidence from the Philippines (Unpublished Master's thesis). Tokyo, Japan: The University of Tokyo.",,2016,http://www.pp.u-tokyo.ac.jp/wp-content/uploads/2016/02/1.31-MB-1.pdf,"December 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The objective of the study was to examine the impact of the Philippines conditional cash transfer program, Pantawid Pamilyang Pilipino Program (4Ps), on child labor and school attendance.
The authors used a nonexperimental design to compare the outcomes of children ages 12-14 that received the conditional cash transfers with those who did not, based on data from the 2011 Annual Poverty Indicator Survey. Using several demographic characteristics, they created a matched comparison group of families who did not receive the benefit to assess the effectiveness of the cash transfer program.
The study found that receipt of the cash transfer was significantly associated with increased school attendance but not significantly associated with child labor.
The quality of causal evidence presented in this report is low because the authors did not account for the outcomes of schooling and work at baseline. This means we are not confident that the estimated effects are attributable to the 4Ps program; other factors are likely to have contributed.","The Pantawid Pamilyang Pilipino Program (4Ps) conditional cash transfer (CCT) program was initiated in 2008, which was targeted to poor households with children and pregnant women. The program's goals were to provide education and health cash grants and provide human capital investments to combat intergenerational poverty. In 2011, the education grant for children ages 6 to 14 was $300 PHP per month (7 USD) per child for 10 months; up to 3 children were allowed to register for the grant. In 2012, the program was expanded to include children ages 15 to 17. As a requirement of the program, children must have 85 percent school attendance (children ages 3 to 5 needed day care with 85% attendance). The health grant required children 0-5 to receive preventative health check-ups, monitoring of growth, and vaccines. Pregnant women must get prenatal care and have a health worker professional with them during child birth, and mothers who have given birth need post-natal care as part of the Department of Health protocol. Additionally, children ages 6-14 need de-worming pills twice a year. Participants chosen for the 4Ps occurred through geographic and household targeting. Municipalities that had more than 50 percent poverty were considered ""very poor"" and those with less were “moderate poor”. A means tests was conducted to estimate household income compared to a threshold.","The authors used a nonexperimental design to compare the outcomes of children ages 6-17 that received the conditional cash transfers with those who did not, based on data from the 2011 Annual Poverty Indicator Survey. Using several demographic characteristics, they created a matched comparison group of families who did not receive the benefit to assess the effectiveness of the cash transfer program. The authors used logit regression to estimate the propensity scores, based on covariates and conducted a balance check on the matching quality. This study focused on children ages 12 to 14, resulting in a sample size of N=5,878.","Employment/Child labor
The study found no statistically significant relationships between 4Ps and child labor.
Education (School participation/enrollment)
The study found a positive relationship between the program and school attendance (96 percent of 4Ps participants attended school versus 89 percent for non-4Ps participants).","The authors created a matched group of non-participating households to compare to 4Ps participants. However, the authors did not account for the outcomes at baseline, such as previous school attendance or child labor. Preexisting differences between the groups—and not the program/intervention— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-No impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Limiting child labor through behavior-based income transfers: An experimental evaluation of the PETI program in rural Brazil (Yap et al. 2009)","Limiting child labor through behavior-based income transfers: An experimental evaluation of the PETI program in rural Brazil (Yap et al. 2009)","Study Type: Causal Impact Analysis","Yap, Y. T., Sedlacek, G., & Orazem, P. F. (2009). Limiting child labor through behavior-based income transfers: An experimental evaluation of the PETI program in rural Brazil. In P. F. Orazem, G. Sedlacek, & Z. Tzannatos (Eds.), Child Labor and Education in Latin America (pp. 147-165). New York, NY: Palgrave Macmillan. doi.org/10.1057/9780230620100_10",,2009,,"December 2018","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of the Programa de Erradicacao do Trabalho Infantil (PETI) or Program to Eradicate Child Labor on child labor and hours in school.
The authors used an experimental design to compare the outcomes of children ages 7 to 14 residing in municipalities who received income transfers with those residing in municipalities who did not receive income transfers. Based on data from Brazil’s national household survey, the authors used statistical models to assess the impact of the program in each group.
The PETI program was significantly associated with an increase in time in school and less time in work for participating children.
The quality of causal evidence presented in this report is low because it is a randomized controlled trial with unknown attrition and the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the PETI program; other factors are likely to have contributed.","The Programa de Erradicacao do Trabalho Infantil (PETI) or Program to Eradicate Child Labor was launched in poor rural states in Brazil in 1996. The program offers monthly income transfers to households with a per capita income of below one-half of the minimum wage on the condition that the child(ren) in the household attends school at 80 percent of the time and attends an afterschool program, which doubles the school day. The program requires parents to sign a contract that their children will not work.","The authors used data from six municipalities in each of three states in Brazil: Bahia, Sergipe, and Pernambuco, where three municipalities were in the treatment and three in control groups. Data were collected on 3,564 households with 6,772 children ages 7 to 14. The authors limited the sample to the poorest household, resulting in 2,864 households and 5,611 children. Each of the three states implemented the program in different time periods, Pernambuco for three years, Sergipe for two years, and Bahia for one year. The model specifications include the impact of PETI on the time in school and time in the labor market for children residing in participating municipalities versus those in control municipalities.","Employment/Child labor
The PETI program was significantly associated with a decrease in time in the labor force for participating children from 5 to 6 percentage points in Pernambuco, 18 percentage points in Bahia, and 4.5 percentage points in Sergipe.
Education (School participation/enrollment) 
The PETI program was significantly associated with an increase in school attendance for participating children from 40 to 50 percent in Bahia and Pernambuco and 5 to 23 percent in Sergipe.","The authors used experimental data from a randomized sample of municipalities in Brazil. However, they did not include information on attrition or account for the outcomes at baseline, such as previous school attendance or child labor. Preexisting differences between the groups—and not the program/intervention—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"International, Rural","Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Do conditional cash transfers reduce child labor?: Evidence from the Philippines (Galang 2016)","Do conditional cash transfers reduce child labor?: Evidence from the Philippines (Galang 2016)","Study Type: Causal Impact Analysis","Galang, I. M. (2016). Do conditional cash transfers reduce child labor?: Evidence from the Philippines (Unpublished Master's thesis). Tokyo, Japan: The University of Tokyo.",,2016,http://www.pp.u-tokyo.ac.jp/wp-content/uploads/2016/02/1.31-MB-1.pdf,"December 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The objective of the study was to examine the impact of the Philippines conditional cash transfer program, Pantawid Pamilyang Pilipino Program (4Ps), on child labor and school attendance.

The authors used a nonexperimental design to compare the outcomes of children ages 12-14 that received the conditional cash transfers with those who did not, based on data from the 2011 Annual Poverty Indicator Survey. Using several demographic characteristics, they created a matched comparison group of families who did not receive the benefit to assess the effectiveness of the cash transfer program.
The study found that receipt of the cash transfer was significantly associated with increased school attendance but not significantly associated with child labor.
The quality of causal evidence presented in this report is low because the authors did not account for the outcomes of schooling and work at baseline. This means we are not confident that the estimated effects are attributable to the 4Ps program; other factors are likely to have contributed.","The Pantawid Pamilyang Pilipino Program (4Ps) conditional cash transfer (CCT) program was initiated in 2008, which was targeted to poor households with children and pregnant women. The program's goals were to provide education and health cash grants and provide human capital investments to combat intergenerational poverty. In 2011, the education grant for children ages 6 to 14 was $300 PHP per month (7 USD) per child for 10 months; up to 3 children were allowed to register for the grant. In 2012, the program was expanded to include children ages 15 to 17. As a requirement of the program, children must have 85 percent school attendance (children ages 3 to 5 needed day care with 85% attendance). The health grant required children 0-5 to receive preventative health check-ups, monitoring of growth, and vaccines. Pregnant women must get prenatal care and have a health worker professional with them during child birth, and mothers who have given birth need post-natal care as part of the Department of Health protocol. Additionally, children ages 6-14 need de-worming pills twice a year. Participants chosen for the 4Ps occurred through geographic and household targeting. Municipalities that had more than 50 percent poverty were considered ""very poor"" and those with less were “moderate poor”. A means tests was conducted to estimate household income compared to a threshold.","The authors used a nonexperimental design to compare the outcomes of children ages 6-17 that received the conditional cash transfers with those who did not, based on data from the 2011 Annual Poverty Indicator Survey. Using several demographic characteristics, they created a matched comparison group of families who did not receive the benefit to assess the effectiveness of the cash transfer program. The authors used logit regression to estimate the propensity scores, based on covariates and conducted a balance check on the matching quality. This study focused on children ages 12 to 14, resulting in a sample size of N=5,878.","Employment/Child labor 
The study found no statistically significant relationships between 4Ps and child labor.
Education (School participation/enrollment)
The study found a positive relationship between the program and school attendance (96 percent of 4Ps participants attended school versus 89 percent for non-4Ps participants).","The authors created a matched group of non-participating households to compare to 4Ps participants. However, the authors did not account for the outcomes at baseline, such as previous school attendance or child labor. Preexisting differences between the groups—and not the program/intervention— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-No impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"The impact of Brazil's Bolsa Familia Program on school attendance, age-grade discrepancy, and child labor (de Lima et al. 2014)","The impact of Brazil's Bolsa Familia Program on school attendance, age-grade discrepancy, and child labor (de Lima et al. 2014)","Study Type: Causal Impact Analysis","de Lima Amaral, E. F., Goncalves, G. Q., & Weiss, C. (2014). The impact of Brazil's Bolsa Familia Program on school attendance, age-grade discrepancy, and child labor. Journal of Social Science for Policy Implications, 2, 101-125",,2014,,"December 2018","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The purpose of the study was to examine the impact of the Brazilian conditional cash transfer program, Bolsa Familia, on child labor and school enrollment.
The authors used a nonexperimental design to compare the outcomes of children ages 7 to 14 who resided in families who met the income threshold for the Bolsa Familia program compared with those who had not, based on data from the 2010 Brazilian Census.
The study found that receipt of the cash transfer was significantly associated with an increase in school enrollment. However, receipt of the transfer was also associated with significantly higher rates of work.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to Bolsa Familia program; other factors are likely to have contributed.","Established in 2003, the Bolsa Familia program is a conditional cash transfer program aimed to increase human capital among impoverished households. Families were eligible to receive Bolsa Familia if they had monthly household per capita earnings below 70 Reais (Brazilian currency) or if they had a monthly household per capital income between 70.01 and 140 Reais and had children 17 and under residing in the household. Participating households must follow certain conditions to receive the financial benefit including attending nutritional and parental counseling sessions, getting the children vaccinated, and enrolling children in school.","The authors implemented a nonexperimental design to examine whether there were differences in child labor and school enrollment between those individuals who received the Bolsa Familia benefit and those who did not. The data were drawn from the 2010 Brazilian Demographic Census, which is administered by the Brazilian Institute of Geography and Statistics. The study included households with children aged 7 to 14 years. The treatment and comparison groups were formed by selecting households that received the Bolsa Familia benefit for the treatment group, and selecting households that did not receive the benefit for the comparison group. This was determined by one question on the Census. It asked whether or not the household received Bolsa Familia benefits. The question also asked if households received benefits from a different program to eradicate child labor (Programa de Erradicação do Trabalho Infantil – PETI). Households with an income up to 140 Brazilian Reais were eligible for Bolsa Familia; those with an income above 140 Braziliam Reais with a child engaged in work were eligible for PETI. The authors attempted to control for this by disaggregating by household income. The authors examined these differences across three economic groups, which was determined by creating cut-offs based on the maximum threshold of per capita household income of below 1) 70 Brazilian Reais, 2) 140 Brazilian Reais, and 3) 280 Brazilian Reais. The outcomes included whether or not the child worked and whether or not the child was enrolled in school. Analyses included means tests and logistic regression with controls for household and child characteristics.","Employment/Child labor 
Participation in Bolsa Familia was associated with significantly higher rates of work when compared to comparison group households across the three income levels. Children in households receiving the Bolsa Familia were 10 to 17 percent more likely to work compared to children in the comparison households.
Education (School participation/enrollment)
The study found that receipt of the Bolsa Familia benefit was significantly related to higher rates of school enrollment across the three income levels. Children in households receiving the Bolsa Familia were twice as likely to be enrolled in school compared to children in the comparison households.","The study used data from the Brazilian Census to identify children who received the program. The question asked if the household received benefits from two conditional cash transfers, Bolsa Familia or PETI. Therefore, it is unclear if the Bolsa Familia program was impacting the child labor and school enrollment outcomes.
The authors used a nonexperimental design and controlled for age, gender, and mother's education in the regression models. However, the authors did not account for the outcomes at baseline, such as previous school attendance or child labor. Preexisting differences between the groups—and not the program/intervention— could explain the observed differences in outcomes.",,,," Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Child labor","Child labor-Low-Unfavorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Fighting for Education: Financial Aid and Non-traditional Students (Barr 2015)","Fighting for Education: Financial Aid and Non-traditional Students (Barr 2015)","Study Type: Causal Impact Analysis","Barr, A. C. (2015). Fighting for Education: Financial Aid and Non-traditional Students. (Doctoral dissertation). Online Archive of University of Virginia Scholarship. https://doi.org/10.18130/V35Z6S.",,2015,https://doi.org/10.18130/V35Z6S,"January 2020",Veterans,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the effects of Post-9/11 GI Bill on veterans’ college enrollment and degree attainment. The author investigated similar research questions in another study, the profile of which is available [here].
The study used regression methods in a nonexperimental analysis, drawing on data from Defense Manpower Data Center, the National Student Clearinghouse, and the Integrated Postsecondary Education Data System. The author used two statistical models to compare veterans eligible for the benefits with veterans who were not eligible.
The study found a statistically significant positive relationship between the Post-9/11 GI Bill benefits and education for veterans who were eligible for the benefits and likely to take them up compared with veterans who were not eligible for the benefits and with veterans who were eligible but unlikely to take up the benefits.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Post-9/11 GI Bill; other factors are likely to have contributed.","This study focuses on the expansion of education benefits for veterans under the Post-9/11 GI Bill. The bill, which took effect in August 2009, retroactively provided additional education benefits to people who served in active duty after September 11, 2001. The Post-9/11 GI Bill roughly doubled the average maximum benefit amounts from the prior iteration of the GI Bill. Benefits under the bill include in-state tuition, fees, a housing allowance, and a stipend for books. Maximum benefit levels vary by state and are based on the highest tuition and fee level of any public institution in the veteran’s state of residence. Veteran students who are enrolled for more than half-time are also eligible for a monthly housing allowance.","The study used statistical models to estimate the effects of the Post-9/11 GI Bill in a nonexperimental analysis. The author used two models to compare college enrollment and degree attainment of veterans eligible for the Post-9/11 GI Bill benefits with veterans who were not eligible for the benefits. First, the author compared educational attainment of the treatment group (veterans eligible for the benefits who exited the military from 2007 to 2008 and were more likely to take up the benefits) with that of a comparison group (veterans who exited the military from 2002 to 2004 who were eligible for the benefits but much less likely to take advantage of the GI Bill benefits because the bill was not passed until 2009, many years after their separation with the military). By the time the bill took effect in 2009, the members of the comparison group likely had either enrolled in school already or moved on to other professions by the time the benefits were available, so they were not interested in returning to school at that time. Second, the author compared veterans eligible for the benefits with veterans who were not eligible for the benefits because they received a dishonorable discharge from the military. For the second comparison, the author examined the differential trends in the outcomes for the two groups, before and after the Post-9/11 GI Bill took effect, covering the period of 2002 to 2008. The author assessed outcomes five and six years following the veteran’s separation from the military.
For both models, the author used data from the Defense Manpower Data Center, the National Student Clearinghouse, and the Integrated Postsecondary Education Data System. There were 14,224 people in the first estimation strategy and 15,457 people in the second estimation strategy. The sample was limited to people ages 22 to 39 who exited the military from 2002 to 2008 with less than a bachelor's degree at the date of military separation.","Education
The study found that veterans who exited the military from 2007 to 2008 were 5 to 7 percentage points more likely to obtain an associate’s degree or higher within either five or six years of military separation compared with veterans who exited from 2002 to 2004 (who were eligible for the benefits but unlikely to take them up). The study also found that veterans eligible for the GI benefits were 5 to 6 percentage points more likely to obtain an associate’s degree or higher within either five or six years of military separation compared with veterans who were ineligible for the benefits.
The study found that veterans eligible for the Post-9/11 GI Bill benefits who exited the military from 2007 to 2008 were 4 to 5 percentage points more likely to obtain a bachelor’s degree or higher within five or six years of military separation compared with veterans who exited from 2002 to 2004 (who were eligible for the benefits but unlikely to take them up). The study also found that veterans eligible for the GI benefits were 3 to 5 percentage points more likely to obtain a bachelor’s degree or higher within either five or six years of military separation compared with veterans who were ineligible for the benefits.
The study found an 8 percentage point increase in college enrollment before and after the Post-9/11 GI Bill for veterans eligible for the benefits compared with veterans who were not eligible for the benefits.","In both models, the author controls for age, race, and gender and accounts for education before the introduction of the Post-9/11 GI Bill by limiting the sample to those with a high school degree but without a college degree. But the author does not account for socioeconomic status measured before introduction of the Post-9/11 GI Bill.
In addition, for the second model, the study used veterans who did not receive an honorable discharge—and hence were ineligible for the Post-9/11 GI Bill benefits—as a comparison group. There were likely differences in unobservable characteristics between veterans who received an honorable discharge and those who did not, and these characteristics could affect their educational trajectories.
Finally, the implementation of the Post-9/11 GI Bill corresponded with the onset of the Great Recession, which might have altered the enrollment behaviors of the treatment and comparison groups, because it was more challenging to find a job, and, therefore, attending college might have been more preferable.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Other training and education","Education and skills gains-Low-Favorable impacts","Education and skills gains","Veterans Review Protocol"
"From the battlefield to the schoolyard: The short-term impact of the Post-9/11 GI Bill (Barr 2015)","From the battlefield to the schoolyard: The short-term impact of the Post-9/11 GI Bill (Barr 2015)","Study Type: Causal Impact Analysis","Barr , A. (2015). From the battlefield to the schoolyard: The short-term impact of the Post-9/11 GI Bill. Journal of Human Resources, 50(3), 580-613.",,2015,http://jhr.uwpress.org/content/50/3/580.full.pdf,"January 2020",Veterans,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the effects of Post-9/11 GI Bill on veterans’ college enrollment. The author investigated similar research questions in another study, the profile of which is available [here].
The study used regression methods in a nonexperimental analysis, drawing on data from the American Community Survey and Current Population Survey. The study compared college enrollment of veterans eligible for the GI Bill benefits with civilians, who were not eligible to receive the benefits.
The study found that veterans eligible for the Post-9/11 GI Bill benefits were significantly more likely to enroll in college after the introduction of the bill compared with civilians.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Post-9/11 GI Bill; other factors are likely to have contributed.","This study focuses on the expansion of education benefits for veterans under the Post-9/11 GI Bill. The bill, which took effect in August 2009, retroactively provided additional education benefits to people who served in active duty after September 11, 2001. The Post-9/11 GI Bill roughly doubled the average maximum benefit amounts from the prior iteration of the GI Bill. Benefits under the bill include in-state tuition, fees, a housing allowance, and a stipend for books. Maximum benefit levels vary by state and are based on the highest tuition and fee level of any public institution in the veteran’s state of residence. Veteran students who are enrolled for more than half-time are also eligible for a monthly housing allowance.","The study used regression methods to estimate the effects of the Post-9/11 GI Bill in a nonexperimental analysis. The author compared college enrollment of veterans before and after the bill was implemented relative to college enrollment of nonveteran civilians over the same period.
Data came from the 2006 to 2011 samples of the American Community Survey and the October Current Population Survey supplement on education. The sample included people who were born and reside in the United States, were ages 23 to 28, and had at least a high school or equivalent diploma but not a bachelor’s degree. In the Current Population Survey data, there were 1,285 veterans and 29,684 nonveterans observed; in the American Community Survey data, there were 39,937 veterans and 587,345 nonveterans.","Education
The study found that veterans increased college enrollment by about 1.7 percentage points before and after the introduction of the Post-9/11 GI Bill compared with civilians over the same time period.
The study found that the separated veterans (those no longer in the military) increased college enrollment by 4 to 6 percentage points (depending on data source) before and after the introduction of the post-9/11 GI Bill compared with civilians over the same time period.","In the analysis, the author controls for age, race, and gender and accounts for education before the introduction of the Post-9/11 GI Bill by limiting the sample to those with a high school degree but without a college degree. But the author does not account for socioeconomic status measured before introduction of the Post-9/11 GI Bill.
In addition, the study used nonveterans as a comparison group for veterans. Although the study demonstrated that the trends in college enrollment were similar between veterans and nonveterans in the period before the enactment of the Post-9/11 GI Bill, there were still likely unobservable differences between the two groups that would affect their educational trajectories.
Finally, the implementation of the Post-9/11 GI Bill corresponded with the onset of the Great Recession, which could have altered the enrollment behaviors of both veterans and nonveterans because it was more challenging to find a job, and, therefore, attending college might have been more preferable.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Other training and education","Education and skills gains-Low-Favorable impacts","Education and skills gains","Veterans Review Protocol"
"The effect of veterans benefits on education and earnings (Angrist 1993)","The effect of veterans benefits on education and earnings (Angrist 1993)","Study Type: Causal Impact Analysis","Angrist, J. D. (1993). The effect of veterans benefits on education and earnings. Industrial and Labor Relations Review, 46(4), 637-652.",,1993,https://doi.org/10.1177%2F001979399304600404,"January 2020",Veterans,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the effects of veteran benefits on the post-service education and earnings of soldiers who served in the Vietnam era and in the first years of the All-Volunteer Forces (AVF).
The study used statistical tests in a nonexperimental analysis to evaluate the effects of veteran benefits. The data came from the U.S. Census Bureau’s 1987 Survey of Veterans.
The study found a statistically significant positive relationship between veteran benefits and education and earnings outcomes.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to veteran benefits; other factors are likely to have contributed.","This study focuses on veteran benefits under the post-Korean G.I. Bill and the Veterans Education Assistance Program (VEAP). Veterans on active duty for more than 180 consecutive days from January 31, 1955, to January 1, 1977, including Vietnam veterans, were eligible for benefits under the post-Korean G.I. Bill for up to 10 years after exiting the military. Benefits under this bill included monthly payments that varied by type of schooling and the number of dependents.
VEAP replaced the post-Korean G.I. Bill. Those who entered the military from December 31, 1976, to July 1, 1985, including those who served during the early AVF period, were eligible for VEAP. VEAP was a contributory program in which participants contributed monthly to a personal VEAP fund for at least one year, which the government matched when benefits were paid out. VEAP benefits were payable for up to 10 years after discharge from military service.
VEAP provided less generous benefits than the post-Korean G.I. Bill. Under the post-Korean G.I. Bill, a veteran with no dependents in 1978 would have been eligible for 45 monthly payments of $311, and, under VEAP, a veteran could receive a maximum of 36 monthly payments of $150. To address the perceived decline in the number and quality of recruits after VEAP was implemented, individual military services later also offered enhancements, known as kickers, to the basic VEAP program. Kickers could add up to $6,000 to an individual’s VEAP fund and were available to eligible recruits with a high school diploma who entered the military after January 1, 1979.","The author used statistical models to estimate the effects of veteran benefits on post-military increases in educational attainment and earnings. The author estimated effects on earnings by comparing veterans who did and did not take up veteran benefits (through either the post-Korean G.I. Bill or VEAP). This analysis used a sample of men only and accounted for individual background characteristics including age, race, years of service, and marital status.
Effects on educational attainment were estimated using a different model that accounted for people choosing to receive veteran benefits. This model examined the differences in education outcomes among those who served during the Vietnam era (August 1964 to May 1975) and were eligible for the more generous G.I. Bill and those who served during the early AVF period (May 1975 to September 1980) and were eligible for the less generous VEAP. The model compared these differences for veterans with and without a high school diploma at entry because the latter were eligible for fewer education benefits after the transition from the G.I. Bill to VEAP.
Data came drawn from the U.S. Census Bureau’s 1987 Survey of Veterans. The sample included men who served in the Vietnam or early AVF periods, were age 30 to 54 in 1987, had 1 to 15 years of active duty service and hence were eligible for veteran benefits, and had more than 9 years of schooling when entering the military. Among the 3,337 veterans in the Survey of Veterans with service in the Vietnam or AVF eras, 2,388 met the criteria for inclusion in the sample.","Education
The study found that veteran benefits significantly increased post-service education by 1.54 grades, from a mean of 12.5 years at the time of military entry.
Earnings
The study found that veteran benefits significantly increased earnings by 5.8 percent.","The author did not account for existing differences in education and earnings between the treatment and comparison groups before the take-up of the veteran benefits. These existing differences between the groups—and not veteran benefits—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Other wages and benefits, Other training and education","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Earnings and wages, Education and skills gains","Veterans Review Protocol"
"Veterans Homelessness Prevention Demonstration evaluation: Final report (Cunningham et al. 2015)","Veterans Homelessness Prevention Demonstration evaluation: Final report (Cunningham et al. 2015)","Study Type: Causal Impact Analysis","Cunningham, M. K., Biess, J., Emam, D., & Burt., M. R. (2015). Veterans Homelessness Prevention Demonstration evaluation: Final report. Washington, DC: Urban Institute.",,2015,http://apps.urban.org/features/homeless-veterans/,"January 2020",Veterans,,"Low Causal Evidence",Pre/Post,,"The study examined the effect of the Veterans Homelessness Prevention Demonstration (VHPD) program on the employment, earnings, public benefit receipt, and education and training outcomes of veterans at risk of homelessness.
The study uses a nonexperimental design to examine whether there were changes in veterans’ outcomes from before the program began to six months after the program ended using self-reported data directly from the veterans.
The study found statistically significant positive relationships between the VHPD program and veterans’ employment and earnings six months after the program ended compared with before the program. The study found mixed relationships between the VHPD program and public benefit receipt: significantly fewer veterans in the study received several types of public benefits, such as Temporary Assistance for Needy Families (TANF) or the Supplemental Nutrition Assistance Program (SNAP). Significantly more veterans in the study, however, received Veterans Pension benefits after the program compared with before it.
The quality of causal evidence presented in this report is low because the authors did not account for trends in outcomes before the intervention. This means we are not confident that the estimated effects are attributable to the VHPD program; other factors are likely to have contributed.","VHPD was designed to serve veterans who were homeless or at risk for homelessness, with an emphasis on serving female veterans, veterans with children, and veterans who served in Afghanistan and Iraq under Operation Enduring Freedom, Operation Iraqi Freedom, or Operation New Dawn.
As part of a three-year pilot program, five sites received funding from the U.S. Departments of Housing and Urban Development and Veterans Affairs (VA). The VHPD sites were required to spend 65 percent of their funding on housing assistance, which included short- or medium-term housing assistance, short-term financial assistance including moving cost, utilities, and arrears assistance. Sites had discretion to fund other services based on local need, including collaborating with U.S. Department of Labor One-Stop Career Centers to provide employment services (although only one site actually worked with a local One-Stop Career Center). Sites could also provide case management and assistance with child care, credit repair, and transportation. Finally, the program worked with local VA health care providers to improve health care access for participants.","The study was a nonexperimental study that used a pre-post study design to compare outcomes of veterans before and six months after they participated in the VHPD program. The outcomes were collected directly from veterans through two surveys: one survey was conducted before the VHPD program, and the second survey was conducted six months after the VHPD program. In total, 509 veterans were enrolled in the study, and 315 of them completed both surveys and were included in the analyses.
To be eligible for the program, people had to have veteran status, be eligible for VA health care, have household income below 50 percent of the median income in the area, and be at risk of homelessness or homeless for 90 days or less.
Nearly all of the participants were veterans from the Armed Forces of the United States, and 55 percent of the sample served in Iraq and Afghanistan. About half the sample (51 percent) was White, 43 percent was African American, and 10 percent was Hispanic. About a quarter of the participants were female, and 46 percent of the sample members were ages 25 to 40.
The authors used a statistical model to compare veterans’ employment, earnings, public benefit receipt, and education and training outcomes before and after the program.","Employment
There was a statistically significant positive relationship between the VHPD program and employment. The study found that the proportion of veterans in the study who were employed full-time increased by 14 percentage points after the VHPD program compared with before it. The study also found that 43 percent of the sample members were employed at the time of follow-up compared with 25 percent before the intervention; the authors did not report the significance of this finding.
Earnings
There was a statistically significant positive relationship between the VHPD program and earnings. The study found that the veterans in the study earned $525 more after the VHPD program than before it.
Public benefit receipt
The study found mixed relationships between the VHPD program and public benefit receipt. Significantly fewer veterans in the study received TANF, SNAP, Medicare, Children’s Health Insurance Program, and unemployment insurance after the VHPD program compared with before it. But significantly more veterans in the study received Veterans Pension benefits after the intervention than before.
Education and training
The study found that 29 percent of the sample members were engaged in training or education at follow-up compared with 28 percent before the intervention. The authors did not report the significance of this finding.","The authors compared the outcomes of participants measured before and after they participated in the intervention. For these types of designs, the authors must observe outcomes for multiple periods before the intervention to rule out the possibility that participants had increasing or decreasing trends in the outcomes examined before enrollment in the program. That is, if participants who had increasing employment rates or earnings tended to enroll in the program, we would anticipate further increases over time, even if they did not participate in the program. Without knowing the trends before program enrollment, we cannot rule out this possibility. Therefore, the study receives a low causal evidence rating.",,"Five military bases and their surrounding communities participated in the study:
Camp Pendleton in San Diego, California
Fort Hood in Killeen, Texas
Fort Drum in Watertown, New York
Joint Base Lewis-McChord in Tacoma, Washington
MacDill Air Force Base in Tampa, Florida.",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Low income, Veteran or military, Homeless","Job search assistance and supportive services, Other training and education","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts, Public benefits receipt-Low-Mixed impacts","Earnings and wages, Education and skills gains, Employment, Public benefits receipt","Veterans Review Protocol"
"The differences in first-year retention rates for adult and traditional community college students enrolled and not enrolled in learning communities (Budd 2017)","The differences in first-year retention rates for adult and traditional community college students enrolled and not enrolled in learning communities (Budd 2017)","Study Type: Causal Impact Analysis","Budd, L. B. (2017). The differences in first-year retention rates for adult and traditional community college students enrolled and not enrolled in learning communities. (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 10639485)",,2017,https://search.proquest.com/docview/2007545357,"January 2020","Community College",,"Low Causal Evidence",,,"The study’s objective was to determine the impact of participation in the learning communities on retention rates for community college students.
The author used a nonexperimental design to compare the enrollment rates of first-time community college students who enrolled in learning communities to those who did not, including one semester and one year after initial enrollment.
The study found that students who participated in learning communities were significantly more likely to enroll in college one year later.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the learning communities; other factors are likely to have contributed.","The community college included in this study is located in the Mid-Atlantic region in a suburban community and serves approximately 65,000 credit and non-credit seeking students. The community college implemented its first set of learning communities (LCs) in 2000, and has expanded the initiative over time. Originally, the community college's developmental LCs paired courses in developmental reading, writing, or math with general education courses. At the time of this study, the LC program had expanded and was offering 25 LCs that included general education, the honors program, and English as a second language in addition to the original LCs for developmental education. Each LC typically enrolls between 15 and 20 students, for a total of approximately 450 students annually.","The author compared the outcomes of community college students who participated in learning communities to students who did not. Data from the college’s Office of Planning and Research was obtained for 252 first-time students who enrolled in the fall semesters from 2010-2015 (126 community college students in each group). T-tests were used to determine whether students who chose to participate in learning communities had higher rates of enrollment one semester and one year later than students who did not participate in learning communities.","Education and skills gain
The study found a significant relationship between participation in learning communities and enrollment rates one year later with 67% of learning community students enrolling the following year compared to 40% of comparison group students.
However, no significant relationship was found between participation in learning communities and enrollment rates one semester later.","The author did not account for self-selection into the learning communities or preexisting differences between the groups before participation, such as degree of financial disadvantage or measure of academic achievement. These preexisting differences between the groups—and not the intervention— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Developmental students' persistence towards graduation in paired coursework programs among African American and Latino males in community colleges (Davis 2018)","Developmental students' persistence towards graduation in paired coursework programs among African American and Latino males in community colleges (Davis 2018)","Study Type: Causal Impact Analysis","Davis, E. (2018). Developmental students' persistence towards graduation in paired coursework programs among African American and Latino males in community colleges. (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database.",,2018,https://jscholarship.library.jhu.edu/handle/1774.2/59950,"January 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of accelerated paired courses on male African American and Latino students’ academic persistence.
The study used a nonexperimental design to compare the outcomes of students who completed paired coursework to students who did not. The author examined enrollment and course completion using college administrative records.
The study found that students who completed paired coursework were significantly more likely to enroll in college the following semester than students in the comparison group.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the paired coursework; other factors are likely to have contributed.","The community college system in Houston, Texas implemented two types of accelerated paired courses designed to increase the rate of developmental course completion. Accelerated course pairing allowed students to complete the developmental course while also completing the college-level version of the course (paired course). Mainstream ‘paired with support’ allowed students to complete the college-level course while also completing a developmental support course (such as a mandatory class or a lab session) or other learning supports. The college system was interested in assessing an accelerated approach for their male, African American, and Latino students to increase their rate of completing developmental coursework. Accordingly, the participants in the study were all male, African Americans or Latinos, who were referred for developmental education in their first or second year.","The study took place at six campuses in the Houston community college system. The author used a nonexperimental design to assess the impact of paired coursework on academic persistence (defined as re-enrollment in the spring semester). Study participants included 102 students enrolled in paired coursework (treatment group), and 817 students enrolled only in the traditional developmental course (comparison group). College administrative records provided data on academic outcomes, including completion of the initial developmental course and enrollment in the subsequent course in the following semester. The author compared outcomes between the treatment and comparison groups, but did not control for any student characteristics.","Education and skills gain
The study found a significant relationship between paired coursework enrollment and academic persistence with 79% of students enrolled in paired coursework re-enrolling in the spring semester compared to 68% students in the traditional developmental course.","The author did not account for preexisting differences between the groups before intervention participation. These preexisting differences between the groups—and not the paired coursework—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Male, Black or African American, Hispanic of any race","Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Maintaining success rates: Does Statway® sustain its impact as it scales to new classrooms and institutions? (Huang & Yamada 2017)","Maintaining success rates: Does Statway® sustain its impact as it scales to new classrooms and institutions? (Huang & Yamada 2017)","Study Type: Causal Impact Analysis","Huang, M., & Yamada, H. (2017). Maintaining success rates: Does Statway® sustain its impact as it scales to new classrooms and institutions? Carnegie Foundation for the Advancement of Teaching: Stanford, CA.",,2017,https://eric.ed.gov/?id=ED582441,"January 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Statway® math program on community college students’ earning of math credits.
The authors used a nonexperimental design to compare the outcomes of students who were in the Statway® program to a matched comparison group.
When compared to non-participating students, the study found that Statway® program participation was associated with a significantly higher likelihood of earning college math credit.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Statway® program; other factors are likely to have contributed.","The Statway® program was developed by the Carnegie Foundation for the Advancement of Teaching with a goal of improving success rates for students in developmental math. Statway® provides an accelerated program to allow students to meet math requirements by providing a two-course program, intended for students to take consecutively. The program includes elementary and intermediate algebra as well as college-level statistics for students pursing non-STEM majors. The curriculum is grounded in evidence-based practices to promote learning such as productive struggle, explicit connections, and deliberate practice. Socioemotional supports are integrated into the program with productive persistence to help support and build student confidence in math. Language and literacy supports provide students with information on math vocabulary and core concepts and skills to help with critical thinking and reasoning. Finally, the program includes faculty professional development and a ""networked improvement community"" (NIC) to allow stakeholders to collaborate across institutions.","The authors used a nonexperimental design to compare the outcomes of students who were in the Statway® program with students who had taken traditional developmental math. The study included students from two Statway® cohorts. The Year 3 Cohort (Fall 2013) included students from 19 community colleges and three 4-year universities across seven states (California, Connecticut, Florida, Minnesota, New York, Texas, and Washington). The Year 4 Cohort (Fall 2014) included students from 22 community colleges and four 4-year universities across 10 states (including the addition of Hawaii, Massachusetts, and Wyoming). The authors received student demographic, prior course enrollment, and performance data from each institution. The authors used a hierarchical linear model framework when creating propensity scores to match students on 32 characteristics. Students in the Year 3 cohort were matched to students from Fall 2012 and those in the Year 4 Cohort were matched to students from Fall 2013. The Year 3 sample included 1,353 students in the treatment group and 5,853 in the comparison group. The Year 4 sample included 1,254 students in the treatment group and 4,240 in the comparison group.","Education and skills gain
The study found a significant relationship between program participation and earning of math credits, where the odds of earning college math credit for Statway® students were almost five times higher than for the comparison group.","The authors created a matched group of non-participating students to compare to Statway® students. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the Statway® program—could explain the observed differences in outcomes.",,"California
Connecticut
Florida
Hawaii
Massachusetts
Minnesota
New York
Texas
Washington
Wyoming",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"The effect of participation in a Title V program on Latinx student success at a community college (Lacagnino 2019)","The effect of participation in a Title V program on Latinx student success at a community college (Lacagnino 2019)","Study Type: Causal Impact Analysis","Lacagnino, S. N. (2019). The effect of participation in a Title V program on Latinx student success at a community college. (Unpublished doctoral dissertation). Seton Hall University, South Orange, NJ.",,2019,https://scholarship.shu.edu/dissertations/2642/,"January 2020","Community College",,"Low Causal Evidence",,,"The study’s objective was to determine the impact of participation in the Project Accel program on education outcomes for community college students.
The study used a nonexperimental design to compare the outcomes of students who participated in the Project Accel program with those who did not, based on administrative data from the community college. Using several demographic characteristics, the author created a matched comparison group of students who were not in the program to assess the effectiveness of the program on course completion, program persistence, and fall-to-fall retention.
For the accelerated English as a Second Language (ESL) program, the study found that program participation was significantly related to higher completion, persistence, and retention rates. For the accelerated developmental English program, the study found that program participation was significantly related to higher completion and persistence rates.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Project Accel program; other factors are likely to have contributed.","A mid-Atlantic public community college, federally designated Hispanic-Serving Institution (HSI), received a five-year Title V grant to fund accelerated developmental English and English as a Second Language (ESL) programs, called Project Accel. The Title V grants, also called “Developing Hispanic-Serving Institutions Programs”, are competitive grants designed to help HSIs increase student success by addressing a challenge facing their institution. Project Accel was designed to implement accelerated developmental English and accelerated ESL programs.
Students were eligible for the accelerated developmental English program if they were first-time, full-time students in the fall term and placed into developmental English based on the ACCUPLACER college placement exam. The comparison group students had to meet the same criteria of being a first-time, full-time student in the fall and placed into developmental English. Students were eligible for the ESL program if they were full time in the fall term and placed into level 5 ESL based on a college placement exam or if they completed the level 4 ESL program. The comparison group students had to meet the same criteria of being full-time in the fall and placed into ESL courses.","The author matched Project Accel participants to similar nonparticipants using propensity scores developed from baseline demographic information. The treatment group included 319 students in the English program and 76 in the ESL program. The comparison group included 2,446 students who enrolled in the developmental English courses and 537 students who enrolled in ESL courses. The author received demographic and academic record data from the community college's research office. The outcomes included course completion rates (first attempt grades of D or better in the developmental English and ESL courses), fall-to-fall retention rates (re-enrollment in the fall semester of the next academic year), and student persistence rates (enrollment and successful completion of each course in the developmental English or ESL course sequence). The author used chi-square and t-test statistics to examine differences between the groups.","Education and skills gain
For the accelerated developmental English program, the study found a significant relationship between program participation and course completion with 83.1% of treatment students completing the course compared to 66.1% of comparison students.
The study also found a significant relationship between program participation and persistence rates with 81.5% of students in the accelerated developmental English group persisting in the program compared to 55.8% of students in the comparison group. No statistically significant relationship was found for retention.
For the accelerated ESL program, the study found a significant relationship between program participation and course completion with 96.1% of treatment students completing the course compared to 62.5% of comparison students.
The study found a significant relationship between participation in the accelerated ESL program and persistence rates (93.4% of treatment students and 57.9% of comparison students) and retention rates (73.1% of treatment students and 50% of comparison students).","While the study used a matched comparison group design, the author did not account for other factors that could have affected the difference between the treatment and comparison groups. The author matched on age, gender, race/ethnicity, and financial aid but did not match on pre-intervention academic achievement or control for it in the analyses. These preexisting differences between the groups—and not the intervention— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Hispanic of any race","Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Accelerating developmental math students in California community colleges: A comparative assessment of two acceleration models (Martinez 2018)","Accelerating developmental math students in California community colleges: A comparative assessment of two acceleration models (Martinez 2018)","Study Type: Causal Impact Analysis","Martinez, A. F. (2018). Accelerating developmental math students in California community colleges: A comparative assessment of two acceleration models. (Unpublished doctoral dissertation). California State University, Long Beach, CA.",,2018,https://search.proquest.com/openview/adffc87f79e0373131993e9c4e2f5070/1?pq-orig…,"January 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of two accelerated pathway models (course redesign and compression) on completion rates for community college students. This summary focuses on the compression accelerated model.
This study used a nonexperimental design to compare accelerated and non-accelerated pathways to college-level math by tracking the progress of students placed in developmental math courses.
The study found that participation in the compression accelerated program was significantly related to higher college-level and developmental math completion rates for students placed in mid-level remediation.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the accelerated pathway programs; other factors are likely to have contributed.","A community college in Southern California implemented the Compression Accelerated Model for its developmental math courses. Compression accelerated models aim to restructure or compress multiple sequential courses in a single term. For example, compressing two developmental math courses into a single 16-week semester by extending classroom time. These compressed courses typically do not incorporate changes to curriculum, pedagogy, or student support.
The college offered acceleration courses to students placing two- and four-levels below a college-level math course. The pre-requisite for the acceleration course for students placed two-levels below was either Pre-Algebra or placement via the college’s assessment office. There was no pre-requisite for the acceleration course for students placed four-levels below. Taking the two accelerated math courses in sequence served as an alternative pathway to the traditional, non-accelerated four-course pathway through remediation or developmental math.
The colleges’ developmental math programs consisted of two non-exclusive pathways: a traditional four-level sequential pathway or an accelerated pathway of two sequential courses. The traditional pathway included Basic Math, Pre-Algebra, Elementary Algebra and Intermediate Algebra. The accelerated pathway courses included Combined Basic Math and Pre-Algebra (Math A) which had no pre-requisite and Combined Elementary and Intermediate Algebra (Math B), which had one of three pre-requisites: pre-algebra, Combined Basic Math and Pre-Algebra, or direct placement.
Students placing four-levels below a college-level math course (low-level remediation) were placed in either an accelerated pathway, a traditional pathway, or a mixed pathway (accelerated and traditional). Students placed two-levels below a college-level math course (mid-level remediation) were placed in either an accelerated or traditional pathway.","This nonexperimental study compared the outcomes of students who participated in the accelerated pathway to those who participated in the traditional pathway. All students included in the study were in below college-level math courses and enrolled in these courses for the first time. The sample included 2,529 students. Using secondary student data from the college’s Institutional Research Office, the author conducted statistical models to examine differences between the groups. Outcomes included completion rates for both developmental and college-level math courses, and the time in years to complete the courses.","Education and skills gain
The study found that enrollment in the Compression Accelerated Model program was significantly related to decreased time to complete both developmental and college-level math courses.
The study found that enrollment in the Compression Accelerated Model program was significantly related to higher college-level and developmental math completion rates for students placed in mid-level remediation.","The author did not account for preexisting differences between the groups before program participation. These preexisting differences between the groups—and not the Compression Accelerated Model — could explain the observed differences in outcomes. Also, the author did not account for self-selection into the accelerated learning pathway. Students who are high achievers or students who have exhausted the number of attempts in the traditional pathway may be more prevalent in the accelerated cohorts. These differences were not accounted for in the study.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers","Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Accelerating developmental math students in California community colleges: A comparative assessment of two acceleration models (Martinez 2018)","Accelerating developmental math students in California community colleges: A comparative assessment of two acceleration models (Martinez 2018)","Study Type: Causal Impact Analysis","Martinez, A. F. (2018). Accelerating developmental math students in California community colleges: A comparative assessment of two acceleration models. (Unpublished doctoral dissertation). California State University, Long Beach, CA.",,2018,https://search.proquest.com/openview/adffc87f79e0373131993e9c4e2f5070/1?pq-orig…,"January 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to compare the impact of two accelerated pathway models (course redesign and compression) on completion rates for community college students. This summary focuses on the course redesign accelerated model.
This study used a nonexperimental design to compare accelerated and non-accelerated pathways to college-level math by tracking the progress of students placed in developmental math courses.
The study found that participation in the course redesign accelerated program was significantly related to higher college-level and developmental math completion rates for first-generation students and students placed in both low-level and mid-level remediation.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the course redesign accelerated model; other factors are likely to have contributed.","A community college in Southern California implemented a course redesign accelerated model for its developmental math courses. Course redesign accelerated models not only compress courses but also implement structural and pedagogical changes from traditional math programs. They seek to consolidate the content from the different courses, eliminate redundancy, and incorporate new pedagogical strategies.
The college offered acceleration courses to students placing two- and four-levels below a college-level math course. The pre-requisite for the acceleration course for students placed two-levels below was either Pre-Algebra or placement via the college’s assessment office. There was no pre-requisite for the acceleration course for students placed four-levels below. Taking the two accelerated math courses in sequence served as an alternative pathway to the traditional, non-accelerated four-course pathway through remediation or developmental math.
The colleges’ developmental math programs consisted of two non-exclusive pathways: a traditional four-level sequential pathway or an accelerated pathway of two sequential courses. The traditional pathway included Arithmetic, Pre-Algebra, Elementary Algebra and Intermediate Algebra. The accelerated pathway courses included Basic Math (Math A), which had no pre-requisite, and General Algebra (Math B), which had one pre-requisite: pre-algebra, Basic Math, or direct placement.
Students placing four-levels below a college-level math course (low-level remediation) were placed in either an accelerated pathway, a traditional pathway, or a mixed pathway (accelerated and traditional). Students placed two-levels below a college-level math course (mid-level remediation) were placed in either an accelerated or traditional pathway.","This nonexperimental study compared the outcomes of students who participated in the accelerated pathway to those who participated in the traditional pathway. All students included in the study were in below college-level math courses and enrolled in these courses for the first time. The sample included 6,251 students. Using secondary student data from the college’s Institutional Research Office, the author conducted statistical models to examine differences between the groups. Outcomes included completion rates for both developmental and college-level math courses, and the time in years to complete the courses.","Education and skills gain
The study found that enrollment in the Course Redesign Accelerated Pathway was significantly related to decreased time to complete both developmental and college-level math courses.
The study found that enrollment in the Course Redesign Accelerated Pathway was significantly related to higher college-level and developmental math completion rates for first-generation students and students placed in both low-level and mid-level remediation.","The author did not account for preexisting differences between the groups before program participation. These preexisting differences between the groups—and not the Course Redesign Accelerated Pathway —could explain the observed differences in outcomes. Also, the author did not account for self-selection into the accelerated learning pathway. Students who are high achievers or students who have exhausted the number of attempts in the traditional pathway may be more prevalent in the accelerated cohorts. These differences were not accounted for in the study.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers","Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Does AVID Higher Education (AVID HE) increase student term-to-term progression, persistence toward credited classes and social capital for first-generation college students placing into developmental education: A mixed methods study (Plinski 2018)","Does AVID Higher Education (AVID HE) increase student term-to-term progression, persistence toward credited classes and social capital for first-generation college students placing into developmental education: A mixed methods study (Plinski 2018)","Study Type: Causal Impact Analysis","Plinski, C. M. (2018). Does AVID Higher Education (AVID HE) increase student term-to-term progression, persistence toward credited classes and social capital for first-generation college students placing into developmental education: A mixed methods study. (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 10825692) doi: 10.15760/etd.6311",,2018,http://dx.doi.org/10.15760/etd.6311,"January 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of the Advancement via Individual Determination Higher Education (AVID HE), a developmental education program, on first generation college students’ educational persistence.
The study used a nonexperimental design to compare the outcomes of students who participated in AVID HE and those who did not. Using data from college institutional records, the author tested for group differences in educational persistence.
The study found that AVID HE participation was significantly related to higher cumulative credits and higher enrollment rates in a credit-earning writing course.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to AVID HE; other factors are likely to have contributed.","The Advancement via Individual Determination Higher Education (AVID HE) is a program designed to help students with low academic and college readiness skills close the academic achievement gap. AVID HE provides training in non-cognitive strategies to increase academic success such as time-management, study skills, note-taking skills, and how to think analytically and critically. The program includes the development of supports, such as student/teacher relationships and learning communities. AVID HE also trains faculty to use more effective classroom instructional strategies and provide student-centered learning environments. The program focuses on encouraging academic progress, tracking data on student learning, and enhancing collaboration within the college.","The study took place at a community college in Multnomah County, Oregon. The author used a nonexperimental design to compare the outcomes of first generation college students in developmental education who participated in the AVID HE program to those who did not. Study participants included 787 students who participated in the program and 1,413 students in the comparison group. Using college institutional records, the author examined group differences in student enrollment and completion of two introductory, credit-earning courses: one in writing and one in math. The author also assessed differences in persistence (defined as enrollment in the second term and the number of credits earned over four terms). The author used chi-square statistics to examine differences between the groups but did not control for differences in student characteristics.","Education and skills gain
The study found a significant relationship between AVID HE participation and credits attempted and earned in the writing course. A higher percentage of students in AVID HE, compared to students who were not in the AVID HE program, enrolled in (+16%) and passed the writing course (+11%). However, no significant relationship was found between program participation and credits attempted or earned in the math course.
A higher percentage of students (+9%) who were in AVID HE enrolled in the next term after beginning the program than did students who were not AVID HE. However, the author did not report the statistical significance of this difference.
The study found a significant relationship between program participation and cumulative credits, with students in AVID HE having a higher number of cumulative credits than the comparison students across the four terms (ranging from 1 to 2 credits per semester).
Similarly, students in AVID HE earned a higher number of credits in each of the four terms. However, the author did not report the statistical significance of the differences.","The author did not account for preexisting differences between the groups before program participation. These preexisting differences between the groups—and not AVID HE—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers","Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Assessing the First Two Years’ Effectiveness of Statway® (Yamada & Bryk 2016)","Assessing the First Two Years’ Effectiveness of Statway® (Yamada & Bryk 2016)","Study Type: Causal Impact Analysis","Yamada, H., & Bryk, A. S. (2016). Assessing the First Two Years’ Effectiveness of Statway®. Community College Review, 44(3), 179–204. doi: 10.1177/0091552116643162",,2016,https://doi.org/10.1177%2F0091552116643162,"January 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the impact of Statway® on community college students’ earning of math course credits.
The study used a nonexperimental design to compare the outcomes of students who were in the Statway® program to a matched comparison group. Using institutional research data from the community colleges, the authors conducted statistical models to examine differences between the groups.
When compared to non-participating students, the study found that Statway® program participation was associated with a significantly higher likelihood of earning college math credit.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to Statway®; other factors are likely to have contributed.","Statway® is a one-year intensive community college course designed to help developmental math students obtain college-level credit in statistics. The program provides an alternative to the traditional developmental mathematics sequence at community colleges that reduces structural barriers to college math achievement while meeting the requirements for introductory college-level math. It combines college-level statistics with skills and concepts from more rudimentary mathematics courses and supplements it with faculty support and instruction. The program is grounded in research-based principles that underscore connecting mathematical concepts to real-world problems, employing repetitive practice to increase students’ understanding of core concepts, fostering active student engagement, and reducing language and literacy barriers. Moreover, Statway® provides professional development support for faculty to ensure high-quality teaching is maintained and facilitates continuous improvement in teaching and learning.","The authors used a nonexperimental design to compare the outcomes of students who were in the Statway® program with students who had taken traditional developmental math. The study included students from two Statway® cohorts. The Year 1 Cohort (Fall 2011) included students from 17 community colleges across 5 states (California, Connecticut, Florida, Texas, and Washington). The Year 2 Cohort (Fall 2012) included students from 15 community colleges in the same states. The authors received student demographic, prior course enrollment, and performance data from each community college. The authors used a hierarchical linear model framework when creating propensity scores to match students on 44 characteristics. Statway® students were matched with non-Statway® students who had similar demographic characteristics and similar past academic performance prior to the start of the study. The Year 1 sample included 928 students in the treatment group and 4,549 in the comparison group. The Year 2 sample included 771 students in the treatment group and 3,583 in the comparison group. The author conducted statistical models to examine group differences in math credit accumulation for the first cohort only.","Education and skills gain 
The study found a significant relationship between program participation and earning of math credits, with Statway® students accumulating 5.57 college credits one year after program completion versus 4.08 credits for comparison students.","The authors created a matched group of non-participating students to compare to Statway® students. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the Statway® program—could explain the observed differences in outcomes.",,"California
Connecticut
Florida
Texas
Washington",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers","Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Assessing the effectiveness of Quantway®: A multilevel model with propensity score matching (Yamada et al. 2018)","Assessing the effectiveness of Quantway®: A multilevel model with propensity score matching (Yamada et al. 2018)","Study Type: Causal Impact Analysis","Yamada, H., Bohannon, A. X., Grunow, A., & Thorn, C. A. (2018). Assessing the effectiveness of Quantway®: A multilevel model with propensity score matching. Community College Review, 46(3), 257-287. doi: 10.1177/0091552118771754",,2018,https://doi.org/10.1177%2F0091552118771754,"January 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Quantway® 1 math program on community college students’ developmental math completion rates.
The authors used a nonexperimental design to compare the outcomes of students who were in the Quantway® 1 program to a matched comparison group.
When compared to non-participating students, the study found that Quantway® 1 program participation was associated with a significantly higher likelihood of successfully completing the developmental math course.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Quantway® 1 program; other factors are likely to have contributed.","The Quantway® program was developed as part of the Carnegie Foundation for the Advancement of Teaching to improve mathematics sequencing. Quantway® is intended for non-STEM students placed two levels below college math and focuses on quantitative literacy. The program includes an accelerated, single term developmental math course (Quantway® 1) as well as a college level math course (Quantway® 2). Quantway® 1 is organized around everyday themes (including citizenship, health care, and financial literacy), and utilizes real world examples in the curriculum as a way to engage students. The program also provides two levels of student supports. One is ""productive persistence"" which is designed to help students persist through complex coursework grounded in social psychology. The second is language and literacy supports in understanding math concepts and vocabulary. The program also includes a faculty development program with online resources, mentorships, and workshops as well as the ability for faculty to participate in collaborative learning community.
The program was first implemented in Spring 2012 in eight community colleges across three states (California, Connecticut, Florida, Minnesota, New York, Texas, and Washington). By 2016, the program was implemented in 14 colleges across eight states (Georgia, Minnesota, New Jersey, New York, Ohio, Washington, West Virginia, and Wisconsin).","The study used a nonexperimental design to compare the outcomes of students who were in the Quantway® 1 program to those who were not. The authors received student demographic, course enrollment, and course performance data from each institution. The authors used a hierarchical linear model framework when creating propensity scores to match students on 37 characteristics. The sample included data from 10 colleges for students who enrolled in the course from Spring 2012 to Fall 2014, resulting in a sample of 4,243 Quantway® 1 students. The comparison group included 83,887 students at the same institutions during the timeframe who did not participate in the program. The sample size was reduced due to the availability of data for the full sample, and the final analytic sample was 3,992 Quantway® 1 students and 12,448 comparison students.","Education and skills gain
The study found a significant relationship between program participation and course completion, where Quantway® 1 students were twice as likely to successfully complete the developmental math course than students in the comparison group.","The authors created a matched group of non-participating students to compare to Quantway® 1 students. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not Quantway® 1—could explain the observed differences in outcomes.",,"Georgia
Minnesota
New Jersey
New York
Ohio
Washington
West Virginia
Wisconsin",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers","Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Life after service for post-9/11 veterans: Data, methods, and policy impacts (Gardner 2016)","Life after service for post-9/11 veterans: Data, methods, and policy impacts (Gardner 2016)","Study Type: Causal Impact Analysis","Gardner, J. S. (2016). Life after service for post-9/11 veterans: Data, methods, and policy impacts (Doctoral dissertation). Retrieved from https://digitalscholarship.unlv.edu/cgi/viewcontent.cgi?article=3867&context=thesesdissertations",,2016,https://digitalscholarship.unlv.edu/cgi/viewcontent.cgi?article=3867&context=th…,"March 2020",Veterans,,"Low Causal Evidence",,,"The study’s objective was to examine the effects of the 2008 GI Bill on post-9/11 veterans’ employment and educational attainment.
The study used a statistical model to compare outcomes between veterans from before and after the passing of the 2008 GI Bill using data from the American Community Survey.
The study found statistically significant relationships between the 2008 GI Bill and the education or employment of veterans.
The quality of causal evidence presented in this report is low because the author did not observe outcomes for multiple demonstrations of the intervention. This means we are not confident that the estimated effects are attributable to the post-9/11 GI Bill; other factors are likely to have contributed.","The 2008 G.I. Bill was a modification of the Montgomery G.I. Bill (1985) that substantially expanded educational benefits to recipients—compared with bills enacted through previous service eras—to increase opportunities for veterans so they could obtain higher education degrees. Benefits for eligible post-9/11 era veterans vary based on geographic location and can amount to more than $170,000 (in 2013 dollars). Similar to the Montgomery GI Bill, the 2008 GI Bill provides living expense stipends in addition to educational benefits. Veterans are eligible for these benefits if they were discharged honorably or for service-related disabilities.","The author compared the outcomes of post-9/11 veterans before and after the passing of the 2008 GI Bill. The study used regression methods to estimate the effects of the bill using data from the American Community Survey-Public Use Microdata Sample from 2005 to 2015. The statistical model included veterans’ race, sex, and location.
The sample included 183,027 post-9/11 veterans ages 18 to 69 who were not on active duty or in the active reserve or National Guard. Study participants were mostly male (85 percent), mostly White (78 percent), and mostly employed.","Employment 
The study found a statistically significant negative relationship between the 2008 GI Bill and the employment outcomes of veterans.
Education 
The study found a statistically significant positive relationship between the 2008 GI Bill and the educational attainment of veterans.","The author compared the education and employment outcomes of post-9/11 veterans measured before and after the passing of the 2008 GI Bill. For these types of designs, the author must observe outcomes for multiple periods before the intervention to rule out the possibility that these veterans had increasing or decreasing trends in the outcomes examined before the 2008 GI Bill. That is, if the veterans who had increasing trends in education or employment outcomes were veterans after the passing of the 2008 GI Bill, we would anticipate further increases over time, even if they didn’t receive benefits through the 2008 GI Bill. Without knowing the trends before the 2008 GI Bill, we cannot rule this out. In addition, the passage of the 2008 GI Bill coincided with the beginning of the Great Recession, and the observed effects on education and employment could be attributable to the recession rather than the bill. Therefore, the study receives a low causal evidence rating.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Veterans' reemployment, Other training and education","Education and skills gains-Low-Favorable impacts, Employment-Low-Unfavorable impacts","Education and skills gains, Employment","Veterans Review Protocol"
"Consortium for Healthcare Education Online: Final evaluation report (Edwards et al 2016)","Consortium for Healthcare Education Online: Final evaluation report (Edwards et al 2016)","Study Type: Causal Impact Analysis","Edwards, R., McKay, H., Mattoon, A., & Yang, S. (2016). Consortium for Healthcare Education Online: Final evaluation report. Piscataway, NJ: Rutgers Education and Employment Research Center.",,2016,https://epe.rutgers.edu/sites/default/files/images/Research_Documents/cheo_fina…,"April 2020","Community College",,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Consortium for Healthcare Education Online (CHEO) on education, earnings, and employment outcomes. This summary contains the findings from Pueblo Community College.
Using institutional, employment, and wage data, the authors conducted a nonexperimental design to compare the outcomes of students who were enrolled in the CHEO program to a historic comparison group of students from previous enrollment years.
The study found that CHEO program participation was significantly associated with earning more course credits, a higher probability of passing courses and receiving a credential, greater likelihood of pursuing further education, and higher rates of employment after program completion.
The quality of causal evidence presented in this study is low because the authors used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the CHEO program; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
TAACCCT funds were used to establish the Consortium for Healthcare Education Online (CHEO) program, which was an interstate consortium consisting of eight community colleges across five states (Colorado, Wyoming, South Dakota, Montana, and Alaska). The CHEO program was created to assist with developing and redesigning allied health programs as well as create hybrid and online delivery options. Strategies included online/hybrid course formats; incorporating the North American Network of Science Labs Online (NANSLO) allowing students to complete science lab work remotely; career coaches to assist students from registration through graduation; building/expanding relationships with employers and local workforce representatives; stackable credential programs with viable career pathways; student portals to assist in planning future careers in healthcare; and the creation and use of open education resources.","The study took place at Pueblo Community College (PCC) in Colorado. The authors used a nonexperimental design to compare students enrolled in the CHEO program to a historic cohort of students from PCC enrolled in the same programs of study. The treatment group included 4,375 PCC students enrolled during the CHEO grant period from Spring 2013 through Spring 2016. The comparison group included a historic cohort of 5,099 PCC students enrolled in the same or similar courses from Fall 2008 through Fall 2012. The study used multiple data sources including institutional and self-reported data from PCC and unemployment and wage data from the state. The authors examined differences in outcomes between the groups using statistical models with controls for demographic, academic, and employment information. Outcomes included academic performance (number of credits and pass rates), program completion (credentials earned, CHEO credential, and credential type earned), pursuing further education, earnings, and employment post-completion.","Education and skills gain 
The study found a significant relationship between program participation and number of credits earned, with CHEO participants earning 5 more course credits than the comparison group.
The study found a significant relationship between program participation and the probability of pursuing further education; CHEO participants were over 19% more likely to continue education relative to the comparison group.
The study also found a significant relationship between program participation and passing rates, with CHEO participants having a 19% higher probability of passing a CHEO course.
Relative to the comparison group, the study found that CHEO program participants were 11% more likely to receive a CHEO credential and 6% more likely to earn any credential.
Earnings and wages
The study did not find a significant association between the CHEO program and post-completion wages for both incumbent and non-incumbent workers.
Employment 
The study found a significant relationship between program participation and post-completion employment rates for incumbent workers; CHEO participants were 1% more likely to be retained in a job relative to the comparison group.
For non-incumbent workers, the study found a significant association between the CHEO program and post-completion employment rates, where nearly 9% of program participants were more likely to gain employment after earning a CHEO credential.","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college or labor market) and not the program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) TAACCCT IV evaluation final report (Foster et al 2018)","Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) TAACCCT IV evaluation final report (Foster et al 2018)","Study Type: Causal Impact Analysis","Foster, L. R., Staklis, S., Ott, N. R., & Moyer, R. (2018). Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) TAACCCT IV evaluation final report. Raleigh, NC: RTI International.",,2018,https://www.skillscommons.org/bitstream/handle/taaccct/18334/KanTRAIN_TAACCCT_I…,"April 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) program on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare education, earnings, and employment outcomes of KanTRAIN participants to a matched comparison group.
The study found KanTRAIN participants were significantly more likely to earn more credits per semester, pass a higher percentage of the credits attempted, attain more credentials, and earn a nondegree credential or a college degree, and receive a wage increase than the comparison group.
The quality of causal evidence presented in this study is low because the authors used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the KanTRAIN program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) program was funded by a Round IV TAACCCT grant and implemented by five community and technical colleges in Kansas. The main objectives of the KanTRAIN program were to expand institutional capacity and develop career pathway programs in health care and advanced manufacturing. The program sought to increase employer engagement, facilitate outreach to veterans, and upgrade technical skill training programs by aligning these pathways with statewide initiatives. To expand and strengthen existing career pathways, KanTRAIN incorporated several evidence-based strategies. These strategies included design and delivery of industry-focused and employer-driven workforce training models, integration of industry-recognized and stackable credentials into career pathway programs, strengthened case management and increased support for participants.","The study used a nonexperimental design to compare the outcomes of students who participated in the KanTRAIN program to students who did not participate. The authors matched KanTRAIN students to similar students using propensity scores developed from 21 demographic and academic variables. Across the five colleges, study participants included 4,617 students. The treatment group included 1,168 students who participated in the KanTRAIN program and were enrolled between fall of 2015 and spring of 2017. The comparison group was a historical cohort composed of 3,449 students that were enrolled between fall 2011 and summer 2015 in the same or similar programs (prior to KanTRAIN implementation). Using data from the KanTRAIN program and the Kansas Board of Regents, the authors conducted statistical models to examine differences in outcomes. Outcomes included number of credits earned, proportion of credits passed, nondegree credential attainment, college degree attainment, employment status, quarterly wages, and wage difference.","Education and skills gain 
The study found a significant relationship between program participation and general credits earned and passed, with KanTRAIN students earning 1.5 more general credits per semester, and passing a higher percentage than students in the comparison group (87% versus 83%).
The study found a significant relationship between program participation and technical credits earned and passed, with KanTRAIN students earning 2.1 more technical credits per semester, and passing a higher percentage than students in the comparison group (91% versus 88%).
The study found that KanTRAIN students were significantly more likely to earn a earn a nondegree credential (72%) and earn a college degree (11%) when compared to students in the comparison group (62% and 9%, respectively).
Earnings and wages
For study participants with pre-enrollment data, the study found that the likelihood of a wage increase was 6% higher for KanTRAIN students compared to students in the comparison group.
The study found no other statistically significant relationships between the KanTRAIN program and wages.
Employment 
The study found no statistically significant relationship between the KanTRAIN program and employment.","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the program. Also, the employment data used in the analyses were all self-reported and only available for participants employed or unemployed in Kansas or Missouri. Participants employed or unemployed outside of these states were not included in the employment outcome analysis of the study. Finally, the study did not account for changes in state or regional labor markets, thus, a broader economic context may have impacted the estimated effects of the KanTRAIN program on employment and earnings outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"Flint Hills Technical College in Emporia, Kansas
Garden City Community College in Garden City, Kansas
Washburn Institute of Technology in Topeka, Kansas
Washburn University Washburn University in Topeka, Kansas
Wichita State University Campus of Applied Sciences and Technology in Wichita, Kansas",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final evaluation report: Summative evaluation TAACCCT round 2 accelerated, blended – online, 2 + 2 TAACCCT training project (Franco 2016)","Final evaluation report: Summative evaluation TAACCCT round 2 accelerated, blended – online, 2 + 2 TAACCCT training project (Franco 2016)","Study Type: Causal Impact Analysis","Franco, E. O. (2016). Final evaluation report: Summative evaluation TAACCCT round 2 accelerated, blended – online, 2 + 2 TAACCCT training project. Cayey, PR: Colegio Universitario de San Juan.",,2016,https://www.skillscommons.org/handle/taaccct/15668,"April 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Colegio Universitario de San Juan’s (CUSJ) modified electronics, engineering, and technology programs on education outcomes.
The study used a nonexperimental design to compare education outcomes of CUSJ participants to a comparison group. Using CUSJ’s embedded course assessments, the author conducted statistical tests to examine differences between the groups.
The study found that CUSJ participation was significantly associated with higher approval rates (defined as passing a course).
The quality of causal evidence presented in this report is low because the author did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to the CUSJ course modifications; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Colegio Universitario de San Juan (CUSJ) in Puerto Rico used TAACCCT funds to improve technological infrastructure; create online and blended courses in Instrumentation Technology, Electric Power Technology, and Electronics Engineering Technology; add virtual educational modules; improve faculty's use of technology; improve student services and counseling; and implement online tutoring. The program was aimed at TAA-eligible workers, unemployed workers, and other university students.","The study used a nonexperimental design to compare the outcomes of students who participated in the CUSJ modified courses to students who did not participate. The treatment group included 192 CUSJ students who enrolled in and rated select modified core courses from the Instrumentation Technology, Electric Power Technology, and Electronics Engineering Technology programs from 2013-2016. The comparison group included 2,495 students who enrolled in and rated the traditional formats. The outcome of interest was course approval rates defined as passing a course. Using embedded assessment data from CUSJ’s courses, the author used statistical models to examine differences between the treatment and comparison groups.","Education and skills gain
Relative to the comparison group, the study found that participation in CUSJ modified courses was significantly associated with higher approval rates (passing rates) in five courses: Physics I Lab (78% vs. 62%), Pre-Calculus I (51% vs. 25%), Alternate Current (AC) Lab (93% vs. 83%), Introduction to Electronics Lab (92% vs. 83%), and Advance Electronics (100% vs. 67%).","The author did not account for preexisting differences between the groups before program participation or include sufficient control variables, such as age, race/ethnicity, gender, a pre-intervention measure of financial disadvantage, or a pre-intervention measure of education. These preexisting differences between the groups—and not CUSJ modified courses—could explain the observed differences in the outcome. Additionally, the author does not provide information on when the comparison group completed the courses, does not account for self-selection into TAACCCT-modified courses, and conducted a one-tailed t-test as opposed to a two-tailed t-test when assessing statistical significance. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Unemployed","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Final evaluation Golden Triangle Modern Manufacturing Project (Harpole 2017)","Final evaluation Golden Triangle Modern Manufacturing Project (Harpole 2017)","Study Type: Causal Impact Analysis","Harpole, S. (2017). Final evaluation Golden Triangle Modern Manufacturing Project. SHH Consulting, LLC",,2017,https://www.skillscommons.org/handle/taaccct/15669,"April 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Golden Triangle Modern Manufacturing Project on education, earnings, and employment outcomes.
The author used a nonexperimental design to compare outcomes of Golden Triangle Modern Manufacturing Project participants to a matched comparison group.
The study found that program participation was significantly associated with increases in program retention, credential attainment, and earnings.
The quality of causal evidence presented in this study is low because the author used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to Golden Triangle Modern Manufacturing Project; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
East Mississippi Community College (EMCC) received a TAACCCT grant to implement the Golden Triangle Modern Manufacturing Project. Based on the model from the 2012 Missouri Manufacturing Workforce Innovation Networks TAACCCT project, the Golden Triangle Modern Manufacturing Project aimed to prepare adult students for careers in advanced manufacturing. Features included building programs that meet industry needs, enhancing career pathway options for learners and workers, accelerating and improving certification and employment attainment, and strengthening online and technology-enabled learning. The project added three credential- and craft-based technician programs, created a new career and technical education (CTE) developmental program, and engaged local manufacturers and workforce partners in these programs’ design and curriculum to concurrently meet industry demands and the needs of the target population, including veterans, TAA-eligible workers, and other unemployed or unskilled adults. These enhanced programs helped fill gaps in existing education and career training programs and relied on incorporating the National Association of Manufactures (NAM) endorsed credentials. More specifically, the project modernized EMCC's manufacturing-related CTE programs by stacking industry-recognized credentials into six CTE programs, having CTE instructors leveraging advanced technology, prioritizing soft skill development in their instruction delivery, and requiring all program participants to complete a Silver Career Readiness Certificate (CRC).","The study used a nonexperimental design to compare the outcomes of students who participated in the Golden Triangle Modern Manufacturing Project to those who did not. The comparison group included first-time students who were enrolled in comparable CTE programs offered at EMCC prior to October 2013. The author matched Golden Triangle Modern Manufacturing participants to similar nonparticipants using propensity scores developed from demographic, economic, and program-based characteristics. Study participants included 373 students in the treatment group and 548 in the comparison group. Student program performance and baseline data was provided and collected from October 2013 to September 2016 for the treatment group and from October 2009 to September 2013 for the comparison group. Using performance and baseline data, the author conducted statistical models to examine differences in outcomes. The outcomes included program credential completion and retention, enrollment in further education, employment of non-incumbent workers, job retention, and increase in wages for incumbent workers.","Education and skills gain 
The study found that participation in the Golden Triangle Modern Manufacturing Project was significantly associated with increased program retention, with higher rates of program retention among participating students (85%) compared to students in the comparison group (54%).
The study also found that Golden Triangle Modern Manufacturing students were significantly more likely to attain a credential (90%) when compared to students in the comparison group (46%).
The study did not find a significant relationship between participation in the Golden Triangle Modern Manufacturing Project and further education in a non-TAACCCT funded program of study.
Earnings and wages 
The study found that Golden Triangle Modern Manufacturing students were significantly more likely to have an earnings increase after program completion (21%) when compared to students in the comparison group (20%).
Employment
The study did not find a significant relationship between enrollment in the Golden Triangle Modern Manufacturing Project and the employment rate one quarter after program exit or the job retention rate two quarters after program exit.","Though baseline equivalency was achieved between the two research groups prior to the program intervention, there was a confounding factor present in this study. Given that the author used a cohort from previous enrollment years as the comparison group, differences in outcomes could be due to time-varying factors (such as overall changes at the community college or local labor market changes) and not the program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Unemployed, Low-skilled, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Evaluation of Clark State Community College’s Advanced Manufacturing to Compete in a Global Economy (AMCGE) training program final report (Haviland et al 2018)","Evaluation of Clark State Community College’s Advanced Manufacturing to Compete in a Global Economy (AMCGE) training program final report (Haviland et al 2018)","Study Type: Causal Impact Analysis","Haviland, S. B., Van Noy, M., Kuang, L., Vinton, J., & Pardalis, N. (2018). Evaluation of Clark State Community College’s Advanced Manufacturing to Compete in a Global Economy (AMCGE) training program final report.",,2018,http://www.skillscommons.org/bitstream/handle/taaccct/17927/Clark%20State_Final…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Advanced Manufacturing to Compete in a Global Economy (AMCGE) workforce training program on education outcomes.
The authors used a nonexperimental design to compare the outcomes of students participating in the AMCGE program to a matched comparison group using data from college administrative records.
The study found a significant relationship between AMCGE program participation and increased number of credits earned and increased program completion rates in the first year.
The quality of causal evidence presented in this report is low because the authors used a comparison group from a different academic program presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the AMCGE program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Clark State Community College in Ohio received a TAACCCT grant to implement the Advanced Manufacturing to Compete in a Global Economy (AMCGE) workforce training program. The AMCGE program included five certificate programs: Additive Manufacturing, Computer Numerical Control, Industrial Maintenance, Supervisory Control and Data Acquisition, and Welding. The certificate programs were credit-bearing and stackable and could be applied towards earning an associate’s degree in Manufacturing Engineering Technology or Industrial Technology. As part of the grant, the program added hands-on equipment for instruction, offered a career navigator, and redesigned the manufacturing coursework.","The authors used a nonexperimental design to compare the outcomes of students in the AMCGE program to students who were not in the program. The study only included participants who were first time college students. Participants in the treatment group were enrolled in an AMCGE course or certificate program, between Fall 2015 and Fall 2017. Participants in the comparison group were enrolled in an Information Technology (IT) program at the college and started college between Fall 2013 and Fall 2017. The authors matched AMCGE participants to similar nonparticipants using propensity scores developed from demographic and education information. There were 305 students in the treatment group and 547 students in the comparison group. Using college administrative systems data and data from the state of Ohio, the authors compared outcomes between the treatment and comparison groups.","Education and skills gain
The study found a significant positive relationship between program participation and program completion within one year, with a higher percentage of AMCGE students completing the program (6%) compared to IT students (1%). However, no significant relationship was found between program participation and program completion within two years.
Similarly, the study found a significant positive relationship between program participation and total number of credits earned within one year, with AMCGE students earning more credits than IT students (12.43 credits versus 10.16 credits). However, no significant relationship was found in the second year.
The study found no significant relationship between program participation and retention rates within one or two years.","Participants in the comparison group were in an IT program, whereas those in the treatment group were in the Advanced Manufacturing program. Because the two groups were enrolled in different programs, differences in outcomes could be due to varying factors in the programs of study (such as required coursework) and not the AMCGE program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Alliance evaluation: Final evaluation report (Ho 2016)","Alliance evaluation: Final evaluation report (Ho 2016)","Study Type: Causal Impact Analysis","Ho, H. Y. (2016). Advanced Manufacturing Education (AME) Alliance evaluation: Final evaluation report (Grant Number TC-23753-12-60-A-27). Denver, CO: McREL International.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/10174/DOLAME_Y4_FinalEva…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"This study’s objective was to examine the impact of the Advanced Manufacturing Education (AME) Alliance on education, earnings, and employment outcomes.
The author used a nonexperimental design to compare education, employment, and earnings outcomes of AME participants to a matched comparison group.
The study found that AME participants were significantly more likely to complete a program of study relative to the comparison group.
The quality of casual evidence provided in this study is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the enhanced AME program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Advanced Manufacturing Education (AME) Alliance was formed with three community colleges and one state college in Minnesota upon receipt of a four-year TAACCCT grant. The AME redesigned advanced manufacturing degree programs at these colleges to better meet the needs of their students by developing a hybrid curriculum, providing additional support mechanisms to students, collaborating with employer and workforce partners, and utilizing technology across programs. This included purchasing technology for students, developing four hybrid courses and five modularized courses, hiring six educational and employment advisors, establishing a website providing general information about grant programs for students, and expanding employment partners.","The nonexperimental study was conducted across the AME alliance colleges in Minnesota (Central Lakes College, Pine Technical and Community College, Saint Cloud Technical and Community College, and Minnesota’s 360 Center for Manufacturing Excellence). The author matched participants enrolled in enhanced AME courses to nonparticipants, using propensity scores developed from baseline demographic information. Study participants included 180 participants enrolled in an enhanced AME program in the fall of 2014, as well as 180 comparison students who attended a non-enhanced AME or similar program from 2005-2010. Using data from student database records and Minnesota’s Department of Employment and Economic development, the author used statistical models to examine differences in outcomes between the groups. Outcomes included degree completion, and wage increases and employment six months after completion of an AME program.","Education and Skills Gain 
The study found that AME participation was significantly associated with program completion, with program participants being 1.6 times more likely to complete their program of study than the comparison group.
Earnings and Wages 
The study found no significant relationship between participation in the AME-enhanced program and earnings increase 6 months after completing an AME program.
Employment 
The study found no significant relationship between participation in the AME-enhanced program and employment 6 months after completing an AME program.","The author created a matched group of non-participating students to compare to students who were enrolled in an AME-enhanced program. However, the author did not appropriately control for other factors that could have affected the differences between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the enhanced AME program—could explain the observed differences in outcomes. Additionally, the author used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Project IMPACT: Innovative multi-industry partnership and career training project final evaluation report. Gateway Community Technical College (Jensen et al. 2017)","Project IMPACT: Innovative multi-industry partnership and career training project final evaluation report. Gateway Community Technical College (Jensen et al. 2017)","Study Type: Causal Impact Analysis","Jensen, J., Horohov, J., & Waddington, J. (2017). Project IMPACT: Innovative multi-industry partnership and career training project final evaluation report. Gateway Community Technical College. Lexington, KY: University of Kentucky, College of Education, Educational Policy Studies & Evaluation.",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/15674/IMPACT%20Final%20E…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of Project IMPACT on education, earnings, and employment outcomes.
Using academic records from the college as well as data obtained from the Kentucky Center for Workforce Statistics, the authors used a nonexperimental design to compare outcomes of Project IMPACT participants to outcomes of historical cohort students who were enrolled in the Project IMPACT programs of study prior to grant implementation.
The study found that participating in Project IMPACT was significantly associated with a higher likelihood to take courses, pass courses, earn credits, and receive credentials. The study also found that participating in Project IMPACT was significantly associated with higher earnings in the quarter following enrollment.
The quality of causal evidence presented in this study is low for the education and employment outcomes because the authors used a comparison group from previous enrollment years presenting a confounding factor and because the authors did not ensure the groups being compared were similar before the intervention, respectively. This means we are not confident that the estimated effects are attributable to Project IMPACT; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2013, Gateway Community and Technical College (GCTC) in Kentucky received a Trade Adjustment Assistance Community College Career Training (TAACCCT) grant to implement the Innovative Multi-Industry Partnership and Career Training (IMPACT) program. The purpose of the IMPACT project was to enhance career preparation in logistics, manufacturing, heating and cooling, and energy fields. The project provided a combination of academic and career coaching in cooperation with workforce development partners, community agencies, and industry employers. The program targeted eligible workers, veterans and underemployed adults who were pursuing careers in logistics, manufacturing, energy, and heating and cooling fields. Key components of Project IMPACT included academic and career coaching in partnership with workforce development partners, community agencies, and industry employers.","The study was conducted at Gateway Community Technical College in Florence, Kentucky. The authors compared outcomes of Project IMPACT participants to outcomes of historical cohort students who were enrolled in the Project IMPACT programs of study before implementation of the grant. The authors used a nonexperimental design with propensity score matching to ensure participants in both groups were similar on baseline characteristics. After the matching procedure, there were 96 students who had received the full Project IMPACT program treatment, and there were 93 students in the comparison group.","Education 
The study found that participating in Project IMPACT was significantly associated with taking and passing more courses. Project IMPACT students took an average of 20.4 courses and passed 18.2 of them; students in the comparison group took an average of 5.0 courses and passed 3.2.
The study found that participating in Project IMPACT was significantly associated with earning more credits, with higher proportions of Project IMPACT students earning credits (93%) than comparison students (73%).
The study found that participating in Project IMPACT was significantly associated with earning more credentials, with higher proportions of Project IMPACT students earning credentials (62%) than comparison students (15%).
Employment 
The study did not find a significant relationship between Project IMPACT and employment rates in the quarter after enrollment.
Earnings
The study found that participating in Project IMPACT was significantly associated with an increase in earnings in the quarter after enrollment. Project IMPACT students had nearly double the mean quarterly wages as the comparison students ($9,100 and $4,413, respectively).","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Additionally, the authors created a matched group of a historical cohort to compare to Project IMPACT students. However, for the employment outcomes, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as a pre-intervention measure of employment. These preexisting differences between the groups—and not Project IMPACT —could explain the observed differences in outcomes. Finally, the authors also note that employment data was only available for students one month after completion and therefore, students who were employed and also still enrolled in the program, were not included in the sample. They also note that the location of the program on the border of Kentucky and Ohio may have misrepresented the employment data as individuals who found work in Ohio would not have been captured within the Kentucky data (although most of the students did have Kentucky addresses).",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers, Veteran or military","Employer partnerships, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final evaluation report: Linn-Benton Community College (Kelley-Smith et al. 2017)","Final evaluation report: Linn-Benton Community College (Kelley-Smith et al. 2017)","Study Type: Causal Impact Analysis","Kelley-Smith, A., Schoeph, K., Hamai, T., & Bishop, M. (2017). Final evaluation report: Linn-Benton Community College. Albany, OR: LB iLearn Campus.",,2017,https://www.skillscommons.org/handle/taaccct/15675,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of degree and certificate programming offered at Linn-Benton Community College’s (LBCC) iLearn campus compared to traditional programming offered at the community college on various educational, employment, and earnings outcomes.
The study used a nonexperimental design to compare the outcomes of students enrolled in the Linn-Benton (LB) iLearn campus to students enrolled in traditional programming.
The study found that program participation was significantly associated with increased rates of program completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the LB iLearn campus program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Linn-Benton Community College (LBCC) in Oregon was awarded a TAACCCT grant in October 2013. LBCC used the TAACCCT funds to design and operate a virtual college called LB iLearn campus with the goal being to help dislocated workers, veterans, and other disadvantaged adults complete education and training so they can successfully obtain employment in Oregon's in-demand industries. These include healthcare, accounting, business and office administration, communications, and marketing. The virtual campus design included incorporating stacked and latticed credentials, using a competency-based framework that could accelerate learning, and offering extensive student support services through an online platform that used education modules for instruction. Specifically, LB iLearn campus staff offered customizable, online educational programming that was student-centric and allowed students to complete the program at their own pace. Additionally, this online platform provided academic and non-academic support to students through dedicated program personnel like Student Navigators, Content Experts, and Assessment Evaluators.","The authors used a nonexperimental design to compare the outcomes of students in the LB iLearn campus program to students attending the traditional programming offered at the LBCC campus. Study participants were comprised of students enrolled with their first credit during the grant period in a TAACCCT-funded LB iLearn campus program (i.e., treatment group) or in a traditional program comparable to a TAACCCT-funded LB iLearn campus program (i.e., comparison group). The grant period for the analysis included 15 consecutive quarters from Fall 2014 to Winter 2017. The authors matched program participants to similar nonparticipants using propensity scores developed from demographic and academic information. After matching. study participants included 193 students in the treatment group and 386 students in the comparison group. Using data from LBCC’s existing program review process, the National Student Clearinghouse, and the Oregon Employment Department, the authors used statistical models to examine differences in outcomes between the groups. Outcomes included program completion, credits earned, transfer to a 4-year institution, retention in the LB iLearn or traditional campus, employment placement, and wage upon completion of the program.","Education
The study found that LB iLearn campus students were 37.21 times more likely to complete the program than traditional comparison students and this result was statistically significant.
The study found that there was no statistically significant relationship between enrollment in the LB iLearn campus programs and number of credits earned, the likelihood of transferring to a four-year academic institution, or the likelihood of program retention.
Employment
The study found that there was no statistically significant relationship between enrollment in the LB iLearn campus programs and improved employment status upon program completion.
Earnings and Wages
The study found that there was no statistically significant relationship between enrollment in the LB iLearn campus programs and improved wages upon program completion.","The authors created a matched group of non-participating students to compare to LB iLearn campus students. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the LB iLearn campus program—could explain the observed differences in outcomes. Moreover, the authors noted that they could not find comparable comparison students for two programs (Computed Tomography and Social Media Specialist). Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers, Dislocated or displaced worker, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Butler Community College TAACCCT final evaluation report (Kansas State University Office of Educational Innovation and Evaluation 2017)","Butler Community College TAACCCT final evaluation report (Kansas State University Office of Educational Innovation and Evaluation 2017)","Study Type: Causal Impact Analysis","Kansas State University Office of Educational Innovation and Evaluation. (2017). Butler Community College TAACCCT final evaluation report. Manhattan, KS: Kansas State University.",,2017,https://www.skillscommons.org/handle/taaccct/15664,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Butler Community College Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant on education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the outcomes of students who were in the Butler TAACCCT Information Technology (IT) program to a comparison group.
The study found that students in the Butler TAACCCT IT program were significantly more likely to complete their program, be retained in their program, and earn credentials than their counterparts in other selected Career and Technical Education (CTE) courses.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention or include sufficient controls. This means we are not confident that the estimated effects are attributable to the Butler TAACCCT IT program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Butler Community College in El Dorado, Kansas received a TAACCCT grant to increase the number of students with credentials in the Information Technology (IT) field. Butler created the Butler IT Institute and created a core set of courses for IT students to develop their skills and knowledge. The program was designed to target TAA-eligible, veterans, and other dislocated workers. The project also integrated academic and employment supportive services for students and developed IT-related degrees, credentials, and certifications in Cyber Security, Database Administration, Digital Media, Interactive & 3D Technologies, Internetworking Management, Software Development, Web Development, and Windows Administration.","The study used a nonexperimental design to determine the impact of the Butler Community College TAACCCT IT program on education, earnings, and employment outcomes. The treatment group consisted of 657 students taking TAACCCT IT program courses between 2013 and 2010, while the comparison group consisted of 88 students taking courses in the CTE degree tracks of Engineering Graphics Technology, Composite Engineering Technology, Marketing and Management, and Welding during the same time. Using data from Butler College's administrative records, the authors compared the differences in the proportion of students who achieved education, earnings, and employment outcomes between the treatment and comparison groups.","Education and skills gain 
The study found that students who participated in the Butler TAACCCT IT program were significantly more likely than comparison group students to complete a program of study (23.0% vs. 4.5%, respectively), be retained in a program of study (29.2% vs. 4.5%), and earn credentials (23.0% vs. 4.5%).
The study found no statistically significant relationships between participation in the Butler TAACCCT IT program and credit hour completion or the likelihood of pursuing further education.
Earnings and wages
The study found no statistically significant relationship between participation in the Butler TAACCCT IT program and a wage increase post-enrollment, for students who were employed at the beginning of the program.
Employment
The study found no statistically significant relationships between participation in the Butler TAACCCT IT program and the likelihood of being employed or retaining employment upon program completion.","The authors do not establish whether the treatment and comparison groups are significantly different at baseline. They provide information indicating there may be differences in characteristics such as average age, race, incumbent worker status, part-time status, and other demographic variables, but do not report whether these variations are statistically significant. Also, the authors did not control for any baseline differences as they used chi-square analyses. These preexisting differences between the groups—and not the TAACCCT IT program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Veteran or military","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"An evaluation of the manufacturing advancement and assessment center (MAAC) program, final report (Kundin & Dretzke 2016)","An evaluation of the manufacturing advancement and assessment center (MAAC) program, final report (Kundin & Dretzke 2016)","Study Type: Causal Impact Analysis","Kundin, D.M., & Dretzke, B.J. (2016). An evaluation of the manufacturing advancement and assessment center (MAAC) program, final report. St. Paul, MN: University of Minnesota, Center for Applied Research and Educational Improvement, College of Education and Human Development.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Manufacturing Assessment and Advancement Centers (MAAC) program on educational outcomes.
The authors used a nonexperimental design to compare the outcomes of students who were enrolled in the MAAC program to a historic comparison group of students from previous enrollment years.
The study found a significant relationship between program participation and increased credential attainment.
The quality of causal evidence presented in this study is low because the authors used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the MAAC program; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories. federal grant.
In 2012, Hennepin Technical College (HTC) in Minnesota was awarded a TAACCCT grant to establish the Manufacturing Assessment and Advancement Centers (MAAC) program to address a demand in the Minneapolis-St. Paul metro area for more highly skilled manufacturing workers. The MAAC program was designed to assist TAA-eligible and other adult or non-traditional learners, in earning credentials, become employed, receive wage increases, and retain employment in the manufacturing industry. The MAAC program’s primary goals were to establish an accelerated learning pathway for students and assist HTC in their ability to increase the number of students who receive credentials, complete programs, and retain employment. HTC awarded credit for prior learning assessments and implemented online learning opportunities. They also provided students with access to hands-on simulation technology (e.g., virtual welders, CNC machines, and 3D printing), career counseling, and strong organizational support with increased resources.","The nonexperimental study compared students enrolled in the MAAC program to a historic cohort of students from HTC enrolled in the same programs of study. The treatment group consisted of 220 HTC students who participated in the MAAC program (51 participants were enrolled in 2014 and 169 were enrolled in 2015). The comparison group also consisted of 220 HTC students from the same manufacturing programs, but who were enrolled in 2011 and 2012 prior to the implementation of the MAAC program. The authors matched MAAC students to similar students using propensity scores developed from demographic and academic variables. Using institutional data and self-reported data from HTC, the authors conducted statistical models to examine differences between the groups on program completion (measured as earning an Associate’s degree, a certificate or a diploma), earned credit hours, and the aggregate number of earned credentials.","Education and skills gain
The study found a significant relationship between program participation and earning an Associate’s degree or a certificate or diploma, with a higher proportion of MAAC program participants from the 2014 cohort earning a credential (41%) than the comparison group (16%). However, there was no significant relationship found between the MAAC program 2015 cohort and credential attainment.
The study also found a significant relationship between program participation and the aggregate number of earned credentials, with MAAC participants earning more credentials than the comparison group. However, there was no significant relationship found between the MAAC program 2015 cohort and the total number of obtained credentials. • The study found no significant association between MAAC program participation and earning course credits for both 2014 and 2015 cohort comparisons.","The authors used a cohort of HTC students from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the MAAC program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Compliance, Education and skills gains","Community College Review Protocol"
"Transitioning students from adult education to postsecondary education through co-enrollment career pathways model (Maisak 2017)","Transitioning students from adult education to postsecondary education through co-enrollment career pathways model (Maisak 2017)","Study Type: Causal Impact Analysis","Maisak, N. (2017). Transitioning students from adult education to postsecondary education through co-enrollment career pathways model. (Unpublished doctoral dissertation). Charlotte, NC: Wingate University.",,2017,https://eric.ed.gov/?id=ED580629,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the impact of the Pathways to Careers program on community college students’ education outcomes.
The author used a nonexperimental design to compare the outcomes of students enrolled in the Pathways to Careers program to students enrolled in a traditional Adult Secondary Education (ASE) program.
The study found that participation in the Pathways to Careers program was significantly related to increased retention rates and a higher likelihood of transition to postsecondary education.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Pathways to Careers program; other factors are likely to have contributed.","The Pathways to Careers Program was offered at a large, urban community college in North Carolina and integrated a career pathways model. The program allowed students to work towards post-secondary credentials while obtaining high school equivalency or a high school diploma. The goal of the program and model was to move students to post-secondary education and/or employment faster. An essential element of the program was co-enrollment, because the program organized a series of manageable steps that led to the completion of a high school credential as well as an occupational or industry recognized certification. This was accomplished by integrating basic education, academic skills development, occupational training, work and college readiness skills, and occupational-specific training through college-level courses. The Pathways program also offered support services to students that included life skills assistance, college adjustment and retention services, supportive services, case management, career assistance, mentoring, and counseling.","The study used a nonexperimental design to compare the outcomes of students who participated in the Pathways to Careers program to those who did not. Participants included adult basic education students from the North Carolina Community Colleges System (NCCCS) in the college and career readiness (CCR) department. Students were in traditional Adult Basic Education (ABE) and Adult Secondary Education (ASE) programs during the 2013-2014 and 2014-2015 academic years. The author used secondary data to identify the treatment and comparison groups. The treatment group included students who were enrolled in the Pathways to Careers program and had low basic education skills in writing, reading, math, and computer literacy. The comparison group included CCR students who had an ASE placement, but were not enrolled in the Pathways to Careers program. The 2013-2014 study cohort included 239 students in the treatment group and 974 students in the comparison group. The 2014-2015 study cohort included 210 students in the treatment group and 983 students in the comparison group. The author conducted chi-square analyses to examine differences in enrollment in postsecondary education and retention between the treatment and comparison groups.","Education and skills gain
The study found that participation in the Pathways to Careers program was significantly related to increased retention rates for both cohorts. For the 2013-2014 study cohort, 37% of treatment students were retained compared to 10% of comparison students. For the 2014-2015 study cohort, 61% of treatment students were retained compared to 44% of comparison students.
The study found a significant relationship between student enrollment in the Pathways to Careers program and the transition to postsecondary education for both cohorts. For the 2013-2014 study cohort, 29% of treatment students transitioned to postsecondary education compared to 9% of comparison students. For the 2014-2015 study cohort, 29% of treatment students transitioned to postsecondary education compared to 9% of comparison students.","The author did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not the Pathways to Careers program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"Urban, United States","Causal Impact Analysis",Adult,"Basic skills, Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Kansas City Kansas Community College: Training for Employment (T4E) program (Martin & Melzer 2016)","Kansas City Kansas Community College: Training for Employment (T4E) program (Martin & Melzer 2016)","Study Type: Causal Impact Analysis","Martin, G., & Melzer, B. A. (2016). Kansas City Kansas Community College: Training for Employment (T4E) program. Kansas City, MO: Evalytics, LLC.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Kansas City Kansas Community College’s Training for Employment (T4E) program on student education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the education, earnings, and employment outcomes of T4E students to a comparison group. Using data from the college’s student database and the American Job Link Alliance, the authors conducted statistical models to examine differences in outcomes between the groups.
The study found that participation in the T4E program was significantly associated with a higher likelihood of certificate completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the T4E program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Kansas City Kansas Community College’s (KCKCC) Training for Employment (T4E) program was developed with TAACCCT funds to increase student skills, knowledge, and employment outcomes in the construction and advanced manufacturing fields. The T4E program, piloted from 2014-2016, included six primary course pathways: 1) Building and Property Management Technology, 2) Construction Technology, 3) Electrical Technology, 4) Heating and Refrigeration, 5) Machine Technology, and 6) Welding Technology. In addition, students could enroll in a green construction program called Green-Up. The T4E program targeted TAA-eligible workers and other community college students in the advanced manufacturing and construction industries and strategies included creating stacked and latticed industry credential pathways that moved students toward employment and further education, alternative technology-based teaching and instruction, basic adult education and technical training, financial literacy, employability skills, and entrepreneurship training.","The study used a nonexperimental design to compare the outcomes of students who participated in the T4E program to students who did not participate. KCKCC’s main campus is located in Kansas City, Kansas, while its two extension campuses are located in Leavenworth, Kansas. The treatment group included 153 students enrolled in one of the six T4E treatment programs or the Green-Up program and exited in 2014. The comparison group included 117 students who took one of the six T4E programs (excluding Green-Up) and exited the program in fall 2011, spring 2012, or summer 2012. Outcomes included certificate completion, program completion, withdrawal or dropout status, employment retention, and earnings. Using data from KCKCC's Ellucian student database and the American Job Link Alliance databases, the authors used statistical tests to examine differences in the outcomes between treatment and comparison groups.","Education and skills gain 
The study found that T4E participation was significantly associated with higher rates of certificate completion (59%) than comparison group students (44%).
The study found no significant relationship between T4E participation and program completion or withdrawal/dropout rates.
Employment
The study found that T4E participation was associated with higher employment retention two quarters after exit than the comparison group (90% vs. 87%, respectively). However, no tests of significance were performed.
Earnings and wages
The study found that T4E participation was associated with lower average earnings over two quarters than the comparison group ($12,826 vs. $14,341, respectively). However, no tests of significance were performed.","When reporting employment and earnings outcomes, the authors noted that the comparison group had been in the workforce longer, so differences in average earnings may represent a difference in pay related to length of employment. The authors also did not perform tests of statistical significance on employment and earnings data. Additionally, the authors used a singular cohort from previous enrollment years as the comparison group and compared them to a three-cohort treatment group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy or course difficulty) and not the intervention. Lastly, the authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not T4E—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Basic skills, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Third-party evaluation of the outcomes and impact of the National Information, Security & Geospatial Technologies Consortium (NISGTC) (Bridges et al. 2015)","Third-party evaluation of the outcomes and impact of the National Information, Security & Geospatial Technologies Consortium (NISGTC) (Bridges et al. 2015)","Study Type: Causal Impact Analysis","Bridges, K., Bishop, C., & Giani, M. (2015). Third-party evaluation of the outcomes and impact of the National Information, Security & Geospatial Technologies Consortium (NISGTC). Champaign, IL: Office of Community College Research and Leadership, University of Illinois at Urbana-Champaign.",,2015,https://occrl.illinois.edu/docs/librariesprovider4/nisgtc/nisgtc-impact.pdf,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of TAACCCT-funded programming offered at National Information Security & Geospatial Technology Consortium (NISGTC) colleges on educational and employment outcomes.
Using student-level data collected by the colleges’ Institutional Research offices and states’ Unemployment Insurance data systems, the study used a nonexperimental design to compare the outcomes of students enrolled in NISGTC programs with students enrolled in similar programs of study prior to the grant.
The study found that students in the NISGTC-affected programs were significantly more likely to attain credentials and have higher average quarterly earnings than students in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analysis for the education outcomes. This means we are not confident that the estimated effects are attributable to the NISGTC-affected programs; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The National Information Security & Geospatial Technology Consortium (NISGTC) was awarded a TAACCCT grant in 2011 to improve its colleges’ programs to better assist trade-impacted and low-skilled workers in completing post-secondary education and obtaining employment in the IT industry. NISGTC is comprised of seven community colleges and their affiliates located across the U.S. Federal funding from the TAACCCT grant was distributed across these sites to facilitate the implementation of enhanced education and training strategies to help prepare students for high-wage IT jobs. The objectives of the grant included working with national businesses to identify competencies to help students gain employment, align curriculum to cover these knowledge areas, provide virtual IT laboratory access and equipment to students, offer curriculum through online modules, and provide tailored tutoring, career coaching, mentoring, and placement services.","The authors used a nonexperimental design to compare the outcomes of students who participated in NISGTC-affected programs of study with students enrolled in earlier cohorts of similar programs prior to NISGTC implementation. Student demographic and educational outcomes data was provided by staff at the colleges’ Institutional Research offices and employment-based data was provided by states’ unemployment insurance (UI) data systems. Of the seven NISGTC co-grantee colleges, only Moraine Valley Community College in Palos Hills, Illinois and Collin College in Frisco, Texas were able to provide student-level data for a comparison cohort to use in the impact analysis. For Moraine Valley Community College, the treatment group was comprised of students enrolled in any Local Area Networking (LAN) course during the fall 2012 semester. For Collin College, the treatment group included students enrolled in its Cisco Certified Network Associate (CCNA) certificate program during the fall 2012 semester. The comparison groups for these two co-grantee colleges represented the same type of students, at each college respectively, but were enrolled in these programs during the fall 2009 semester (prior to NISGTC full implementation). The authors used several statistical models to assess NISGTC-affected programs’ impact on the following outcomes: credential attainment, employment retention, and average quarterly wages. For the first outcome, data from both community colleges were combined and analyzed and the total analytical sample included 244 students in the comparison group (i.e., retrospective students) and 377 in the treatment group (i.e., NISGTC students). For the earnings and employment outcomes, only data from Moraine Valley Community College was available, and its final analytical sample included 72 retrospective students and 210 NISGTC students.","Education and skills gain
The study found a statistically significant relationship between NISGTC participation and credential attainment, where NISGTC students were more likely to attain a credential than those in the comparison cohort (33 percentage point difference).
Earnings and wages
The study found a statistically significant relationship between average quarterly wage and program participation in which NISGTC students were estimated to earn $1,772.09 more than students in the comparison cohort on average.
Employment
The study did not find a statistically significant relationship between program participation and employment.","Across all outcomes, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. The authors used statistical methods to ensure students in the research groups were similar in terms of baseline demographic characteristics; however, for the education outcomes, they did not include a pre-intervention measure of education/training in their education models. These preexisting differences between the groups—and not the NISGTC program—could explain the observed differences in education outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other, Low-skilled","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final external evaluation report: Trade Adjustment and Assistance Community College and Career Training Program Clovis Community College (Caffey 2016)","Final external evaluation report: Trade Adjustment and Assistance Community College and Career Training Program Clovis Community College (Caffey 2016)","Study Type: Causal Impact Analysis","Caffey, D. L. (2016). Final external evaluation report: Trade Adjustment and Assistance Community College and Career Training Program Clovis Community College. Fresno, CA: Clovis Community College.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of Clovis Community College’s (CCC) inclusive Nursing and enhanced Radiologic Technology (RADT) programs on education outcomes.
The study used a nonexperimental design to compare the outcomes of students who were in the inclusive Nursing or enhanced RADT programs to a comparison group of students.
The study found that participation in the Nursing and enhanced RADT program was significantly associated with higher program retention rates.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention and did not include sufficient controls. This means we are not confident that the estimated effects are attributable to the inclusive Nursing or enhanced RADT program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Clovis Community College (CCC) received a TAACCCT grant aimed at improving their Associate of Applied Science in Nursing and Radiologic Technology (RADT) programs. The enhanced “inclusive” Nursing program was developed as an alternative to the “traditional” nursing program in order to target students coming from high school and wanting to complete the nursing curriculum without interruption. It provided an accelerated program, allowing students to simultaneously take general education/prerequisite courses and nursing courses in order to complete the program more quickly. The RADT Digitization of Imaging program integrated new lab imaging equipment to improve the curriculum and allow students to better learn relevant laboratory skills.","The study used a nonexperimental design to compare the outcomes of students who participated in the inclusive Nursing program or enhanced RADT program to students who did not. For the Nursing program, the treatment group included 27 students who enrolled in the inclusive Nursing program in the 2013 school year and the comparison group included 36 students who enrolled in the traditional Nursing program during the same year. For the enhanced RADT program, the treatment group included 26 students who enrolled in the enhanced program in the 2013 and 2014 school years and the comparison group included 36 students who enrolled in the 2010 and 2011 school years. Outcomes included program completion, program retention, and credit hour completion. Using CCC’s institutional data system and data from department staff, the author conducted statistical tests to examine differences in outcomes between the groups.","Education and skills gain
The study found that enhanced RADT participation was significantly associated with higher retention rates, with 96% of treatment students retained in the program after one year compared to 76% of comparison students.
However, the study found no significant relationship between enhanced RADT participation and program completion or credit hour completion.
For the Nursing program, the study did not find any significant relationships between inclusive program participation and the education outcomes.","The author did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not the inclusive Nursing or RADT programs—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Final evaluation report Round 3 TAACCCT grant: Mission critical operations (NC State Industry Expansion Solutions 2017)","Final evaluation report Round 3 TAACCCT grant: Mission critical operations (NC State Industry Expansion Solutions 2017)","Study Type: Causal Impact Analysis","NC State Industry Expansion Solutions. (2017). Final evaluation report Round 3 TAACCCT grant: Mission critical operations. Raleigh, NC: NC State Industry Expansion Solutions.",,2017,http://www.skillscommons.org/handle/taaccct/13778,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the effects of the National Consortium for Mission Critical Operations (MCO) program on educational outcomes. This summary contains the findings from Nash Community College.
The authors used a nonexperimental design to compare the outcomes of students enrolled in the MCO program to a matched comparison group of students enrolled in the same courses of study prior to the implementation of the MCO program.
The study found that participation in the MCO program was significantly associated with higher rates of retention and program progress.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the MCO program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Mission Critical Operations (MCO) program was funded by the TAACCCT grant to build institutional capacity to increase credentials in a range of industries focusing on electrical/electronic, industrial, and information technology in response to the disappearance of traditional manufacturing and skilled service jobs. The MCO program had three main objectives: 1) to better prepare TAA eligible and other adults with high-skill, high wage employment or re-employment by increasing the attainment of certificates, diplomas, and other industry related credentials; 2) to improve learning outcomes and retention rates for TAA workers and other adults by redesigning curriculum with innovative and effective methods to address specific industry needs; and 3) demonstrate for TAA workers in particular improved employment outcomes. A consortium of colleges was formed to implement and monitor the MCO program. The National Consortium for Mission Critical Operations consisted of the following five colleges: Cleveland Community College, Nash Community College, Wake Technical Community College, Moultrie Technical College, and the University of North Carolina at Charlotte. Each college in the consortium implemented various intervention activities, including articulation, collaboration, course design, credential development, instruction/support, scorecard development, telepresence, and tracking/evaluation.","The nonexperimental study took place at Nash Community College (NCC) in Rocky Mount, North Carolina. The authors compared the education outcomes of students who enrolled in the MCO program to students enrolled in the same courses of study prior to the implementation of the MCO program. The treatment group consisted of 724 students who participated in the MCO program after Fall 2014. The comparison group included 953 students enrolled between Fall 2010 and Summer 2013, in the same courses that were later impacted by the MCO funds. The authors matched MCO participants to similar nonparticipants using propensity scores developed from baseline demographic information. Using data from NCC’s Student Information System, the authors conducted statistical models and analyses to examine differences in education outcomes between the groups. Education outcomes included program retention rates and program progress (measured as earning grades of A, B, or C in courses).","Education and skills gain
The study found a significant relationship between MCO program participation and retention rates, with MCO participants approximately twice as likely to be retained than students in the comparison group.
The study also found a significant association between MCO program participation and higher rates of course progression than students in the comparison group.","The authors used propensity score matching to create a comparison group; however, they did not account for preexisting differences between the groups in financial disadvantage or baseline education outcomes. These preexisting differences between the groups—and not the MCO program—could explain the observed differences in outcomes. In addition, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Final evaluation report Round 3 TAACCCT grant: Mission critical operations (NC State Industry Expansion Solutions 2017)","Final evaluation report Round 3 TAACCCT grant: Mission critical operations (NC State Industry Expansion Solutions 2017)","Study Type: Causal Impact Analysis","NC State Industry Expansion Solutions. (2017). Final evaluation report Round 3 TAACCCT grant: Mission critical operations. Raleigh, NC: NC State Industry Expansion Solutions.",,2017,http://www.skillscommons.org/handle/taaccct/13778,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the effects of the National Consortium for Mission Critical Operations (MCO) program on educational outcomes. This summary contains the findings from Wake Tech Community College.
The authors used a nonexperimental design to compare the outcomes of students enrolled in the MCO program to a matched comparison group of students enrolled in the same courses of study prior to the implementation of the MCO program.
The study found that participation in the MCO program was significantly associated with higher rates of retention and program progress.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the MCO program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Mission Critical Operations (MCO) program was funded by the TAACCCT grant to build institutional capacity to increase credentials in a range of industries focusing on electrical/electronic, industrial, and information technology in response to the disappearance of traditional manufacturing and skilled service jobs. The MCO program had three main objectives: 1) to better prepare TAA eligible and other adults with high-skill, high wage employment or re-employment by increasing the attainment of certificates, diplomas, and other industry related credentials; 2) to improve learning outcomes and retention rates for TAA workers and other adults by redesigning curriculum with innovative and effective methods to address specific industry needs; and 3) demonstrate for TAA workers in particular improved employment outcomes. A consortium of colleges was formed to implement and monitor the MCO program. The National Consortium for Mission Critical Operations consisted of the following five colleges: Cleveland Community College, Nash Community College, Wake Technical Community College, Moultrie Technical College, and the University of North Carolina at Charlotte. Each college in the consortium implemented various intervention activities, including articulation, collaboration, course design, credential development, instruction/support, scorecard development, telepresence, and tracking/evaluation.","The nonexperimental study took place at Wake Tech Community College in Raleigh, North Carolina. The authors compared the education outcomes of students who enrolled in the MCO program to students enrolled in the same courses of study prior to the implementation of the MCO program. The treatment group consisted of 404 students who participated in the MCO program after Fall 2014. The comparison group included 1,392 students enrolled between Fall 2010 and Summer 2013, in the same courses that were later impacted by the MCO funds. The authors matched MCO participants to similar nonparticipants using propensity scores developed from baseline demographic information. Using data from Wake Tech's Student Information System, the authors conducted statistical models and analyses to examine differences in education outcomes between the groups. Education outcomes included retention rates, program progress (measured as earning grades of A, B, or C in courses), and program completion.","Education and skills gain 
The study found a significant relationship between MCO program participation and retention rates, with MCO participants approximately twice as likely to be retained than students in the comparison group.
The study also found a significant association between MCO program participation and higher rates of course progression than students in the comparison group.
However, the study did not find a significant association between MCO program participation and program completion.","The authors used propensity score matching to create a comparison group; however, they did not account for preexisting differences between the groups in financial disadvantage or baseline education outcomes. These preexisting differences between the groups—and not the MCO program—could explain the observed differences in outcomes. In addition, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Third-party evaluation of DOL Round 4 TAACCCT grant Mechatronics Re-envisioned: Final evaluation report (North Carolina State University 2018)","Third-party evaluation of DOL Round 4 TAACCCT grant Mechatronics Re-envisioned: Final evaluation report (North Carolina State University 2018)","Study Type: Causal Impact Analysis","North Carolina State University. (2018). Third-party evaluation of DOL Round 4 TAACCCT grant Mechatronics Re-envisioned: Final evaluation report. Raleigh, NC: North Carolina State University Industry Expansion Solutions.",,2018,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of the Mechatronics Re-envisioned (MRE) program on education outcomes.
Using college administrative data, the authors conducted a nonexperimental study to compare the differences in outcomes between students in the MRE program and a matched comparison group.
The study found a significant relationship between MRE program participation and higher rates of program retention and completion, relative to the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the MRE program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
A TAACCCT grant funded the Mechatronics Re-envisioned (MRE) program at Central Piedmont Community College in Charlotte, North Carolina. The MRE program’s objective was to train workers for employment in mechatronics, an area of local need. Through the MRE program, the college expanded its existing mechatronics Associate’s degree program to add web-based curricula and a fast-track certificate program. The new program curriculum was designed to reflect employer input and prepare students for local industry careers. Students received support services including a success coach who helped them with academic and career planning. In addition, some students could participate in apprenticeships as part of their training.","The study used a nonexperimental design to compare the outcomes of students in the MRE program to a matched comparison group. The treatment group included 110 students who were enrolled in an MRE course from Fall 2015 to Fall 2017. The comparison group included 392 students who took the same courses from Fall 2012 to Fall 2014, before the program enhancements were implemented. Using college administrative data, the authors assessed differences in program retention and completion between the two groups using a statistical model that controlled for differences in age, gender, and race/ethnicity.","Education and Skills Gain 
The study found a significant relationship between MRE program participation and program retention, with higher proportions of students in the treatment group retained in their program than students in the comparison group.
The study also found that MRE program participation was significantly associated with higher rates of program completion relative to the comparison group.","The authors did not account for preexisting differences between the groups before program participation. Specifically, the authors did not account for differences in financial disadvantage or baseline education outcomes. These preexisting differences between the groups—and not the MRE program—could explain the observed differences in outcomes. In addition, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Evaluation of the Idaho Center of Excellence Healthcare Partnership (ICE): Final report (Negoita et al. 2018)","Evaluation of the Idaho Center of Excellence Healthcare Partnership (ICE): Final report (Negoita et al. 2018)","Study Type: Causal Impact Analysis","Negoita, M., Paprocki, A., & Gutierrez, I. (2018). Evaluation of the Idaho Center of Excellence Healthcare Partnership (ICE): Final report. Oakland, CA: Social Policy Research Associates.",,2018,http://www.skillscommons.org/bitstream/handle/taaccct/18548/ICE%20Final%20Evalu…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Idaho Center of Excellence Healthcare Partnership (ICE) on education outcomes.
The authors used a nonexperimental design to compare outcomes of students who enrolled in ICE-enhanced courses to a matched comparison group.
The study found a significant positive association between enrollment in ICE-enhanced courses and prerequisite course completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the ICE enhancements; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Idaho Center of Excellence Healthcare Partnership (ICE) was launched after receipt of a TAACCCT grant and is comprised of three community colleges in Idaho: Northern Idaho College (NIC), Lewis-Clark State College (LCSC), and Idaho State University College of Technology (ISU). The partnership was designed to enhance student services, create more accessible career pathways via creation of additional credit and noncredit programs, increase access by adding online and technology-enabled courses, and accelerate the path for students looking to earn a credit or degree. While ICE was launched in 2014, the implementation effects began occurring in early 2016. Students were eligible to receive benefits if they were enrolled in one of the three community colleges beginning in 2016 and took select ICE-enhanced courses. In particular, the grant aimed to impact students who would eventually enter the healthcare sector (including career pathways in diagnostic services, health informatics, and therapeutic services).","The nonexperimental study was conducted with students from NIC and LCSC. The authors compared the outcomes of students who enrolled in ICE’s redesigned prerequisite and non-credit courses to students from previous enrollment years who did not take the redesigned courses. Study participants included 13,345 students taking a redesigned prerequisite course and 7,330 students in the comparison group. For non-credit courses, study participants included 1,078 students taking a redesigned course and 176 comparison students. Using administrative data collected by ICE as well as college institutional records, the authors used statistical models to examine group differences in course completion and course passing rates for prerequisite and non-credit courses.","Education and Skills Gain 
The study found a significant relationship between ICE-enhanced course enrollment and course completion for prerequisite courses, with students enrolled in ICE-enhanced courses being 3.5% more likely to complete the prerequisite course than comparison students.
However, the study did not find a significant relationship between ICE-enhanced course enrollment and passing rates for prerequisite courses.
The study also did not find a significant relationship between ICE-enhanced course enrollment and course completion for non-credit courses.","The authors did not account for preexisting differences between the groups before ICE was implemented or include sufficient control variables. Though the authors controlled for age, gender, and enrollment dates, they did not establish baseline equivalence for the degree of financial disadvantage or pre-intervention measure of education or training. When analyzing impacts for noncredit programs, the authors further acknowledge that they did not statistically control for potential confounders. These preexisting differences between the groups—and not the ICE enhancements—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Evaluation of MoHealthWINS outcomes and impact (Cosgrove et al. 2015)","Evaluation of MoHealthWINS outcomes and impact (Cosgrove et al. 2015)","Study Type: Causal Impact Analysis","Cosgrove, J. J., Cosgrove, M. S., & Bragg, D. D. (2015). Evaluation of MoHealthWINS outcomes and impact. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/5125/SEPTEMBER%202015%20FINAL%20VERSION%20OF%20MHW%20OUTCOME%20IMPACT%20EVALUATION%20REPORT.pdf?sequence=1&isAllowed=y",,2015,https://www.skillscommons.org/bitstream/handle/taaccct/5125/SEPTEMBER%202015%20…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the relationship between the MoHealthWINS program and education and employment outcomes.
The study used a nonexperimental design, where the program participants were compared to a matched comparison group. Using college administrative data, state employment data, and employer data, the authors compared the program completion and employment rates between the treatment and comparison group.
The study found a significant relationship between program participation and the likelihood to complete the program and be employed post-program completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention or include sufficient control variables. This means we are not confident that the estimated effects are attributable to MoHealthWINS program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
TAACCCT funded a program called MoHealthWINS that provided training for healthcare careers. The Missouri Community College Association (MCCA) led the project and implemented it in the 12 community colleges in Missouri and in one technical college in the state. The program targeted populations that were under-skilled, unemployed, veterans, and TAA-eligible, however, these characteristics were not required for participation in the program. During the grant, the colleges revised and created new healthcare training programs, with a focus on those that were non-credit. The programs offered stacked credentials designed to be relevant to jobs with local employers, and students could receive support services. Most colleges implementing the program offered contextualized instruction, online learning, and accelerated, modular courses.","The study used a nonexperimental design that compared outcomes between the treatment group and a matched comparison group. The impact analyses included students who were in for-credit programs. Students in the treatment group were enrolled from 2011 to 2014, and a comparison group enrolled in Fall 2009. The study included 4,251 students in the treatment group, and 1,704 in the comparison group across the 13 colleges. The authors used college administrative data, state employment data, and obtained information from employers to augment missing employment data. The authors compared education and employment outcomes between the treatment and comparison groups, using a statistical model controlling for gender, age, race, and employment status when starting the program, and whether they had at least some college experience before starting the program.","Education and skills gains
The study found a significant relationship between MoHealthWINS participation and program completion, with a higher percentage of MoHealthWINS participants completing the program (66%) versus students in the comparison group (21%).
Employment
The study found a significant relationship between MoHealthWINS participation and employment, with a higher percentage of MoHealthWINS participants being employed after the program (80%) than students in the comparison group (47%).","The authors created a matched group of non-participating students to compare to MoHealthWINS participants. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the MoHealthWINS program—could explain the observed differences in outcomes. Additionally, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Unemployed, Low-skilled, Veteran or military","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Community College Review Protocol"
"Third party evaluation of MoManufacturingWINs: Implementation, outcomes, and impact (Cosgrove et al. 2016)","Third party evaluation of MoManufacturingWINs: Implementation, outcomes, and impact (Cosgrove et al. 2016)","Study Type: Causal Impact Analysis","Cosgrove, J. J., Cosgrove, M. S., & Bragg, D. D. (2016). Third party evaluation of MoManufacturingWINs: Implementation, outcomes, and impact. St. Louis, MO: Cosgrove & Associates and Bragg & Associates.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/10116/ThirdPartyFinalEva…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of Missouri’s Manufacturing Workforce Innovation Networks(MMW) program on student education and employment outcomes.
The study used a nonexperimental design to compare the outcomes of students who were in MMW to a comparison group of students in other manufacturing-technology related programs.
The study found that MMW participation was significantly associated with higher retention and employment rates.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention and they did not include sufficient controls. This means we are not confident that the estimated effects are attributable to MMW; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Eight Missouri colleges and the State's technical college formed the Missouri’s Manufacturing Workforce Innovation Networks (MMW) consortium in order to use TAACCCT funds to assist and bolster students in manufacturing related career pathways. The program targeted low-skilled, unemployed/underemployed, TAA-eligible students, veterans, and other students interested in the manufacturing industry. To better prepare students for the workforce and enhance educational and employment outcomes, MMW colleges developed a stackable certificate model that directly mapped to industry certifications. The colleges also combined coursework with basic academic skills, provided intensive academic and employment support, and developed multiple entrance and exit points into the Manufacturing Career Pathways program.","The study used a nonexperimental design to compare the outcomes of students who participated in the MMW program to students who did not participate. The treatment group included 663 credit-seeking students who first enrolled in one of the consortium colleges' Manufacturing Career Pathways program in Fall 2013 and were assessed through 2016. The comparison group included 411 credit-seeking, first-time students who enrolled in a manufacturing-technology related program in a consortium college in Fall 2011 and were assessed through 2014. Outcomes included program retention rates and employment rates upon program completion. Using data from college databases, the Missouri Division of Workforce Development, direct follow-up engagement with students, and DOL-approved follow-up methods including employee paystubs and letters from employers, the authors used statistical models to examine differences in the outcomes between treatment and comparison groups.","Education and skills gain
The study found that participation in the TAACCCT-enhanced MMW program was significantly associated with program completion as treatment group participants were 4.4 times more likely than comparison group participants to complete their program.
Employment
The study found that participation in the TAACCCT-enhanced MMW program was significantly associated with employment rates as treatment group participants were 9.7 times more likely than comparison group participants to be employed upon program completion.","The authors note that some of the observed outcomes could be attributed to the accelerated nature and timeframe of MMW programs versus the traditional manufacturing technology programs. Furthermore, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Lastly, although the authors controlled for several demographic variables in their regression analysis, they did not account for certain factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the MMW program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"East Central College (ECC) in Union, Missouri
Metropolitan Community College (MCC) in Kansas City, Missouri
Mineral Area College (MAC) in Park Hills, Missouri
North Central Missouri College (NCMC) in Trenton, Missouri
Ozarks Technical Community College (OTC) in Springfield, Missouri
St. Charles Community College (SCC) in Cottleville, Missouri St.
Louis Community College (STLCC) in St. Louis, Missouri
State Fair Community College (SFCC) in Sedalia, Missouri
State Technical College of Missouri (STCM) in the Osage County, Missouri",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Unemployed, Other, Low-skilled, Veteran or military","Basic skills, Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Community College Review Protocol"
"Third-party evaluation of implementation & impact of the Southwest Missouri Public Safety and Emergency Medical Initiative TAACCCT grant (Cosgrove & Associates 2016)","Third-party evaluation of implementation & impact of the Southwest Missouri Public Safety and Emergency Medical Initiative TAACCCT grant (Cosgrove & Associates 2016)","Study Type: Causal Impact Analysis","Cosgrove & Associates. (2016). Third-party evaluation of implementation & impact of the Southwest Missouri Public Safety and Emergency Medical Initiative TAACCCT grant. St. Louis, MO: Cosgrove & Associates.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9128/Crowder%20College_P…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of Crowder College’s enhanced Public Safety and Emergency Medical Initiative (PSP) program on education and employment outcomes.
The study used a nonexperimental design to compare the outcomes of students who were in Crowder College’s enhanced PSP program to a comparison group of students who were not enrolled in the program.
The study found that participation in the PSP program was significantly associated with higher program completion and employment rates than the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to Crowder College’s enhanced PSP program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Crowder College in Neosho, Missouri used their TAACCCT grant to expand the capacity of their Criminal Justice, Fire Science, and Emergency Medical Services courses within their Public Safety and Emergency Medical Initiative (PSP), and developed new instructional and student support strategies. They sought to expand and build programs to provide needed skills and credentials within the industry, improve achievement rates and/or time to completion, facilitate completion for low-skilled workers, implement work-based teaching strategies, and accommodate student work schedules. The program targeted TAACCCT-eligible individuals, those without previous college, academically low-skilled adults, and under or unemployed adults in the region in the in-demand areas of criminal justice, fire science, and emergency medical services.","The study used a nonexperimental design to compare the outcomes of students who participated in PSP to students who did not participate. The authors assessed 272 treatment group participants who enrolled in TAACCCT-enhanced Criminal Justice, Fire Science, and Emergency Medical Services courses from 2013-2015 and compared them to a retrospective comparison group sample of 1,232 students who, during the same time period, did not take enhanced courses. Both cohorts of students were new to the college. Outcomes included completion rates and employment rates upon program completion. Using data from Crowder College's data systems, supplemental data collection tools developed by the PSP grant team and Cosgrove & Associates, and state unemployed insurance (UI) databases, the authors used statistical models to examine differences in the outcomes between the treatment and comparison groups.","Education and skills gain
The study found that participation in the PSP program was significantly associated with higher program completion (13.92 times more likely to complete their program) than the comparison group.
Employment
The study found that participation in the PSP program was significantly associated with higher employment (17.52 times more likely to be employed) than participation in the comparison group.","The authors do not provide detail on which programs comparison group students are enrolled in; student self-selection into public safety courses versus other college courses may present a confound. In addition, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Furthermore, the authors do not report baseline equivalence or controls for race/ethnicity, student’s pre-intervention degree of financial disadvantage, and student’s pre-intervention education/training or employment. These potential preexisting differences between the groups—and not PSP program—could explain the observed differences in the outcome. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers, Unemployed, Low-skilled","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Community College Review Protocol"
"Third-party evaluation of MoSTEMWINs: Implementation, outcomes, and impact (Cosgrove & Cosgrove 2018)","Third-party evaluation of MoSTEMWINs: Implementation, outcomes, and impact (Cosgrove & Cosgrove 2018)","Study Type: Causal Impact Analysis","Cosgrove, J., & Cosgrove, M. (2018). Third-party evaluation of MoSTEMWINs: Implementation, outcomes, and impact. St. Louis, MO: Cosgrove & Associates, LLC.",,2018,http://www.skillscommons.org/bitstream/handle/taaccct/10845/MoSTEMWINs%20Mid-Po…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Missouri STEM Workforce Innovation Networks (MoSTEMWINs) program on education and employment outcomes.
The authors used a nonexperimental design to compare education and employment outcomes of MoSTEMWINs participants to a historical comparison group.
The study found that MoSTEMWINs participants were significantly more likely to complete the program and attain employment than the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the MoSTEMWINs program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2014, a consortium of 12 Missouri community colleges and the state's technical college was formed under the Missouri Community College Association. Missouri's STEM Workforce Innovation Networks program (MoSTEMWINs) was a TAACCCT funded program that provided training for STEM related careers. The MoSTEMWINs program, targeting adult learners, long-term unemployed, displaced workers, TAA eligible, and veterans, sought to address barriers to entering and completing STEM training within the target populations. The primary goals of the program were accelerated entry into career programs, clear paths to STEM careers, and improved employment attainment. MoSTEMWINs forty instructional programs of study provided basic academic skills, contextualized learning, industry-recognized credentials, stackable credentials, and student support services.","The study used a nonexperimental design to compare the outcomes of students who participated in the MoSTEMWINs program to students who did not participate. The authors compared MoSTEMWINs program participants with a historical sample of participants in non-MoSTEMWINs programs at colleges within the consortium. The sample for analysis included 1,459 first-time college students enrolled in credit bearing programs (551 MoSTEMWINs participants and 908 comparison group participants). Data sources included college databases, a statewide ETO database, Missouri Division of Workforce Development, and participant follow-up surveys. Statistical analyses were used to examine impacts on program completion and attainment of employment following program completion.","Education and skills gain
The study found that MoSTEMWINs participation was significantly associated with program completion, with program participants being six times more likely to complete their program of study than the comparison group.
Employment
The study found that MoSTEMWINs participation was significantly associated with employment following program completion, with program participants being 18 times more likely to attain employment than the comparison group.","The authors did not account for preexisting differences between the groups before participation, such as participants’ degree of financial disadvantage. These preexisting differences between the groups—and not the MoSTEMWINs program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Unemployed, Veteran or military","Basic skills, Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Education and skills gains, Employment","Community College Review Protocol"
"WIOA impacts on community college student success (Doctoral dissertation) (Crumpton 2019)","WIOA impacts on community college student success (Doctoral dissertation) (Crumpton 2019)","Study Type: Causal Impact Analysis","Crumpton, J. A. (2019). WIOA impacts on community college student success (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 13861007).",,2019,https://search.proquest.com/openview/f270d29734cd670349c3aed2866ce41a/1?pq-orig…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of a workforce development program administered by a community college under the Workforce Innovation and Opportunity Act (WIOA) on education outcomes. This summary focuses on the comparison between WIOA program participants and non-participants. 
The author used a nonexperimental design to compare WIOA program student education outcomes (measured by degree or certificate attainment) to the outcomes of students with similar backgrounds (Pell Grant recipients) who were not enrolled in WIOA. Data from the community college were used to examine education success rates over the course of three years (2014-2017).
The study found that WIOA program participants were significantly more likely to attain a degree or certificate than students who were not enrolled in the WIOA program.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention.This means we are not confident that the estimated effects are attributable to the WIOA Program; other factors are likely to have contributed","The WIOA program featured in this study was administered by an unidentified rural community college located in southeastern United States and covers three counties. The community college began providing administrative services for the workforce development program ten years ago under the WIOA’s predecessor, the Workforce Investment Act (WIA), after counties in the service area called for changes to the struggling program. In its administrative role, the community college has been expected to work closely with the workforce system in the area to prepare low-income students to meet the training and employment needs of local industry. Its activities extend beyond core academics to include adult basic education and assisting students with overcoming their barriers to employment. Support services provided to students through the program included tuition assistance, financial support for books and supplies, child care and transportation assistance, tutoring, job placement assistance, and career counseling, which includes resume building and job development services.","The author used a nonexperimental design to compare the outcomes of WIOA program participants at the community college to students enrolled in the community college who were Pell Grant recipients. Study participants included 299 students in the treatment group and 7,916 in the comparison group. Using data from the community college, the author conducted t-tests to examine differences between the groups in degree or certificate attainment over the course of three years (2014-2017).","Education and skills gain 
The study found that the community college students who were enrolled in the WIOA program were significantly more likely to attain a degree or certificate compared to students were not enrolled in the WIOA program (45% versus 20%).","The author did not account for preexisting differences between the groups before participation, such as students’ age, race/ethnicity, or degree of financial disadvantage. These preexisting differences between the groups—and not the WIOA program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"Rural, United States","Causal Impact Analysis","Adult, Low income","Basic skills, Capacity building programs, Community college education and other classroom training, Workforce Investment Act (WIA)/Workforce Innovation and Opportunity Act (WIOA) training programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Cuyahoga Community College Trade Adjustment Assistance Community College Career Training grant: Final report. (The New Growth Group, LLC 2016)","Cuyahoga Community College Trade Adjustment Assistance Community College Career Training grant: Final report. (The New Growth Group, LLC 2016)","Study Type: Causal Impact Analysis","The New Growth Group, LLC. (2016). Cuyahoga Community College Trade Adjustment Assistance Community College Career Training grant: Final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/9890/Tri-C%20Final%20Report.pdf?sequence=1&isAllowed=y",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9890/Tri-C%20Final%20Rep…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the impact of the TAACCCT-enhanced Additive Manufacturing (AM) programs or courses on education outcomes.
Using data from the College Student Information System, the authors conducted a nonexperimental study to compare the educational outcomes of students who were enrolled in AM programs to a matched comparison group.
The study found a significant association between AM program participation and increased credit hour accumulation.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the TAACCCT-enhanced Additive Manufacturing programs or courses; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2012, Cuyahoga Community College (Tri-C) in Ohio was awarded a TAACCCT grant to train workers in high-demand, technical occupations in Additive Manufacturing (AM) to meet evolving workforce needs in Northeast Ohio. Tri-C leveraged the grant to streamline AM programs and create stacked credentials to improve program completion and meet local employers' needs. This involved developing five new Certificate of Achievement programs, creating a new one-year Certificate program, and enhancing an existing Associate degree program. Tri-C redesigned its assessment and placement processes for these programs and embedded developmental math, reading, and writing skills into them. If students were unprepared for the embedded content, Tri-C established a technology-enhanced accelerated bridge program to help students develop these skills. Tri-C also incorporated new technology into this grant-enhanced programming to more effectively assess students’ career interests, assist with career coaching and academic advising, identify students that are struggling in the various programs, and ensure that students are awarded for prior learning credits, if applicable. Lastly, the program developed partnerships with local one-stop shops to help TAA-eligible students with resume development, job placement, and other employment-based services.","The study used a nonexperimental design to compare the outcomes of students who participated in the AM program to students who did not. Students in the treatment group were enrolled in the 3D Digital Design and Manufacturing Technology program. Due to low enrollment, the treatment group was expanded to include students who enrolled in AM courses. Students in the comparison group were enrolled in the Computer Numerical Control (CNC) Machining and Composites Manufacturing program, a program that was comparable to the treatment program in terms of program length, number of students enrolled, and student demographics. The authors matched AM participants to similar nonparticipants using propensity scores developed from demographic information. Study participants included 204 students in the treatment group and 171 in the comparison group. Using data from the College Student Information System, the authors conducted statistical models to assess differences between the groups in program completion and credit hour attainment.","Education and skills gain
The study found that participating in the grant-enhanced AM programs/courses was significantly related to credit hour accumulation, with students in the treatment group attaining and average of eight more credit hours than students in the comparison group.
The study found no significant relationship between participating in the grant-enhanced AM programs/courses and program completion.","Although the authors conducted propensity score matching, they did not account for baseline education outcomes as required in the protocol. These preexisting differences between the groups—and not the grant-enhanced AM program—could explain the observed differences in outcomes. Also, the authors noted that the definition of “grant participant"" for the treatment group was expanded to include any students enrolled in grant-enhanced AM courses, not just those enrolled in the grant-enhanced AM programs. This shift occurred at the end of Year 3 of the grant and complicated isolating the impact of the intervention since students in the treatment group experienced varying levels of the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Basic skills, Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"WIOA impacts on community college student success (Doctoral dissertation) (Crumpton 2019)","WIOA impacts on community college student success (Doctoral dissertation) (Crumpton 2019)","Study Type: Causal Impact Analysis","Crumpton, J. A. (2019). WIOA impacts on community college student success (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 13861007).",,2019,https://search.proquest.com/openview/f270d29734cd670349c3aed2866ce41a/1?pq-orig…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of a workforce development program administered by a community college under the Workforce Innovation and Opportunity Act (WIOA) on education outcomes. This summary focuses on the comparison between the community college WIOA program participants and the state workforce WIOA program participants.
The author used a nonexperimental design to compare WIOA program student education outcomes (measured by degree or certificate attainment) to the outcomes of participants in WIOA programs administered by entities other than community colleges (throughout the state). Data from the community college and the State Board for Workforce Development were used to examine education success rates over the course of three years (2014-2017).
The study found that WIOA program participants were significantly more likely to attain a degree or certificate than participants who entered the WIOA program through a state workforce development board.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention.This means we are not confident that the estimated effects are attributable to the WIOA program; other factors are likely to have contributed","The WIOA program featured in this study was administered by an unidentified rural community college located in southeastern United States and covers three counties. The community college began providing administrative services for the workforce development program ten years ago under the WIOA’s predecessor, the Workforce Investment Act (WIA), after counties in the service area called for changes to the struggling program. In its administrative role, the community college has been expected to work closely with the workforce system in the area to prepare low-income students to meet the training and employment needs of local industry. Its activities extend beyond core academics to include adult basic education and assisting students with overcoming their barriers to employment. Support services provided to students through the program included tuition assistance, financial support for books and supplies, child care and transportation assistance, tutoring, job placement assistance, and career counseling, which includes resume building and job development services.","The author used a nonexperimental design to compare the outcomes of WIOA program participants at the community college to participants who entered the WIOA program through the state workforce development board at the local education agency (LEA). Study participants included 299 students in the treatment group and 2,507 in the comparison group. Using data from the community college and the State Board for Workforce Development, the author conducted t-tests to examine differences between the groups in degree or certificate attainment rates over the course of three years (2014-2017).","Education and skills gain
The study found that the community college students who were enrolled in the WIOA program were significantly more likely to attain a degree or certificate compared to participants who entered the WIOA program through the workforce development board (45% versus 38%).","The author did not account for preexisting differences between the groups before participation, such as participants’ age, race/ethnicity, or degree of financial disadvantage. These preexisting differences between the groups—and not the WIOA program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"Rural, United States","Causal Impact Analysis","Adult, Low income","Basic skills, Capacity building programs, Community college education and other classroom training, Workforce Investment Act (WIA)/Workforce Innovation and Opportunity Act (WIOA) training programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"TAACCCT building Illinois' bio-economy (BIB) consortium final evaluation report (The New Growth Group 2018)","TAACCCT building Illinois' bio-economy (BIB) consortium final evaluation report (The New Growth Group 2018)","Study Type: Causal Impact Analysis","The New Growth Group. (2018). TAACCCT building Illinois' bio-economy (BIB) consortium final evaluation report. Cleveland, OH: New Growth Group.",,2018,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Building Illinois’ Bio-economy (BIB) consortium on education outcomes.
The authors used a nonexperimental design to compare the outcomes of students in BIB grant-affected programs to a matched comparison group of students in non-BIB grant-affected programs.
The study found a significant positive relationship between BIB program participation and program completion rates at two of the five colleges.
The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to participation in BIB grant-affected programs; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2014, the Building Illinois’ Bio-economy (BIB) consortium was awarded TAACCCT grant funding to train workers (including veterans and TAA-eligible workers) in bio-economy. The consortium was comprised of five open access community colleges in south-central Illinois. Targeted training areas included bioprocessing, biofuel technology, water management, restorative ecology, and agricultural watershed management. There were 48 certificate and degree programs through the consortium. Some new programs were created while others that previously existed at the colleges were modified. Programs used multiple strategies, including prior learning assessment, remedial instruction, employer engagement and partnerships, applied learning, internships, hands-on learning, competency-based learning, online and hybrid instruction with simulations and game design, and student support services.","The study used a nonexperimental design to compare educational outcomes of participants in the BIB grant-affected programs to participants in non-BIB grant-affected programs. The comparison groups at Lewis and Clark Community College, Lincoln Land Community College, and Southern Illinois University Edwardsville included students in comparable programs to the treatment programs, whereas a historical comparison group was used at Carl Sandburg College and Southeastern Illinois College. The authors matched BIB participating students to similar non-participating students using propensity scores developed from demographic information for each college. Across the five colleges, study participants included 1,231 students in the treatment group and 571 students in the comparison group. The study used data from College Student Information Systems, the Participant Intake form, and participant surveys completed after the conclusion of programs. The authors conducted statistical models individually for each college to compare differences in program completion between the treatment and comparison groups.","Education and skills gain
The study found a significant relationship between BIB program participation and increased odds of program completion at LCCC and SIUE only.","For each college, the authors used matching to ensure students in the research groups were similar in terms of baseline demographic characteristics. They found that the composition of the treatment and comparison groups varied by pre-intervention degree of financial disadvantage at some colleges but did not control for it in the analyses. These preexisting differences between the groups—and not the BIB grant-affected programs—could explain the observed differences in outcomes. Moreover, because participants in the treatment and comparison groups were enrolled in different programs or at different times, differences in outcomes could be due to time-varying or varying factors in the programs of study (such as required coursework) and not the BIB grant-affected programs. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"Carl Sandburg College (CSC) in Galesburg, Illinois
Lewis and Clark Community College (LCCC) in Godfrey, Illinois
Lincoln Land Community College (LLCC) in Springfield, Illinois
Southeastern Illinois College (SIC) in Harrisburg, Illinois
Southern Illinois University Edwardsville (SIUE) in Edwardsville, Illinois",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Veteran or military","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report. (Corporation for a Skilled Workforce & The New Growth Group 2016)","Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report. (Corporation for a Skilled Workforce & The New Growth Group 2016)","Study Type: Causal Impact Analysis","Corporation for a Skilled Workforce & The New Growth Group. (2016). Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report. Ann Arbor, MI: Corporation for a Skilled Workforce (CSW); Cleveland, OH: The New Growth Group, LLC.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9692/MSAMC%20Final%20Eva…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Multi-State Advanced Manufacturing Consortium (M-SAMC) program participation on education outcomes. This summary focuses on the findings at Bridge Valley Community and Technical College in West Virginia.
The authors used a nonexperimental design to compare outcomes of M-SAMC program participants with a matched comparison group.
The study found that the odds of program completion were significantly higher for participants in the M-SAMC programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the M-SAMC; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Multi-State Advanced Manufacturing Consortium (M-SAMC) was a consortium of 13 community colleges across 10 states designed to meet the local needs of the manufacturing industry. M-SAMC set out to develop and improve curricula, credentials, instructional design and delivery to contextualize learning, student support, student success, job placement, and partnerships with employers, community-based organizations, and unions. Competency-based education (CBE) was a major effort of the consortium including use of common industry-respected simulators and consortium-based models. All schools had access to industry standard equipment; the Integrated Manufacturing Systems Trainer (IMST), and the first level of consortium developed and delivered equipment-use training in a manufacturing industry-like instructional environment. All schools participated in at least one structured workgroup to develop the models that were later provided to all schools for adoption or adaptation based on their local context and needs.","The nonexperimental study took place at Bridge Valley Community and Technical College in Kanawha County, West Virginia. The authors compared program completion among 77 M-SAMC program participants and a matched comparison group of 765 participants. Bridge Valley Community and Technical College was unable to provide a suitable group of participants for comparison in the analysis; therefore, the authors used a comparison group from another consortium school. Gadsden State Community College in Gadsden, Alabama was selected because their program types and duration were comparable to those offered at Bridge Valley. Treatment participants at Bridge Valley were enrolled between Fall 2012 and Spring 2016. The comparison group of participants were enrolled at Gadsden between the Fall 2010 and Fall 2012. The authors matched Bridge Valley participants to similar Gadsden State nonparticipants using propensity scores developed from baseline demographic information. Data sources included participant intake forms, college databases, state earning records, and post-program completion participant surveys. The authors used a statistical model to examine the odds of program completion for treatment participants versus participants in the comparison group.","Education and skills gain
The study found that M-SAMC participation was significantly related to higher odds of program completion, where treatment participants were 5.5 times more likely to complete the program than participants in the comparison group.","The authors created a matched group of non-participants to compare to M-SAMC participants. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention education/training as required by the protocol. These preexisting differences between the groups—and not the M-SAMC program—could explain the observed differences in outcomes. Additionally, the authors used a cohort from previous enrollment years at a different college as the comparison group. Because the outcome data on the treatment and comparison groups were collected from participants at different times and at different colleges, differences in outcomes could be due to time-varying factors or college varying characteristics, and not the M-SAMC program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report (Corporation for a Skilled Workforce & The New Growth Group 2016)","Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report (Corporation for a Skilled Workforce & The New Growth Group 2016)","Study Type: Causal Impact Analysis","Corporation for a Skilled Workforce & The New Growth Group. (2016). Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report. Ann Arbor, MI: Corporation for a Skilled Workforce (CSW); Cleveland, OH: The New Growth Group, LLC",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9692/MSAMC%20Final%20Eva…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Multi-State Advanced Manufacturing Consortium (M-SAMC) program participation on education outcomes. This summary focuses on the findings at Henry Ford College in Michigan.
The authors used a nonexperimental design to compare outcomes of M-SAMC program participants with a matched comparison group.
The study found that the odds of program completion were significantly higher for participants in the M-SAMC programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the M-SAMC; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Multi-State Advanced Manufacturing Consortium (M-SAMC) was a consortium of 13 community colleges across 10 states designed to meet the local needs of the manufacturing industry. M-SAMC set out to develop and improve curricula, credentials, instructional design and delivery to contextualize learning, student support, student success, job placement, and partnerships with employers, community-based organizations, and unions. Competency-based education (CBE) was a major effort of the consortium including use of common industry-respected simulators and consortium-based models. All schools had access to industry standard equipment; the Integrated Manufacturing Systems Trainer (IMST), and the first level of consortium developed and delivered equipment-use training in a manufacturing industry-like instructional environment. All schools participated in at least one structured workgroup to develop the models that were later provided to all schools for adoption or adaptation based on their local context and needs.","The nonexperimental study took place at Henry Ford College in Dearborn, Michigan. The authors compared program completion among 1,132 M-SAMC program participants and a historical comparison group of 2,450 participants. The comparison group of participants were enrolled previously in the same programs of study three-years prior to the implementation of the grant funds. The authors matched M-SAMC program participants to similar nonparticipants using propensity scores developed from baseline demographic information. Data sources included participant intake forms, college databases, state earning records, and post-program completion participant surveys. The authors used a statistical model to examine the odds of program completion for treatment participants versus participants in the comparison group.","Education and skills gain
The study found that M-SAMC participation was significantly related to higher odds of program completion, where 49% of treatment participants completed the program compared to 4% of participants in the comparison group.","The authors created a matched group of non-participants to compare to M-SAMC participants. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention education/training as required by the protocol. These preexisting differences between the groups—and not the M-SAMC program—could explain the observed differences in outcomes. Additionally, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the treatment and comparison groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the community college) and not the M-SAMC program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report (Corporation for a Skilled Workforce & The New Growth Group 2016)","Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report (Corporation for a Skilled Workforce & The New Growth Group 2016)","Study Type: Causal Impact Analysis","Corporation for a Skilled Workforce & The New Growth Group. (2016). Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report. Ann Arbor, MI: Corporation for a Skilled Workforce (CSW); Cleveland, OH: The New Growth Group, LLC.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9692/MSAMC%20Final%20Eva…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Multi-State Advanced Manufacturing Consortium (M-SAMC) program participation on education outcomes. This summary focuses on the findings at Rock Valley College in Illinois.
The authors used a nonexperimental design to compare outcomes of M-SAMC program participants with a matched comparison group.
The study found that the odds of program completion were significantly higher for participants in the M-SAMC programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the M-SAMC; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Multi-State Advanced Manufacturing Consortium (M-SAMC) was a consortium of 13 community colleges across 10 states designed to meet the local needs of the manufacturing industry. M-SAMC set out to develop and improve curricula, credentials, instructional design and delivery to contextualize learning, student support, student success, job placement, and partnerships with employers, community-based organizations, and unions. Competency-based education (CBE) was a major effort of the consortium including use of common industry-respected simulators and consortium-based models. All schools had access to industry standard equipment; the Integrated Manufacturing Systems Trainer (IMST), and the first level of consortium developed and delivered equipment-use training in a manufacturing industry-like instructional environment. All schools participated in at least one structured workgroup to develop the models that were later provided to all schools for adoption or adaptation based on their local context and needs.","The nonexperimental study took place at Rock Valley College in Rockford, Illinois. The authors compared program completion among 114 M-SAMC program participants and a matched comparison group of 765 participants. Rock Valley College was unable to provide a group of participants for comparison in the analysis due to their student record confidentiality policy. Therefore, the authors used a comparison group from another consortium school. Gadsden State Community College in Gadsden, Alabama was selected because their program types and duration were comparable to those offered at Rock Valley. Treatment participants at Rock Valley were enrolled between Spring 2014 and Fall 2015. The comparison group of participants were enrolled at Gadsden between Fall 2010 and Fall 2012. The authors matched Rock Valley participants to similar Gadsden State nonparticipants using propensity scores developed from baseline demographic information. Data sources included participant intake forms, college databases, state earning records, and post-program completion participant surveys. The authors used a statistical model to examine the odds of program completion for treatment participants versus participants in the comparison group.","Education and skills gain
The study found that M-SAMC participation was significantly related to higher odds of program completion, where 80% of treatment participants completed the program compared to 10% of participants in the comparison group.","The authors created a matched group of non-participants to compare to M-SAMC participants. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention education/training as required by the protocol. These preexisting differences between the groups—and not the M-SAMC program—could explain the observed differences in outcomes. Additionally, the authors used a cohort from previous enrollment years at a different college as the comparison group. Because the outcome data on the treatment and comparison groups were collected from participants at different times and at different colleges, differences in outcomes could be due to time-varying factors or college varying characteristics, and not the M-SAMC program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Southwest Arkansas Community College Consortium: Trade Adjustment Assistance Community College Training grant final report. (TAACCCT enhanced training 2017)","Southwest Arkansas Community College Consortium: Trade Adjustment Assistance Community College Training grant final report. (TAACCCT enhanced training 2017)","Study Type: Causal Impact Analysis","TAACCCT enhanced training. (2017). Southwest Arkansas Community College Consortium: Trade Adjustment Assistance Community College Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/15686/SWACCC_Final_Report_9_27_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/15686/SWACCC_Final_Repor…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the impact of the South West Arkansas Community College Consortium’s (SWACCC) grant-enhanced programs on education outcomes. This summary contains the findings from Cossatot Community College of the University of Arkansas. 
The author used a nonexperimental design to compare the outcomes of students enrolled in the grant-enhanced Industrial Technology program to those enrolled in the Automotive Service Technology program at the same community college.
The study found that participation in the grant-enhanced Industrial Technology treatment group was significantly associated with a higher likelihood of program completion relative to the Automotive Service Technology comparison group.
The quality of causal evidence presented in this study is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to SWACCC’s grant-enhanced program; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Upon receipt of a TAACCCT grant, seven colleges within the South West Arkansas Community College Consortium (SWACCC) implemented several strategies to upskill advanced manufacturing workers while engaging new employer relationships across the community colleges. Strategies included enhancing stacked and latticed credentials and certifications at each college, creating on-site work-based learning opportunities for students, and enhancing career counseling and advising opportunities while partnering with employers.","The study took place at Cossatot Community College of the University of Arkansas (UA Cossatot) in De Queen, Arkansas. The author used a nonexperimental design to compare the outcomes of students enrolled in the grant-enhanced program to those enrolled in a comparable program at UA Cossatot. The grant-enhanced program was matched to a comparable program at UA Cossatot that was similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and its students having a similar demographic composition. The treatment group included 163 students enrolled in the Industrial Technology program between 2014 and 2017. The comparison group included 41 students enrolled in the Automotive Service Technology program during the same time period. Data sources included institutional data from UA Cossatot, data from the Arkansas Research Center, and intake and survey forms given to students in cases of missing data. The author used a statistical model with controls for demographic and employment information to examine differences in the odds of program completion between the treatment and comparison groups.","Education and skills gain 
The study found that SWACCC grant-enhanced program participation was significantly related to higher odds of program completion, where treatment participants were 5.4 times more likely to complete the program than participants in the comparison group.","Although the author accounted for baseline demographic characteristics, the author did not account for a pre-intervention measure of education which is required by the review protocol. The preexisting differences between the groups on this variable—and not the SWACCC grant-enhanced program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Southwest Arkansas Community College Consortium: Trade Adjustment  Assistance Community College Training grant final report (TAACCCT enhanced training 2017)","Southwest Arkansas Community College Consortium: Trade Adjustment  Assistance Community College Training grant final report (TAACCCT enhanced training 2017)","Study Type: Causal Impact Analysis","TAACCCT enhanced training. (2017). Southwest Arkansas Community College Consortium: Trade Adjustment Assistance Community College Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/15686/SWACCC_Final_Report_9_27_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/15686/SWACCC_Final_Repor…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the impact of the South West Arkansas Community College Consortium’s (SWACCC) grant-enhanced programs on education outcomes. This summary contains the findings from National Park College.
The author used a nonexperimental design to compare the outcomes of students enrolled in the grant-enhanced Aerospace Fabrication and Assembly, Industrial Technology, and Welding Technology programs to those enrolled in the Marine Repair Technology, HVAC, and Automotive Service Technology programs at the same community college.
The study found that participation in the grant-enhanced program treatment group was significantly associated with a higher likelihood of program completion relative to the comparison group.
The quality of causal evidence presented in this study is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to SWACCC’s grant-enhanced program; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Upon receipt of a TAACCCT grant, seven colleges within the South West Arkansas Community College Consortium (SWACCC) implemented several strategies to upskill advanced manufacturing workers while engaging new employer relationships across the community colleges. Strategies included enhancing stacked and latticed credentials and certifications at each college, creating on-site work-based learning opportunities for students, and enhancing career counseling and advising opportunities while partnering with employers.","The study took place at National Park College (NPC) in Hot Springs, Arkansas. The author used a nonexperimental design to compare the outcomes of students enrolled in the grant-enhanced programs to those enrolled in comparable programs at NPC. Each grant-affected program was matched to a comparable program at NPC that was similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and its students having a similar demographic composition (Aerospace Fabrication and Assembly versus Marine Repair Technology, Industrial Technology versus HVAC, and Welding Technology versus Automotive Service Technology). The treatment group included 122 students pursuing certificates in Aerospace Fabrication and Assembly, Industrial Technology, or Welding Technology between 2014 and 2017. The comparison group included 63 students pursing certificates in Marine Repair Technology, HVAC, or Automotive Service Technology during the same time period. Data sources included institutional data from NPC, data from the Arkansas Research Center, and intake and survey forms given to students in cases of missing data. The author used a statistical model with controls for demographic and employment information to examine differences in the odds of program completion between the treatment and comparison groups.","Education and skills gain 
The study found that SWACCC grant-enhanced program participation was significantly related to higher odds of program completion, where treatment participants were 2.4 times more likely to complete the program than participants in the comparison group.","Although the author accounted for baseline demographic characteristics, the author did not account for a pre-intervention measure of education which is required by the review protocol. The preexisting differences between the groups on this variable—and not the SWACCC grant-enhanced program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Evaluation of the Pennsylvania’s Advanced Training and Hiring Program: Final report (Davis & Bill 2018)","Evaluation of the Pennsylvania’s Advanced Training and Hiring Program: Final report (Davis & Bill 2018)","Study Type: Causal Impact Analysis","Davis, S., & Bill, N. (2018). Evaluation of the Pennsylvania’s Advanced Training and Hiring Program: Final report. Washington, DC: Impaq International.",,2018,https://www.impaqint.com/work/case-studies/evaluation-pennsylvania%E2%80%99s-ad…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of the Pennsylvania’s Advanced Training and Hiring (PATH) initiative on education outcomes.
Using college administrative data, the authors conducted a nonexperimental study to compare education outcomes of students in the PATH program to a matched comparison group.
The study found that PATH participation was significantly associated with higher rates of program completion and academic progress.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention.This means we are not confident that the estimated effects are attributable to the PATH initiative; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Pennsylvania’s Advanced Training and Hiring (PATH) initiative was funded by TAACCCT and operated from 2014-2018 at three community colleges in Pennsylvania: Northampton Community College (NCC), Lehigh Carbon Community College (LCCC), and Luzerne County Community College (Luzerne). The program was created to train local workers for employment within in-demand industries of manufacturing, healthcare, and logistics/transportation. To do this, the program created and revised curricula to align with local employer workforce needs, while also providing more hands-on learning opportunities and updated equipment and classroom space for instruction. The program expanded opportunities for students to earn credentials, by allowing students to stack credentials and by incorporating industry certifications into the programs. While participating in the program, students could receive various support services, such as career planning, job placement, job preparation, and referrals for other services within the college.","The nonexperimental study was conducted with students from NCC and Luzerne. The authors compared the education outcomes of students who enrolled in the PATH program to students in similar academic programs at the same colleges. Study participants included 753 students in the treatment group and 1,659 students in the comparison group. The study used college administrative data to measure education outcomes in both groups. After creating a matched comparison group based on students’ age, gender, and race, the authors compared the proportion that achieved educational outcomes between the treatment and comparison groups.","Education and skills gain 
The study found a significant relationship between PATH participation and program completion, with a larger proportion of PATH students completing their program (20 percentage points more) than students in the comparison group.
The study also found a significant relationship between PATH participation and academic progress, with a larger proportion of PATH students making normal academic progress (20 percentage points more) than students in the comparison group.","The authors did not account for preexisting differences between the groups before program participation. Specifically, they did not account for differences in financial disadvantage or baseline education outcomes when creating the matched comparison group. These preexisting differences between the groups—and not the PATH initiative—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Southwest Arkansas Community College Consortium: Trade Adjustment Assistance Community College Training grant final report (TAACCCT enhanced training 2017)","Southwest Arkansas Community College Consortium: Trade Adjustment Assistance Community College Training grant final report (TAACCCT enhanced training 2017)","Study Type: Causal Impact Analysis","TAACCCT enhanced training. (2017). Southwest Arkansas Community College Consortium: Trade Adjustment Assistance Community College Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/15686/SWACCC_Final_Report_9_27_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/15686/SWACCC_Final_Repor…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the impact of the South West Arkansas Community College Consortium’s (SWACCC) grant-enhanced programs on education outcomes. This summary contains the findings from South Arkansas Community College.
The author used a nonexperimental design to compare the outcomes of students enrolled in the grant-enhanced Industrial Technology/Mechatronics or Process Technology programs to those enrolled in the Automotive Service Technology program at the same community college.
The study found that participation in the grant-enhanced Industrial Technology/Mechatronics or Process Technology treatment group was significantly associated with a higher likelihood of program completion relative to the Automotive Service Technology comparison group.
The quality of causal evidence presented in this study is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to SWACCC’s grant-enhanced program; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Upon receipt of a TAACCCT grant, seven colleges within the South West Arkansas Community College Consortium (SWACCC) implemented several strategies to upskill advanced manufacturing workers while engaging new employer relationships across the community colleges. Strategies included enhancing stacked and latticed credentials and certifications at each college, creating on-site work-based learning opportunities for students, and enhancing career counseling and advising opportunities while partnering with employers.","The study took place at South Arkansas Community College (SouthArk) in El Dorado, Arkansas. The author used a nonexperimental design to compare the outcomes of students enrolled in the grant-enhanced program to those enrolled in a comparable program at SouthArk. The grant-enhanced program was matched to a comparable program at SouthArk that was similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and its students having a similar demographic composition. The treatment group included 236 students enrolled in the Industrial Technology/Mechatronics or Process Technology programs between 2014 and 2017. The comparison group included 92 students enrolled in the Automotive Service Technology program during the same time period. Data sources included institutional data from SouthArk, data from the Arkansas Research Center, and intake and survey forms given to students in cases of missing data. The author used a statistical model with controls for demographic and employment information to examine differences in the odds of program completion between the treatment and comparison groups.","Education and skills gain
The study found that SWACCC grant-enhanced program participation was significantly related to higher odds of program completion, where treatment participants were 11.6 times more likely to complete the program than participants in the comparison group.","Although the author accounted for baseline demographic characteristics, the author did not account for a pre-intervention measure of education which is required by the review protocol. The preexisting differences between the groups on this variable—and not the SWACCC grant-enhanced program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"2016 I-AM annual evaluation report: Iowa Advanced Manufacturing Statewide Consortium (de la Mora et al. 2016)","2016 I-AM annual evaluation report: Iowa Advanced Manufacturing Statewide Consortium (de la Mora et al. 2016)","Study Type: Causal Impact Analysis","de la Mora, A., Kemis, M., Callen, E., & Starobin, S. (2016). 2016 I-AM annual evaluation report: Iowa Advanced Manufacturing Statewide Consortium. Ames, IA: Research Institute for Studies in Education (RISE), Iowa State University.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Iowa Advanced Manufacturing Consortium (I-AM) on welding student’s education, earnings, and employment outcomes.
The study used a nonexperimental design to compare outcomes of students in the I-AM program to a matched comparison group of students.
The study found that participating in the I-AM program was significantly associated with higher rates of credential attainment, larger numbers of certificates and welding awards earned, higher employment rates, and higher earnings.
The quality of causal evidence presented in this study is low because the authors used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to I-AM; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Iowa Advanced Manufacturing Consortium (I-AM) was comprised of all 15 community colleges in the state. In 2012, I-AM received a TAACCCT grant that allowed the colleges to expand their marketing efforts to attract students, enroll more students, provide enrolled students with advising and career counseling, and improve their programs (e.g., by offering courses aligned to industry needs, new industry-recognized certifications, online and hybrid courses, and updating their facilities). I-AM focused on seven programs: Welding Technician/Technology, Computer Numerical Control (CNC)/Tool and Die Machining, Industrial Maintenance, Industrial Automation, Manufacturing Technology, Robotics, and Transportation and Logistics. The study authors did not indicate if there were eligibility criteria to enter the I-AM program.","The nonexperimental study was conducted at 13 of the 15 community colleges in Iowa and compared students who participated in the I-AM program (specifically the for-credit welding program) to students who did not. The authors matched participants enrolled in the I-AM program to similar nonparticipants, using propensity scores developed from demographic, education, employment, and wage information. The treatment group consisted of 364 students who enrolled in an I-AM for-credit welding program between fall 2013 and spring 2014. The comparison group consisted of 1,207 students who enrolled in a for-credit welding program between fall 2010 and spring 2012, with 364 students included after matching. Data sources included the I-AM student database, the Iowa Department of Education, and the Iowa Workforce Development Agency. The authors used t-tests to compare the means of students in the treatment and comparison groups for the educational, employment, and earnings outcomes reported in the study.","Education and Skills Gain
The study found a significant relationship between I-AM program participation and credential attainment, with higher proportions of I-AM students than comparison students earning at least one certificate (9% vs. 4%) or a diploma (19% vs. 1%).
The study also found that I-AM program participation was significantly related to the number of certificates earned and the number of welding awards earned, with treatment students earning more certificates and welding awards than students in the comparison group.
Earnings and Wages
The study found that I-AM program participation was significantly associated with higher average wages one quarter after completing one year in the program, with students in the I-AM program earning an average of $3,968 compared to an average of $3,285 for comparison group students.
Employment
The study found that I-AM program participation was significantly related to employment, with students in the I-AM program being more likely to have a job one quarter after completing one year in the program than students in the comparison group.","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Iowa Western 2017 ITC2 annual  evaluation report: Information technology - Credentials to Careers (ITC2), Iowa Western Community College (de la Mora et al. 2017)","Iowa Western 2017 ITC2 annual  evaluation report: Information technology - Credentials to Careers (ITC2), Iowa Western Community College (de la Mora et al. 2017)","Study Type: Causal Impact Analysis","de la Mora, A., Callen, E., Kemis, M., & Abraham, W. (2017). Iowa Western 2017 ITC2 annual evaluation report: Information technology - Credentials to Careers (ITC2), Iowa Western Community College. Arnes, IA: Research Institute for Studies in Education.",,2017,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Information Technology – Credentials to Careers (ITC2) program on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare the outcomes of students who were enrolled in Computer Information Technology (CIT) programs after the initiation of ITC2 programming to those who were enrolled in CIT programs prior to the start of the ITC2 program.
The study found that ITC2 program participation was associated with a significant increase in certificate attainment.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention and did not include sufficient controls. This means we are not confident that the estimated effects are attributable to the ITC2 program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Information Technology – Credentials to Careers (ITC2)program began in 2014 and was implemented in Computer Information Technology (CIT) programs at Iowa Western Community College (IWCC). This included programs in computer science, application and web programming, cyber security, management information systems, network and system administration, programming, and graphic communications. The ITC2 program included curriculum development, increased course offerings, faculty/staff training, facilities improvement, and links to industry and 4-year institutions.","The study used a nonexperimental design to compare the outcomes of students who participated in the ITC2 program (starting in fall 2014) to students who were enrolled in CIT programs prior to the start of the ITC2 funding (fall 2011- spring 2013). Study participants included 52 students in the treatment group and 49 in the comparison group. Using data from Iowa Western Community College and Iowa Workforce Development, the authors conducted statistical tests to examine differences in outcomes between the groups. Outcomes included certificates awarded, number of Associate degrees earned, number of jobs, wages earned, and difference in wages earned over two time periods. The authors analyzed data from participants one quarter prior to enrollment at IWCC (time 1), and data from the second full quarter after two years of enrollment at IWCC (time 2; this is the time it takes to receive an Associate degree plus one quarter).","Education and skills gain
The study found that participation in the ITC2 program was significantly associated with certificate attainment, with ITC2 students earning 10% more certificates than students in the comparison group.
However, the study did not find a significant relationship between ITC2 program participation and the number of associate degrees earned.
Earnings and wages
The study did not find a significant relationship between ITC2 program participation and wages earned at time 2.
The study also did not find a significant relationship between ITC2 program participation and differences in wages from time 1 to time 2.
Employment
The study did not find a significant relationship between ITC2 program participation and the average number of jobs held at time 2.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not the ITC2 program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Summative evaluation report: Training for Regional Energy in North Dakota (TREND) (Dervarics et al. 2018)","Summative evaluation report: Training for Regional Energy in North Dakota (TREND) (Dervarics et al. 2018)","Study Type: Causal Impact Analysis","Dervarics, C., Evans, W. D., Bucci, P. T., Fox, D. A., Bishop, K., & Balraj, N. (2018). Summative evaluation report: Training for Regional Energy in North Dakota (TREND). Washington, DC: PTB and Associates",,2018,https://www.skillscommons.org/bitstream/handle/taaccct/18122/FINAL%20TREND%20Ro…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Training for Regional Energy in North Dakota (TREND) program on education outcomes.
The authors used a nonexperimental design to compare students enrolled in the TREND program to a matched comparison group of students at a community college in North Dakota.
Using secondary data, the authors conducted statistical models to examine differences between the groups.
The study found that TREND program participation was significantly related to higher credential attainment and higher rates of retention.
The quality of causal evidence presented in this study is low because the authors used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the TREND program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories. From 2014-2018, Bismarck State College and three postsecondary institutions received grant funding to provide additional resources on training for in-demand jobs in the state’s energy sector, primarily in oil and gas. The Training for Regional Energy in North Dakota (TREND) program aimed at preparing more adults and recent high school graduates for employment with the goal of increasing student completion and success by combining job-specific training with strong advising. The grant funds were used to support learning by hiring faculty and purchasing new equipment, software, and licensing. Students had access to advising support in their classes as well as assistance with job interviews and employment. An emphasis was placed on career pathways and stackable credentials and students were provided with a flexible technology-enabled learning environment, both in-person and online.","The nonexperimental study was conducted at Bismarck State College in Bismarck, North Dakota. The authors matched TREND program participants to similar nonparticipants using propensity scores developed from socio-demographic information. The total number of study participants included 2,160 students. The 1,094 TREND participants were newly enrolled beginning in Fall 2014 through 2018. The comparison group was a historical cohort composed of 1,066 students who were enrolled in similar programs from 2009 through 2012 (prior to the implementation of the TREND program). Using secondary data from the college, the authors conducted statistical models to examine differences in credential attainment and retention.","Education and skills gains
The study found a significant relationship between TREND participation and credential attainment, with TREND participants having an average of 14% more completed credentials than the comparison group.
The study also found that participation in the TREND intervention was significantly associated with an 11.7% increase in retention.","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Northwest State Community College of Ohio Industrial Automation Manufacturing innovative Strategic Training Achieving Results (IAM iSTAR) Initiative (TAACCCT Grant TC-26481-14-60-A-39) (Dockery et al. 2018)","Northwest State Community College of Ohio Industrial Automation Manufacturing innovative Strategic Training Achieving Results (IAM iSTAR) Initiative (TAACCCT Grant TC-26481-14-60-A-39) (Dockery et al. 2018)","Study Type: Causal Impact Analysis","Dockery, J., Bottomley, M., Murray, C., Tichnell, T., Stover, S., Schroeder, N., Hance, D., Fruchey, K., & Franco, S. (2018). Northwest State Community College of Ohio Industrial Automation Manufacturing innovative Strategic Training Achieving Results (IAM iSTAR) Initiative (TAACCCT Grant TC-26481-14-60-A-39). Dayton, OH: Wright State University, Applied Policy Research Institute.",,2018,https://www.skillscommons.org/bitstream/handle/taaccct/18570/2018%20IAM%20iSTAR…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Industrial Automation Manufacturing innovative Strategic Training Achieving Results (IAM iSTAR) initiative on education outcomes.
The authors used a nonexperimental design to compare education outcomes of students enrolled in the IAM iSTAR program to a historic cohort of students enrolled in the same programs of study prior to the implementation of the IAM iSTAR program.
The study found that IAM iSTAR program participation was significantly associated with higher numbers of credit hours attempted and completed per semester.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the IAM iSTAR initiative; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Northwest State Community College (NSCC) in Ohio used TAACCCT funding to implement the Industrial Automation Manufacturing innovative Strategic Training Achieving Results (IAM iSTAR) program. The grant funds were awarded to the college to assist in the conversion of traditional industrial technology courses to an intensive modularized hybrid format. The industrial technologies division at NSCC trains industrial automation maintenance technicians in the advanced manufacturing industry. The IAM iSTAR program included several components and strategies which included: developing new and modified competency-based curriculum, transforming the academic process to accelerate student learning and completion (e.g., virtual trainers), creating innovative instructional media and technology to accelerate learning (e.g., online classes and open labs), and redesigning student support services to increase student success (e.g., career coaching).","The study was a nonexperimental design to compare the education outcomes of students enrolled in the IAM iSTAR program to a historic cohort of students attending NSCC. The treatment group consisted of 589 NSCC students who were enrolled in the IAM iSTAR program. The comparison group consisted of 233 students enrolled in the same programs of study between fall 2014 and spring 2016. Data sources included the NSCC Banner System and self-reported data for the treatment group and the Higher Education Information (HEI) records for the comparison group. The authors conducted repeated measures ANOVAs to examine differences in education outcomes (credit hours attempted and credit hours completed) between the treatment and comparison groups.","Education and skills gain
The study found a significant relationship between the IAM iSTAR program and credit hours attempted, with treatment students attempting more credit hours each semester than students in the comparison group.
The study also found a significant relationship between the IAM iSTAR program and credit hours completed, with treatment students completing more credit hours than students in the comparison group.","The authors used a historical cohort of NSCC students as the comparison group. However, these students were not matched to treatment students, and control variables were not included in the analyses to account for preexisting differences between the groups. These preexisting differences between the groups—and not the IAM iSTAR program—could explain the observed differences in outcomes. Additionally, because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the community college) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"The impact of a cohort-based learning model on student success within vocational technical  certificates at a community college (Doctoral dissertation, Northeastern University) (Oldham 2017)","The impact of a cohort-based learning model on student success within vocational technical  certificates at a community college (Doctoral dissertation, Northeastern University) (Oldham 2017)","Study Type: Causal Impact Analysis","Oldham, T. M. (2017). The impact of a cohort-based learning model on student success within vocational technical certificates at a community college (Doctoral dissertation, Northeastern University). Retrieved from https://repository.library.northeastern.edu/files/neu:cj82pw09w/fulltext.pdf",,2017,https://repository.library.northeastern.edu/files/neu:cj82pw09w/fulltext.pdf,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of a cohort-based learning model within a certificate program at a community college on the students’ education completion rate.
Using community college data, the study used a nonexperimental design to compare students in the cohort-based learning model with those in the traditional program.
The author found a statistically significant association between the cohort-learning model and education completion rates where students in the cohort model were more likely to complete the program than students in the traditional model.
The quality of causal evidence presented in this report is low because the authors did not include sufficient control variables. This means we are not confident that the estimated effects are attributable to the cohort-based learning model; other factors are likely to have contributed.","The cohort-learning model included the following cohort design elements of: block scheduling, students in each cohort taking all classes together, limited faculty teaching the cohort, peer and faculty supports in place, and active job placement services upon program completion. The setting for the study was a public community college in upstate New York where students were enrolled in a Precision Machining vocational technical certificate program using a 22-week instructional model with a cohort model or in a traditional ""cafeteria style"" non-cohort model.","The population for this study included students who were enrolled in a vocational technical certificate program at a community college. The author assessed the association between the program type (traditional or cohort) and the completion rate (pass versus no pass) using a nonexperimental design. The analysis sample included 69 in the treatment group and 44 in the traditional (comparison) group.","Education and skills gain
In comparing the traditional versus cohort model for the precision machining certificate program, the study found that students in the cohort model were 3.6 times more likely to complete the program than students in the traditional non-cohort model, and the result was statistically significant.","The author did not include sufficient control variables such as a pre-intervention measure of education and training or a degree of financial disadvantage. These preexisting differences between the groups—and not the cohort-based learning model—could explain the observed differences in outcomes. The author also notes that this is a retrospective study.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Final evaluation report Trade adjustment Assistance Community  College and Career Training (TAACCCT) Grant: Round 2 CyberSecurity Career Pathways Program (Pacific Research and Evaluation 2016)","Final evaluation report Trade adjustment Assistance Community  College and Career Training (TAACCCT) Grant: Round 2 CyberSecurity Career Pathways Program (Pacific Research and Evaluation 2016)","Study Type: Causal Impact Analysis","Pacific Research and Evaluation. (2016). Final evaluation report Trade Adjustment Assistance Community College and Career Training (TAACCCT) Grant: Round 2 CyberSecurity Career Pathways Program. Portland, OR: Pacific Research & Evaluation, LLC.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the CyberSecurity Career Pathways Program (CSCPP) on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare the outcomes of students who were in the CSCPP to a historical comparison group.
When compared to a historical cohort, the study found a significant relationship between CSCPP participation and high rates of program completion and retention, and wage increases.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before program participation or include sufficient control variables. This means we are not confident that the estimated effects are attributable to CSCPP; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The CyberSecurity Career Pathways Program (CSCPP) was a TAACCCT-funded program at Utah Valley University in Orem, Utah, created to fill local industry and training gaps in the area. Efforts of CSCPP focused on developing and modifying courses and programs in information systems and technology from entry-level to post-baccalaureate certificates, and Associate (AS, AAS) and Bachelor (BS) degree programs. This included AS in Information Technology, AAS in Information Technology, BS in Information Technology, and BS in Information Systems. The CSCPP also enhanced student support services (e.g., advising, mentoring, tutoring, funding for industry certification exams, career guidance), and strengthened partnerships with local industry, including formation of an advisory board.","The nonexperimental study compared the outcomes of students who were in CSCPP to a historical comparison group. The treatment group included 288 students enrolled in the four CSCPP credential programs during the first year of the grant (2012-13 academic year). The comparison group included 291 students enrolled in one of the same four credentials in the 2008-09 academic year, prior to grant implementation. The authors examined program completion, program retention, credentials earned, wage increases, and employment attained and retained. Using institutional research data from Utah Valley University and data from the Utah Department of Workforce Services, the authors conducted chi-square analyses to examine differences between the groups, but did not control for differences in student characteristics.","Education and skills gain
The study found a significant relationship between CSCPP participation and program completion, with higher program completion rates in the treatment group (42%) than the comparison group (30%).
The study also found that CSCPP participation was significantly associated with retention rates, with higher retention rates in the treatment group (28%) than the comparison group (16%).
The study found no statistically significant relationship between participation in CSCPP and credentials earned.
Earnings and wages
The study found a significant relationship between CSCPP participation and wage increases for incumbent workers, with higher proportions of the treatment group receiving wage increases (79%) than the comparison group (70%).
Employment
The study found no significant relationships between participation in CSCPP and entering or retaining employment.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not CSCPP—could explain the observed differences in outcomes. The authors also used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final evaluation report Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant: Round 3 Kapiʻolani Community College (Pacific Research and Evaluation 2017)","Final evaluation report Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant: Round 3 Kapiʻolani Community College (Pacific Research and Evaluation 2017)","Study Type: Causal Impact Analysis","Pacific Research & Evaluation, LLC. (2017). Final evaluation report Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant: Round 3 Kapiʻolani Community College. Portland, OR: Pacific Research & Evaluation, LLC.",,2017,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of the grant-funded Implementing an Island-wide Strategy for Industry Cluster Formation and Community Revitalization in Hawai'i (C3T) project on education outcomes.
The authors used a nonexperimental design to compare the outcomes of students enrolled in C3T programs and courses with students enrolled in comparable programs offered at the same community college.
When compared to nonparticipating students, the study found that C3T participation was significantly associated with higher program completion and retention rates for students enrolled in the hospitality program.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the C3T project; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Kapi’olani Community College (KCC) in Hawai’i leveraged a TAACCCT grant to fund the Implementing an Island-wide Strategy for Industry Cluster Formation and Community Revitalization in Hawai'i (C3T) Project. This project created or modified six courses and programs offered at KCC by providing workforce development and training for the local culinary, hospitality, and health industries. These C3T courses and programs that were developed or enhanced offered additional certifications, pathways for further degree completion, or training that addressed local industry demand. Instructional and curricular modifications were made with the help of local industry partners and the project provided academic and career guidance to program participants in the form of job fairs, field trips, informal advising, and work-based learning opportunities. Moreover, these programs and courses implemented new delivery methods by leveraging technology that fostered distance learning and used new equipment to enhance the students’ learning experience.","The study used a nonexperimental design to compare the outcomes of students who participated in C3T to those who did not. Students enrolled in the following C3T-enhanced programs or courses comprised the treatment cohorts: Hospitality Operations Management (HOST APC) program (one cohort - Fall 2016), HOST 101 course with gamification (two cohorts - Fall 2016 and Spring 2016), or School Health Aid (SHA) Level 1 training program (three cohorts - Summer 2014, Fall 2014, and Spring 2015). Each program or course in the treatment cohort were compared to a respective comparison cohort based on program characteristics (including learning objectives, credential attainment, and certification outcomes). Students who participated in comparable programs or courses that ran in tandem with those in the treatment cohorts were part of the comparison cohorts. Study participants included 294 students in the treatment group and 234 in the comparison group. The authors conducted statistical analyses to examine differences between the treatment and comparison groups in program completion, earning of credentials, and retention rate.","Education and skills gain
The study found that there was a significant relationship between C3T participation and program completion for the HOST 101 gamified treatment cohorts. Specifically, program completion for these treatment cohorts was 18 and 21 percentage points higher than their respective comparison cohorts.
The study found no significant relationship between C3T participation and program completion for all three of the SHA treatment cohorts or the HOST APC treatment cohort.
The study found that there was a significant relationship between C3T participation and retention rates for the Fall 2016 HOST 101 gamified treatment cohort, with the retention rate being 30 percentage points higher for treatment students than comparison students. However, the study found no significant relationship between C3T participation and retention rates for the Spring 2016 HOST 101 gamified cohort.
The study found no significant relationship between C3T participation and earning credentials.","The authors did not account for other factors that could have affected the difference between the treatment and comparison cohorts prior to the intervention. No demographic information for the comparison cohorts is provided in the report. Thus, preexisting differences between the respective group pairings—and not the C3T-enhanced programs or courses—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Saddleback College TAACCCT grant final evaluation report (Pacific Research and Evaluation 2017)","Saddleback College TAACCCT grant final evaluation report (Pacific Research and Evaluation 2017)","Study Type: Causal Impact Analysis","Pacific Research and Evaluation, LLC. (2017). Saddleback College TAACCCT grant final evaluation report. Portland, OR: Pacific Research and Evaluation, LLC.",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/13850/Saddleback%20Colle…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of Saddleback College’s Fast Track to Success program on student education outcomes.
The study used a nonexperimental design to compare the outcomes of students who were in the Medical Assistant (MA) and Registered Nurse (RN) tracks of the Fast Track to Success program to a comparison group.
The study found that MA and RN students in Fast Track to Success programming were significantly more likely to earn certificates, degrees, and credentials in the MA and RN areas.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention or include sufficient control variables. This means we are not confident that the estimated effects are attributable to the Fast Track to Success program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Saddleback College’s Health Sciences and Human Services (HSHS) division received a TAACCCT grant in 2013 that targeted dislocated TAA-affected workers and other students. Their program, Fast Track to Success, improved the University’s current health career curriculum, created new healthcare programs, integrated technology-enhanced learning opportunities, and worked with local workforce development systems and employers to facilitate student employment.","The authors used a nonexperimental design to determine the impact of Saddleback College’s Fast Track to Success Program at a singular campus location in Mission Viejo, California. The treatment group consisted of 113 students in the Medical Assistant (MA) program and 225 students in the Registered Nurse (RN) program in the 2014 cohort. The comparison group included 165 students in the MA program and 227 students in the RN program who enrolled in 2011, before the Fast Track to Success program was implemented. The authors conducted statistical analyses to examine differences in outcomes between the groups using South Orange County Community College District databases. Outcomes included MA students who earned a certificate in less than one year; RN students who earned a certificate in more than one year; MA students who earned a degree; RN students who earned a degree; and number of combined credentials earned by MA and RN students.","Education and skills gain
The study found that Fast Track to Success participants were significantly more likely than comparison group students to earn an MA certificate in less than one year (71% vs. 44%), earn an RN certificate in more than one year (84% vs. 22%), earn an MA degree (11% vs. 1%), earn an RN degree (84% vs. 24%), and earn 1, 2, or 3 credentials as an MA or RN (82% vs. 34%).","The authors used MA and RN cohorts from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors and not the intervention. Also, the authors did not account for factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage and pre-intervention degree of education. These preexisting differences between the groups—and not the Fast Track to Success program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Greater Memphis Alliance for a Competitive Workforce: TAACCCT round 4 grant, impact evaluation final report (Panaik 2018)","Greater Memphis Alliance for a Competitive Workforce: TAACCCT round 4 grant, impact evaluation final report (Panaik 2018)","Study Type: Causal Impact Analysis","Patnaik, A. (2018). Greater Memphis Alliance for a Competitive Workforce: TAACCCT round 4 grant, impact evaluation final report. Austin, TX: Ray Marshall Center for the Study of Human Resources, The University of Texas at Austin, Lyndon B. Johnson School of Public Affairs.",,2018,https://raymarshallcenter.org/files/2018/10/FINAL-GMACW-TAACCCT-Impact-Evaluati…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Greater Memphis Alliance for a Competitive Workforce (GMACW) Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program on credential attainment.
Using a nonexperimental design, the study author compared students in the program to a matched historical comparison group.
The study found that students in the GMACW TAACCCT program were significantly more likely to achieve a credential than students in the comparison group.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the GMACW TAACCCT; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The TAACCCT program funded the Greater Memphis Alliance for a Competitive Workforce (GMACW) to create or improve programs in advanced manufacturing as well as transportation, distribution, and logistics (TDL) industries in Memphis and its surrounding areas. The grant period of performance was October 1, 2014 through March 31, 2018. The program included partnerships with employers to effectively train students. The Alliance included a consortium of four colleges: Arkansas State University Mid-South (ASU Mid-South) as the lead college, Southwest Tennessee Community College (Southwest), William R. Moore College of Technology (Moore Tech), and Tennessee College of Applied Technology (TCAT-Memphis).","The nonexperimental study was conducted at ASU Mid-South, Moore Tech, and TCAT Memphis. The author compared students in the GMACW TAACCCT program to a matched historical comparison group. The treatment group included 1,409 students who enrolled in manufacturing/TDL programs between Fall 2015 and Fall 2017. The comparison group included 508 students who enrolled in the same programs in Fall 2014 (one year before program implementation). The author matched GMACW TAACCCT participants to similar nonparticipants using propensity scores developed from demographic and education information. Data sources included a common Salesforce database, institutional research data systems, Unemployment Insurance records, and National Student Clearinghouse data. The author used statistical models to examine differences in credential achievement between the groups.","Education and skills gain
The study found that GMACW TAACCCT program participation was significantly associated with increased credential achievement, with 60% of the treatment group achieving a credential versus 46% of the comparison group.","The author created a matched group of non-participating students to compare to GMACW TAACCCT students. However, the author did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not GMACW TAACCT —could explain the observed differences in outcomes. Additionally, the author used a historical cohort from previous years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Retraining the Gulf Coast through Information Technology Pathways: Final  impact evaluation report (Patnaik & Prince 2016)","Retraining the Gulf Coast through Information Technology Pathways: Final  impact evaluation report (Patnaik & Prince 2016)","Study Type: Causal Impact Analysis","Patnaik, A., & Prince, A. (2016). Retraining the Gulf Coast through Information Technology Pathways: Final Impact evaluation report. Austin, TX: The University of Texas at Austin, Lyndon B. Johnson School of Public Affairs.",,2016,https://raymarshallcenter.org/files/2016/11/Retraining-the-Gulf-Coast-through-I…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Retraining the Gulf Coast Workforce through Information Technology Pathways Consortium (Gulf Coast IT Pathways) grant program on education outcomes.
Using a nonexperimental design with institutional data from the college systems, the study authors compared students in the program to a matched historical cohort comparison group.
The study found that the Gulf Coast IT Pathways program was significantly related to higher credential, certificate, or a degree attainment and higher credit hour accumulation.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Gulf Coast IT Pathways TAACCCT; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Retraining the Gulf Coast Workforce through Information Technology Pathways Consortium was funded by a four-year TAACCCT grant. Funded in 2012, the Consortium of nine colleges across Louisiana and Mississippi was led by Bossier Parish Community College. The project was designed to meet a workforce demand for skilled workers in Information Technology (IT), and specifically in cyber security, health information technology, and industrial information IT. The program's objective was to target TAA eligible workers, veterans, and others who require basic skills training for jobs. The project included the following five career pathways strategies: an evidence-based, integrated career pathway in IT; a continuum of completion for stacking credentials; integrated hybrid and online learning models; integrated transferrable/portable credentials with degree articulation; and programs in alignment with partners to meet the needs of the IT workforce.","The study used a nonexperimental design to compared students in the Gulf Coast IT Pathways program to a matched historical comparison group. The participants in the treatment group attended and participated in the program at one of the nine community colleges in the first 3 years of the grant program. The comparison group included students who were enrolled in similar programs in the year prior to program implementation. The authors matched Gulf Coast IT Pathways program participants to similar nonparticipants using propensity scores developed from demographic characteristics. The treatment group included 1,774 participants, and the comparison group included 5,017 participants. Using institutional research data systems at each of the colleges in the consortium, the authors conducted statistical tests to examine differences in outcomes between the groups.","Education and skills gain
The study found that participation in the Gulf Coast IT Pathways program was significantly associated with an increased likelihood of credit hour accumulation, with the treatment group accumulating more credit hours (12 credits) versus the comparison group (11.3 credits).
The study also found that participation in the Gulf Coast IT Pathways program was significantly associated with an increased likelihood of credential, certificate, or a degree attainment. Relative to the comparison group, higher proportions of treatment students earned a credential (26% vs. 18%), a certificate (18% vs. 11%), or a degree (14% vs. 9%).","The authors did not account for other factors that could have affected the difference between the treatment and comparison groups such as a pre-intervention measure of the degree of financial disadvantage and a pre-intervention measure of education. The study also uses a historical comparison group to the year prior to the intervention and external, unobserved factors may have contributed to the findings. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"South Louisiana Community College in Lafayette, Louisiana
Bossier Parish Community College in Bossier City, Louisiana
Delgado Community College in New Orleans, Louisiana
Louisiana Delta Community College in Monroe, Louisiana
Copiah-Lincoln Community College in Wesson, Mississippi
Mississippi Delta Community College in Moorhead, Mississippi
Pearl River Community College in Poplarville, Mississippi
Meridian Community College in Meridian, Mississippi
Northeast Mississippi Community College in Booneville, Mississippi",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Low-skilled, Veteran or military","Basic skills, Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Northeast Resiliency Consortium final evaluation report (Price et al. 2017)","Northeast Resiliency Consortium final evaluation report (Price et al. 2017)","Study Type: Causal Impact Analysis","Price, D., Childress, L., Sedlak, W., & Roach, R. (2017). Northeast Resiliency Consortium final evaluation report. Indianapolis, IN: DVP-PRAXIS LTD.",,2017,https://www.dvp-praxis.org/wp-content/uploads/2015/05/Northeast-Resiliency-Cons…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Northeast Resiliency Consortium (NRC) on education, earnings, and employment outcomes. This summary focuses on the continuing education to credit-based educational pathways strategy. 
The authors used a nonexperimental design to compare the outcomes of NRC students who enrolled in continuing education to credit pathways to a matched comparison group.
The study found a significant association between NRC program participation and increased credit accumulation, matriculation, and employment attainment and retention.
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the NRC program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Northeast Resiliency Consortium (NRC) was comprised of seven community colleges across four northeastern states. The consortium received grant funds in 2013, under TAACCCT, to close skill gaps through enhanced and expanded programing in healthcare, information technology, hospitality, and environmental science. As a result, NRC colleges offered 84 programs of study (44 continuing education and workforce development programs and 40 credit-bearing programs). NRC’s goals included: (1) accelerating skills, competencies, and credential acquisition; (2) incorporating advanced technology to increase access to classes and accelerate learning; (3) increasing employer engagement; and (4) providing comprehensive outreach, assessment, and student supports. The NRC program included two primary strategies. The first strategy focused on continuing education to credit-based educational pathways, including stacked and latticed credentials, advanced technology, and work-based learning opportunities. The second strategy included comprehensive support services (career, personal, and academic) framed to support resiliency.","The study used a nonexperimental design to compare the outcomes of NRC program participants who enrolled in continuing education to credit pathways to students at the same colleges who did not. The authors matched NRC participants to similar nonparticipants using propensity scores developed from demographic information, education, employment status, and state. Across the seven colleges, study participants included 1,594 students in the treatment group and 1,594 students in the comparison group. The authors conducted statistical analyses to examine differences in outcomes between the groups using community college administrative data, a centralized database, and state workforce data. Outcomes included program completion rates, attainment of credentials, credit accumulation, employment attainment and retention, and earnings increases.","Education and skills gain
The study found a significant relationship between enrollment in continuing education to credit pathways and credit accumulation, with 41% of treatment students earning or banking credits for continuing education compared to 14% of comparison students.
The study also found that enrollment in continuing education to credit pathways was significantly associated with matriculation, with higher proportions of treatment students transitioning to credit-bearing college-level programs than comparison students (26% versus 14%).
The study did not find significant relationships between enrollment in continuing education to credit pathways and program completion or credential attainment.
Earnings and wages
The study did not find a significant relationship between enrollment in continuing education to credit pathways and increased earnings for incumbent workers.
Employment
The study found a significant relationship between enrollment in continuing education to credit pathways and employment for students who were unemployed upon entry into NRC programs, with higher employment rates for treatment students than comparison students (36% versus 27%).
Enrollment in continuing education to credit pathways was also significantly associated with higher rates of employment retention, with 66% of treatment students being employed three quarters after program exit compared to 51% of comparison students.","The authors created a matched group of non-participating individuals to compare to students who were enrolled in an NRC program. However, the authors did not appropriately control for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the NRC program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"Atlantic Cape Community College in Hamilton, New Jersey
Passaic County Community College in Paterson, New Jersey
Bunker Hill Community College in Boston, Massachusetts
Capital Community College in Hartford, Connecticut
Housatonic Community College Bridgeport, Connecticut
Kingsborough Community College in Brooklyn, New York
LaGuardia Community College in Queens, New York",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Northeast Resiliency Consortium final evaluation report (Price et al. 2017)","Northeast Resiliency Consortium final evaluation report (Price et al. 2017)","Study Type: Causal Impact Analysis","Price, D., Childress, L., Sedlak, W., & Roach, R. (2017). Northeast Resiliency Consortium final evaluation report. Indianapolis, IN: DVP-PRAXIS LTD.",,2017,https://www.dvp-praxis.org/wp-content/uploads/2015/05/Northeast-Resiliency-Cons…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Northeast Resiliency Consortium (NRC) on education, earnings, and employment outcomes. This summary focuses on the comprehensive support services strategy.
The authors used a nonexperimental design to compare the outcomes of NRC students who received comprehensive support services to a matched comparison group.
The study found a significant association between receipt of support services and increased program completion, credential attainment, credit accumulation, earnings for incumbent workers, and employment attainment and retention.
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the NRC program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Northeast Resiliency Consortium (NRC) was comprised of seven community colleges across four northeastern states. The consortium received grant funds in 2013, under TAACCCT, to close skill gaps through enhanced and expanded programing in healthcare, information technology, hospitality, and environmental science. As a result, NRC colleges offered 84 programs of study (44 continuing education and workforce development programs and 40 credit-bearing programs). NRC’s goals included: (1) accelerating skills, competencies, and credential acquisition; (2) incorporating advanced technology to increase access to classes and accelerate learning; (3) increasing employer engagement; and (4) providing comprehensive outreach, assessment, and student supports. The NRC program included two primary strategies. The first strategy focused on continuing education to credit-based educational pathways, including stacked and latticed credentials, advanced technology, and work-based learning opportunities. The second strategy included comprehensive support services (career, personal, and academic) framed to support resiliency.","The study used a nonexperimental design to compare the outcomes of continuing education students who received comprehensive support services to students at the same colleges who did not. The authors matched NRC participants to similar nonparticipants using propensity scores developed from demographic information, education, employment status, and state. Across the seven colleges, study participants included 1,396 students in the treatment group and 1,396 students in the comparison group. The authors conducted statistical analyses to examine differences in outcomes between the groups using community college administrative data, a centralized database, and state workforce data. Outcomes included program completion rates, attainment of credentials, credit accumulation, employment attainment and retention, and earnings increases.","Education and skills gain
The study found a significant relationship between receipt of comprehensive support services and program completion, with 82% of treatment students completing their program compared to 44% of comparison students.
The study also found a significant relationship between receipt of comprehensive support services and credential attainment, with higher proportions of treatment students earning a credential than comparison students (74% versus 37%).
Receipt of comprehensive support services was significantly associated with higher rates of credit accumulation, with 41% of treatment students earning or banking credits for continuing education compared to 24% of comparison students.
The study did not find a significant relationship between receipt of comprehensive support services and transition to a credit program.
Earnings and wages 
The study found a significant relationship between receipt of comprehensive support services and increased earnings for incumbent workers, with higher proportions of treatment students having an increase in earnings than comparison students (87% versus 76%).
Employment
The study found a significant relationship between receipt of comprehensive support services and employment for students who were unemployed upon entry into NRC programs, with higher employment rates for treatment students than comparison students (41% versus 19%).
Receipt of comprehensive support services was also significantly associated with higher rates of employment retention, with 67% of treatment students being employed three quarters after program exit compared to 50% of comparison students.","The authors created a matched group of non-participating individuals to compare to students who were enrolled in an NRC program. However, the authors did not appropriately control for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the NRC program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"Atlantic Cape Community College in Hamilton, New Jersey
Passaic County Community College in Paterson, New Jersey
Bunker Hill Community College in Boston, Massachusetts
Capital Community College in Hartford, Connecticut
Housatonic Community College Bridgeport, Connecticut
Kingsborough Community College in Brooklyn, New York
LaGuardia Community College in Queens, New York",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Trade Adjustment Assistance Community College Career Training grant final report. (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Trade Adjustment Assistance Community College Career Training grant final report. (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Study Type: Causal Impact Analysis","Mississippi River Transportation, Distribution & Logistics Consortium. (2017). Trade Adjustment Assistance Community College Career Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Report_9_30_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Re…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium’s grant-funded programs on education outcomes. This summary contains the findings from Delgado Community College.
The authors used a nonexperimental design to compare the outcomes of students enrolled in programs enhanced through grant funding to those enrolled in a similar program at the same community college.
The study found that the odds of program completion were significantly higher for participants in the grant-funded programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the MRTDL Consortium’s grant-funded programs; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium is comprised of nine community colleges across eight states (Illinois, Louisiana, Mississippi, Arkansas, Minnesota, Missouri, Tennessee, and Kentucky). In 2013, the MRTDL consortium was awarded a TAACCCT grant to train workers for jobs in the transportation, distribution, and logistics sector. Building on previous funding from the American Association of Community Colleges grant, the project continued efforts to enhance economic development along the Mississippi River. Project goals included building and improving sector partnerships among community colleges, employers, workforce agencies, and other relevant stakeholders; revamping postsecondary programs by aligning program content and capacity with employer and industry needs; incorporating stacked and latticed credentials in programming that served the needs of TAA-eligible workers; and sharing and realizing the benefits of working in a consortium to foster collaboration and innovation. At the various community colleges in the MRTDL Consortium, the project either developed new degree pathways or improved existing ones in areas related to truck driving, aviation maintenance, and logistics technicians among others. Depending on the community college and the grant-funded program, screening was conducted on prospective participants to determine program eligibility. TAACCCT funding at Delgado improved the Steersman Apprentice Mate and Licensed Mariner/Wheelman certification programs and developed new Shipyard and Forklift Operator/Dock Personnel certification programs.","The study took place at Delgado Community College (Delgado) in New Orleans, Louisiana. The authors used a nonexperimental design to compare the outcomes of students enrolled in the grant-funded programs to those enrolled in a comparable program at Delgado. The grant-funded programs were matched to a comparable program that was similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and similar student demographic composition. The treatment group included 768 students enrolled in grant-funded programs between 2014 and 2016. The comparison group included 206 students enrolled in short-term, non-credit certificate programs such as that for Customer Service Representatives during the same time period. Data sources included institutional data from the College Study Information System and self-reported data from participant intake forms and post-program completion surveys. The authors used a statistical model with controls for demographic and employment information to examine differences in program completion between the groups.","Education and skills gain
The study found that grant-funded program participation was significantly related to higher odds of program completion, where treatment participants were 2.6 times more likely to complete the program than participants in the comparison group.","The authors created a matched group of non-participating students to compare to participants enrolled in grant-funded programs at Delgado. However, the authors did not appropriately control for other factors that could have affected the difference between the treatment and comparison groups, such as a pre-intervention degree of financial disadvantage as required by the protocol. This preexisting difference between the groups—and not the MRTDL Consortium’s grant-funded programs—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Study Type: Causal Impact Analysis","Mississippi River Transportation, Distribution & Logistics Consortium. (2017). Trade Adjustment Assistance Community College Career Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Report_9_30_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Re…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium’s grant-funded programs on education outcomes. This summary contains the findings from Hinds Community College. 
The authors used a nonexperimental design to compare the outcomes of students enrolled in programs enhanced through grant funding to those enrolled in a similar program at the same community college.
The study found that the odds of program completion were significantly higher for participants in the grant-funded programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors used a comparison group from different academic programs presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the MRTDL Consortium’s grant-funded programs; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium is comprised of nine community colleges across eight states (Illinois, Louisiana, Mississippi, Arkansas, Minnesota, Missouri, Tennessee, and Kentucky). In 2013, the MRTDL consortium was awarded a TAACCCT grant to train workers for jobs in the transportation, distribution, and logistics sector. Building on previous funding from the American Association of Community Colleges grant, the project continued efforts to enhance economic development along the Mississippi River. Project goals included building and improving sector partnerships among community colleges, employers, workforce agencies, and other relevant stakeholders; revamping postsecondary programs by aligning program content and capacity with employer and industry needs; incorporating stacked and latticed credentials in programming that served the needs of TAA-eligible workers; and sharing and realizing the benefits of working in a consortium to foster collaboration and innovation. At the various community colleges in the MRTDL Consortium, the project either developed new degree pathways or improved existing ones in areas related to truck driving, aviation maintenance, and logistics technicians among others. Depending on the community college and the grant-funded program, screening was conducted on prospective participants to determine program eligibility. TAACCCT funding at Hinds improved a Truck Driving certification program and developed a new Shipyard certification program.","The study took place at Hinds Community College (Hinds) in Jackson, Mississippi. The authors used a nonexperimental design to compare the outcomes of students enrolled in the grant-funded programs to those enrolled in comparable programs at Hinds. The grant-funded programs were matched to comparable programs that were similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and similar student demographic composition. The treatment group included 2,644 students enrolled in grant-funded programs between 2014 and 2016. The comparison group included 522 students enrolled in a short-term Phlebotomy or Emergency Medical Technician (EMT) certification program during the same time period. Data sources included institutional data from the College Study Information System and self-reported data from participant intake forms and post-program completion surveys. The authors used a statistical model with controls for demographic and employment information to examine differences in program completion between the groups.","Education and skills gain
The study found that grant-funded program participation was significantly related to higher odds of program completion, where treatment participants were 3.4 times more likely to complete the program than participants in the comparison group.","The authors created a matched group of non-participating students to compare to participants enrolled in grant-funded programs at Hinds. However, the treatment and comparison group participants were enrolled in different programs. For example, the outcomes of participants in the Certified Driver’s License (CDL) certification program were compared to that of the Emergency Medical Technician (EMT) certification program. Because the two groups were enrolled in different programs, differences in outcomes could be due to varying factors in the programs of study (such as required coursework) and not the MRTDL Consortium’s grant-funded programs. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Study Type: Causal Impact Analysis","Mississippi River Transportation, Distribution & Logistics Consortium. (2017). Trade Adjustment Assistance Community College Career Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Report_9_30_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Re…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium’s grant-funded programs on education outcomes. This summary contains the findings from Lewis & Clark Community College.
The authors used a nonexperimental design to compare the outcomes of students enrolled in programs enhanced through grant funding to those enrolled in a similar program at the same community college.
The study found that the odds of program completion were significantly higher for participants in the grant-funded programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the MRTDL Consortium’s grant-funded programs; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium is comprised of nine community colleges across eight states (Illinois, Louisiana, Mississippi, Arkansas, Minnesota, Missouri, Tennessee, and Kentucky). In 2013, the MRTDL consortium was awarded a TAACCCT grant to train workers for jobs in the transportation, distribution, and logistics sector. Building on previous funding from the American Association of Community Colleges grant, the project continued efforts to enhance economic development along the Mississippi River. Project goals included building and improving sector partnerships among community colleges, employers, workforce agencies, and other relevant stakeholders; revamping postsecondary programs by aligning program content and capacity with employer and industry needs; incorporating stacked and latticed credentials in programming that served the needs of TAA-eligible workers; and sharing and realizing the benefits of working in a consortium to foster collaboration and innovation. At the various community colleges in the MRTDL Consortium, the project either developed new degree pathways or improved existing ones in areas related to truck driving, aviation maintenance, and logistics technicians among others. Depending on the community college and the grant-funded program, screening was conducted on prospective participants to determine program eligibility. TAACCCT funding at Lewis & Clark Community College improved the Automotive Technology, Process Operations, and Welding Technology degree pathways and developed new Truck Driving and Logistics Technician degree pathways.","The study took place at Lewis & Clark Community College (L&C) in Godfrey, Illinois. The authors used a nonexperimental design to compare the outcomes of students enrolled in the grant-funded programs to those enrolled in comparable programs at L&C. The grant-funded programs were matched to comparable programs that were similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and similar student demographic composition. The treatment group included 677 students enrolled in grant-funded programs between 2014 and 2016. The comparison group included 1,088 students enrolled in comparable programs such as those that led to Associate’s degrees in CAD Technology or that resulted in short-term certificates in Human Resources during the grant period. Data sources included institutional data from the College Study Information System and self-reported data from participant intake forms and post-program completion surveys. The authors used a statistical model with controls for demographic and employment information to examine differences in program completion between the groups.","Education and skills gain
The study found that grant-funded program participation was significantly related to higher odds of program completion, where treatment participants were 4 times more likely to complete the program than participants in the comparison group.","The authors created a matched group of non-participating students to compare to participants enrolled in grant-funded programs at L&C. However, the authors did not appropriately control for other factors that could have affected the difference between the treatment and comparison groups, such as a pre-intervention degree of financial disadvantage as required by the protocol. This preexisting difference between the groups—and not the MRTDL Consortium’s grant-funded programs—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Study Type: Causal Impact Analysis","Mississippi River Transportation, Distribution & Logistics Consortium. (2017). Trade Adjustment Assistance Community College Career Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Report_9_30_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Re…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium’s grant-funded programs on education outcomes. This summary contains the findings from Southwest Tennessee Community College.
The authors used a nonexperimental design to compare the outcomes of students enrolled in programs enhanced through grant funding to those enrolled in a similar program at the same community college.
The study found that the odds of program completion were significantly higher for participants in the grant-funded programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the MRTDL Consortium’s grant-funded programs; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium is comprised of nine community colleges across eight states (Illinois, Louisiana, Mississippi, Arkansas, Minnesota, Missouri, Tennessee, and Kentucky). In 2013, the MRTDL consortium was awarded a TAACCCT grant to train workers for jobs in the transportation, distribution, and logistics sector. Building on previous funding from the American Association of Community Colleges grant, the project continued efforts to enhance economic development along the Mississippi River. Project goals included building and improving sector partnerships among community colleges, employers, workforce agencies, and other relevant stakeholders; revamping postsecondary programs by aligning program content and capacity with employer and industry needs; incorporating stacked and latticed credentials in programming that served the needs of TAA-eligible workers; and sharing and realizing the benefits of working in a consortium to foster collaboration and innovation. At the various community colleges in the MRTDL Consortium, the project either developed new degree pathways or improved existing ones in areas related to truck driving, aviation maintenance, and logistics technicians among others. Depending on the community college and the grant-funded program, screening was conducted on prospective participants to determine program eligibility. TAACCCT funding at Southwest improved an Industrial Technology training program and developed a new Advanced Industrial Logistics Technology degree pathway.","The study took place at Southwest Tennessee Community College (Southwest) in Memphis, Tennessee. The authors used a nonexperimental design to compare the outcomes of students enrolled in the grant-funded programs to those enrolled in a comparable program at Southwest. The grant-funded programs were matched to a comparable program that was similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and similar student demographic composition. The treatment group included 497 students enrolled in grant-funded programs between 2014 and 2016. The comparison group included 111 students enrolled in a short-term, non-credit certificate program that led to an Associate’s degree in Mechanical Engineering Technology during the same time period. Data sources included institutional data from the College Study Information System and self-reported data from participant intake forms and post-program completion surveys. The authors used a statistical model with controls for demographic and employment information to examine differences in program completion between the groups.","Education and skills gain
The study found that grant-funded program participation was significantly related to higher odds of program completion where treatment participants were almost 4 times more likely to complete the program than participants in the comparison group.","The authors created a matched group of non-participating students to compare to participants enrolled in grant-funded programs at Southwest. However, the authors did not appropriately control for other factors that could have affected the difference between the treatment and comparison groups, such as a pre-intervention measure of degree of financial disadvantage as required by the protocol. This preexisting difference between the groups—and not the MRTDL Consortium’s grant-funded programs—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Study Type: Causal Impact Analysis","Mississippi River Transportation, Distribution & Logistics Consortium. (2017). Trade Adjustment Assistance Community College Career Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Report_9_30_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Re…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium’s grant-funded programs on education outcomes. This summary contains the findings from St. Louis Community College.
The authors used a nonexperimental design to compare the outcomes of students enrolled in programs enhanced through grant funding to those enrolled in a similar program at the same community college.
The study found that the odds of program completion were significantly higher for participants in the grant-funded programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors used a comparison group from different academic programs presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the MRTDL Consortium’s grant-funded programs; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium is comprised of nine community colleges across eight states (Illinois, Louisiana, Mississippi, Arkansas, Minnesota, Missouri, Tennessee, and Kentucky). In 2013, the MRTDL consortium was awarded a TAACCCT grant to train workers for jobs in the transportation, distribution, and logistics sector. Building on previous funding from the American Association of Community Colleges grant, the project continued efforts to enhance economic development along the Mississippi River. Project goals included building and improving sector partnerships among community colleges, employers, workforce agencies, and other relevant stakeholders; revamping postsecondary programs by aligning program content and capacity with employer and industry needs; incorporating stacked and latticed credentials in programming that served the needs of TAA-eligible workers; and sharing and realizing the benefits of working in a consortium to foster collaboration and innovation. At the various community colleges in the MRTDL Consortium, the project either developed new degree pathways or improved existing ones in areas related to truck driving, aviation maintenance, and logistics technicians among others. Depending on the community college and the grant-funded program, screening was conducted on prospective participants to determine program eligibility. TAACCCT funding at STLCC improved an Aviation Maintenance degree pathway and developed new Truck Driving, Logistics, and Avionics certification programs.","The study took place at St. Louis Community College (STLCC) in St. Louis, Missouri. The authors used a nonexperimental design to compare the outcomes of students enrolled in the grant-funded programs to those enrolled in comparable programs at STLCC. The grant-funded programs were matched to comparable programs that were similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and student demographic composition. The treatment group included 447 students enrolled in grant-funded programs between 2014 and 2016. The comparison group included 262 students enrolled in a short-term Certified Production Technician program, an Associate’s degree program in Electrical/Electronic Engineering Technology, or a short-term certification program in Line Worker Training during the grant period. during the same time period. Data sources included institutional data from the College Study Information System and self-reported data from participant intake forms and post-program completion surveys. The authors used a statistical model with controls for demographic and employment information to examine differences in program completion between the groups.","Education and skills gain
The study found that grant-funded program participation was significantly related to higher odds of program completion, where treatment participants were 7.2 times more likely to complete the program than participants in the comparison group.","The authors created a matched group of non-participating students to compare to participants enrolled in grant-funded programs at STLCC. However, the treatment and comparison group participants were enrolled in different programs. For example, the outcomes of participants in the short-term program for Truck Driving (CDL-A) certification were compared to that of the short-term Line Worker Training certification program. Because the two groups were enrolled in different programs, differences in outcomes could be due to varying factors in the programs of study (such as required coursework) and not the MRTDL Consortium’s grant-funded programs. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Trade Adjustment Assistance Community College Career Training grant final report (Mississippi River Transportation, Distribution & Logistics Consortium 2017)","Study Type: Causal Impact Analysis","Mississippi River Transportation, Distribution & Logistics Consortium. (2017). Trade Adjustment Assistance Community College Career Training grant final report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Report_9_30_2017.pdf?sequence=1&isAllowed=y",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/14261/MRTDL%20Final%20Re…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium’s grant-funded programs on education outcomes. This summary contains the findings from West Kentucky Community and Technical College.
The authors used a nonexperimental design to compare the outcomes of students enrolled in programs enhanced through grant funding to those enrolled in a similar program at the same community college.
The study found that the odds of program completion were significantly higher for participants in the grant-funded programs compared to participants in the comparison group.
The quality of causal evidence presented in this report is low because the authors used a comparison group from different academic programs presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the MRTDL Consortium’s grant-funded programs; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Mississippi River Transportation, Distribution and Logistics (MRTDL) Consortium is comprised of nine community colleges across eight states (Illinois, Louisiana, Mississippi, Arkansas, Minnesota, Missouri, Tennessee, and Kentucky). In 2013, the MRTDL consortium was awarded a TAACCCT grant to train workers for jobs in the transportation, distribution, and logistics sector. Building on previous funding from the American Association of Community Colleges grant, the project continued efforts to enhance economic development along the Mississippi River. Project goals included building and improving sector partnerships among community colleges, employers, workforce agencies, and other relevant stakeholders; revamping postsecondary programs by aligning program content and capacity with employer and industry needs; incorporating stacked and latticed credentials in programming that served the needs of TAA-eligible workers; and sharing and realizing the benefits of working in a consortium to foster collaboration and innovation. At the various community colleges in the MRTDL Consortium, the project either developed new degree pathways or improved existing ones in areas related to truck driving, aviation maintenance, and logistics technicians among others. Depending on the community college and the grant-funded program, screening was conducted on prospective participants to determine program eligibility. TAACCCT funding at WKCTC improved Marine Yard, Marine Technology, Logistics & Operations Management, and Welding certification programs.","The study took place at West Kentucky Community and Technical College (WKCTC) in Paducah, Kentucky. The authors used a nonexperimental design to compare the outcomes of students enrolled in the grant-funded programs to those enrolled in comparable programs at WKCTC. The grant-funded programs were matched to comparable programs that were similar in terms of being in the same department, having the same credit/non-credit status, having a similar program duration, and similar student demographic composition. The treatment group included 2,037 students enrolled in grant-funded programs between 2014 and 2016. The comparison group included 4,621 students enrolled in short-term programs leading to an Associate’s degree in Criminal Justice, an Associate’s degree in Business Administration, or a certificate in Diesel Technology during the grant period. Data sources included institutional data from the College Study Information System and self-reported data from participant intake forms and post-program completion surveys. The authors used a statistical model with controls for demographic and employment information to examine differences in program completion between the groups.","Education and skills gain
The study found that grant-funded program participation was significantly related to higher odds of program completion, where treatment participants were 15 times more likely to complete the program than participants in the comparison group.","The authors created a matched group of non-participating students to compare to participants enrolled in grant-funded programs at WKCTC. However, the treatment and comparison group participants were enrolled in different programs. For example, the outcomes of participants in certification programs that led to an Associate’s degree in Marine Technology were compared to those for certification that led to an Associate’s degree in Criminal Justice. Because the two groups were enrolled in different programs, differences in outcomes could be due to varying factors in the programs of study (such as required coursework) and not the MRTDL Consortium’s grant-funded programs. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Trade Adjustment Assistance Community College Career Training Grant (TAACCCT): Final evaluation report (The Greater Cincinnati Supply Chain Career Development Center (SCCDC) 2018)","Trade Adjustment Assistance Community College Career Training Grant (TAACCCT): Final evaluation report (The Greater Cincinnati Supply Chain Career Development Center (SCCDC) 2018)","Study Type: Causal Impact Analysis","The Greater Cincinnati Supply Chain Career Development Center (SCCDC). (2018). Trade Adjustment Assistance Community College Career Training Grant (TAACCCT): Final evaluation report. Cincinnati, OH: Author.",,2018,https://www.skillscommons.org/handle/taaccct/17943,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of The Greater Cincinnati Supply Chain Career and Development Center’s (SCCDC) Supply Chain Management and Materials Handling & Logistics programs on education outcomes.
The study used a nonexperimental design to compare the program completion of students enrolled in SCCDC programs to students in a matched comparison group.
The study found that participation in the Supply Chain Management program, one of the SCCDC programs, was associated with a significant increase in the likelihood of program completion.
The quality of causal evidence presented in this report is low because the authors used a comparison group from different academic programs presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the SCCDC programs; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Using TAACCCT funds, Cincinnati State Technical and Community College developed The Greater Cincinnati Supply Chain Career Development Center (SCCDC) project. The project recruited TAA workers, veterans, and other adult learners who reside in counties across Ohio, Kentucky, and Indiana. In 2014, the college was awarded another TAACCCT grant to enhance its Supply Chain Management and Materials Handling & Logistics Technologies programs and train additional workers in order to meet local employer needs. The five SCCDC programs included Supply Chain Management, Forklift Essentials, Global Logistics Associate, Commercial Driver's License, and Manufacturing Skill Standards Council-Certified Logistics Associate and Certified Logistics Technician. The project enhanced several accelerated training programs by modernizing the curriculum, using technology-based learning modules, and linking programming to industry-recognized credentials. Additionally, SCCDC expanded the student services available under the Pathway to Employment Center (PTEC)—career interest and talent assessments, academic advising and tutoring, and career coaching and resources—to those enrolled in SCCDC programs.","The study used a nonexperimental design to compare program completion rates of students who participated in the SCCDC programs to students who did not participate. Each SCCDC program was matched to a comparison program also offered at Cincinnati State Technical and Community College, including Automotive Service Management, Occupational Safety and Health Administration (OSHA) 40-hr, OSHA 24-hr, and Chemical Operator. Programs were matched to a “best match” based on the similarity of program structure including department, credit/noncredit status, program size, and program duration. Next, the authors matched students enrolled in SCCDC programs to similar students in “best match” programs using propensity scores developed from demographic variables. Study participants included 389 students in the treatment group and 705 in the comparison group. Data sources included administrative data provided by the college, data from the Ohio Department of Job and Family Services, participant intake forms completed at baseline, and study participant follow-up surveys. The authors conducted statistical models to examine differences in the odds of program completion between the two groups overall and by SCCDC program.","Education and skills gain
The study found that there was no overall statistically significant relationship between being enrolled in an SCCDC program and program completion.
However, the study found that Supply Chain Management program participation was significantly associated with program completion. Students in the Supply Chain Management program were more than twice as likely to complete their program compared to students in the Automotive Service Management program.","The authors created a matched group of non-participating students to compare to students enrolled in SCCDC programs based on demographic characteristics. However, the treatment and comparison group students were enrolled in different programs. For example, students from the Forklift Essentials program were compared to that of the Occupational Health and Safety Administration program. Because the two groups were enrolled in different programs, differences in outcomes could be due to varying factors in the programs of study (such as required coursework) and not the SCCDC programs. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Veteran or military","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Get IT project evaluation: Final report (Singer 2018)","Get IT project evaluation: Final report (Singer 2018)","Study Type: Causal Impact Analysis","Singer, S. (2018). Get IT project evaluation: Final report. Syracuse, NY: Hezel Associates.",,2018,http://www.skillscommons.org/bitstream/handle/taaccct/18125/HezelAssoc_GetITEva…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the enhanced Growing the Economy through Information Technology (Get IT) curricula on student education outcomes.
The study used a nonexperimental design to compare the outcomes of students in enhanced Get IT courses to a comparison group of students not taking enhanced IT courses. Using administrative records, the author conducted statistical tests to examine the differences between groups.
The study found that students in enhanced Get IT courses were significantly more likely to remain in school than students who were not in enhanced Get IT courses.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention or include sufficient control variables. This means we are not confident that the estimated effects are attributable to NHTI’s enhanced Get IT program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Upon receipt of a TAACCCT grant in 2015, NHTI expanded and improved some of their Growing the Economy through Information Technology (Get IT) courses. The improved courses included: 1) the development of common core IT courses that can be stacked and transferred across the Community College System of New Hampshire; 2) new IT-related certificate programs that are relevant to employers; 3) the implementation of programs such as career counseling, latticed programming, and improved prior learning assessment (PLA) procedures to facilitate faster IT course completion; and 4) the development of new Digital Fabrication Lab that combines IT and industrial design. The program was designed to target Trade Adjustment Assistance-eligible workers and other adults. Additional services made available to students through the newly developed program included a dedicated career specialist to assist students with job and internship placement, accelerated courses, the integration of a soft skill ""Mindful Communication"" program, and alternate avenues for students to earn credits for previous experiences. Aspects of some Get IT courses were not enhanced using grant funds (“non-TAACCCT”).","The author used a nonexperimental design to assess retention and certificate or degree attainment at NHTI Community College in Concord, New Hampshire. When measuring retention, 716 students in TAACCCT programs were compared to 4,498 students in non-TAACCCT programs. All students first enrolled in the Fall 2015 to Spring 2018 semesters at NHTI. When measuring credential attainment, 293 students TAACCCT programs were compared to 1,979 students in non-TAACCCT programs. Certificate students included those who first enrolled in the Spring 2017 semester or earlier, while degree attainment students included those who first enrolled in Spring 2016 or earlier. Using data from NHTI administrative records, the author compared retention and certificate or degree attainment between the treatment and comparison groups and conducted statistical tests on retention outcomes.","Education and skills gain
The study found that TAACCCT Get IT participation was significantly associated with a higher likelihood of remaining in the program one year after entering than students in non-TAACCCT courses (1.4 times more likely).
The study found that TAACCCT Get IT participation was associated with a higher percentage of Associate degree or certificate awards (31%) than comparison group participants (23%). Tests of significance were not reported.","The author did not provide results of the regression model that controlled for TAACCCT status, age, and GPA, and did not account for other factors that could have affected the difference between the treatment and comparison groups, such as race/ethnicity, gender, a pre-intervention measure of degree of financial disadvantage, and a pre-intervention measure of education/training. These preexisting differences between the groups—and not the Get IT program—could explain the observed differences in outcomes. Also, the author did not provide information on the types of courses that were enhanced using TAACCCT funds and those that were not enhanced. Thus, the treatment and comparison courses may differ in systematic and unobserved ways, affecting the observed outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"PluggedIn and WorkREADY! at Southwest Virginia Community College: 2018 final report (Styers et al. 2018)","PluggedIn and WorkREADY! at Southwest Virginia Community College: 2018 final report (Styers et al. 2018)","Study Type: Causal Impact Analysis","Styers, M., Haden, C., Cosby, A., & Peery, E. (2018). PluggedIn and WorkREADY! at Southwest Virginia Community College: 2018 final report. Charlottesville, VA: Magnolia Consulting, LLC.",,2018,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the PluggedIn and WorkREADY! advanced manufacturing training program on education, employment, and earnings outcomes.
The study used a nonexperimental design to compare the outcomes of students enrolled in the PluggedIn and WorkREADY! program with students enrolled at Southwest Virginia Community College (SWCC) in comparable majors.
The study found a significant positive relationship between PluggedIn and WorkREADY! program participation and program completion, credential attainment, number of credentials attained, and employment and a significant negative relationship with program dropout.
The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to the PluggedIn and WorkREADY! program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories. Southwest Virginia Community College (SWCC) was awarded a TAACCCT grant in October 2014 to improve its PluggedIn and WorkREADY! six-month advanced manufacturing program. The grant allowed the program to develop new courses in machining/ Computer numerical control (CNC) and mechatronics and re-design courses in carpentry and welding to meet Virginia industry demands. The program offers industry-specific workforce classes, tutoring, academic counseling, adult education classes in English and mathematics, career-ready courses, and professional development support to program participants. Students in the program are involved in applied learning experiences, learn and work with the same cohort of students, complete a Capstone project towards the end of the program, and are monitored and supported by program staff throughout the program using online databases.","The authors used a nonexperimental design to compare the outcomes of adult students who participated in the PluggedIn and WorkREADY! program to those who attended SWCC but did not participate in the program. The comparison group included students who were on the Associate’s degree track and were enrolled in welding, carpentry, machining/CNC, or mechatronics. Data sources included SWCC institutional data, statewide employment and wage data, student and staff focus group outcomes data, and data from two study participant surveys for 16 program cohorts from January 2015 to July 2017. The authors conducted statistical models to examine differences in outcomes between the two groups. The outcomes included the attainment of at least one third-party credential, the average number of third-party credentials obtained, program completion, program dropout, employment one month after program exit, employment retention, and wage increases. The final analytical sample included 251 students in the treatment group and 145 in the comparison group.","Education and skills gain
The study found a significant relationship between program participation and academic completion, where the odds of program completion were almost four times higher and the odds of dropping out of the program were 67% lower for the treatment than the comparison group.
The study also found that program participation was significantly associated with credential attainment, where the odds of obtaining at least one third-party credential was eight times higher for the treatment than the comparison group and treatment group students received more third-party credentials than comparison group students (3.53 versus 1.66).
Earnings and wages
The study did not find a significant relationship between program participation and earnings outcomes.
Employment
The study found a significant relationship between program participation and finding employment within one month of program exit, where the odds of finding employment was almost three times higher for the treatment than the comparison group.
The study did not find a significant relationship between program participation and employment retention.","The authors used statistical tests to ensure students in the treatment and comparison groups were similar in terms of baseline demographic characteristics. They found that the composition of the treatment and comparison groups varied by gender, veteran status, and age. They controlled for veteran status and age but did not control for gender in the analyses. This preexisting difference between the groups—and not the PluggedIn and WorkREADY! program—could explain the observed differences in outcomes. Moreover, participants in the treatment and comparison groups were enrolled in different programs, presenting a confounding factor. The program characteristics of the comparison group greatly differed from that of the treatment group in terms of program length, course load, class schedule, cohort structure, and availability of support services. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Information technology career preparation for rural areas (Habs-Vaughn et al. 2019)","Information technology career preparation for rural areas (Habs-Vaughn et al. 2019)","Study Type: Causal Impact Analysis","Hahs-Vaughn, D. L., Swan, B., & Clark, M. H. (2019). Information technology career preparation for rural areas. Community College Journal of Research and Practice, 1-13.","Swan, B., Hahs‐Vaughn, D., Fidanzi, A., Serpa, A., DeStefano, C., & Clark, M. H. (2017). Florida XCEL‐IT: Information Technology Careers for Rural Areas final evaluation report. (Rep. No. CFTAACCCT2013.Y4F). Orlando, FL: University of Central Florida, Program Evaluation and Educational Research Group (PEER).",2019,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Florida Information Technology Careers for Rural Areas (XCEL-IT) program on education, employment, and earnings outcomes.
The authors used a nonexperimental design to compare the differences in outcomes between XCEL-IT program participants and a comparison group.
The study found that XCEL-IT program participation was significantly related to program completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the XCEL-IT program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Florida Information Technology Careers for Rural Areas (XCEL-IT) grant was funded through the TAACCCT program. The award was to a single-state consortium led by the College of Central Florida to include seven colleges. The community colleges represent 22 of the state's 67 counties with 15 counties located in critical economic rural areas. The program provided training in IT for manufacturing, logistics and supply chain management, and cyber security; and it also focused on entrepreneurship. The program targeted adults who were TAA eligible, or over age 24 and were unemployed or underemployed, including Veterans. Additionally, the program refined its career pathways for non-credit courses in the Bachelor of Applied Science degree in IT, cybersecurity, logistics, manufacturing, entrepreneurship, and developmental education. The program also created an Employment Results Scorecard to offer information on employment for students; and improved techniques for conducting outreach to eligible workers in rural areas.","The study used a nonexperimental design to assess the differences between the treatment group and a matched comparison group. The treatment group included students who enrolled in XCEL-IT courses during year 2 of the project (October 2014-September 2015). The comparison group was selected by each institution based on a ""best-match"" to the XCEL-IT programs. The comparison program could be a similar program but not one that was funded with other grant monies that occurred in former years or concurrently with the grant. Of the seven colleges in the consortium, one did not have a comparable program and did not provide a comparison group of students. The authors matched XCEL-IT program participants to similar nonparticipants using propensity scores developed from demographic information. The treatment group and the comparison group both had 394 students each in the analysis. Each institution provided student records with educational outcomes data and student demographic information. Each institution also provided linked unemployment wage data provided by the local workforce investment board or workforce partner. The authors tested program impacts with statistical tests on the following outcomes: program completion, continued enrollment, employment, and wage increase.","Education and skills gain
The study found that participants in the XCEL-IT program were eight times more likely to complete the program than students the comparison group, and the results were statistically significant.
There was no statistically significant association between the XCEL-IT program and the likelihood to continue enrollment in other education.
Earnings and wages
The study did not find a statistically significant relationship between the XCEL-IT program and the likelihood to receive a wage increase.
Employment
The study did not find a statistically significant relationship between the XCEL-IT program and the likelihood to enter employment for those who were not employed prior to the program.","Although the authors accounted for baseline demographic characteristics, they did not control for a preintervention measure of education in their analytic model which is required by the review protocol. The preexisting differences between the groups on this variable—and not the XCEL-IT program—could explain the observed differences in outcomes. Also, students were not matched within their colleges; the comparison group used for matching was pooled across the schools due to data issues and missing data from one college. Finally, according to the authors, due to the variety of programs offered through the XCEL-IT program (18 programs), colleges were not provided with the types of programs that would be comparable to the XCEL-IT program. These inherent differences in the comparison programs present a confound. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"College of Central Florida in Ocala, Florida Eastern
Florida State College in Cocoa, Florida
Florida Southwestern State College in Fort Myers, Florida
North Florida Community College in Madison, Florida
Palm Beach State College in Palm Beach County, Florida
South Florida State College in Avon Park, Florida
St. John’s River State College in Palatka, Florida",,"Absence of conflict of interest.",,"Rural, United States","Causal Impact Analysis","Adult, Unemployed, Veteran or military","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"M-PATH: Advanced  manufacturing final evaluation report. (Rep. No. 33VCDOLTACT2014.Y4S) (Swan et al. 2018)","M-PATH: Advanced  manufacturing final evaluation report. (Rep. No. 33VCDOLTACT2014.Y4S) (Swan et al. 2018)","Study Type: Causal Impact Analysis","Swan, B., Clarke, M. H., Serpa, A., DeStefano, C., & Hahs-Vaughn (2018). M-PATH: Advanced manufacturing final evaluation report. (Rep. No. 33VCDOLTACT2014.Y4S). Orlando, FL: University of Central Florida, Program Evaluation and Educational Research Group (PEER).",,2018,http://www.skillscommons.org/bitstream/handle/taaccct/18579/TC-26475-14-60-A-12…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the M-PATH: Advanced Manufacturing Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program on education, earnings, and employment outcomes.
Using a nonexperimental design, the authors compared students in the M-PATH TAACCCT program to a matched comparison group.
The study found that participation in the M-PATH TAACCCT program was significantly related to increased program completion and decreased continued education in another certificate program. The study also found that non-incumbent workers who completed the program were significantly more likely to enter employment than the comparison group.
The quality of causal evidence presented in this report is low because the authors did not include sufficient control variables. This means we are not confident that the estimated effects are attributable to the M-PATH TAACCCT program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Valencia College received a TAACCCT grant for the M-PATH: Advanced Manufacturing (M-PATH) program to offer additional specialized advanced manufacturing jobs. The goals of the M-PATH program were to increase attainment of certifications, certificates, and relevant credentials for workers in ""growth industry sectors""; provide innovative curriculum that meets industry needs and increases learning and retention for workers; and show that program participants improve employment outcomes as a result of the M-PATH program. The program was designed to provide TAA-eligible workers and other adults for employment in the targeted industry sectors. The College partnered with regional manufacturing, CareerSource (local workforce investment board), and other industry relevant organizations to develop the educational pathway program which included for-credit programs as well as continuing education. In the four years of the grant, 14 programs were developed or improved to include 14 that provided industry relevant certificates. The programs included 10 ""flexible"" labs to support hands-on training and instruction. Finally, the program included student-tailored support services and advising, career coaching, non-traditional recruitment methods, and tutoring to students.","The study took place at Valencia College and used a nonexperimental design to compare the outcomes of students who participated in the M-PATH TAACCCT program to students who did not participate. The comparison group included students who were enrolled or formerly enrolled in a similar program of study. The authors matched M-PATH students to similar students using propensity scores developed from demographic variables. Study participants included 147 students in the treatment group and 244 students in the comparison group. Using data from institutional records and Unemployment Insurance data from the local workforce investment board, the authors conducted statistical tests to examine differences in outcomes between the groups.","Education and skills gain
The study found that students in the M-PATH TAACCCT program were significantly more likely than comparison group students to complete their program of study (4,000 times more likely).
However, the study found that students in the M-PATH TAACCCT program were significantly less likely than comparison group students to continue education in another certificate program (50 times less likely).
Earnings and wages
The study did not find a significant relationship between participation in the M-PATH TAACCCT program and receipt of a wage increase.
Employment
The study found that non-incumbent workers who completed the program were significantly more likely to enter employment than the comparison group (26 times more likely).","The authors created a matched group of comparison students to compare to M-PATH students. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage which is required by the review protocol. These preexisting differences between the groups—and not the M-PATH TAACCCT program—could explain the observed differences in outcomes. Additionally, the authors used a single regression to impute data for missing data on some of their control variables. The authors also note that 40% of students were enrolled in the M-PATH program before students enrolled in the comparison group, which could have affected the outcome of program completion. Finally, some of the comparisons had no or few students in some of the classifications, and the authors state that this could have affected the accuracy and generalizability of the study as odds ratios may be inflated. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Job training for a knowledge-based economy: The Quincy College Biotechnology and Compliance program (Van Dyke et al. 2016)","Job training for a knowledge-based economy: The Quincy College Biotechnology and Compliance program (Van Dyke et al. 2016)","Study Type: Causal Impact Analysis","Van Dyke, B., Bayala, I., & Cutaia, K. (2016). Job training for a knowledge-based economy: The Quincy College Biotechnology and Compliance program. Quincy, MA: Quincy College.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/15632/Quincy%20College%2…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Quincy College Biotechnology and Compliance program on education, earnings, and employment outcomes.
The study used a nonexperimental design to assess the outcomes of students enrolled in the Biotechnology and Compliance program to a comparison group.
The study found that Biotechnology and Compliance program participation was associated with higher rates of employment, degree attainment, and pursuit of further education, but lower earnings. However, the study did not include tests of statistical significance.
The quality of casual evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Biotechnology and Compliance program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Upon receipt of a TAACCCT grant, Quincy College developed two new pathways related to Biotechnology and Compliance, an Associate’s degree and a credential program. In addition, the program included a state-of-the-art biomanufacturing training lab, an upgraded curriculum, an online virtual laboratory, bridge classes, access to professional development opportunities, and additional staff who were hired specifically for these programs. Students were eligible to benefit if they enrolled in either of these programs beginning in the fall of 2013.","The study used a nonexperimental design to compare the outcomes of students who participated in the Biotechnology and Compliance Associate’s degree program to students who did not participate. The comparison group included students who were enrolled in the Medical Technology Associate’s degree program during the same time period. Study participants included 11 students in the treatment group and 11 students in the comparison group. Using student surveys, the authors examined differences in program completion, credit hours received, employment retention, gainful employment, and wage increases. However, no tests of statistical significance were provided.","Education and skills gain
The study found that participation in the Biotechnology and Compliance program was related to increased credit attainment, degree completion, and enrollment in further education, relative to the comparison group.
Earnings and wages
When compared to students in the comparison group, the study found that participation in the Biotechnology and Compliance program was related to lower earnings after program completion.
Employment
The study found that participation in the Biotechnology and Compliance program was related to increased employment after program completion and employment retention six months after completion, relative to the comparison group.","The authors did not provide demographic information about the comparison group and did not account for preexisting differences between the treatment and comparison groups before program participation. These preexisting differences between the groups—and not the Biotechnology and Compliance program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-Unfavorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Evaluation of the Illinois Network for Advanced Manufacturing: Final Report (Westat 2016)","Evaluation of the Illinois Network for Advanced Manufacturing: Final Report (Westat 2016)","Study Type: Causal Impact Analysis","Westat. (2016). Evaluation of the Illinois Network for Advanced Manufacturing: Final Report. Rockville, MD: Westat & GEM Software Development, Inc.",,2016,https://www.researchgate.net/publication/315664265_Evaluation_of_the_Illinois_N…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Illinois Network for Advanced Manufacturing (INAM) program on education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the outcomes of students who were in the INAM program to a comparison group of students who took similar courses before the creation of INAM. Using data from college databases and the Illinois Department of Employment Security, the authors conducted statistical models to examine differences in outcomes between the groups.
The study found that participation in the INAM program was significantly associated with improved rates of certificate or degree completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the INAM program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories
The Illinois Network for Advanced Manufacturing (INAM) was developed using TAACCCT funds and consisted of 21 colleges throughout the state. INAM aimed to prepare participants for advanced manufacturing careers in six areas: certified production technicians, mechatronics, precision machining (CNC or computer numerical control), industrial maintenance, welding, and metalworking. The programs targeted TAA workers, veterans, incumbent workers, and the unemployed, as well as other students who applied. To achieve their goal, INAM: (1) developed student educational plans; (2) improved curricula to facilitate attainment of industry credentials; (3) obtained industry-relevant equipment; (4) developed online and technology-based learning; (5) recognized prior student academic achievements to accelerate program completion; (6) created internships and on-the-job training opportunities; (7) instituted career placement services; and (8) created articulation agreements with higher learning institutions.","The nonexperimental study was conducted at the colleges that formed INAM, although several colleges were excluded due to the strategies used to recruit treatment and comparison group students. The treatment group for educational outcomes consisted of 2,645 students taking INAM courses who were recruited into the INAM program by education planners from summer 2013 through 2015. The planners intentionally excluded some students who did not fit the program intent, were not intending on taking further courses, had poor grades, or had already taken other INAM courses. For employment and wage outcomes, this sample was restricted to only students in one of four chosen colleges who completed a certificate. The comparison group consisted of students enrolled in similar courses before the creation on INAM, including those in INAM courses but not in the INAM program. Furthermore, the comparison group for employment outcomes was limited to 549 students from four colleges where wage data was available.
Outcomes included completion rates, incumbent workers with wage increases, incumbent workers with no wage increases, wage declines after enrollment, non-incumbent workers with increased wages, non-incumbent workers with zero wages before and after college entrance, predicted wages in the twelfth quarter, and employment. Using data from college databases and the Illinois Department of Employment Security, the authors conducted statistical models to examine differences in outcomes between treatment and comparison groups.","Education and skills gain
The study found that participation in the INAM program was significantly associated with improved rates of completion; students in the treatment group were 1.4 times as likely to complete a certificate or degree than students in the comparison group.
Earnings and wages
The study found no significant relationship between participation in the INAM program and predicted wages in the twelfth quarter.
Employment
The study found no significant relationship between participation in the INAM program and employment.","When creating the treatment and comparison groups, education planners selectively approached students in INAM courses with information on enrolling in the program. Potentially inconsistent selection of students, as well as student self-selection into the INAM program, represents a confound. Additionally, the authors used students from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Lastly, the authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. The authors did not provide demographic information about the comparison group. These preexisting differences between the groups—and not the INAM program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Unemployed, Other, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"East Los Angeles College: Technology & Logistics program TAACCCT final evaluation report (Wiima 2016)","East Los Angeles College: Technology & Logistics program TAACCCT final evaluation report (Wiima 2016)","Study Type: Causal Impact Analysis","Wijma, C. (2016). East Los Angeles College: Technology & Logistics program TAACCCT final evaluation report. San Francisco, CA: WestEd.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the enhanced East Los Angeles College’s (ELAC) Technology and Logistics program on student’s education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the education, earnings, and employment outcomes of students in the ELAC Technology and Logistics program to a matched comparison group of students in the Automotive Technology program.
The study found that participation in the ELAC Technology and Logistics program was significantly associated with improved course and degree completion.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the enhanced ELAC Technology and Logistics program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The East Los Angeles College’s (ELAC) Technology and Logistics program used TAACCCT funds to fulfill employer needs in the in-demand logistics industry. To do this, they improved their Logistics program by creating and improving courses, implementing an Associate of Science degree in Technology and Logistics, and providing supports for students in the Logistics program, including online student coaching; career/job coaching; life/academic coaching with career guidance counselor assistants; and academic counseling.","The study used a nonexperimental design to compare the outcomes of students who participated in the ELAC Technology and Logistics program to students who did not participate. The treatment group included 504 students who enrolled in the ELAC Technology and Logistics program during the 2012, 2013, and 2014 school years. The comparison group were students from the Automotive Technology (AutoTech) program enrolled during the same time period. Treatment students were matched with comparison students using age group, ethnicity, education level, educational goal attainment, and financial aid variables. Outcomes included course and certificate/degree completion rates, employment rates, median and mean quarterly wages, and mean weekly wage. Using data from ELAC's student database, departmental student data, the State of California’s Employment Development Department, and student exit surveys, the author used statistical models to examine differences in the outcomes between treatment and comparison groups. However, the study did not include tests of statistical significance for the employment and earnings outcomes.","Education and skills gain
The study found that participation in the ELAC Technology and Logistics program was significantly associated with improved course (51.1% vs 25.5%) and degree (37.1% vs 12.2%) completion than participation in the comparison program.
Employment
The study found that participation in the ELAC Technology and Logistics program was associated with lower rates of employment than participation in the comparison program (58.8% vs. 63.7%). However, the study did not include tests of statistical significance.
Earnings and wages
The study found that participation in the ELAC Technology and Logistics program was associated with higher wages than participation in the comparison program: median quarterly wages were $6,540 for the treatment vs. $4,332 for the comparison group; mean quarterly wages were $7,512 for the treatment vs. $5,658 for the comparison group; and, average weekly wages were $343 for the treatment vs. $277 for the comparison group. However, the study did not include tests of statistical significance.","Although the author reports differences in employment rates and wages between the treatment and comparison groups, no tests of significance were performed on these outcomes. Furthermore, the author did not control for gender in the matching analysis as required by the protocol. Lastly, the author did not report the final number of AutoTech students included in the analysis sample for the comparison group, nor information about their demographics. Because the author did not ensure groups were similar at baseline, these differences—and not the ELAC Technology and Logistics program—could explain the observed differences in the outcome. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Weber State University Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant-round 3: Final third party evaluation report (Wixom, 2017)","Weber State University Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant-round 3: Final third party evaluation report (Wixom, 2017)","Study Type: Causal Impact Analysis","Wixom, G. (2017). Weber State University Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant-round 3: Final third party evaluation report. Orem, UT: Education Matters.",,2017,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the enhanced Health Information Technology Career Mobility (HITCM) program on education outcomes.
The author used a nonexperimental design to compare the outcomes of students who were in the enhanced HITCM program to a matched comparison group of students in the non-enhanced HITCM program.
The study found that enhanced HITCM program participation was significantly associated with more credits earned and higher graduation rates.
The quality of causal evidence presented in this study is low because the author used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the enhanced HITCM program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Weber State University (WSU) received a TAACCCT grant to enhance their Health Information Technology Career Mobility (HITCM) program, designed to target TAACCCT candidates and adult workers. The HITCM track consists of the Health Information Management (HIM) Bachelor of Science Degree, Health Information Technology (HIT) Associate of Applied Science Degree, and Healthcare Coding Certificate and Network Technology and Multimedia (NTM) Network Management/Network Security certificate, Associate of Applied Science Degree and Bachelor of Science Degree programs. Changes were made to increase student access to HITCM courses, which included adding online education technology/delivery and partnering with other higher education institutions to allow Weber State students to attend HITCM courses at their campuses.","The nonexperimental study was conducted at WSU’s campus in Ogden, Utah and compared students who participated in the enhanced HITCM program to students who did not participate. The author matched HITCM participants to similar nonparticipants using demographic information. The treatment group included 347 students who participated in the HITCM program and were enrolled between spring of 2015 and spring of 2017. The comparison group was a historical cohort composed of 760 students that were enrolled in the original HITCM program in the five years before TAACCCT-funded enhancements were made. Using WSU’s databases, the author conducted statistical models to examine differences in the outcomes between the treatment and comparison groups. Outcomes included the number of credits earned and graduation rates.","Education and skills gain
The study found that enhanced HITCM program participation was significantly associated with higher credit attainment, with students in the treatment group earning five more credits than students in the comparison group.
The study also found a significant relationship between enhanced HITCM program participation and graduation rates, with 23% of students in the treatment group graduating with an Associate’s degree and 19% with a Bachelor’s degree (as compared to 0.9% and 0.8% of students in the comparison group, respectively).","The author used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the HITCM program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Johnson County Community College (JCCC) TAACCCT final evaluation report (York 2018)","Johnson County Community College (JCCC) TAACCCT final evaluation report (York 2018)","Study Type: Causal Impact Analysis","York, V. (2018). Johnson County Community College (JCCC) TAACCCT final evaluation report. Office of Educational Innovation and Evaluation. Manhattan, KS: Kansas State University.",,2018,https://www.skillscommons.org/bitstream/handle/taaccct/17872/JCCC%20TAACCCT%20F…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the effects of Johnson County Community College’s (JCCC) Accelerated, Collaborative Technology Training Services (ACTTS) project on education outcomes.
The author used a nonexperimental design to compare cohorts of students enrolled in ACTTS and comparison programs on five educational outcomes of interest.
The study found that ACTTS students were significantly more likely than comparison group participants to complete their program of study, be retained in their program of study, complete credit hours, and earn credentials.
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention or include sufficient controls. This means we are not confident that the estimated effects are attributable to ACTTS; other factors are likely to have contributed.","The U.S. Department of Labor's Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program was a multi-year (2011 to 2018), $1.9 billion initiative to build human capital around workforce development, particularly in the following areas: manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, the Department of Labor awarded a total of 256 TAACCCT grants to approximately 800 educational institutions, most of which were community colleges., received TAACCCT funding to implement the Accelerated, Collaborative Technology Training Services (ACTTS) project. The project was created to expand JCCC’s capacity to provide information technology (IT) training to meet growing IT sector needs in the Kansas City Metropolitan Area. The program was designed to serve TAA-eligible workers, eligible veterans and spouses, and other adults. The goals of the ACTTS project were to: 1) create an IT curriculum to better meet students’ unique scheduling and learning needs; 2) engage local employers to assist in curriculum development, professional development, and job forecasting; and 3) provide enhanced student support services for ACTTS programs. The ACTTS programs included four career pathways (Computer Information Systems/Programming, Information Technology Network, Web Development and Digital Media, and Health Information Systems), whereby students could pursue eight certificates and six associate degrees. Students enrolled in an ACTTS program were provided with career coaches to create customized educational plans and professional development opportunities.","The nonexperimental study was conducted at Johnson County Community College (JCCC) in Overland Park, Kansas. The author compared the outcomes of students who participated in the ACTTS program to students who did not. The treatment condition included 396 students who participated in one of the four career pathways in the ACTTS program and were enrolled between September 2015 and March 2018. The comparison group was composed of 685 students enrolled between August 2015 and January 2018 in one of two IT programs similar to the career pathways in the ACTTS program, but not included in the ACTTS project. Data sources included JCCC's banner system, the ACTTS student intake forms, the National Student Clearinghouse, and Kansas Department of Commerce's KANSASWorks data system. The author conducted chi-square analyses to examine differences between the groups on five educational outcomes (program completion, retention, completed credit hours, earned credentials, and pursued further education).","Education and skills gain
The study found that ACTTS students were significantly more likely than comparison group students to complete their program of study (a difference of 12 percentage points).
The study found that more than students in the ACTTS programs were significantly more likely than comparison group students to be retained in their program of study (a difference of 37 percentage points).
The study found that ACTTS students were significantly more likely than comparison group students to complete more credit hours (a difference of 8 percentage points) and earn more credentials (a difference of 12 percentage points).
However, the study found that ACTTS program participation was significantly associated with lower rates of further education, with fewer treatment students pursuing further education after completing their program than comparison students (a difference of 14 percentage points).","The composition of the treatment and comparison groups varied significantly by gender, race, school enrollment status, and Pell grant status. However, the authors did not control for the differences between the groups. These preexisting differences between the groups—and not the ACTTS program—could explain the observed differences in outcomes. Additionally, the author used a comparison group of students enrolled in different IT programs than those supported by the ACTTS project. Although the comparison group students attended the same college, the program-varying differences between the participants can lead to a confound, which could also impact the results. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Veteran or military","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"University of the District of Columbia Community College TAACCCT round 3 final evaluation report (Takyi-Laryea at el. 2017)","University of the District of Columbia Community College TAACCCT round 3 final evaluation report (Takyi-Laryea at el. 2017)","Study Type: Causal Impact Analysis","Takyi-Laryea, A., Passa, K., & Gall, A. (2017). University of the District of Columbia Community College TAACCCT round 3 final evaluation report. Fairfax, VA: ICF.",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/15691/UDC-CC%20TAACCCT%2…,"August 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the grant-funded DC Construction Academy (DCCA) program on education outcomes.
The study used a nonexperimental design to compare the education outcomes of DCCA program participants to a matched historical comparison group of students enrolled in the same construction programs prior to TAACCCT funding.
The study found that DCCA program participants were significantly more likely to attain a credential than the comparison group.
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the DCCA program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The University of the District of Columbia Community College (UDC-CC) received TAACCCT grant funds to enhance programs in the construction and hospitality sectors and reduce unemployment and underemployment in the District of Columbia by enhancing the skills of residents. These two industries were identified as high-growth, high-demand sectors. UDC-CC implemented the DC Construction Academy and the DC Hospitality Academy to address these labor-market needs. An academy approach was added to existing curricula for the hospitality and construction programs offered at UDC-CC. Features of the new program included expanded online programming, new curricula, latticed and stackable programming, learning assessments, student supports, and integrated teaching.","The study took place on the campus of UDC-CC in Washington, District of Columbia and focused on the impact of the DC Construction Academy (DCCA). The authors used a nonexperimental design to compare the outcomes of students enrolled in the grant-funded DCCA program to a matched comparison group of students enrolled in construction programs prior to the TAACCCT funding. The treatment group consisted of 266 students that were enrolled in the DCCA grant-funded program from Summer 2015-Spring 2017 and received grant-funded services. The comparison group consisted of 266 students who were enrolled from Summer/Fall 2014-Spring 2015 in the same construction programs and did not receive grant-funded services. The authors matched DCCA program participants to similar nonparticipants using propensity scores developed from demographic information. Using administrative, and baseline and follow-up survey data, the study examined differences in completion rates and credential attainment between the treatment and historical matched comparison groups.","Education and skills gains
The study found a significant relationship between the grant-funded DCCA program and credential attainment, with a higher percentage of students enrolled in the DCCA program (51%) earning a credential compared to the students in the comparison group (22%).
The study did not find a significant relationship between the grant-funded DCCA program and completion rates.","Although the authors created a matched comparison group using propensity score matching, they did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention measures of financial disadvantage or education as required in the protocol. These preexisting differences between the groups—and not the DCCA program—could explain the observed differences in outcomes. Additionally, because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the community college) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"Heroes for Hire (H4H) program evaluation final report (Horwood et al. 2018)","Heroes for Hire (H4H) program evaluation final report (Horwood et al. 2018)","Study Type: Causal Impact Analysis","Horwood, T., Campbell, J., McKinney, M., & Bishop, M. (2018). Heroes for Hire (H4H) program evaluation final report. Fairfax, VA: ICF.",,2018,,"August 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Heroes for Hire (H4H) program on education and earnings outcomes.
The authors used a nonexperimental design to compare the education outcomes of H4H program participants to a matched comparison group of students enrolled at the same college, at the same point of entry. and in similar programs of study. The authors also compared the earnings outcomes of students before and after participating in the H4H program.
The study found that H4H participants were significantly more likely to complete the program and attain credit hours than the comparison group. In addition, participation in the H4H program was significantly associated with higher post-enrollment salaries compared to initial enrollment salaries.
The quality of causal evidence presented in this study is low for the education outcomes because the authors did not ensure that the groups being compared were similar before the intervention nor include sufficient control variables. The quality of causal evidence is also low for the earnings outcomes because the authors did not account for trends in outcomes before the intervention. This means we are not confident that the estimated effects are attributable to the H4H program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
A three college consortium in West Virginia (Mountwest Community and Technical College, Blue Ridge Community and Technical College, and Southern West Virginia Community and Technical College) received TAACCCT funds to administer the Heroes for Hire program. The program was designed to help veterans and trade-affected workers receive training and upgrade their skills for employment in various fields. The consortium targeted multiple occupations tailored to each college's job market. These occupations included 38 industry-certified credentials across the healthcare sector and manufacturing service industries with four specific curriculum pathways: Health Information Management (i.e., Medical Billing and Coding), Health Professions (i.e., Patient Care Technician, EKG and Phlebotomy, EMT, and Paramedic Science), Chemical Technology, and Geospatial Technologies. An important element of the H4H program was hands-on applied learning. A variety of professional development was also offered to the faculty and grant staff throughout the life of the grant.","The study took place across the three colleges in a consortium in West Virginia. The authors used a nonexperimental design to compare the education outcomes of students enrolled in one of the H4H pathways programs to a concurrent cohort of students. The treatment group consisted of 444 students that were enrolled in a H4H Pathway program from Fall 2015 through Spring 2018 and received grant funded services. The comparison group consisted of a cohort of 444 students who were enrolled at the same colleges, during the same entry point between Fall 2015-Spring 2018, in similar programs of study, but presumably did not receive grant funded curriculum, resources, or student support services. Treatment students were matched with comparison students using demographic characteristics. The study used administrative data records to examine the impact of the H4H program on educational outcomes by comparing differences between the treatment and comparison cohorts. To examine the earnings outcome, the authors compared pre- and post-enrollment data for the treatment group only.","Education and skills gains
The study found that H4H participation was significantly associated with credit hour attainment, with program participants earning more credit hours than students in the comparison group (43.1 vs. 41.1).
The study also found that H4H participation was significantly related to program completion, with program participants having higher completion rates than comparison students (20.2% vs. 12.2%).
Earnings and wages
The study found that on average, salaries significantly increased by over $6,000 for students enrolled in the H4H pathway programs when comparing their pre- and post-program enrollment salaries.","The authors used propensity score matching to create a comparison group; however, they did not account for preexisting differences between the groups in baseline education outcomes or include sufficient control variables as outlined in the protocol. These preexisting differences between the groups—and not the H4H program—could explain the observed differences in outcomes. Moreover, the authors compared the earnings outcome of participants measured once before and once after they participated in the H4H program. CLEAR’s guidelines require that the authors observe outcomes for multiple periods before the intervention to rule out the possibility that participants had increasing or decreasing trends in the outcomes examined before enrollment in the program. Without knowing the trends before program enrollment, we cannot rule this out. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines. ",,"United States","Causal Impact Analysis","Adult, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Earnings and wages, Education and skills gains","Community College Review Protocol"
"Maine is IT! program evaluation final report (Horwood et al. 2017)","Maine is IT! program evaluation final report (Horwood et al. 2017)","Study Type: Causal Impact Analysis","Horwood, T., Usher, K.., McKinney, M., & Passa, K. (2017). Maine is IT! program evaluation final report. Fairfax, VA: ICF.",,2017,https://www.skillscommons.org/handle/taaccct/13794,"August 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the effects of the Maine is IT! program on education outcomes.
The authors used a nonexperimental design to compare the outcomes of students enrolled in the Maine is IT! program to a matched comparison group of students enrolled at the same college and at the same point of entry in business administration programs of study. Using student tracking data, the authors conducted statistical models to examine differences between the groups.
The study found that compared to non-participating students, students enrolled in the Maine is IT! program were significantly more likely to complete credit hours, be retained in their program of study, and earn a degree.
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Maine is IT! program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
A seven college consortium located in the state of Maine was awarded TAACCCT grant funds to develop the Maine is IT! program. The program was designed to address employment needs in the information technology (IT) field by improving access to IT training opportunities. The Maine is IT! program was comprised of 12 degree programs, 9 certificate programs, and 4 advanced certificate programs. Maine is IT! implemented several evidence-based strategies: 1) creating and enhancing industry recognized certifications, 2) increasing the number and type of stackable credentials, 3) building bridges between non-credit and credit courses, 4) standardizing and expanding credit for prior learning, and 5) introducing innovative approaches such as online and technology enabled competency-based learning strategies and accelerated time to completion strategies.","The nonexperimental study took place across seven community colleges in Maine. The authors matched students enrolled in the Maine is IT! program to similar non-participating students using demographic and enrollment characteristics. The treatment group consisted of 812 students that were enrolled in the Maine is IT! program between Fall 2014 and Spring 2017 and received grant funded services. The comparison group consisted of 812 students who were enrolled in Business Administration programs of study at the same colleges, during the same entry point between Fall 2014-Spring 2017, but did not receive grant funded services. Using student tracking and longitudinal administrative data, the authors conducted statistical models to compare the groups on program retention, credit hour completion, and degree/credential attainment.","Education and skills gain
The study found that students in the Maine is IT! program were significantly more likely than comparison group students to be retained in their program of study (1.34 times more likely).
The study also found that program participation was significantly related to higher odds of credit hour completion, where treatment participants were two times more likely to complete credit hours than participants in the comparison group.
The study found that participation in the Maine is IT! program was significantly associated with an increased likelihood of degree attainment, with higher proportions of treatment students earning a degree relative to the comparison group (18.5% vs. 13.5%).
However, the study did not find a significant relationship between Maine is IT! participation and credential attainment.","Although the authors created a matched group of non-participating students to compare to the Maine is IT! students, they did not account for other factors that could have affected the differences between the treatment and comparison groups, such as a pre-intervention measure of financial disadvantage. These preexisting differences between the groups—and not the Maine is IT! program—could explain the observed differences in outcomes. Additionally, the authors used a comparison group composed of a concurrent cohort of students that attended the same colleges, but were enrolled in different programs of study. The program varying characteristics of the comparison groups creates a confound. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"Central Maine Community College (CMCC) in Auburn, Maine
Eastern Maine Community College (EMCC) in Bangor, Maine
Kennebec Valley Community College (KVCC) in Fairfield, Maine
Northern Maine Community College (NMCC) in Presque Isle, Maine
Southern Maine Community College (SMCC) in South Portland, Maine
Washington County Community College (WCCC) in Calais, Maine
York County Community College (YCCC) in Wells, Maine",,"Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines. ",,"United States","Causal Impact Analysis",Adult,"and Math (STEM) programs, Engineering, Science, Technology, Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Hendricks et al. 2018)","University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Hendricks et al. 2018)","Study Type: Causal Impact Analysis","Hendricks, A., Mitran, A., & Ferroggiaro, E. (2018). University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Contract #: PO-GF-2015-C-0134-DJ). Fairfax, VA: ICF.",,2018,,"August 2020","Community College",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of the University of the District of Columbia Community College (UDC-CC) grant-funded healthcare and hospitality programs on education, earnings, and employment outcomes. This summary focuses on the healthcare program. 
The authors used a randomized controlled trial to compare the education, employment, and earnings outcomes of participants enrolled in healthcare pathway courses and received the contextualized learning program, compared to participants enrolled in the same program, who received the traditional curriculum.
The study found that program participation was significantly related to increased credential attainment.
The quality of causal evidence presented in this report is low because it was based on a randomized controlled trial with unknown attrition and the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the UDC-CC grant-funded healthcare program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The University of the District of Columbia Community College (UDC-CC) developed a program using TAACCCT grant funds to implement training pathways within the framework of the healthcare and hospitality industries. The goal was to develop a program in these high-growth, high-demand industries that addressed the needs of District employers. UDC-CC had previously received a TAACCCT grant that provided improved training pathways, focusing on low-skilled adults, both incumbent and new workers. The currently funded program expanded on the hospitality and healthcare programs by promoting lifelong learning and attainment of stackable credentials to assist participants in pursuing promising career pathways. Enhancements included contextualized and work-based learning, accelerated classes, technology-enabled classes, improved computer labs, staff and instructor development initiatives, and an improved student intake process.","The study took place on the campus of the UDC-CC in Washington, District of Columbia. The authors used a randomized controlled trial to compare the outcomes of the treatment group to the control group. Participants included students enrolled between Summer 2016 and Spring 2018 in one of the seven courses in the healthcare pathway (Electronic Health Records, First Aid & CPR, Introduction to Healthcare, Medical Billing & Coding, Medical Office Administrative Assistant, Medical Terminology & Anatomy, and Nursing Assistant). All participants had the same recruitment and intake process, including the Comprehensive Adult Participant Assessment Systems exam, document check, and discussions with career advisors. The treatment group consisted of 291 students that were enrolled in the healthcare pathway program courses and received contextualized learning. The control group consisted of 188 students who were enrolled in the same healthcare course but received the traditional curriculum. The study used UDC-CC administrative records, follow-up surveys, and unemployment insurance data to examine the impact of the grant-funded healthcare program on education, earnings, and employment outcomes.","Education and skills gains
The study found a significant relationship between grant-funded healthcare program participation and credential attainment, with higher proportions of students in the treatment group earning a certificate in a healthcare program (66%) than students in the control group (55%).
Earnings and wages
The study did not find a significant relationship between grant-funded healthcare program participation and quarterly wages.
Employment
The study did not find any statistically significant relationships between grant-funded healthcare program participation and full- or part-time employment.","The study was a randomized controlled trial with unknown attrition in each group. In cases of high or unknown attrition, a study can receive a moderate causal evidence rating if the analysis controls for possible differences in background characteristics of the treatment and control groups. However, the authors in this study did not control for any such differences between the groups due to the empirical approach used. These preexisting differences between the groups—and not the grant-funded healthcare program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating. Finally, the authors recognize that some contamination occurred between the treatment and control groups, leading to some control participants receiving elements of the treatment curriculum and thus potentially affecting the observed differences between the groups.",,,,"Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.",,"United States","Causal Impact Analysis","Adult, Low-skilled","Basic skills, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Improving the labor market outcomes of U.S. veterans: The long-run effect of the Transition Assistance Program (Li 2018)","Improving the labor market outcomes of U.S. veterans: The long-run effect of the Transition Assistance Program (Li 2018)","Study Type: Causal Impact Analysis","Li, X. (2018). Improving the labor market outcomes of U.S. veterans: The long-run effect of the Transition Assistance Program. New York: Syracuse University.",,2018,,"May 2020",Veterans,,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact the Transitional Assistance Program (TAP) had on outcomes related to employment, earnings, and education.
The study used statistical models to compare veterans who had been offered and received TAP to veterans who were not offered TAP. Data came from the veterans supplement of the Current Population Survey for survey years 1995 to 2010.
The study found positive associations between participation in TAP and employment, earnings, and education.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to TAP; other factors are likely to have contributed to the findings.","Six states pilot tested TAP in military installations in 1990 to respond to job readiness and career development needs increasing because of downsizing military personnel. The U.S. Congress authorized TAP in 1991 as part of the National Defense Authorization Act. It took several years to roll out TAP, beginning in large states and military installations first. Initial rollout began in June 1991 with implementation at 50 installations and over time expanding to 226 installations by September 1993.
Multiple agencies, including the U.S. Departments of Defense, Labor, and Veterans Affairs (VA) oversee TAP, each responsible for delivering specific aspects of the program. Overall, TAP’s intent was to help veterans search for jobs. TAP offers a three- to four-day program, consisting of three parts:
A pre-separation counseling session focused on transitioning and developing personal and professional goals
A three-day employment workshop covering job search assistance, resume writing, and mock interviews
A briefing session to discuss benefits available through the VA, including the GI Bill, Disability Compensation, health care services, and additional career assistance services
An additional workshop is available for veterans with a service-connected disability. This workshop includes more information about Disability Compensation and health care services, as well as Vocational Rehabilitation and Employment programs. Each military branch has the authority and flexibility to adjust services offered within TAP to meet the needs of its personnel.
People who are eligible for TAP include those who have been honorably discharged from having been on active duty for more than 180 days, or less than 180 days if separation was the result of a health condition. People can pursue TAP beginning six months before separation or one year before retirement.","The study used statistical models to compare veterans who had been offered and received TAP to veterans who were not offered TAP. Using an approach called instrumental variables, among other approaches, the author estimated statistical models of measures of employment, earnings, and education that took into account demographic and military characteristics, state of residence, year of separation from the military, and year of the survey. Outcomes were measured at the time of the survey, 10 years after separation.
The study was restricted to male veterans, ages 26 to 64 at the time of the survey, who were separated from the military from 1990 to 1993. Of those veterans who were offered and accepted TAP, the average age was about 41, 20 percent were African American, and all had a high school diploma. Of those veterans who were not offered TAP, the average age was about 40, 12 percent were African American, and 98 percent had a high school diploma.","Employment 
The study found that veterans who participated in TAP were significantly more likely to be employed 10 years after separation from the military than veterans not offered TAP by 7.3 percentage points.
Earnings and wages
The study found that veterans who participated in TAP were significantly more likely to have an annual family income exceeding $40,000 than veterans not offered TAP by about 12 percentage points.
Education and skills gain
The study found that veterans who participated in TAP were significantly more likely to seek an education or training program than veterans not offered TAP by 15.4 percentage points.
The study found that veterans who participated in TAP were significantly more likely to have some college education than veterans not offered TAP by about 15 percentage points.","The author did not account for preexisting differences between the groups before program participation. These preexisting differences between the groups—and not TAP—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Dislocated or displaced worker, Veteran or military","Job search assistance and supportive services, Veterans' reemployment, Other training and education","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-Favorable impacts","Earnings and wages, Education and skills gains, Employment","Veterans Review Protocol"
"Going to war and going to college: Did World War II and the G.I. Bill increase educational attainment for returning veterans? (Bound & Turner 2002)","Going to war and going to college: Did World War II and the G.I. Bill increase educational attainment for returning veterans? (Bound & Turner 2002)","Study Type: Causal Impact Analysis","Bound, J., & Turner, S. (2002). Going to war and going to college: Did World War II and the G.I. Bill increase educational attainment for returning veterans? Journal of Labor Economics, 20(4), 784-815.",,2002,https://www.journals.uchicago.edu/doi/abs/10.1086/342012,"May 2020",Veterans,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the World War II (WWII) G.I. Bill on years of college completed and college completion.
The authors used nonexperimental analyses to compare education outcomes for those eligible and not eligible for the WWII G.I. Bill benefits using data from the 1970 U.S. Census.
The study found that being eligible for WWII G.I. Bill benefits was associated with completing more years of college and a greater likelihood of completing college.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the WWII G.I. Bill; other factors are likely to have contributed to the findings.","The WWII G.I. Bill was passed in 1944. All veterans who served in the military from September 1940 through July 1947 were eligible for the benefits. Benefits covered at least one year of education for all eligible people and one additional month of coverage for each month of active duty, up to 48 months. The benefits covered three or four full years of postsecondary education for most participants. Participants received full payment for tuition, books, and supplies at any U.S. higher education institution along with a stipend to cover living expenses that varied by size of the veteran’s family.","The authors used multiple nonexperimental analyses to estimate the effect of a person’s serving in WWII, meaning the person was eligible for the WWII G.I. Bill benefits, on postsecondary educational attainment. In all analyses, the authors analyzed a sample of White men using a national sample from the 1970 U.S. Census. In the first analytic approach, the authors compared education outcomes (specifically, years of college completed and completing a college degree) for veterans of WWII, who were eligible for GI Bill benefits, with those of nonveterans, who were not eligible for the benefits, born from 1923 to 1928. The second analytic approach compared differences in outcomes with differences in the fraction of the population serving in the military across birth year cohorts, because the changing manpower demands of WWII led to changes in the proportion of the population serving in the military as each cohort turned 18. In this approach, the authors’ preferred analytic models included veterans born from 1923 to 1932 and those born from 1923 to 1938. Across all analyses, the authors did not provide the exact sample sizes.","Education and skill gains
Across both model specifications, eligibility for WWII G.I. Bill benefits was associated with completing more years of college and a higher likelihood of completing college. In the first analytic approach, veterans eligible for the benefits completed 0.52 more years of college and were 10 percentage points more likely to complete college than nonveterans, who were not eligible for the benefits. In the second analytic approach, eligibility for benefits was associated with 0.23 to 0.28 more years of college enrollment and 5 to 6 percentage points higher likelihood of completing college (ranging from 0.30 to 0.52 years, depending on the model specification); with this analytic approach, eligibility was also associated with a higher likelihood of completing college (between 4 and 10 percentage points, depending on the model specification).","The authors accounted for race and gender in their analyses by restricting their sample to White men, but they did not account for age or pre-intervention measures of education and socioeconomic status. Relevant for the first analytic strategy (comparing veterans with nonveterans), most men during the WWII period were enlisted to serve in the military, but those that were not enlisted typically received a deferment for physical or mental disabilities. As a result, veterans and nonveterans during this period might not be comparable, and the authors did not account for these potential differences between the groups.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Veterans' reemployment, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Veterans Review Protocol"
"An evaluation of an urban community college Single Parent and Displaced Homemaker Program  (Doctoral dissertation) (Rice 1999)","An evaluation of an urban community college Single Parent and Displaced Homemaker Program  (Doctoral dissertation) (Rice 1999)","Study Type: Causal Impact Analysis","Rice, L. M. (1999). An evaluation of an urban community college Single Parent and Displaced Homemaker Program  (Doctoral dissertation). Old Dominion University. DOI: 10.25777/tqh7-3c76",,1999,https://digitalcommons.odu.edu/urbanservices_education_etds/50/,"January 2021","Community College",,"Low Causal Evidence",Nonexperimental,,"The study's objective was to examine the impact of the Single Parent and Displaced Homemaker Program on education outcomes by students at an urban community college.
The study used a nonexperimental design to compare the outcomes of program participants to those on the waiting list. Using data from the community college’s Students Information System, the author conducted statistical tests to examine the differences between groups.
The study found that participation in the Single Parent and Displaced Homemaker Program was significantly related to increased retention rates and number of credits taken.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Single Parent and Displaced Homemaker Program; other factors are likely to have contributed.","Single parents, particularly women living in urban areas, experience difficulties in entering and continuing college. In the mid-1980s, the federal government set aside funding for Single Parent Displaced Homemaker programs that provided single parents with supportive services to address their difficulties. An urban community college built a Regional Women's Center in 1994 after receiving Single Parent and Displaced Homemaker Program funds. The Center provided mostly female single parents, single pregnant women, and displaced homemakers, direct support, such as support groups, financial assistance, and crisis intervention, and indirect support, such as job placement and financial aid.","The nonexperimental study was conducted in an urban community college with campuses in five southeastern cities: Portsmouth, Norfolk, Chesapeake, Virginia Beach, and Suffolk. The author used a population of students who had applied to and entered the Center’s Single Parent and Displaced Homemaker Program and a population of students who had applied and were placed on a waiting list between Spring 194 and Fall 1997. To be eligible for the program, students needed to be single mothers or displaced homemakers (although one male was included). All program participants had completed a GED or equivalent, while most had at least one child and were on public assistance.
Random sampling was used to select program participants and students on the waiting list for evaluation. Students who were placed on the waiting list met all the criteria for entrance into the program. The treatment condition was a sample of 100 students who had applied to and been accepted into the Single Parent and Displaced Homemaker program. The comparison group was a sample of 100 students who had applied to and had been placed on the waiting list. Using data from college’s Students Information System, the author compared the outcomes of program participants to those on the waiting list using statistical tests. Outcomes included the number of consecutive semesters completed and number of credits taken.","Education and skills gain
The study found a statistically significant relationship between participation in the Single Parent and Displaced Homemaker Program and college retention rates. Participants were retained an average of 3.29 consecutive semesters after program enrollment, compared to 1.05 semesters for the comparison group.
The study found a statistically significant relationship between participation in the Single Parent and Displaced Homemaker Program and credits taken. Participants took an average of 30.02 credits after program enrollment, compared to 25.9 credits semesters for the comparison group.","The author did not account for preexisting differences between the treatment and comparison groups before program participation, such as age, race/ethnicity, gender, a pre-intervention measure of financial disadvantage, or a pre-intervention measure of education. These preexisting differences between the groups—and not the Single Parent and Displaced Homemaker Program—could explain the observed differences in the outcome. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"Urban, United States","Causal Impact Analysis","Adult, Other barriers, Female, Parent","Employment and Training Services, Training and Education, Capacity building programs, Community college education and other classroom training, Other training and education","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"The heterogeneous impact of conditional cash transfers (Galiani & McEwan 2013)","The heterogeneous impact of conditional cash transfers (Galiani & McEwan 2013)","Study Type: Causal Impact Analysis","Galiani, S., & McEwan, P. J. (2013). The heterogeneous impact of conditional cash transfers. Journal of Public Economics, 103, 85-96. doi:10.1016/j.jpubeco.2013.04.004","Glewwe, P., & Olinto, P. (2004). Evaluating the impact of conditional cash transfers on schooling: An experimental analysis of Honduras’ PRAF program. Unpublished manuscript, University of Minnesota and IFPRI-FCND. Retrieved from http://pdf.usaid.gov/pdf_docs/PNADT588.pdf",2003,https://doi.org/10.1016/j.jpubeco.2013.04.004,"February 2021","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of the Honduran Programa de Asignación Familiar (PRAF-II) conditional cash transfer program on child labor and school enrollment.
The authors used census data from a randomized sample of municipalities to compare the outcomes of children who received the conditional cash transfer to those who did not. The authors used statistical models to estimate program effects.
The study found a significant relationship between the PRAF-II and child labor, with work outside the home decreasing by 3 percentage points and work inside the home decreasing by 4 percentage points. Program participation was also significantly related to an 8-percentage point increase in school enrollment.
The quality of causal evidence presented in this report is low because it is a randomized controlled trial with unknown attrition and the authors did not account for preexisting differences between the groups before program participation. This means we are not confident that the estimated effects are attributable to the Programa de Asignación Familiar; other factors are likely to have contributed.","The Honduran Programa de Asignación Familiar (PRAF), or Family Allowance Program, was a conditional cash transfer program that began in the early 1990s. The second phase of the program, PRAF-II, began in the late 1990s with support from the Inter-American Development Bank. PRAF-II provided two types of cash transfers to households. The education transfer of $50 USD per year was paid for children aged 6-12 who enrolled and regularly attended grades 1-4. The health transfer of $40 USD per year was paid for children under 3 or a pregnant mother if they regularly visited a health center. Households could receive up to three education transfers and two health transfers per year.
In addition to conditional cash transfers paid to households, PRAF-II planned to invest in schools and health centers in the municipalities. Parent associations in primary schools were to receive approximately $4,000 USD per year (dependent on school size) conditional on obtaining legal status and preparing a quality improvement plan. Local health centers were to receive $6,000 USD per year (dependent on their client base) conditional on forming a health team and preparing a budget and proposal. At the end of 2002, however, only 7 percent of the education and 17 percent of the health funds were distributed to municipalities.","The study used groups based on a randomized controlled trial (Glewwe & Olinto, 2004). The 298 municipalities in Honduras were rank ordered, from lowest to highest, based on the average height-for-age z-scores (HAZ scores) of first graders in the country. The HAZ score served as a proxy for poverty. Of the 298 municipalities, 70 were included in the study. The 70 municipalities were stratified based on the HAZ scores (14 municipalities in five quintiles) and randomly assigned to one of three treatment groups or a control group. Group 1 municipalities received the household cash transfers. Group 2 municipalities received the household cash transfers and the school and health center funds. Group 3 municipalities only received the school and health center funds. Group 4, the control group, received no funds. Since the municipality school and health center funds were not distributed as planned, the authors combined groups 1 and 2 (the household cash transfer treatment group – 40 municipalities) and groups 3 and 4 (the control group – 30 municipalities).
The authors used census data that was collected from all 298 municipalities after the program was implemented. The authors merged individual and household data with municipal-level data. Data were analyzed for 120,411 children across the treatment and control groups using regression models. The sample consisted of children 6 to 12 years of age who had not yet completed fourth grade. Child labor was measured by asking if the child (ages 7 and older) worked outside the home the week before the census data were collected or exclusively inside the home. School enrollment was defined as being enrolled in school at the time of the census.","Working children/Child labor
The study found a significant relationship between the PRAF-II and child labor, with work outside the home decreasing by 3 percentage points and work inside the home decreasing by 4 percentage points for children participating in the program.
Education (School participation/enrollment)
The study found that participation in PRAF-II was significantly associated with an 8-percentage point increase in school enrollment.","The authors used data from a randomized sample of municipalities in Honduras. However, they did not provide attrition information and the study was treated as a nonexperimental design for this review. The authors controlled for age, gender, and mother's education in the regression models but did not account for the outcomes at baseline, such as previous school attendance or child labor. These preexisting differences between the groups—and not PRAF-II—could explain the observed differences in outcomes.",,,,"Absense of Conflict of Interest.",,International,"Causal Impact Analysis","Other, Low income","Compensation and Workplace Conditions, Other Worker Protections, Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"The impact of Ethiopia’s Productive Safety Net Program on schooling and child labor (Hoddinott et al. 2009)","The impact of Ethiopia’s Productive Safety Net Program on schooling and child labor (Hoddinott et al. 2009)","Study Type: Causal Impact Analysis","Hoddinott, J., Gilligan, D. O., & Taffesse, A. S. (2009). The impact of Ethiopia’s Productive Safety Net Program on schooling and child labor. Retrieved from http://ssrn.com/abstract=1412291",,2009,http://ssrn.com/abstract=1412291,"February 2021","Child Labor",,"Low Causal Evidence","Comparison Group Design",,"The purpose of the study was to assess the impact of Ethiopia’s Productive Safety Net Program (PSNP) on child labor and school attendance, along with the potential added benefit of participation in the Other Food Security Program (OFSP). This summary focuses on the comparison between the group receiving a certain amount of the PSNP benefit and the comparison group.
The study used a nonexperimental design to compare the outcomes of children ages 6-16 that received the cash transfers with those who did not, based on data from the Food Security Program Survey. Using several demographic characteristics, the authors created a matched comparison group of households who did not receive the benefit to assess the effectiveness of the cash transfer program.
The study found that for boys ages 6-16, there was a significant relationship between households receiving a certain amount of cash transfer and increased school attendance rates.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Productive Safety Net Program; other factors are likely to have contributed.","Ethiopia’s Productive Safety Net Program (PSNP) was implemented to improve food security, and maintain household and community financial assets. The program operated in 262 districts in four regions of Ethiopia: Amhara, Oromia, Tigray, and the Southern Nations, Nationalities and Peoples Region (SNNPR). In the program, household members could earn money (6 birr or $.61 USD) for each day they work on community development projects, such as building roads. Payments were typically paid monthly, but in some areas they were only given once the community development project was complete. The Other Food Security Program (OFSP) was also offered in some localities at the same time as the PSNP. The OFSP offered additional benefits to support food security such as advice on farming crops and access to credit. Communities largely selected households to participate in the PSNP based on whether they were poor and had received food aid in the past, although the specific selection process varied by locale.","The study used a matched comparison group design to understand the effects of the PSNP program on school attendance and child labor for children ages 6-16. Since the OFSP program was expected to also help improve food security, the authors were also interested in assessing the combined effect of participating in both the PSNP and OFSP program. About 18 months after the PSNP began, the Food Security Program Survey was administered. The survey asked households about participation in public safety net programs such as the PSNP and OFSP, child and household characteristics, and children’s schooling and labor outcomes. The survey included questions about the households at the current time of the survey, and also included retrospective questions about the households prior to the initiation of the PSNP. Using propensity score matching, the authors created comparison groups matched on pre-program characteristics such as age, gender, asset levels, and schooling. The study included three treatment groups and comparison groups based on survey responses about household participation in the programs:
Treatment group 1 included households who had received any payment from the PSNP program within the past 12 months. A matched comparison group was developed for that treatment group that included any households that had not participated in PSNP or had participated but not received any payment through that program.
Treatment group 2 included households that received at least 90 birr per household member (or about $9 USD) through the PSNP. The matched comparison group developed for that group included households who did not participate in the PSNP program.
Treatment group 3 included households that received any payment from PSNP and received any type of benefit from OFSP. The matched comparison group included households that did not receive any OFSP benefits, and that was either not a participant of PSNP or was a PSNP participant who received no payments from the program.
All treatment and comparison groups excluded households who received Direct Support, which was a program that provided transfers to the elderly or people with disabilities. After the matched comparison groups were formed, the authors compared school attendance and child labor at 18 months between three treatment groups of those participating in the PSNP and those in a corresponding matched comparison groups who did not participate in the PSNP. Child labor was measured on the survey by asking households for the number of hours that children worked. School attendance was measured by asking households whether or not children attended school at the time the survey was completed. Analyses were separated to determine program effects by gender and child age (ages 6-10, 11-16, and 6-16).","Working children/Child labor
For boys and girls of all ages, the study did not find a significant relationship between PSNP benefit receipt of at least 90 birr per household member and the number of hours worked in domestic labor, agricultural labor or in total labor including paid labor outside the home.
Education (School participation/enrollment)
For boys ages 6-16, the study found that PSNP benefit receipt of at least 90 birr per household member was significantly associated with an increase in school attendance rates.
For girls ages 6-16, the study did not find a significant relationship between PSNP benefit receipt of at least 90 birr per household member and school attendance rates.","While the study used a matched comparison group design, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as prior child labor participation. Also, the matched comparison group was formed using data collected 18 months after the intervention through retrospective survey items; households may not have reported accurately on their baseline characteristics given how much time had passed since the program began.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Compensation and Workplace Conditions, Other Worker Protections, Child labor","Child labor-Low-No impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Productive safety net program and children’s time use between work and schooling in Ethiopia (Woldehanna 2010)","Productive safety net program and children’s time use between work and schooling in Ethiopia (Woldehanna 2010)","Study Type: Causal Impact Analysis","Woldehanna T. (2010). Productive safety net program and children’s time use between work and schooling in Ethiopia. In J. Cockburn & J. Kabubo-Mariara (Eds.). Child Welfare in Developing Countries (pp. 157-209). New York, NY: Springer.","Woldehanna, T. (2009). Productive safety net programme and children’s time use between work and schooling in Ethiopia. Retrieved from https://assets.publishing.service.gov.uk/media/57a08b5340f0b652dd000c04/YL-WorkingPaper-40.pdf",2010,https://rd.springer.com/chapter/10.1007/978-1-4419-6275-1_6,"February 2021","Child Labor",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Agricultural Extension Support Program (AEP) on child labor and schooling outcomes in rural Ethiopia.
The study used a matched-comparison group design. Using data from a household survey, the author compared time spent in child labor and schooling among AEP participants and non-participants.
The study found that AEP participation was significantly related to lower amounts of time spent in paid work and in a combined measure of all work, as well as greater amounts of time spent in school.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Agricultural Extension Support Program; other factors are likely to have contributed.","The government of Ethiopia has historically offered social welfare programs to protect citizens from periods of famine and improve food security. In 2005, they began a new social assistance program called the Productive Safety Net Program (PSNP). The Agricultural Extension Support Program (AEP) was an additional program offered that was around before and during PSNP implementation. AEP provides technical assistance on farming practices, farming support, and access to advanced technologies and fertilizer. The AEP program was available to all local farmers with no restrictions, which applied to most of the PSNP program participants.","The author used a nonexperimental design to compare the labor and schooling outcomes of children in households that participated in the AEP with those who did not, using data from the 2006 Young Lives survey. The Young Lives survey provided data on how much time youth spend in different activities, including schooling and paid work. It also asked about household demographics and participation in social programs since 2002; this information was used to identify participants for the treatment and comparison groups. The author created a matched comparison group based on child and household socioeconomic characteristics. The study included only children who were 12 years old when they took the survey. This included a total of 584 children (104 in the treatment group, 480 in the comparison group). Child labor was measured as the number of hours in a day spent in paid work outside the home, unpaid work outside the home, combined time spent in paid and unpaid work outside the home, time spent in child care and household chores, and a combined measure of all hours worked. Schooling was measured as the number of hours spent in school in a day. The author used statistical models to estimate program impacts.","Working children/Child Labor
The study found a statistically significant relationship between AEP participation and a lower amount of time spent doing only paid work outside the home and in a combined measure of all work (including paid and unpaid work outside the home and child care and household chores).
However, there was no a significant relationship between AEP participation and time spent doing only unpaid work outside of the home, in total time spent in work outside home (whether paid or unpaid), or in child care and household chores.
Education (School participation/enrollment)
The study found a statistically significant relationship between AEP participation and a greater amount of time spent in school.","The authors did not account for preexisting differences between the groups in child labor or schooling outcomes before AEP participation. These preexisting differences between the groups—and not the AEP— could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Compensation and Workplace Conditions, Other Worker Protections, Child labor","Child labor-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"
"Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report (Corporation for a Skilled Workforce & The New Growth Group 2016)","Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report (Corporation for a Skilled Workforce & The New Growth Group 2016)","Study Type: Causal Impact Analysis","Corporation for a Skilled Workforce & The New Growth Group. (2016). Multi-State Advanced Manufacturing Consortium (M-SAMC) TAACCCT Round II grant final evaluation report. Ann Arbor, MI: Corporation for a Skilled Workforce (CSW); Cleveland, OH: The New Growth Group, LLC.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9692/MSAMC%20Final%20Eva…,"April 2021","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Multi-State Advanced Manufacturing Consortium (M-SAMC) program participation on education outcomes. This summary focuses on the findings at Rhodes State College in Ohio.  
	The authors used a nonexperimental design to compare outcomes of M-SAMC program participants with a matched comparison group. 
	The study found that the odds of program completion were significantly higher for participants in the M-SAMC programs compared to participants in the comparison group.
	The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the M-SAMC; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.

The Multi-State Advanced Manufacturing Consortium (M-SAMC) was a consortium of 13 community colleges across 10 states designed to meet the local needs of the manufacturing industry. M-SAMC set out to develop and improve curricula, credentials, instructional design and delivery to contextualize learning, student support, student success, job placement, and partnerships with employers, community-based organizations, and unions. Competency-based education (CBE) was a major effort of the consortium including use of common industry-respected simulators and consortium-based models. All schools had access to industry standard equipment; the Integrated Manufacturing Systems Trainer (IMST), and the first level of consortium developed and delivered equipment-use training in a manufacturing industry-like instructional environment. All schools participated in at least one structured workgroup to develop the models that were later provided to all schools for adoption or adaptation based on their local context and needs.","The nonexperimental study took place at Rhodes State College in Lima, Ohio. The authors compared program completion among 298 M-SAMC program participants and a matched comparison group of 478 participants enrolled concurrently in similar programs of study. Participants in both groups were enrolled between Fall 2012 and Spring 2016. The authors matched M-SAMC program participants to similar nonparticipants using propensity scores developed from baseline demographic information. Data sources included participant intake forms, college databases, state earning records, and post-program completion participant surveys. The authors used a statistical model to examine the odds of program completion for treatment participants versus participants in the comparison group.","Education and skills gain

The study found that M-SAMC participation was significantly related to higher odds of program completion, where treatment participants were twice as likely to complete the program than participants in the comparison group.","The authors created a matched group of non-participants to compare to M-SAMC participants. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention education/training as required by the protocol. These preexisting differences between the groups—and not the M-SAMC program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Employment and Training Services, Training and Education, Community college education and other classroom training","Education and skills gains-Low-Favorable impacts","Education and skills gains","Community College Review Protocol"
"The impact of the Child Grant Programme on child labour and education in Lesotho (Universita di Roma Tor Vergata et al. 2017)","The impact of the Child Grant Programme on child labour and education in Lesotho (Universita di Roma Tor Vergata et al. 2017)","Study Type: Causal Impact Analysis","Universita di Roma Tor Vergata, Centre for Economic and International Studies, the International Labour Organisation, UNICEF and the World Bank. (2017). The impact of the Child Grant Programme on child labour and education in Lesotho. Understanding Children's Work (UCW) Working Paper Series. Rome, Italy: UCW.",,2017,,"May 2021","Child Labor",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of the Child Grant Programme (CGP) on child labor and education outcomes in Lesotho. 
	The study used a randomized controlled trial to compare outcomes between children in households that received the cash transfer with children in households that did not.
	The study found that participation in the CGP was significantly associated with increased school enrollment two years after the introduction of the program. 
	The quality of causal evidence presented in this report is low because it was based on a randomized controlled trial with high attrition and the treatment group received a second grant payment, the Food Emergency Grant, which presents a confound. This means we are not confident that the estimated effects are attributable to Child Grant Programme (CGP); other factors are likely to have contributed.","Established in 2009, the Child Grant Programme (CGP) provided unconditional cash transfers to poor households with children in Lesotho, South Africa. The purpose of the CGP was to reduce child poverty and improve living standards, including reducing malnutrition, increasing school enrollment, and improving the health of orphans and other vulnerable children. The cash transfer was a quarterly payment of the equivalence of US $50-$105 (M360-M750) to eligible households. The payment represented about 16 percent of household expenditures. Program eligibility was determined by two factors: 1) Proxy Means Testing using high vulnerability economic indicator ratings from the National Information System for Social Assistance (NISSA) survey, and 2) community validation. The NISSA divided households into five groups: ultra-poor, very poor, poor, less poor, and better off. Community members selected households from the ultra-poor and very poor NISSA categories with at least one child under 17 to be eligible for the program.","The study was a randomized controlled trial in which 96 electoral divisions (EDs) from 10 Community Counsels were randomly assigned to either the treatment (48 EDs) or control group (48 EDs). Eligible households in the treatment EDs received the unconditional cash transfer quarterly payments of US $50 (M360) regardless of the size of the household and the number of children. For the last quarterly payment, the payment scheme changed so that payments were based on the number of children in the household: US $50 (M360) for 1-2 children; US $84 (M600) for 3-4 children; and US $105 (M750) for 5 and more children. In the control EDs, eligible households would not receive the transfers until after the study ended. 

The study used National Information System for Social Assistance (NISSA) data, household survey data, and community evaluation data. The authors used a stratified random sample for the household survey. The baseline survey was conducted in June-August 2011 and the follow-up survey was conducted in June-August 2013. At baseline, there were 747 eligible households sampled in the treatment EDs and 739 eligible households in the control EDs. Any households lost at follow-up due to changes in household structure were replaced with new households with the highest likelihood of retaining the transfer and were matched to the baseline observation. The analytic sample at follow-up included 706 households in the treatment EDs and 648 households in the control EDs. The authors used sampling weights to adjust for high attrition and estimated impacts using statistical models with controls for child and household characteristics and Community Counsel designation. Outcomes included children’s work participation and hours and days worked in the last seven days in any economic activity; children’s work participation; hours and days in farming and livestock activities; time spent in household chores; and school enrollment.","Working children/Child labor

The study found no significant relationship between participation in the CGP and participation in any economic activity or hours or days worked in any economic activity in the last seven days. 
	The study found no significant relationship between participation in the CGP and participation in any farming/livestock economic activities or hours or days worked in farming/livestock economic activity in the last seven days. 
	The study found no significant relationship between participation in the CGP and time spent on household chores.
Education (School participation/enrollment)

The study found that participation in the CGP was significantly associated with increased school enrollment at follow-up (a 4 percent increase).","Although the study was a randomized controlled trial, it had high attrition at the unit of analysis level (household and children). The household numbers at follow-up indicated a 9 percent overall attrition rate and a 7 percent differential attrition rate between the treatment and control groups. Attrition at the unit of assignment (i.e., EDs) was unknown. A confounding factor was also identified. There was a second unrelated grant, the Food Emergency Grant, that was also provided to the treatment group during the course of the study so the program impacts cannot be attributed with certainty to the CGP.  ",,,,"Absence of conflict of interest.",,International,"Causal Impact Analysis","Other, Low income","Compensation and Workplace Conditions, Other Worker Protections, Child labor","Child labor-Low-No impacts, Education and skills gains-Low-Favorable impacts","Child labor, Education and skills gains","Child Labor Review Protocol"