Body,Title,"Study Type",Citation,AdditionalSources,"Original Publication Date","Original Publication URL","Reviewed by CLEAR","Topic Area","Subtopic Area","Study Evidence Rating","Causal Design",Conclusions,Body,"Features of Intervention","Features of the Study",Findings,"Considerations for Interpreting the Findings","Types of and Outcomes","Study Sites",ProfileFootnotes,"Conflict of Interest","Firm Characteristics","Geographic Setting","Study Type","Target Population","Topic Tags","Domain Effectiveness",Outcome,"Review Protocol"
"The Quantum Opportunity Program Demonstration: Short-term impacts (Schirm et al. 2003)","The Quantum Opportunity Program Demonstration: Short-term impacts (Schirm et al. 2003)","Study Type: Causal Impact Analysis","Schirm, A., Rodriguez-Planas, N., Maxfield, M., & Tuttle, C. (2003). The Quantum Opportunity Program Demonstration: Short-term impacts. Washington, DC: Mathematica Policy Research.",,2003,http://www.doleta.gov/reports/papers/QOP_short_term_impacts.pdf,"September 2014","Opportunities for Youth",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The report’s objective was to evaluate the impact of the Quantum Opportunity Program (QOP)—which includes case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives—on educational attainment and labor market participation approximately five years after program enrollment. Other studies examine longer-term impacts.
About 1,100 9th-grade students from 11 high schools were randomly assigned to the treatment group, which could enroll in QOP, or the control group, which could not participate in QOP activities. The study team administered a survey to treatment and control group members about five years after program enrollment (one year after their scheduled graduation from high school) to collect information on high school and postsecondary educational attainment and labor market outcomes.
QOP increased the likelihood that students earned a high school diploma one year after their scheduled graduation by 7 percentage points. No other employment or educational attainment outcomes were statistically significant.
The quality of the causal evidence presented in this report is low. This means we do not have confidence that the estimated effects are attributable to QOP; other factors likely contributed.","QOP was an intensive program comprised of case management and mentoring, education, developmental activities, community service, supportive services, and financial incentives. Services were provided year-round for up to five years to enrollees who had not yet graduated from high school. After high school graduation, participants received some continued mentoring and assistance applying to postsecondary education or training. According to the program model, case managers were to serve 15 to 25 students, and annual participation goals were 750 hours per enrollee who had not yet graduated from high school. For this evaluation, QOP was operated by seven community-based organizations, each affiliated with one to three high schools (11 high schools in total).
Eligibility requirements for students included being in 9th grade for the first time during the 1995-1996 academic year (except at one site, for which the relevant academic school year was 1996-1997) and being in the bottom two-thirds of the GPA distribution for their school in 8th grade. In addition, the students could not be so physically or learning disabled that, according to the school, the program was inappropriate for them.","Students who met eligibility requirements were randomly selected to participate in the study. Those who consented (about 1,100) were randomly assigned to either the treatment or control group. The treatment group was allowed to participate in QOP activities, whereas the control group was not. Participants in both groups were surveyed about one year after their scheduled graduation from high school—when they were entering their late teens—to collect information on their educational and labor market outcomes. High school transcripts for both groups were also collected.","QOP increased the likelihood that students earned a diploma or a general educational development (GED) diploma by 7 percentage points.
No other employment or educational attainment outcomes were statistically significant using regression-adjusted impact estimates.","Although it was based on a randomized controlled trial, the study had high differential attrition across the study groups: the treatment group’s response rate to the follow-up survey was substantially higher than was the control group’s. This made the study ineligible to receive a high causal evidence rating. In addition, the analysis did not demonstrate equivalence of the two groups on a measure of their financial disadvantage, which is required for studies reviewed in this topic area, nor did it include a control for this characteristic. Therefore, the study could not receive a moderate causal evidence rating.",,"Cleveland, Ohio 
Fort Worth, Texas 
Houston, Texas 
Memphis, Tennessee 
Philadelphia, Pennsylvania
Washington D.C.
Yakima, Washington",,"Absence of conflict of interest: This study was conducted by staff from Mathematica Policy Research, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.",,"United States","Causal Impact Analysis",Youth,"Mentoring, Other training and education, Youth programs","Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Education and skills gains, Employment","Opportunities for Youth Review Protocol"
"A promising start: Initial impacts of Year Up on low-income young adults' careers (Roder & Elliot, 2011)","A promising start: Initial impacts of Year Up on low-income young adults' careers (Roder & Elliot, 2011)","Study Type: Causal Impact Analysis","Roder, A., & Elliott, M. (2011). A promising start: Initial impacts of Year Up on low-income young adults' careers. Economic Mobility Corporation.",,2011,,"April 2014","Opportunities for Youth",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"This report presents preliminary findings for the impacts of the Year Up program, which provides technical skills training and internships to low-income adults ages 18 to 24.
The authors randomly assigned eligible applicants to either be invited to participate in the program immediately (treatment group) or gain entry after 10 months (control group). They compared the outcomes of treatment and control group members for approximately two years after random assignment.
The authors reported many statistically significant and positive impacts of the program in the second year after random assignment. These included impacts on annual earnings and hourly wages.
The quality of causal evidence is low because the randomized controlled trial had high attrition and the study did not show that the groups being compared were equivalent prior to the program. This means we are not confident that the estimated effects are attributable to Year Up. Other factors are likely to have contributed.","Year Up is a nonprofit organization headquartered in Boston, Massachusetts. It was founded by a former software entrepreneur with the goal of providing a year of training and work experience to urban young adults. Participants in the Year Up program must be low-income and ages 18 to 24. A chief feature of the program model is six months of technical skills training in information technology and investment operations fields followed by a six-month internship with a top company in the region. The program also includes classes in business writing, communication, and professional skills training. Participants receive a weekly stipend and the support of social workers and mentors throughout the program.","In summer 2007, eligible Year Up applicants were randomly assigned to either be invited to participate in the program immediately (treatment group) or be placed on a waiting list to gain entry to the program after 10 months (control group). In total, 135 applicants were assigned to the treatment group and 60 to the control group. A follow-up survey was administered to both groups to capture post-program earnings and employment outcomes; the last survey was administered 24 to 30 months after random assignment.","During the first year after random assignment, the Year Up group’s earnings lagged those of control group members because most members of the Year Up group were participating in program-related activities. However, by two years after random assignment, Year Up participants were earning more than control group members by a statistically significant margin ($15,082 versus $11,621).
During the second year after random assignment, Year Up and control group members had similar levels of employment, but the Year Up group held higher-paying jobs, on average, than the control group ($12.58 per hour versus $10.32 per hour). 
The two groups were equally likely to be attending college two years after random assignment.","Although this study was based on a randomized controlled trial, which can provide the highest possible causal evidence if well-implemented, the trial suffered from high differential attrition across the study groups. In other words, outcomes were available for a larger proportion of Year Up members than control group members. This could bias the estimated effects of the program if characteristics of the control group are systematically related to responding to the follow-up survey. Therefore, this study cannot receive a high causal evidence rating.
In addition, the study did not demonstrate that the groups being compared were equivalent at the time of program application, nor was it clear from the study whether the authors included controls for baseline characteristics in their analysis approach. Attempts to reach the authors to clarify these issues were unsuccessful. Therefore, this study cannot receive a moderate causal evidence rating in the absence of additional information from the authors.",,"Year Up sites in three cities participated in the evaluation: 
Boston, Massachusetts 
New York City
Providence, Rhode Island",,,"Finance and insurance","United States, Urban","Causal Impact Analysis","Youth, Low income","Mentoring, Work based and other occupational training, Youth programs","Earnings and wages-Low-Mixed impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Opportunities for Youth Review Protocol"
"YO! participants’ employment and earnings (Callahan Consultants, Inc. n.d.)","YO! participants’ employment and earnings (Callahan Consultants, Inc. n.d.)","Study Type: Causal Impact Analysis","Callahan Consultants, Inc. (n.d.). YO! participants’ employment and earnings. Ellicott City, MD: Author.",,2007,http://www.yobaltimore.org/documents/Impact/impact_employment_earnings.pdf,"August 2013","Opportunities for Youth",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to determine the effect of the YO! Baltimore initiative, which included a diverse set of programs offering comprehensive educational, employment preparation, support, and leadership development services to youth ages 14 to 21.
The authors compared employment and earnings outcomes for participants and nonparticipants in YO! Baltimore.
The study found that participants had increased earnings and employment after enrollment.
The quality of the causal evidence presented in this study is low. This means we are not confident that the effects estimated in this study are attributable to the YO! Baltimore program.","YO! Baltimore provided education, employment, support, and leadership development services to out-of-school youth, or those at risk of dropping out of school, between the ages of 14 and 21. Services were provided through locations in five centers and four high schools in Baltimore, Maryland. YO! Baltimore is one of several programs in the Youth Opportunity Grant Initiative funded by the Workforce Investment Act.","The authors assessed the impact of YO! Baltimore by comparing the outcomes of the treatment group, composed of 331 program enrollees who completed at least one program activity and obtained employment after program enrollment, with the outcomes of a comparison group, composed of 794 program enrollees who did not complete any program activities. Members of the treatment and comparison groups lived in the Baltimore Empowerment Zone, making them eligible for YO! Baltimore services.","The study found that the average post-enrollment earnings (constructed as the ratio of total post-enrollment earnings of group members to the total number of group members) were higher for members of the treatment group than for members of the comparison group.
Members of the treatment group worked a greater percentage of quarters after enrollment than members of the comparison group did.
The authors did not conduct testing to determine the statistical significance of these findings.","The treatment and comparison groups were not similar on demographics or educational attainment before program enrollment; the analysis did not account for these differences between the groups, and they could have influenced the findings. The way the comparison group was created—by focusing on youth who were eligible but never received a service—likely introduced bias to the estimates, because individuals who were more motivated or higher achieving were more likely to receive services (which is voluntary) and to have better labor market outcomes even in the absence of the intervention.",,,,,,"United States, Urban","Causal Impact Analysis",Youth,"Basic skills, Job search assistance and supportive services, Other training and education, Work based and other occupational training, Youth programs","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Opportunities for Youth Review Protocol"
"Influencing the disability trajectory for workers with serious mental illness: Lessons from Minnesota’s Demonstration to Maintain Independence and Employment (Linkins et al. 2011)","Influencing the disability trajectory for workers with serious mental illness: Lessons from Minnesota’s Demonstration to Maintain Independence and Employment (Linkins et al. 2011)","Study Type: Causal Impact Analysis","Linkins, K. W., Brya, J. J., Oelschlaeger, A., Simonson, B., Lahiri, S., McFeeters, J., Schutze, M., Jonas, J., & Mowry, M. A. (2011). Influencing the disability trajectory for workers with serious mental illness: Lessons from Minnesota’s Demonstration to Maintain Independence and Employment. Journal of Vocational Rehabilitation, 34(2), 107-118.","Linkins, K. W., Brya, J. J., McFeeters, J., Schutze, M., Oelschlaeger, A., Jonas, J. (2010). Minnesota Demonstration to Maintain Independence and Employment: Final outcome evaluation report. Falls Church, VA: The Lewin Group.",2011,,"February 2015","Disability Employment Policy",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The report’s objective was to examine the impacts of the Demonstration to Maintain Independence and Employment (DMIE) in the Minnesota site. The DMIE was designed to prevent or delay people with disabilities from leaving the workforce and applying for Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI) benefits.
The study was a randomized controlled trial examining DMIE’s impacts on earnings, benefits applications, and activities of daily living (ADLs). Data sources included participant surveys and health care and employment service utilization information. Impacts were evaluated 12 months after sample members’ enrollment.
The study found that program participants did not work more hours per month or earn more over the follow-up period than control group members, but were less likely to apply for SSDI or SSI benefits. The program also decreased the number of ADL limitations.
The quality of causal evidence provided in this study is low. This means that we cannot be confident that the estimated effects are attributable to Minnesota’s DMIE; other factors are likely to have contributed.","The DMIE, which was authorized under the 1999 Ticket to Work and Work Incentives Improvement Act and funded by the Centers for Medicare & Medicaid Services, aimed to delay or prevent reliance on Social Security disability benefits through medical assistance and other supports. Hawaii, Kansas, Minnesota, and Texas received federal funding to design and test innovative programs to obtain this objective. Each state provided health care services beyond existing health insurance coverage, as well as discounted deductibles, premiums, and copayments. DMIE participants also received employment services and a personal case manager.
This profile summarized the 12-month impacts of the Minnesota DMIE—Stay Well, Stay Working (SWSW)—which the Minnesota Department of Human Service in the Twin Cities region (Anoka, Dakota, Hennepin, and Ramsey counties) and Northeast Minnesota (Carlton, St. Louis, Pine, and Lake counties) operated from 2007 through 2009. Eligible participants had to work at least 40 hours a month, have a diagnosis of mental illness from a mental health professional, and not be receiving benefits from or be eligible to participate in any other Minnesota-sponsored public program.","The study was a randomized controlled trial examining DMIE’s impact on earnings, benefits applications, and ADL limitations. The study randomly assigned 1,557 volunteer participants: 1,257 to the treatment group, which could receive SWSW services, and 300 to the control group.
Data for the analysis were drawn from multiple sources, including participant surveys administered by mail at baseline and 12 months after enrollment and health care and employment service utilization data from the Minnesota Medicaid Management Information System. Multivariate regression models estimated SWSW impacts on earnings, SSDI or SSI applications, and difficulty with an ADL.","The study found no statistically significant differences between treatment and control group members in earnings or hours worked per month over the follow-up year.
Among the control group, 14 percent applying for SSDI or SSI benefits during the follow-up year, compared with 7 percent of treatment group members; this difference was statistically significant.
The number of ADL limitations was lower among treatment than control group members.","Although this study was based on a randomized controlled trial, information from supplementary reports shows that the randomization process for Minnesota’s DMIE changed after May 1, 2008. Because the analysis did not account for this change in randomization process, the study could not achieve a high causal evidence rating. In addition, the racial composition of the treatment and control groups differed, and the authors did not control for race in their analyses.
Study results might not generalize to other populations, such as working-age adults diagnosed with mental illness who work fewer than 40 hours per month.",,,,,,"United States","Causal Impact Analysis",Disability,"Health, Disability insurance, Supplemental security income (SSI), Vocational rehabilitation","Earnings and wages-Low-No impacts, Employment-Low-No impacts, Public benefits receipt-Low-Favorable impacts","Earnings and wages, Employment, Public benefits receipt","Disability Employment Policy Review Protocol"
"Using establishment size to measure the impact of Title VII and affirmative action (Carrington et al. 2000)","Using establishment size to measure the impact of Title VII and affirmative action (Carrington et al. 2000)","Study Type: Causal Impact Analysis","Carrington, W.J., McCue, K., & Pierce, B. (2000). Using establishment size to measure the impact of Title VII and affirmative action. Journal of Human Resources, 35(3): 503-523.",,2000,,"August 2015","Employer Compliance",,"Low Causal Evidence","Regression Analysis",,"This study’s objective was to assess the impact of the Civil Rights Act of 1964 (the Civil Rights Act) and subsequent affirmative action executive orders on the employment share of African American employees at larger employers (which were subject to those laws) and the African American–white wage gap.
The authors examined trends in the annual employment shares of African American men, African American women, white men, and white women at employers in different size classes. They also assessed the contributions of individual- and firm-level factors to the gaps in wages earned by African American and white workers over time. Analyses used March Current Population Survey Annual Demographic Survey (1964–1988) linked with County Business Pattern data.
The authors estimated that the Civil Rights Act and affirmative action increased the share of African American men and women employed by large firms. In addition, the authors found that a substantial proportion of African American–white wage convergence during the study period was attributable to African American employees’ growing employment share at large firms.
The quality of causal evidence presented in this report is low. This means that we are not confident that the observed changes in employment are attributable to the Civil Rights Act and affirmative action; other factors are likely to have contributed.","Under Title VII of the Civil Rights Act, employers are forbidden from discriminating against their employees on the basis of race, sex, color, religion, or national origin. The law prohibits discrimination in terms, compensation, working conditions, and other aspects of employment, mandates enforcement by courts, rather than juries, and provides civil penalties for violations, including mandatory remedial hiring policies for employers and reinstatement with back pay awards to victims. It also created the Equal Employment Opportunity Commission to bring class action litigation against employers for discrimination. The Civil Rights Act initially applied to private sector employers with more than 25 employees; since 1972, it has applied to those with more than 15 employees.
Affirmative action Executive Order 11246, signed by President Lyndon B. Johnson in 1965, reinforced prohibitions in the Civil Rights Act of 1964 and earlier executive orders against discrimination based on race, color, religion, and national origin; subsequent Orders and revisions have added sex (1967) and sexual orientation and gender identity (2014) as protected classes. Order 11246 required the federal government and federal contractors with 50 or more employees or $50,000 in contracts to develop written affirmative action plans and ensure equal opportunities for all workers. The Order authorized the Secretary of Labor to establish the Office of Federal Contract Compliance Programs to enforce federal contractors’ compliance.","This study estimated the impact of the Civil Rights Act and affirmative action executive orders, which apply disproportionately to larger employers, on the growing representation of African American and female employees at large firms during the post-civil rights period of 1964–1988. In addition, because large employers typically pay more than small employers, the authors attempted to identify the share of African American–white wage convergence during this period that was attributable to the shifting share of African American employment at large firms.
The authors matched characteristics and counts of private sector, full-time, full-year workers ages 18 to 64, obtained from the March Current Population Survey, to employers by state, industry, and year, as identified from the 1963–1987 County Business Patterns data. They first compared the percentage of employment at small employers (defined as less than a certain threshold, the level of which varied from 20 to 100) relative to large employers, for African American and white men, African American and white women, and white women and men. Analyses estimated employment shares overall and after controlling for differences in employee characteristics, including education and North/South geographic location. They then computed the proporations of the change in wage gaps that were attributable to changes in wage disparities between African American and white workers over time into components explained by changes in human capital, employment at large versus small firms, and other factors.","The study found that, following the Civil Rights Act and affirmative action executive orders, the employment distribution of African American men and women relative to their white counterparts reversed dramatically, from 13 to 16 percent over-representation at small firms in the early 1960s to 10 percent under-representation at small firms by 1975. The reversal was apparent even after controlling for education and potential work experience. This trend was particularly apparent in the South, where pre-1964 employment discrimination was known to be higher. However, the authors did not indicate whether these findings were statistically significant.
In addition, a substantial proportion of African American–white wage convergence during the period could be explained by African American employees’ growing employment share at large firms. For example, an estimated 15 percent of the wage convergence between African American and white men in the South and an estimated 20 percent of the wage convergence between African American and white women in the South were attributable to African American employees’ growing representation at large employers. However, the authors did not indicate whether these findings were statistically significant.","The authors identified the effect of the policies by comparing workers at large employers, which were affected by the policies being examined, to workers at small employers, which were not affected. However, the analysis did not establish that small employers were comparable to large employers before the passage of these laws. Although the authors controlled for worker and firm characteristics, African American and female employment trends at small and large employers may not have been comparable before 1964, so subsequent changes in employment may not have been due solely to the implementation of the policies. The authors also did not address the fact that establishment size was under employers’ control. Some employers (for example, those that found it desirable to have a relatively small share of African American employment) might have kept their establishment size low to avoid being subject to the policies.",,,,,,"United States","Causal Impact Analysis","Black or African American","Affirmative action, Civil Rights Act of 1964, Executive Order 11246 (E.O. 11246)","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Employer Compliance Review Protocol"
"Final evaluation report of the SSI Work Incentives Demonstration Project (Kregel 2006)","Final evaluation report of the SSI Work Incentives Demonstration Project (Kregel 2006)","Study Type: Causal Impact Analysis","Kregel, J. (2006). Final evaluation report of the SSI Work Incentives Demonstration Project. Richmond, VA: State Partnership Systems Change Initiative Project Office, Virginia Commonwealth University.",,2006,http://www.google.com/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=1&cad=rja&ua…,"September 2015","Disability Employment Policy",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to measure employment and earnings impacts for the four State Partnership Initiative (SPI) sites that implemented the Supplemental Security Income (SSI) Work Incentives Demonstration Project, also known as the SSI Waiver Demonstration Project.
The author compared the outcomes of SSI Waiver Demonstration Project participants with those of nonparticipants using data from SPI project offices and Unemployment Insurance (UI) administrative records.
The study found that during the intervention delivery period, SSI Waiver Demonstration Project participants’ earnings increased relative to (1) people at the same sites who chose not to participate in the demonstration and (2) people at other SPI sites that did not implement the demonstration.
The quality of the causal evidence presented in this report is low because the study was a nonexperimental analysis that did not include adequate controls for underlying differences between the groups being compared. This means we are not confident that the estimated effects are attributable to the SSI Waiver Demonstration Project. Other factors are likely to have contributed.","The SSI program provides cash assistance to children and adults with disabilities and senior citizens with very limited incomes and assets. The monthly SSI benefit amount is reduced based on the recipient’s unearned and earned income. The first $20 of unearned income are disregarded; for unearned income, such as from the Social Security Disability Insurance (SSDI) program, beyond that amount, the SSI benefit is reduced by $1 for every $1. Similarly, for earned income, the first $65 of earned income is disregarded, and the SSI benefit is reduced by $1 for every $2 of earned income beyond that amount.
From 1998 to 2004 the Social Security Administration (SSA) and Rehabilitation Services Administration provided five-year funding for 12 SPI states to implement innovative projects to help people with disabilities find jobs. Four of those states—California, New York, Vermont, and Wisconsin—implemented the SSI Work Incentives Demonstration Project (also known as the SSI Waiver Demonstration Project) from May 2001 to September 2004.
The SSI Work Incentives Demonstration Project aimed to improve employment and earnings outcomes among SPI participants who were SSI recipients with disabilities or blindness. The intervention included four waivers to SSI program rules. Participants in each state could volunteer to enroll in the demonstration, making them eligible to take advantage of the waivers offered.

A three-for-four earnings deduction in which, after the $65 earnings disregard, the SSI benefit is reduced by $1 for every $4 of earned income. (Vermont did not offer this waiver.)
The three-for-four earnings deduction was also applied to certain types of unearned income, including UI, worker’s compensation, state disability, and disability-related benefits paid through private insurance plans.
Participants were allowed to keep up to $8,000 in annual earnings (not to exceed 50 percent of gross earnings) in an independence account that was covered by the resource exclusion. There was typically a $2,000 limit for those accounts.
Participants who did not concurrently receive SSDI benefits were not required to complete a medical continuing disability review.","The study included two different nonexperimental analyses to estimate the demonstration’s impact on participants’ employment rates and earnings. For the first analysis, the author tested whether the employment rates and earnings of waiver participants differed from those of waiver nonparticipants in the same states. In the second analysis, the author compared the outcomes of waiver participants to those of SPI project participants in other states that did not offer the waivers. In both analyses, the treatment group was composed of those who volunteered for the waiver program.
For employment outcomes, the study tested for differences between the treatment and comparison groups using a chi-squared statistic. For earnings, the author conducted an analysis of variance to test for differences in wages between the two groups. The model included controls for type of SSA benefit, primary disability, race, previous education and training completed, and employment status at intake.
Most participants’ data were collected through the SPI project offices. Some sites tracked individual employment themselves, whereas others used UI administrative data.","Excluding the New York site, waiver participants experienced an earnings gain of $388.75 over the life of the project, compared with nonparticipants in the same demonstration sites, whose earnings increased by $262.41.
Waiver participants had a wage increase of $387.78 during the project compared with a $313.20 increase for the comparison group of SPI participants in states not participating in the SSI Waiver Demonstration Project.
The authors did not provide enough information to determine the statistical significance of findings on employment.","The author conducted a nonexperimental analysis comparing waiver participants with nonparticipants. It is possible that underlying differences in people’s underlying characteristics could both influence the decision to participate in the waiver program and future employment and earnings outcomes. To guard against this possibility, the Clearinghouse for Labor Evaluation and Research (CLEAR) requires that the analysis includes controls for demographic characteristics, disability type, and earnings history, among others. The author’s analyses did not include controls for previous earnings history, so the analysis cannot receive a moderate causal evidence rating. In addition, it is unclear how the study measured employment, whether the measure was consistent across states, and the study’s follow-up period; this further complicates any interpretation of the study’s findings.",,,,,,"United States","Causal Impact Analysis",Disability,"Supplemental security income (SSI)","Earnings and wages-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Employment","Disability Employment Policy Review Protocol"
"Do disability laws impair firm performance? (Bird & Knopf 2010)","Do disability laws impair firm performance? (Bird & Knopf 2010)","Study Type: Causal Impact Analysis","Bird, R., & Knopf, J. (2010). Do disability laws impair firm performance? American Business Law Journal, 47(1), 145-190.",,2010,,"October 2015","Employer Compliance",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the effect of state disability laws—those related to antidiscrimination and reasonable accommodation for employees with disabilities—on commercial banks’ number of full-time equivalent employees and their average salary.
The authors estimated the impact of antidiscrimination and reasonable accommodation laws using a differences-in-differences regression model with state- and year-fixed effects. Data were obtained from the Federal Reserve Bank of Chicago’s Commercial Bank Database and from Global Insight.
The study found that neither antidiscrimination nor reasonable accommodation laws had a statistically significant effect on bank employment levels during the year the law went into effect or during the subsequent three years. Antidiscrimination laws led to a statistically significant 1.3 percent decline in bank employees’ average salary in the third year after enactment, but not in any other time period.
The quality of causal evidence presented in this report is low. This means we are not confident that the estimated effects are attributable to state disability laws. Other factors are likely to have contributed.","Antidiscrimination laws are those that prohibit employers from discriminating against applicants or employees on the basis of disability. Reasonable accommodation laws require that employers provide reasonable accommodations to help employees with disabilities complete the functions of their jobs, provided the accommodations do not impose undue hardship on the employer. Reasonable accommodations may include adjusting work schedules or equipment, providing qualified readers or interpreters, or modifying facilities to improve accessibility. States differ in how they define disability. By 1980, 38 states had enacted antidiscrimination laws and by 1990, almost all states had enacted such laws. Most also had some form of reasonable accommodation provision by 1990.","This study examined the effect of state antidiscrimination and reasonable accommodation laws for employees with disabilities at commercial banks in all 50 U.S. states from 1976 to 1988. The authors estimated a differences-in-differences regression with state- and year-fixed effects to measure the impact of state disability laws on the number of full-time equivalent employees and the average salary at the firm. Average salary was calculated as total salary plus bonuses paid divided by the number of full-time equivalent employees. The study looked at the effect of states having both antidiscrimination and reasonable accommodation laws and those having only antidiscrimination laws relative to having neither type of law. The authors considered effects at five points in time: the year before the relevant law went into effect, the year the law went into effect, and each of three years after the law went into effect. The authors estimated separate regressions for each outcome variable for each of these points in time. Control variables included time trends by state, an indicator for interstate banking legality, gross state product, and one- and two-year lagged values of the dependent variables.
To be eligible for inclusion in the study, banks must have reported 21 or more full-time employees and complete observations in all sample years. The number of banks in the sample ranged from 5,010 to 8,512 across sample years. Data on banks came from the Commercial Bank Database curated by the Federal Reserve Bank of Chicago. Data on state economic factors were obtained from Global Insight, a private firm.","The study found that neither state antidiscrimination laws nor state reasonable accommodation laws had a statistically significant effect on commercial bank employment levels at any time from 1976 to 1988.
Antidiscrimination laws led to a statistically significant 1.3 percent decline in bank employees’ average salary in the third year after enactment, but not in any other time period.","The authors did not control for average characteristics of banks’ employees, such as disability status, gender, or race, and thus did not establish baseline comparability of banks that were and were not covered by state disability laws. If banks’ workforce profiles differed systematically across states, the outcomes of treatment and comparison banks might have reflected differences in workforce characteristics in addition to the effect of state disability laws.
It is also important to note that the authors conducted five regression analyses for each of four outcome variables without controlling for multiple comparisons. Performing multiple statistical tests on related outcomes makes it more likely that some impacts will be found statistically significant purely by chance and not because they reflect program effectiveness. Thus the statistical significance of the average salary finding is likely overstated.",,,,,"Finance and insurance","United States","Causal Impact Analysis",Disability,"Americans with Disabilities Act (ADA)","Earnings and wages-Low-Unfavorable impacts, Employment-Low-No impacts","Earnings and wages, Employment","Employer Compliance Review Protocol"
"An examination of the utility of training: Relationships with turnover and promotion (Wagner 2004)","An examination of the utility of training: Relationships with turnover and promotion (Wagner 2004)","Study Type: Causal Impact Analysis","Wagner, S. (2004). An examination of the utility of training: Relationships with turnover and promotion. A thesis presented to the Department of Psychology and Special Education, Emporia State University, [1-69].",,2004,https://esirc.emporia.edu/bitstream/handle/123456789/995/Wagner%202004.pdf?sequ…,"December 2015","Low-Income Adults",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to explore the relationship between employee training investment, promotion strategies, and employee turnover in credit unions in the United States.
Using data from a 13-item survey, the author compared employee turnover rates and promotions across organizations with differing human resource management practices, such as employee training programs and promotion strategies.
The study found no statistically significant relationships between employee training investment or turnover and the human resource management practices examined.
The quality of causal evidence presented in this study is low. This means that we would not be confident that any estimated effects would be attributable to the human resource management practices examined. However, the study found no statistically significant effects.","Strategic human resource management involves designing and implementing policies and practices that support employees in achieving an organization’s objectives and, in turn, improving overall organizational health. Such practices are often viewed as an investment in both the individual and the organization. A common strategy involves employee training, an organized effort to encourage learning through instruction. Training programs can take many forms and target a variety of topics, such as general or firm-specific skills. Another such practice is promoting employees from within the company to fill vacancies, rather than searching outside of the company for suitable employees.","The study estimated the relationship between employer human resource management strategies and rates of employee turnover and promotions in 26 credit unions. The author developed a 13-item training and development survey to collect information on training programs and hiring practices used by the credit unions, including frequency and costs of training, yearly turnover rate, and promotion strategies. The author attempted to recruit human resource executives representing 500 credit unions with $100 million or more in total assets throughout the United States. Thirteen of them agreed to participate, completing the survey described above. Due to the low response rate, 200 smaller credit unions ($25 to $50 million in assets) were sampled, and 13 of these sites completed the survey, for a total sample size of 26 credit unions.
The author compared the outcomes of credit unions that had one type of human resource policy with firms that had the other type, as reported on the survey. For instance, the turnover rates of credit unions having a promote-from-within hiring strategy were compared with those of credit unions with hire-from-outside strategies.","The study found no statistically significant relationships between the human resource management practices examined and employee turnover or promotion rates.","In this study, human resource outcomes were evaluated among a sample of credit unions that agreed to participate in the study and disclose information about their training and promotion strategies. Credit unions that chose to participate in the study likely differed systematically from those that chose not to participate. For instance, small credit unions were more likely to participate than large ones. The author did not apply any statistical controls that would account for differences between the credit unions being examined or demonstrate that the groups of credit unions were similar on observed characteristics that might be related to the outcomes. Therefore, the observed differences in outcomes might reflect factors other than the credit unions’ human resource management strategies.",,,,,"Finance and insurance","United States","Causal Impact Analysis",,"Other employer services, Other training and education","Employment-Low-No impacts",Employment,"Low-Income Adults Review Protocol"
"An analysis of the impact of affirmative action programs on self-employment in the construction industry (Blanchflower & Wainwright 2005)","An analysis of the impact of affirmative action programs on self-employment in the construction industry (Blanchflower & Wainwright 2005)","Study Type: Causal Impact Analysis","Blanchflower, D.G., & Wainwright, J. (2005). An analysis of the impact of affirmative action programs on self-employment in the construction industry. Working paper no. 11793. Cambridge, MA: National Bureau of Economic Research.",,2005,http://www.nber.org/papers/w11793.pdf,"December 2015","Employer Compliance",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of City of Richmond v. J.A. Croson Co. (referred to as the Croson decision) in 1989, which weakened affirmative action policies, on self-employment, with a focus on the construction industry, by race, ethnicity, and gender.
The study used a nonexperimental analyses to compare self-employment before and after the Croson decision. The authors used national data from the Current Population Survey (CPS) and the 2000 decennial U.S. Census.
The study found that gender disparities in self-employment rates have declined since the Croson decision in all industries, but have declined most in construction. In contrast, racial and ethnic disparities in self-employment have declined since the Croson decision in all industries except construction.
The quality of causal evidence presented in this study is low because the authors did not control for trends in self-employment or earnings before the Croson decision. This means we are not confident that the estimated effects are attributable to the Croson decision. Other factors are likely to have contributed.","In 1989, the U.S. Supreme Court ruled in the case of City of Richmond v. J.A. Croson Co. This decision instituted more stringent tests for the legitimacy of race- and gender-based affirmative action programs in public contracting. The Croson decision required a “strict scrutiny test” that there was actual discrimination to warrant race-based policies; otherwise, such race-based policies were deemed unconstitutional. Although the Croson decision was specific to Richmond, Virginia, other cities’ race-based affirmative action policies were challenged in court following the Croson decision, with many deemed unconstitutional, including in Philadelphia, Pennsylvania; Columbus, Ohio; Fulton County, Georgia (Atlanta); Dade County, Florida (Miami); Cook County, Illinois (Chicago); the District of Columbia; and the states of Florida and Ohio. The authors argue that such tests dampen the effects of affirmative action programs on promoting business formation by minorities, proxied in this study by self-employment rates.","This study used a nonexperimental design to examine self-employment rates and earnings in the construction industry and on average across all industries by race, ethnicity, and gender. The analysis included noninstitutionalized men and women ages 16 and older in all states and industries who were either self-employed or employed in the private sector. The authors estimated regression models comparing the outcomes for different race, ethnicity, and gender groups before (1979 to 1991) and after (1992 to 2004) the Croson decision, controlling for age, education, industry, state, and year. The authors used data from the CPS March Annual Demographic files to estimate statistical models of earnings, with additional controls included for occupation. Data from the CPS Merged Outgoing Rotation Group files and the 2000 decennial U.S. Census were used to estimate models of self-employment.","Across all industries, disparities in self-employment rates by race and ethnicity declined somewhat following the Croson decision. However, these disparities changed little in the construction industry following the Croson decision. The authors did not report whether the change before and after the Croson decision was statistically significant.
Disparities in self-employment rates by gender declined following the Croson decision, especially in the construction industry. The authors did not report whether the change before and after the Croson decision was statistically significant.
Disparities in self-employment earnings by race and ethnicity fell in the construction industry after the Croson decision, but increased on average across all industries. In contrast, disparities in self-employment earnings by gender increased in the construction industry, but fell on average across all industries. The authors did not report whether the change before and after the Croson decision was statistically significant.","The authors defined the period before the Croson decision as 1979 to 1991, but the Croson decision occurred in 1989. Thus, some of the effect of the decision might be captured in the last two years of the comparison period as defined by the authors. In addition, the authors did not adequately control for pre-decision trends in self-employment and earnings for the groups being examined; therefore, the estimated effects might reflect pre-existing differences in employment and earnings and not the effect of the court ruling.",,,,,Construction,"United States","Causal Impact Analysis",Other,"Affirmative action","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Employer Compliance Review Protocol"
"Recruitment strategies for gender equity: Lessons from cohort 1 and cohort 2 advance institutions (Zajicek et al. 2011)","Recruitment strategies for gender equity: Lessons from cohort 1 and cohort 2 advance institutions (Zajicek et al. 2011)","Study Type: Causal Impact Analysis","Zajicek, A., Morimoto, S., Terdalkar, A., Hunt, V., Rencis, J., & Lisnic, R. (2011). Recruitment strategies for gender equity: Lessons from cohort 1 and cohort 2 advance institutions. Washington, DC: American Society for Engineering Education.",,2011,http://www.asee.org/file_server/papers/attachment/file/0001/1532/Recruitment_St…,"January 2016","Women in Science, Technology, Engineering, & Math (STEM)",,"Low Causal Evidence","Interrupted Time Series Design",,"The study’s objective was to evaluate the effectiveness of the ADVANCE institutional transformation program (ADVANCE-IT), sponsored by the National Science Foundation (NSF), on the recruitment of female engineering faculty at participating U.S. colleges and universities.
The authors used program implementation and outcome data obtained from seven participating colleges and universities to compare the number of female engineering faculty before and after institutions participated in the intervention.
The study found no statistically significant relationships between participation in the ADVANCE-IT program and the number of female assistant professors at those institutions.
The quality of causal evidence presented in this study is low because the authors conducted no tests of statistical significance and did not observe outcomes for multiple periods before the intervention. This means we are not confident that the estimated effects would have been attributable to the ADVANCE-IT program; other factors are likely to have contributed.","Since 2001, the NSF has invested more than $130 million in support of ADVANCE programs nationwide. ADVANCE programs aim to increase the participation and advancement of women in academic science and engineering careers. The ADVANCE-IT program, in particular, consists of five-year, institution-wide programs that aim to transform recruitment policies and practices, as well as the climates and cultures, of colleges and universities to advance women in STEM positions.","The authors compared the number of female faculty at the assistant professor rank in colleges of engineering before and after the ADVANCE-IT grant period to measure the impacts of targeted recruitment practices developed under the grant. Specifically, the study focused on changes in the number of full-time, tenure-track assistant professors from the start to the end of the grant period (2001–2006) and through the three years following the grant period (2006–2009). Associate and full professor ranks were excluded on the basis that any increase in senior female faculty was likely driven by promotions rather than recruitment. The study evaluated seven first-round recipients of ADVANCE-IT grants that offered engineering degrees from 2001 to 2009: Georgia Institute of Technology; New Mexico State University; University of California, Irvine; University of Colorado at Boulder; University of Michigan, Ann Arbor; University of Washington; and University of Wisconsin, Madison.
In addition to examining the net gain or loss of female faculty during the study period, the authors analyzed each institution’s recruitment practices and programs to identify similarities and differences in efforts among those with larger or smaller absolute gains in the number of female faculty. Data on female full-time, tenure-track engineering faculty at the assistant professor rank were obtained from the Profiles of Engineering and Engineering Technology Colleges published by the American Society for Engineering Education. Information on ADVANCE-related recruitment programs was obtained through grant proposals and final reports submitted by the institutions to the NSF and through the ADVANCE portal website.","The study found no statistically significant relationships between participation in the ADVANCE-IT program and the number of female assistant professors at those institutions.","The authors compared the outcomes of participating institutions measured before and after they participated in the ADVANCE-IT program. CLEAR’s guidelines require that the authors observe outcomes for multiple periods before the intervention to rule out the possibility that participating institutions had increasing or decreasing trends in the outcomes examined before enrollment in the program. That is, if institutions that had increasing numbers of female assistant professors tended to enroll in the program, we would anticipate further increases over time, even if they did not participate in the program. Without knowing the trends before program enrollment, we cannot rule this out.
Further, the authors did not conduct any tests of statistical significance to demonstrate that observed changes in the number of female assistant professors were meaningful and not due to chance. Lastly, none of the analyses controlled for institution-level characteristics such as average age and demographic composition, as required by CLEAR.",,,"1National Science Foundation. “ADVANCE at a Glance.” Available at http://www.nsf.gov/crssprgm/advance/.",,,"United States","Causal Impact Analysis","Female, STEM professional","Science, Technology, Engineering, and Math (STEM) programs, Other disparities or discrimination in employment and earnings","Employment-Low-No impacts",Employment,"Women in Science, Technology, Engineering, & Math (STEM) Review Protocol"
"Long-term follow-up and benefit-cost analysis of the Jobs Program (Vinokur et al. 1991)","Long-term follow-up and benefit-cost analysis of the Jobs Program (Vinokur et al. 1991)","Study Type: Causal Impact Analysis","Vinokur, A., van Ryn, M., Gramlich, E., & Price, R. (1991). Long-term follow-up and benefit-cost analysis of the Jobs Program. Journal of Applied Psychology, 76(2), 213-219.","Caplan, R., Vinokur, A., Price, R., & van Ryn, M. (1989). Job seeking, reemployment, and mental health: A randomized field experiment in coping with job loss. Journal of Applied Psychology, 74(5), 759-769.
Vinokur, A., Price, R., Caplan, R., van Ryn, M., & Curran, J. (1995). The JOBS I preventive intervention for unemployed individuals: Short- and long-term effects on reemployment and mental health. In L. Murphy, J. Hurrell, S. Sauter, & G. Keita (Eds.), Job stress interventions (pp. 125-138). Washington, DC: American Psychological Association (NIOSH).",1991,,"March 2016",Reemployment,,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of the Jobs Program on long-term earnings and monthly hours worked.
The authors conducted a randomized controlled trial (RCT). Researchers administered a pre-test two weeks before the intervention and three post-tests at 1, 4, and 30 months after the intervention comparing the outcomes of Jobs Program participants against a control group.
The study found no statistically significant relationships between the Jobs Program and increased long-term earnings or time in work during the follow-up period.
The quality of causal evidence presented in this report is low because the study was an RCT with post-intervention subsampling based on participation in the intervention. In addition, the authors did not include sufficient controls in their analysis. This means we are not confident that any estimated effects would be attributable to the Jobs Program; however, the study found no statistically significant effects.","The Jobs Program consisted of eight three-hour sessions conducted on four mornings of each of two weeks. Sessions were led by male-female pairs of trainers who had received about 80 hours of formal training and were regularly observed by members of the research team. Sessions focused on (1) establishing trust in the trainers, (2) building participants’ self-esteem and motivation, and (3) teaching specific job search skills. Skills targeted included identifying and effectively expressing competencies, using social connections to find job leads, contacting potential employers, preparing applications, and interviewing.
In addition to teaching job search skills, the intervention focused on the mental and emotional well-being of participants. Session time was dedicated to “psychologically inoculating” participants against losing their motivation in the face of prolonged unemployment by teaching problem-solving skills designed to help participants anticipate setbacks and adapt when they arose. Trainers also offered direct encouragement and support during the intervention and provided opportunities for other participants to support one another.","The authors conducted an RCT, randomly assigning 1,122 people to either treatment or control groups. Those in the treatment group were given the opportunity to attend the Jobs Program, whereas those in the control group were provided with self-instructional materials on general job search strategies. Study participants were recruited from four unemployment compensation offices across southern Michigan. The recruitment process limited the study to people who were applying for or receiving unemployment compensation in the state. Individuals were considered ineligible to participate if they were within two years of retirement, expected to be recalled to their previous job, were judged by experimenters as showing clear indications of mental illness, or reported being unemployed for more than four months. Researchers administered a pre-test two weeks before the intervention began and two post-tests one and four months after the intervention ended. Impacts were estimated by comparing unadjusted treatment and control group means.","Earnings and wages
The study found no statistically significant relationships between the Jobs Program and long-term earnings or time in work during the follow-up period.","Attrition is difficult to discern in this study because the authors did not provide sample sizes by treatment condition at each follow-up and presented different sample sizes in different studies describing the same experiment. However, the fact that only a subset of participants who were assigned to the Jobs Program but dropped out were sampled for the follow-up surveys means that random assignment was compromised and the study cannot receive a high causal evidence rating. This is because the decision to participate in Jobs occurred after random assignment; to preserve the integrity of the random assignment process, the authors would have had to attempt to survey all of those randomly assigned to the Jobs Program, including all those who did not ultimately participate in it.
Compromised RCTs can receive a moderate causal evidence rating if they include controls for demographic and pre-intervention variables specified in CLEAR’s topic area review protocol. Because the study did not include all the required controls in the analysis, it cannot receive a moderate causal evidence rating. In particular, the study did not include controls for employment stability, pre-intervention employment, or earnings measured more than one year before random assignment.",,,,,,"United States","Causal Impact Analysis","Dislocated or displaced worker, Unemployed","Job search assistance and supportive services","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Reemployment Review Protocol"
"Employment growth from public support of innovation in small firms (Link & Scott 2012)","Employment growth from public support of innovation in small firms (Link & Scott 2012)","Study Type: Causal Impact Analysis","Link, A., & Scott, J. (2012). Employment growth from public support of innovation in small firms. Economics of Innovation and New Technology, 21(7), 655-678.",,2012,,"April 2016","Entrepreneurship and Self-Employment",,"Low Causal Evidence","Interrupted Time Series Design",,"The study’s objective was to examine the effects of the Small Business Innovation Research (SBIR) program, a competitive program providing small businesses with funding for technological innovations, on employment growth of SBIR-recipient firms.
The study used regression analysis to compare firms’ actual levels of employment in 2005 with the levels of employment predicted by their characteristics before receiving the Phase II award.
The study found no statistically significant relationships between SBIR funding and firms’ employment.
The quality of causal evidence presented in this report is low. This means we are not confident that the estimated effects are attributable to SBIR funding; other factors are likely to have contributed.","Established in 1982, the SBIR program was a competitive awards-based program with the goal of encouraging technological innovation among domestic small businesses. Each year, federal agencies with external research and development budgets that exceeded $100 million were required to allocate 2.8 percent of their research and development budget to their own SBIR programs. The program was structured in three phases. The objective of the first phase was to enable small businesses to evaluate the technical merit, commercial potential, and feasibility of a proposed project. Phase II awards were intended for businesses to continue the efforts begun in Phase I and further develop their proposed projects. Businesses pursued further work and commercialization in Phase III; however, SBIR funds were not involved in this phase. The study examined the impact of receiving a Phase II award on subsequent firm employment.","The study used data from a survey conducted by the National Research Council in 2005 on 11,214 small firms that received SBIR Phase II awards made from 1992 to 2001. Employment data covered two points in time: when the Phase II award was given and when the survey was conducted in 2005. To estimate the impact of receiving a Phase II award, the authors compared firms’ actual levels of employment in 2005 with the levels of employment predicted by their characteristics before receiving the award.","The study found no statistically significant relationships between SBIR funding and firm employment.","The authors examined only firms that received Phase II awards. Instead of comparing these firms to a comparison group of firms that did not receive awards, they used characteristics of the firms before the award to predict how employment at the firms would have changed in the absence of the award. Because the firm served as its own control, the CLEAR guidelines require that authors examine the outcome at multiple time points before the intervention to rule out the possibility of selection into the intervention due to trends in the outcome. This study measured the number of employees at only one time point before award receipt, so it does not meet the criteria for a moderate or high rating.
Furthermore, the analyses included controls for firm characteristics, but these characteristics might have differed among firms receiving Phase II awards and firms not receiving awards, so the characteristics of firms receiving awards might not predict employment at firms not receiving awards. Any findings could reflect differences in these characteristics as well as any effects of the intervention.
The authors noted that the response rates to the survey data used in the analysis were low (23 to 36 percent), although they attempted to control for nonresponse bias in their model. Thus, findings might not be generalizable to the full sample of firms receiving Phase II awards.",,,,,"Small business","United States","Causal Impact Analysis",Self-employed,"Other training and education","Employment-Low-No impacts",Employment,"Entrepreneurship and Self-Employment Review Protocol"
"Temporary help service firms’ use of employer tax credits: Implications for disadvantaged workers’ labor market outcomes. (Hamersma & Heinrich 2008)","Temporary help service firms’ use of employer tax credits: Implications for disadvantaged workers’ labor market outcomes. (Hamersma & Heinrich 2008)","Study Type: Causal Impact Analysis","Hamersma, S., & Heinrich, C. (2008). Temporary help service firms’ use of employer tax credits: Implications for disadvantaged workers’ labor market outcomes. Southern Economic Journal, 74(4), 1123-1148.","Hamersma, S., and Heinrich, C.J. (2008). Temporary help service firms’ use of employer tax credits: Implications for disadvantaged workers’ labor market outcomes. Discussion paper no. 1335-08. Madison, WI: Institute for Research on Poverty.",2008,,"September 2016","Low-Income Adults",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the effect of subsidies paid to employers for hiring disadvantaged workers—specifically, the Work Opportunity Tax Credit and Welfare-to-Work Tax Credit—on workers’ earnings and employment. It also examined the interaction of the employment subsidies and being employed through a temporary help service.
The study compared total earnings and total quarters employed in the first and second years after starting a job for certified workers—those workers for whom employers could receive subsidies—with those of eligible but uncertified workers. In addition, the study compared outcomes of certified workers employed through temporary help services with those of certified workers hired directly by an employer. The authors matched the groups being compared on observable characteristics using administrative data.
The study found no statistically significant relationships between employer subsidies and workers’ earnings or employment.
The quality of causal evidence presented in this report is low because the study did not adequately account for existing differences between the study groups. This means we are not confident that any estimated effects would be attributable to employer subsidies.","At the time of this study, the Work Opportunity Tax Credit and Welfare-to-Work Tax Credit (which was subsequently merged with the Work Opportunity Tax Credit) provided employers with tax credits for hiring disadvantaged workers. The Work Opportunity Tax Credit provided employers with a one-year tax credit of 40 percent of the employee’s wages, up to a maximum of $2,400, for welfare recipients and other target groups working at least 400 hours per year. The Welfare-to-Work tax credit applied to employers hiring workers who had been welfare recipients for at least 18 months and worked at least 400 hours per year. This two-year tax credit was 35 percent of the employee’s wages in the first year, up to a maximum of $3,500, and 50 percent of wages in the second year, up to a maximum of $5,000. Employers had to request certification of their eligible employees to receive the tax credits. To become certified, workers had to complete paperwork confirming their eligibility, which the employer submitted to the state.","This nonexperimental study examined the total earnings and number of quarters employed in the first and second years after a worker started a job. The authors conducted two analyses. First, they analyzed the effect of the tax credits overall by comparing the outcomes of 444 certified workers with those of 3,525 eligible but uncertified workers; all workers were employed through temporary help services. Second, they analyzed the effect of employment by a temporary help service by comparing the outcomes of 314 certified workers employed through a temporary help service with those of 10,642 certified workers hired directly through an employer. Because the temporary help service firms—rather than the end employers—would collect the tax credit, the authors hypothesized that workers employed by temporary help service firms would have different earnings than workers employed directly by an employer. The authors used a propensity-score matching model to ensure that the workers being compared in both analyses were initially similar on race, age, and gender.","The study found no statistically significant relationships between Work Opportunity Tax Credit or Welfare-to-Work Tax Credit certification and earnings or quarters employed for either the first or second year following a worker starting a temporary help service job.
The study also found no statistically significant relationships between being a certified worker employed through a temporary help service and earnings or quarters employed.","Although the authors used propensity-score matching to make the groups as similar as possible given the data available, the matching model did not include controls for previous earnings or employment of the workers. Although some models included a firm-level measure of the previous year’s average quarterly earnings at the firm, CLEAR requires inclusion of an individual-level measure of employment and/or earnings measured more than 12 months before the intervention in order to meet the guidelines for a moderate causal evidence rating. Therefore, the study could not receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis","Low income","Other employment and reemployment","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Low-Income Adults Review Protocol"
"A randomized controlled trial to improve health among women receiving welfare in the U.S.: The relationship between employment outcomes and the economic recession. (Kneipp et al. 2013)","A randomized controlled trial to improve health among women receiving welfare in the U.S.: The relationship between employment outcomes and the economic recession. (Kneipp et al. 2013)","Study Type: Causal Impact Analysis","Kneipp, S., Kairalla, J. & Sheely, A. (2013). A randomized controlled trial to improve health among women receiving welfare in the U.S.: The relationship between employment outcomes and the economic recession. Social Science & Medicine, 80, 130-140.",,2013,,"November 2016","Low-Income Adults",,"High Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of the Public Health Nurse (PHN) intervention on employment outcomes.
The authors conducted a randomized controlled trial. Researchers used self-reported job beginning and ending dates to determine whether a participant was employed over a nine-month period.
The authors found no statistically significant effect of PHN on any employment in the first nine months after random assignment.
The quality of causal evidence presented in this study is high because it was based on a well-implemented randomized controlled trial. This means we would be confident that any estimated effects would be attributable to PHN and not to other factors. However, the study did not find statistically significant effects.","The PHN program aimed to address the chronic health needs of Temporary Assistance for Needy Families (TANF) clients so those clients’ health concerns would not be a barrier to employment. People in the treatment group met with a public health nurse for a comprehensive health assessment after entering the program. The nurse was a case manager who supported and coordinated access to care and helped clients manage and prevent diseases through both primary care or referrals and health education. Participants in the treatment group could also attend a two-hour information session about Medicaid. In addition, treatment group participants followed up with the nurse three, six, and nine months later to check in about their health and case management needs.","The authors conducted a randomized controlled trial, randomly assigning 432 unemployed female TANF recipients ages 18 to 60 with at least one chronic health condition to either the treatment or control groups. Participation in the study was denied to pregnant women and those receiving disability income. Using self-reported job beginning and ending dates, the authors used estimated regression models comparing the outcomes of treatment and control group members on whether a participant was employed over a nine-month period. Participants were recruited from one rural and one urban welfare transition program in north-central Florida.","The authors found no statistically significant effect of PHN on any employment in the first nine months after random assignment.","For the outcome of interest to this review, the authors used data from the three-, six-, and nine-month surveys, so later outcomes are missing for sample members who completed the six-month follow-up but not the nine-month follow-up, for example. The estimates may therefore underestimate employment because non-respondents at nine months may have found work by that time. However, the authors accounted for days of participation in the study to mitigate this concern.",,,,,,"United States","Causal Impact Analysis","Disability, Low income","Other employment and reemployment","Employment-Low-No impacts",Employment,"Low-Income Adults Review Protocol"
"Working toward Wellness: Early results from a telephone care management program for Medicaid recipients with depression. (Kim et al 2009)","Working toward Wellness: Early results from a telephone care management program for Medicaid recipients with depression. (Kim et al 2009)","Study Type: Causal Impact Analysis","Kim, S., LeBlanc, A. and Michalopoulos, C. (2009). Working toward Wellness: Early results from a telephone care management program for Medicaid recipients with depression. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.","Bloom, D., Redcross, C., Hsueh, J., & Martin, V. (2007). Four strategies to overcome barriers to employment: An introduction to the Enhanced Services for the Hard-to-Employ Demonstration and Evaluation project. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.",2009,http://www.acf.hhs.gov/sites/default/files/opre/working_wellness.pdf,"November 2016","Low-Income Adults",,"High Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impacts of a telephone care management program, Working toward Wellness (WtW), on low-income depressed parents’ employment and earnings.
The study used a randomized controlled trial design, assigning eligible parents to either the WtW program or existing services. Authors used data from a six-month follow-up survey to estimate program impacts, accounting for parents’ characteristics before random assignment.
The study did not find any statistically significant effects of the WtW program on employment or earnings.
The quality of causal evidence presented in this report is high because it was based on a well-implemented randomized controlled trial. This means we would be confident that any estimated effects would be attributable to WtW and not to other factors. However, the study did not find statistically significant effects.","The WtW program was implemented in Rhode Island, by United Behavioral Health, a Medicaid provider. People were eligible for the program if they were parents, were on Rhode Island Medicaid, and passed a screener indicating depressive symptoms.
Individuals in the WtW group received intensive outreach from case managers to enter and remain in treatment. Master’s degree-level care managers placed telephone calls to WtW participants to encourage them to seek treatment for their depression. The care managers also helped coordinate health appointments, encouraged and monitored follow-through on appointments and treatment plans, and provided supplemental information and counseling. During the calls, care managers also asked participants about their employment status and goals, and about barriers to employment, such as needing child care. The average participant had almost monthly contact with her care manager and received nine phone calls over the course of the year.
Participants who resisted seeking professional treatment were offered a structured psycho-educational program that the care manager administered over the phone. The goal of the phone program was to establish a positive relationship between care managers and participants and maintain participants’ engagement with WtW.","From November 2004 to October 2006, evaluators randomly assigned 253 parents to the treatment group and 254 to the control group. Authors collected employment and earnings data from a survey administered six months after random assignment (six months before program end). The authors compared the outcomes of treatment and control group members, accounting for their characteristics before random assignment.","The study did not find any statistically significant effects of the WtW program on employment or earnings.",None.,,,,,,"United States","Causal Impact Analysis","Low income, Parent","Other employment and reemployment","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Low-Income Adults Review Protocol"
"What strategies work for the hard-to-employ? Final results of the hard-to-employ demonstration and evaluation project and selected sites from the Employment Retention and Advancement project. [NYC SACM] (Butler et al 2012)","What strategies work for the hard-to-employ? Final results of the hard-to-employ demonstration and evaluation project and selected sites from the Employment Retention and Advancement project. [NYC SACM] (Butler et al 2012)","Study Type: Causal Impact Analysis","Butler, D., Alson, J., Bloom, D., Deitch, V., Hill, A., Hsueh, J., Jacobs, E., Kim, S., McRoberts, R., & Redcross, C. (2012). What strategies work for the hard-to-employ? Final results of the hard-to-employ demonstration and evaluation project and selected sites from the Employment Retention and Advancement project. (OPRE Report 2012-08.) Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. [NYC SACM]","Martinez, J., Azurdia, G., Bloom, D., & Miller, C. (2009). The Employment Retention and Advancement project: Results from the Substance Abuse Case Management program in New York City. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.",2012,http://www.acf.hhs.gov/sites/default/files/opre/strategies_work.pdf,"December 2016","Low-Income Adults",,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Substance Abuse Case Management (SACM) program on employment, earnings, and public assistance receipt after two years.
The authors attempted to implement a randomized controlled trial, but they used a process that resulted in nonrandom assignment. The study estimated impacts by comparing the regression-adjusted outcomes of the treatment and control groups, using data collected from New York City public assistance records and Unemployment Insurance wage records for New York State.
The study found no statistically significant relationships between SACM and employment, earnings, or public assistance receipt.
The quality of causal evidence presented in this report is low because the assignment mechanism was nonrandom and the authors did not demonstrate that the treatment and control groups were similar. This means we would not be confident that any estimated effects would be attributable to the Substance Abuse Case Management program; other factors are likely to have contributed. However, the study did not find statistically siginificant effects.","The goal of the SACM program was to identify public assistance clients with substance abuse problems and provide them with supportive services so that they could find and maintain employment and self-sufficiency. The New York City-based program screened Temporary Assistance for Needy Families (TANF) recipients for substance abuse problems. Clinically oriented program staff such as social workers then conducted a two- to three-hour substance abuse assessments to determine the type of substance abuse treatment needed and the client’s readiness to participate in employment-related services. Case managers referred clients to treatment and supportive services, monitored clients’ progress, and referred clients to welfare-to-work activities when treatment was complete. Participation in the in-depth assessment and substance abuse treatment was mandatory; failure to attend could result in cancellation of benefits.","The authors designed the study as a randomized controlled trial; however, the referral process resulted in nonrandom assignment. More than 8,800 TANF and Safety Net recipients whose application questionnaires indicated possible substance abuse were referred via an electronic scheduling system to New York City’s Human Resources Administration for more detailed assessments. When the SACM program had available slots, the automated scheduling system referred applicants to the program. Otherwise, applicants were referred to services-as-usual treatment. The full sample was 70 percent male, with 80 percent ages 31 or older. Fewer than one-third had been employed in the previous year, and most had received cash assistance (50 percent) or food stamps (62 percent) over that period.
The authors estimated the impact of SACM using regression analysis. They analyzed data collected from New York City public assistance records and Unemployment Insurance wage records for New York State.","The study found no statistically significant relationships between the SACM program and employment, earnings, or public assistance receipt after two years.","The referral process presented a problem for randomization because new SACM slots always became available in the morning. Thus, in instances of systematic differences between applicants who applied for appointments in the morning and those who applied in the afternoon, the assignment mechanism would not generate equivalent groups. For example, those eager to engage in treatment and return to work could be more likely to appear for scheduling early in the morning. In addition, staff might have known that the probability of assignment to the SACM group was higher in the morning and thus encouraged participants, who in their opinion needed extra help, to show up earlier rather than later. The study authors argued that such probabilities were remote but presented evidence of differences between the groups and acknowledged that the assignment process was not entirely random. Because the integrity of the random assignment process was compromised, the study was reviewed as a quasi-experimental comparison group design.",,,,,,"United States, Urban","Causal Impact Analysis","Low income","Other employment and reemployment, Substance abuse recovery","Earnings and wages-Low-No impacts, Employment-Low-No impacts, Public benefits receipt-Low-No impacts","Earnings and wages, Employment, Public benefits receipt","Low-Income Adults Review Protocol"
"Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report no. 08-024). (Hollenback & Wei-Jang 2008)","Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report no. 08-024). (Hollenback & Wei-Jang 2008)","Study Type: Causal Impact Analysis","Hollenbeck, K. & Huang, W-J. (2008). Workforce program performance indicators for the Commonwealth of Virginia. (Upjohn Institute Technical Report no. 08-024). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.","Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D.J. Besharov & P.H. Cottingham (Eds.), The workforce investment act: implementation experiences and evaluation findings (pp. 371-295). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.
United States Department of Labor, Employment and Training Administration. (2010, March 9). Wagner-Peyser/Labor Exchange. Retrieved from doleta.gov/programs/wagner_peyser.cfm.",2008,http://research.upjohn.org/cgi/viewcontent.cgi?article=1027&context=up_technica…,"December 2016",,,"Low Causal Evidence","Interrupted Time Series Design",,"The study’s objective was to examine the impact of the Wagner-Peyser/Employment Service program on outcomes of employment, earnings, and credential completion.
The study used an interrupted time series (ITS) design to compare the outcomes of participants before and after participating in the Wagner-Peyser program. • The study found that the median earnings of Wagner-Peyser participants were $250 lower in the second quarter after program exit compared to their median earnings before program participation.
The quality of causal evidence presented in this report is low because the authors did not observe outcomes for multiple periods before the intervention. This means we are not confident that the estimated effects are attributable to the Wagner-Peyser program; other factors are likely to have contributed.
This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.","The Wagner-Peyser program, also known the U.S. Employment Service, provides services to help job seekers find employment. The services include job search assistance, referrals, placement services, and reemployment for recipients of unemployment insurance. These services can be accessed with or without the assistance of facility staff.","The authors used an ITS design that compared the outcomes of participants before and after they participated in the Wagner-Peyser program. There were 237,000 Virginia residents who exited the program between July 2004 and June 2005 for whom wage records were available for 35 quarters. Males accounted for 53 percent of participants; 44 percent were white and 39 percent were African American. The majority of program participants were 21 to 50 years old (78 percent) and had a high school degree (74 percent).","Employment. The study found no statistically significant relationship between participation in the Wagner-Peyser program and employment rates during the second or fourth quarters after program exit.
Earnings. The study found that the median earnings of Wagner-Peyser participants in the second quarter after program exit were $250 lower than their median earnings before program entry. The study found no statistically significant relationship between participation in the Wagner-Peyser program and earnings in the fourth quarter after program exit.","The authors compared the outcomes of participants measured before and after they participated in the Wagner-Peyser program; there was no comparison group that did not participate in the program and could represent what might have occurred had individuals who chose to participate in the program not done so. The authors did not observe outcomes for multiple periods before the intervention to rule out the possibility that participants had preexisting increasing or decreasing trends in the outcomes examined. That is, if participants who had increasing employment or credential completion rates and/or earnings tended to enroll in the program, we would anticipate further increases over time, even if they didn’t participate in the program. Without knowing the trends before program enrollment, we cannot rule this out. Therefore, the study receives a low causal evidence rating. The authors note that participation in the program was likely to have been driven by the individual’s previous labor market distress.",,,,,,"United States","Causal Impact Analysis",Adult,"Job search assistance and supportive services","Earnings and wages-Low-Unfavorable impacts, Employment-Low-No impacts","Earnings and wages, Employment","Review Protocol"
"Job rationing in recessions: evidence from work-search requirements. (Toohey 2015)","Job rationing in recessions: evidence from work-search requirements. (Toohey 2015)","Study Type: Causal Impact Analysis","Toohey, D. (2015). Job rationing in recessions: evidence from work-search requirements. Newark, Delaware: University of Delaware.","Toohey, D. (2014). Job rationing in recessions: evidence from work-search requirements. Newark, Delaware: University of Delaware.",2015,,"October 2016",Reemployment,,"Low Causal Evidence",,,"The study’s objective was to examine the impact of required contacts with employers on Unemployment Insurance (UI) claimants’ employment and UI benefit receipt.
The study used a nonexperimental design in which the author compared the labor market outcomes of UI claimants in different states who were subject to different job search requirements between 2001 and 2013.
The study found that an increase in the number of required contacts with employers was associated with decreases in the unemployment rate. Accounting for expected employment growth along with changes in the number of required contacts also decreased the unemployment rate. The study found no statistically significant relationships between the number of required contacts and UI claim duration.
The quality of causal evidence presented in this report is low because the author did not account for differences in the age of UI claimants in the analysis. This means we are not confident that the estimated effects are attributable to required contacts with employers; other factors are likely to have contributed.","UI provides monetary assistance to full-time workers who lose employment through no fault of their own. To balance the benefits of UI with the potential risks of incentivizing continued unemployment, UI claimants typically must demonstrate that they are actively searching for employment each week to remain eligible to receive UI benefits. Although search requirements vary by state and time, most states require UI claimants to contact a minimum number of prospective employers per week. In recent decades, many states implemented more stringent employer contact requirements for UI claimants.","The author used a nonexperimental design to examine the impact of changes in the number of required contacts on unemployment and UI claims duration. Specifically, the author used data from workforce agency publications, the Current Population Survey, Department of Labor quarterly reports, Local Area Unemployment Statistics, and Quarterly Workforce Indicators to examine the impact of different contact requirements on labor market outcomes across states from 2001 to 2013. UI claimants eligible for the study were 25 to 65 years old, unemployed, and actively searching for employment. They also had to have lost their employment or been laid off. The author used information on the number of required contacts with employers, which ranged from zero to five during the study period. The author conducted statistical analyses to compare the outcomes of claimants subject to more required contacts with employers to those with fewer or no requirements on reemployment, unemployment, and UI claims duration.","Employment
The study found that areas with an increase in required contacts experienced a 0.2 percentage point decrease in the unemployment rate when unemployment was below the state median in 2001 (the beginning of the study period).
Similarly, the study found that areas with an increase in required contacts experienced a 0.4 percentage point decrease in the unemployment rate when unemployment was below the state median in the period just before the change in requirements.
Public Benefits Receipt
The study found no statistically significant relationships between required contacts with employers and UI claims duration.","Although the statistical analysis accounted for several factors that could influence unemployment and UI claims duration, such as claimants’ occupation, education, and sex, the author did not account for claimants’ age. Differences in claimants’ age and in the age composition of the workforce over time may partly account for observed differences in outcomes between UI claimants who experienced different employer contact requirements. This possibility is a concern because the study covers a relatively long time period (2001–2013) and includes several recession years. Without explicitly including age in the analysis, the estimated impacts may reflect the effect of differences in age across groups on unemployment rates and UI claim duration rather than the effect of employer contact requirements.",,,,,,"United States","Causal Impact Analysis","Dislocated or displaced worker, Unemployed","RESEA, Unemployment Insurance","Employment-Low-No impacts, Public benefits receipt-Low-No impacts","Employment, Public benefits receipt","Reemployment Review Protocol"
"The impact of earnings disregards on the behavior of low‐income families. (Matsudaira & Blank 2014)","The impact of earnings disregards on the behavior of low‐income families. (Matsudaira & Blank 2014)","Study Type: Causal Impact Analysis","Matsudaira, J. D., & Blank, R. M. (2014). The impact of earnings disregards on the behavior of low‐income families. Journal of Policy Analysis and Management, 33(1), 7-35.",,2014,,"December 2016","Low-Income Adults",,"Moderate Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of earnings disregards—the amount of earned income that is not counted when calculating whether someone is eligible for public benefits—for welfare assistance on the employment, earnings, and public assistance receipt of low-skilled single mothers.
The study used a difference-in-differences approach to compare the outcomes of women before and after changes in the disregard rate within their state relative to women in other states over the same time period.
The study did not find any statistically significant effects of earnings disregards on the outcomes of interest.
The quality of causal evidence presented in this report is moderate because it was based on a well-implemented nonexperimental design; this is the highest causal evidence rating possible for a nonexperimental design. This means we would be somewhat confident that any estimated effects would be attributable to earnings disregards, but other factors might also have contributed. However, the study did not find statistically significant effects.","Following welfare reform in 1996, many states opted through their Temporary Assistance for Needy Families programs to disregard a higher share of the earnings of working women used to calculate their welfare benefits. The increase in earnings disregards effectively reduces the implicit tax rate imposed on earnings, which according to economic theory should induce greater labor supply among low-wage workers. Holding earnings constant, higher earnings disregards should also increase income among welfare recipients by allowing them to retain a larger share of their total earnings.","The study used information on mothers’ income sources in the past year from the Current Population Survey and the Survey of Income and Program Participation from 1985 to 2004. The sample consisted of single mothers between the ages of 18 and 54 who had less than a high school education and were living with children age 18 or younger. The sample was on average 32 years old, 30 percent black, 32 percent Hispanic, and had two children living with them.
The authors conducted a difference-in-difference analysis to compare the employment, earnings, and public assistance receipt of women before and after changes in the disregard rate within their state relative to women in other states over the same time period. This model assumes that parallel trends in outcomes would exist across states absent any changes in earnings disregards. The authors showed the trends in women’s employment and earnings and concluded that the trends were fairly similar across states before welfare reform in 1996. In addition, the model took into account several individual-level and state-level time-varying characteristics and state-specific time trends.","The study did not find any statistically significant effects of earnings disregards on low-skilled single mothers’ employment, earnings, or public assistance receipt.","The authors suggested one reason for the lack of statistically significant findings could be that few employed women took advantage of the higher earnings disregards.",,,,,,"United States","Causal Impact Analysis","Female, Parent, Low income","Other employment and reemployment","Earnings and wages-Low-No impacts, Employment-Low-No impacts, Public benefits receipt-Low-No impacts","Earnings and wages, Employment, Public benefits receipt","Low-Income Adults Review Protocol"
"Return on investment analysis of a selected set of workforce system programs in Indiana. [WIA Youth] (Hollenbeck 2009)","Return on investment analysis of a selected set of workforce system programs in Indiana. [WIA Youth] (Hollenbeck 2009)","Study Type: Causal Impact Analysis","Hollenbeck, K. (2009). Return on investment analysis of a selected set of workforce system programs in Indiana. Indianapolis, IN: Indiana Chamber of Commerce Foundation. [WIA Youth]","Hollenbeck, K. (2011). Short-term net impact estimates and rates of return. In D. J. Besharov & P. H. Cottingham (Eds.), The Workforce Investment Act: Implementation experiences and evaluation findings, (pp. 347-370). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.",2009,http://www.indianachamber.com/images/media/studies/ROIanalysisWorkforceDevelopm…,"March 2017",,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Workforce Investment Act (WIA) Youth Program on the employment and earnings of low-income youths in Indiana State.
The author used a nonexperimental method to compare the short-term (three quarters after program exit) and long-term (seven quarters after program exit) employment and earnings between youth who took part in the WIA Youth Program relative to youth who participated in the WorkOne program.
The study found no statistically significant relationships between WIA Youth Program participants and employment and earnings.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups compared were similar before program participation. This means we would not be confident that any estimated effects are attributable to the WIA Youth Program; however, the study did not find statistically significant effects.
This study also examined the effectiveness of other workforce development programs. Please click here to find CLEAR profiles of those studies.","The WIA Youth Program was authorized by Title I of the Workforce Investment Act of 1998 and was superseded by the Workforce Innovation and Opportunity Act (WIOA), effective in July 2015. The Youth Program services, which remained essentially the same under WIOA, were designed to provide quality employment and training services to eligible workers. Administered through local workforce investment areas, the Youth Program served people 14 to 21 years old through core services; these included job placement assistance, skills assessments, and provision of information on the labor market. In addition, those unable to obtain a job through core services alone could receive intensive services—which included counseling and specialized assessments—and vouchers for attending training. Recipients of public assistance and other low-income people received priority for intensive and training services in local workforce investment areas in which program funds were limited. In addition, some local areas provided supportive services such as child care, transportation, and work-related financial assistance to those who qualified.","The author used a nonexperimental statistical approach called propensity score matching to create a comparison group of people who participated in WorkOne and were similar to WIA Youth Program participants in terms of demographic characteristics, including gender, age, educational attainment, race, and employment and earnings history. The author then compared the two groups on employment and quarterly earnings. The author collected Indiana Workforce Intelligence System records for those who had exited the WIA Youth or WorkOne program in fiscal year 2006. The study included a sample of 1,774 workers who received services through the WIA Youth Program and 272,780 workers who received services through the WorkOne program in Indiana.","The study found no statistically significant relationships between WIA Youth Program participants and employment and earnings.","Although the author accounted for many underlying characteristics of the groups being compared, which could also influence their outcomes, the author’s decision to define the groups based on their date of program exit rather than program entry is problematic. For example, suppose that the WIA Youth and WorkOne participants were on identical wage trajectories before receiving services from their respective programs and that the average length of participation in WIA Youth services was six months, whereas that for WorkOne was one month. At the conclusion of participation, they exited the program.
If we compared the groups’ earnings 6 months after their recorded exit dates, we would observe WIA Youth participants’ earnings about 12 months after they started receiving services and WorkOne participants’ earnings about 7 months after they started receiving services. If both programs were completely ineffective and everyone stayed on their original upward-sloping wage trajectory, it would appear as though the WIA Youth participants earned more 6 months after their exit dates. However, this would not be attributable to receiving WIA Youth services; it would be caused by the different elapsed time across the groups (12 months for WIA Youth participants versus 7 months for WorkOne participants). Therefore, studies defining the groups based on exit date, rather than entry date, cannot receive a moderate causal evidence rating.",,,,,,"United States","Causal Impact Analysis",Youth,"Workforce Investment Act (WIA)/Workforce Innovation and Opportunity Act (WIOA), Workforce Investment Act (WIA)/Workforce Innovation and Opportunity Act (WIOA) training programs, Youth programs","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Review Protocol"
"Does trade adjustment assistance make a difference? (Reynolds & Palatucci 2012)","Does trade adjustment assistance make a difference? (Reynolds & Palatucci 2012)","Study Type: Causal Impact Analysis","Reynolds, K.M., & Palatucci, J.S. (2012). Does trade adjustment assistance make a difference? Contemporary Economic Policy, 30(1), 43-59. doi:10.1111/j.1465-7287.2010.00247.x","Reynolds, K., & Palatucci, J. (2008). Does trade adjustment assistance make a difference? Unpublished manuscript.",2012,,"April 2017",,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Trade Adjustment Assistance (TAA) Program on the employment and earnings of displaced workers in the manufacturing sector from 2003 to 2005.
The study uses a nonexperimental design to compare the outcomes of TAA recipients with a comparison group of nonrecipients who were also displaced from manufacturing jobs during the same period as the treatment group.
The study found no statistically significant relationship between TAA participation and employment or earnings when comparing TAA recipients with displaced workers who did not participate in TAA. However, when comparing the TAA recipients who received training with recipients who did not receive training, the study found a significant relationship between training and employment and earnings.
The quality of causal evidence provided in this study is low because the authors did not sufficiently account for potential differences between the study groups. This means we are not confident that estimated effects are attributable to the TAA program; other factors are likely to have contributed.","Established under the Trade Act of 1974 and amended through the Trade Adjustment Assistance Reform Act of 2002, the TAA Program provided aid to adult workers (18 and older) within an industry or group whose employment had been lost or reduced as a result of foreign trade competition. The available services included employment and case management services, job search assistance and allowances, training, relocation allowances, and some cash income supports. Eligible employees must have lost their jobs or have been notified of unemployment risk because of import competition or shifts in foreign production.","The study authors used a statistical approach called propensity-score matching to compare the outcomes of displaced manufacturing workers receiving TAA with a demographically similar group of displaced workers who did not receive services. The variables used for matching included import sensitivity, intra-industry trade, unionization rate, average industry wage, industry layoffs, state unemployment rate, individual age, gender, education, and prior employment length. The authors used a similar approach in a separate subsample analysis to compare the outcomes of TAA recipients who received training with a comparable group of TAA recipients who did not receive training. The authors assessed impacts on both employment and earnings.","Employment. The study found no statistically significant relationship between participation in the TAA program and employment. However, for the subsample of TAA recipients who received training versus recipients who did not receive training, the study found that those who participated in training were 10 to 13 percentage points more likely to be reemployed than those who did not participate in training.
Earnings. The study found no statistically significant relationship between participation in the TAA program and earnings. However, for the subsample of TAA recipients who received training versus recipients who did not receive training, the study found that those who participated in training were 9 to 11 percentage points less likely to have an earnings loss than those who did not participate in training.","Although the authors used propensity-score matching to create treatment and comparison groups with similar characteristics (including age, gender, employment, and industry-level earnings), the authors did not account for potential differences between the groups in terms of race, ethnicity, or earnings before TAA participation. Therefore, the estimated impacts could reflect differences in the characteristics of the groups rather than the true impact of the TAA program itself.",,,,,,"United States","Causal Impact Analysis","Dislocated or displaced worker","Job search assistance and supportive services, Other training and education","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Review Protocol"
"Evaluation of the strengthening the connections between unemployment insurance and the One-Stop Delivery Systems Demonstration Project in Wisconsin (Almandsmith 2006)","Evaluation of the strengthening the connections between unemployment insurance and the One-Stop Delivery Systems Demonstration Project in Wisconsin (Almandsmith 2006)","Study Type: Causal Impact Analysis","Almandsmith, S., Ortiz Adams, L., & Bos, H. (2006). Evaluation of the strengthening the connections between unemployment insurance and the One-Stop Delivery Systems Demonstration Project in Wisconsin. Oakland, CA: Berkeley Policy Associates.",,2006,https://wdr.doleta.gov/research/FullText_Documents/Evaluation%20of%20the%20Stre…,"October 2018",Reemployment,,"Low Causal Evidence","Regression Analysis",,"The study examined the impact of the Worker Profiling Reemployment Services program for Unemployment Insurance (UI) claimants in Wisconsin on several outcomes: the rate at which the UI claimants went back to work, average earnings, and UI benefit receipt.
 The authors used a matching procedure and several statistical models to compare the outcomes of demonstration and comparison group members, using administrative data.
 The study found that demonstration participants had higher average quarterly earnings, shorter durations of UI benefits, smaller proportions of maximum UI benefits paid, and a lower average total payout of UI benefits. The study found no statistically significant relationship between demonstration participation and employment outcomes.
 The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Worker Profiling Reemployment Services program; other factors are likely to have contributed.","The overall goal of the Worker Profiling Reemployment Services program was to better integrate the functions of the Wisconsin Division of Workforce Solutions’ Job Service, which operates the state’s One-Stop Career Centers (called Job Centers), and the state Division of UI. To reach this goal, the program identified claimants who were likely to exhaust their regular UI benefits and who were likely to need job search assistance services to make a successful transition to new employment, and referred those claimants to reemployment services. The demonstration sites provided reemployment services to claimants in their profiling pools in 2004–2005. The reemployment services included workshops on resume preparation and interviewing skills, an “Introduction to Computers” curriculum, networking as a job search strategy, job referrals, and employment plan reviews. Within the demonstration group, claimants were assigned to two subgroups based on their preparedness for conducting a job search, the quality of their resume and interviewing skills, and employment barriers. Those who were less prepared were assigned to the group that received more intense services.","The Worker Profiling Reemployment Services program was offered in three demonstration sites in Wisconsin: one in Fox Valley and two in Milwaukee. Claimants were directed to either a demonstration or comparison site depending upon the zip code of their residence. The authors matched demonstration participants to comparison group members on demographic characteristics and pre-program employment status and earnings criteria, and then used several statistical models to compare the outcomes of demonstration and comparison group members. This review focuses on the model that included all of the UI claimants (2,180 demonstration and 4,193 comparison) who were invited to participate in the program, regardless of their program uptake. The authors estimated another model that examined the impacts of those demonstration participants who actually participated in the study. Of 2,180 total demonstration claimants, 1,005 (46 percent) chose not to participate in the study, leaving a total of 1,175 participants who did participate and received intervention services.","Employment
There were no significant differences between demonstration participants and comparison group members on employment during the study period (77 percent of the demonstration participants went back to work during the study period, compared with 78 percent of comparison group members) or on employment in the first quarter after a new claim (54 percent for the demonstration group compared with 53 percent for the comparison group).
Earnings 
Demonstration participants reported average quarterly earnings of $3,020 compared with $2,892 for comparison group members. This difference was statistically significant.
UI benefit receipt
The duration of UI benefits for demonstration participants was 0.6 weeks shorter than for comparison group members, a statistically significant difference.
Demonstration participants drew significantly smaller percentages of their total available benefits than did comparison group members (a 2.2 percent difference).
The average total payout for demonstration participants was $147 less than for comparison group members, a statistically significant difference.","The authors reported statistically significant differences in the gender and race/ethnicity of demonstration participants and comparison group members, and did not account for these differences in analyses. These existing differences between the groups—and not the Worker Profiling Reemployment Services program—could explain the observed differences in outcomes.
In addition to reporting results from an analysis of a model that included all claimants who were invited to participate in the study, the authors reported results from an analysis of a model that was restricted only to those claimants who participated in the study. The results of the latter model were qualitatively similar to those of the main model, but none of the findings was statistically significant.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Dislocated or displaced worker, Unemployed","RESEA, Job search assistance and supportive services, Unemployment Insurance","Earnings and wages-Low-Favorable impacts, Employment-Low-No impacts, Public benefits receipt-Low-Favorable impacts","Earnings and wages, Employment, Public benefits receipt","Reemployment Review Protocol"
"Labor market shocks and retirement: Do government programs matter? (Coile & Levine 2007)","Labor market shocks and retirement: Do government programs matter? (Coile & Levine 2007)","Study Type: Causal Impact Analysis","Coile, C., & Levine, P. (2007). Labor market shocks and retirement: Do government programs matter? Journal of Public Economics, 91(10), 1902-1919.",,2007,http://academics.wellesley.edu/Economics/Coile/PDF%20files/coile-levine%20JPubE…,"November 2018","Older Workers",,"Low Causal Evidence","Regression Analysis",,"The study examined the impact of the generosity of unemployment insurance (UI) benefits on retirement decisions.
The study used a statistical model and data from the March Current Population Survey for 1980–2004 to estimate impacts.
The study found no relationship between the generosity of UI benefits and the likelihood of retirement or receipt of UI.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before changes in the generosity of UI benefits were observed. This means we are not confident that the estimated effects are attributable to changes in the generosity of UI benefits; other factors are likely to have contributed.","UI programs offer monetary benefits to eligible workers who meet eligibility requirements. Typically, workers who lost a job through no fault of their own, are actively looking for work, and had a sufficient work history before job loss are eligible for UI benefits. States administer UI benefits with oversight from the U.S. Department of Labor and vary in both the eligibility requirements and the benefits offered. On average, benefits replace approximately 35 percent of a worker’s lost earnings.","The authors used a statistical model to estimate the likelihood of transitioning to retirement based on variations in the maximum UI benefit amount and the ease of meeting work history eligibility requirements, which vary by state and year.","Employment
The study found no statistically significant relationships between changes in the maximum UI benefit amount or work history requirements and the likelihood of retirement.
Public benefit receipt
The study also found no statistically significant relationships between changes in the maximum UI benefit amount or work history requirements and receipt of UI for older workers.","Although the study used regression models that controlled for age and race, the authors did not account for other existing differences between groups being compared in different states and years. These existing differences between the groups—and not UI generosity—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Older worker, Unemployed","Unemployment Insurance","Employment-Low-No impacts, Public benefits receipt-Low-No impacts","Employment, Public benefits receipt","Older Workers Review Protocol"
"The impact of enhanced incentives on vocational rehabilitation outcomes for dually diagnosed veterans (Drebing et al. 2005)","The impact of enhanced incentives on vocational rehabilitation outcomes for dually diagnosed veterans (Drebing et al. 2005)","Study Type: Causal Impact Analysis","Drebing, C. E., Van Ormer, E. A., Krebs, C., Rosenheck, R., & Rounsaville, B. (2005). The impact of enhanced incentives on vocational rehabilitation outcomes for dually diagnosed veterans. Journal of Applied Behavioral Analysis, 38(3), 359-372.",,2005,https://files.eric.ed.gov/fulltext/EJ725227.pdf,"January 2019",Veterans,,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study examined the impact of enhanced incentives within an existing Veteran’s Administration (VA) Compensated Work Therapy (CWT) program on sustained abstinence from substances, transitioning to competitive employment, retention in competitive employment, and wages.
The study was a randomized controlled trial that compared outcomes for the intervention and control groups over 16 weeks. The CWT program’s clinical and financial records were used to assess employment and wage outcomes. Urine screens and participant reports were used to measure substance use outcomes.
The study found that total wages for the enhanced incentives group were significantly higher than for the control group. The study also found that the intervention group abstained from substance use for a significantly longer time than the control group.
The quality of causal evidence presented in this report is low for the employment and earnings outcomes because the authors did not establish that the intervention and control groups were equivalent one year before the intervention in terms of earnings or employment. This means we are not confident the estimated effects on these outcomes are attributable to the enhanced incentives in the CWT program; other factors are likely to have contributed.",,"The study was a randomized controlled trial that examined the effects of an enhanced incentives program incorporated into an existing VA CWT program at the Bedford VA Medical Center. Dually diagnosed veterans younger than age 55 who had had competitive employment in the past 3 years and more than 10 years of education were eligible for the study. A dual diagnosis was defined as those with current drug or alcohol dependence and either a diagnosis of schizophrenia; major depression; or posttraumatic stress, bipolar, or other anxiety disorder. Veterans with a history of significant head trauma were excluded from the study.
The intervention group received financial incentives for completing job search activities and for abstaining from drugs and alcohol. Job search incentives were awarded for creating a resume and attending a job interview. Participants received additional incentives for consecutive weeks of competitive employment, with increasing payments for each additional consecutive week of competitive work. Participants also received monetary incentives for passing drug and alcohol screens throughout the study period. Incentives increased for each consecutive week the screen showed no drug or alcohol use. To be eligible to earn any of these incentives, individuals in the intervention group were required to maintain enrollment in CWT. This means individuals were indirectly incentivized to participate in CWT activities and meetings. In total, participants could earn up to $1,006 for completing all of the activities during the 16-week period. The control group participated in the CWT program without enhanced incentives.
The study included 19 eligible participants. The OnTrak TestCup 5 and the OnSite Alcohol Assay (Roche Diagnostics Inc.) urine screens as well as participant reports were used for the drug and alcohol screen. The CWT program’s clinical and financial records were used to assess employment and wages. Pay stubs, or other similar work documentation, were used to determine competitive work and related durations and hours. The authors used a statistical model to compare the outcomes of intervention and control group members.
The study also examined job search intensity, measured by the Job Search Behaviors Scale. This outcome is not a focus of the review.","Employment
The study found no statistically significant relationship between the intervention and weeks or hours of competitive employment.
Earnings
Receiving enhanced incentives during the course of the CWT program was associated with significantly higher wages over the course of the 16-week period ($4,701 for the intervention group versus $2,796 for the control group).
Health
The study found that the intervention group abstained from substance use for a significantly longer time than the control group.","As noted by the authors, the sample size of the study was very small.
The employment and wage outcomes received a low rating because the randomized controlled trial had high attrition and the authors did not show that the intervention and control groups were equivalent one year before the intervention for these two measures. If individuals typically enter training programs after a period of especially poor employment outcomes, even in the absence of a training program, outcomes would probably improve over time. Without knowing study participants’ employment status or wages one year or more before the intervention, we cannot rule this out.",,,,"Absence of conflict of interest.","Health care and social assistance","United States","Causal Impact Analysis","Veteran or military","Health, Substance abuse recovery, Veterans' reemployment","Earnings and wages-Low-Favorable impacts, Employment-Low-No impacts, Health and safety-Low-Favorable impacts","Earnings and wages, Employment, Health and safety","Veterans Review Protocol"
"An experimental evaluation of a nationally recognized employment-focused offender reentry program (Farabee et al. 2014)","An experimental evaluation of a nationally recognized employment-focused offender reentry program (Farabee et al. 2014)","Study Type: Causal Impact Analysis","Farabee, D., Zhang, S. X., & Wright, B. (2014). An experimental evaluation of a nationally recognized employment-focused offender reentry program. Journal of Experimental Criminology, 10(3), 309–322.",,2014,,"March 2019",Reentry,,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study examined the impact of an employment-focused offender reentry program (STRIVE) operated in Southern California on the education, employment, and recidivism outcomes of a sample of people released from prison or jail between 2008 and 2011.
Study members were randomly assigned to either receive STRIVE services (the treatment) or a list of other resources in the community, but not STRIVE services (the control). The primary data sources were interviews conducted at baseline and 12 months after random assignment and administrative recidivism data provided by the California Department of Justice.
The study found no statistically significant relationships between STRIVE and education, employment, or recidivism 12 months after random assignment. The study also found no statistically significant relationship between STRIVE and recidivism two years after random assignment.
The quality of causal evidence presented in this report is low because it was based on a randomized controlled trial with fewer members of the control group participating in the study at follow-up than members of the STRIVE intervention group (thus, the study has high attrition). The authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects would have been attributable to the STRIVE program if the study had found statistically significant effects.","STRIVE is an employment-focused offender reentry program that has been implemented in more than 25 communities in the United States since 1998. The California STRIVE program examined in this study consisted of job readiness classes for 40 hours a week for three to four weeks, employment placement assistance, and access to a computer lab for resume writing and job search activities. In addition, housing was available to those released within the last 24 hours. Forty-seven percent of those assigned to the treatment group graduated from the STRIVE program. STRIVE is an employment-focused offender reentry program that has been implemented in more than 25 communities in the United States since 1998. The California STRIVE program examined in this study consisted of job readiness classes for 40 hours a week for three to four weeks, employment placement assistance, and access to a computer lab for resume writing and job search activities. In addition, housing was available to those released within the last 24 hours. Forty-seven percent of those assigned to the treatment group graduated from the STRIVE program.","The study site was a single community-based organization providing reentry services in an unnamed community in Southern California. The authors used random assignment to create similar treatment and control groups of previously incarcerated people released within the last six months. Study members completed a baseline interview and then drew a poker chip from a small bag containing one red and one white chip. Those who selected a red chip were assigned to the treatment group and were eligible to receive STRIVE program services, and those who selected the white chip were assigned to the control group and received a list of alternate community resources. The authors then compared the educational attainment, employment, and re-incarceration of treatment and control group members up to one year after random assignment. Other outcomes, including housing stability, drug use, legal status, and health status were examined in the study but were not eligible for review by CLEAR.","Education
The study found no statistically significant relationships between the STRIVE program and high-school or postsecondary completion 12 months after random assignment.
Employment
The study found no statistically significant relationships between the STRIVE program and full-time or part-time employment 12 months after random assignment. 
 Recidivism
The study found no statistically significant relationships between the STRIVE program and re-arrest or re-incarceration measured at 12 months and two years after random assignment.","Although the study was a randomized controlled trial, follow-up data were available for fewer members of the control group than for members of the STRIVE group (thus, the study has high attrition). Because of this, the authors must ensure that the groups being compared were similar before the intervention. In this case, the authors did not ensure that the groups at follow-up were similar in characteristics measured before the intervention or account for potential preexisting differences between the group members. These potential preexisting differences—and not the STRIVE program—could explain the observed outcomes. Therefore, the study receives a low causal evidence rating.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Justice-involved,"Reentry, Other training and education","Education and skills gains-Low-No impacts, Employment-Low-No impacts","Education and skills gains, Employment","Reentry Review Protocol"
"Employment outcomes among African American and white women with disabilities: Examining the inequalities. (Balcazar & Suarez-Balcazar 2015)","Employment outcomes among African American and white women with disabilities: Examining the inequalities. (Balcazar & Suarez-Balcazar 2015)","Study Type: Causal Impact Analysis","Balcazar, F. E., & Suarez-Balcazar, Y. (2015). Employment outcomes among African American and white women with disabilities: Examining the inequalities. Women, Gender, and Families of Color, 3(2), 144-164.",,2015,,"September 2019","Older Workers",,"Low Causal Evidence","Regression Analysis",,"The study examined the impact of adult vocational rehabilitation (VR) services on employment outcomes for African American and white women.
The authors used a statistical model and data from a Midwestern state VR system to compare the odds of employment after completing VR services for older versus younger women.
The study found no significant difference in employment outcomes between older and younger women who completed VR services.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before receiving VR services. This means we are not confident that the estimated effects are attributable to participation in VR services; other factors are likely to have contributed.","VR is a federally funded program that provides services to help people with disabilities prepare for employment. The services include vocational evaluation, education, training, and placement.","The authors used a statistical model and data from a Midwestern state VR system to estimate the odds of employment for women who participated in VR services. The sample was limited to African American and white women as well as to those with a closed case at the time of data collection. The model predicted employment status for women ages 51 to 65 compared with those 18 to 35 and controlled for demographic characteristics, services received, and case expenditures. The analyses were based on a sample of 4,158 women ages 51 to 65 and 6,942 women ages 18 to 35.","Employment
The authors found no significant difference in employment for women ages 51 to 65 compared with women ages 18 and 35.","The study does not establish that there are no existing differences in employment between the treatment and comparison group at baseline, and it does not adequately control for this potential difference between the two groups in the regression model. Existing differences between the groups could have contributed to the estimated effects.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Older worker, Female, Black or African American, White, Disability","Vocational rehabilitation","Employment-Low-No impacts",Employment,"Older Workers Review Protocol"
"Job insecurity, unemployment insurance and on-the-job search. Evidence from older American workers. (Gutierrez 2016)","Job insecurity, unemployment insurance and on-the-job search. Evidence from older American workers. (Gutierrez 2016)","Study Type: Causal Impact Analysis","Gutierrez, I. A. (2016). Job insecurity, unemployment insurance and on-the-job search. Evidence from older American workers. Labour Economics, 41, 228-245. doi:10.1016/j.labeco.2016.05.011",,2016,,"September 2019","Older Workers",,"Low Causal Evidence","Instrumental Variable",,"The study examined the impact of the potential wage replacement rate through Unemployment Insurance (UI) on the probability of transitioning to non-employment.
The study used a nonexperimental design and data from the Health and Retirement Study (HRS) from 1996–2006 and 2010–2012 to identify workers who expected job loss during the baseline survey and who experienced job transitions by the follow-up survey two years later.
The study found no statistically significant relationships between potential wage replacement rates and transitions to non-employment.
The quality of causal evidence presented in this report is low because the analyses do not sufficiently account for other factors that might contribute to the estimated effects. This means we are not confident that the estimated effects are attributable to the potential wage replacement rate through UI; other factors are likely to have contributed.",,"The author used a statistical model (an instrumental variable approach) to determine the effect of the potential replacement rate on non-employment. The potential replacement rate is the rate of lost earnings that UI benefits would cover, as determined by state-level policies on the nominal replacement rate as well as the maximum amount of weekly UI benefits an individual could receive in a year. The study included 10,440 employed men and women ages 50 to 62 from the HRS, a nationally representative survey.","Employment
The study found no statistically significant relationships between potential wage replacement rates and transitions to non-employment.","The outcome of the analyses is the likelihood that an employed individual is looking for another job. Two key predictors of this likelihood are the subjective probability of job loss and the potential wage replacement rate. The author used two instrument variables for these two key predictors: (1) whether the respondent’s company downsized recently; and (2) the generosity of the state’s UI benefits. However, the variable of recent firm downsizing does not satisfy the exclusion restriction for the instrument variable, which requires that firm downsizing should only affect the job search and job transition through a respondent’s subjective assessment of the likelihood of job loss. Employees could foresee that a shrinking firm might offer slower wage growth than another firm, and thus start looking for other jobs with higher expected wage growth, even if the employees do not think they are likely to lose their job. Further, the author notes one reason why the analysis found no statistically significant impact on non-employment is that older workers are unlikely to lose their jobs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Older worker, Employed","Older workers' programs, Unemployment Insurance","Employment-Low-No impacts",Employment,"Older Workers Review Protocol"
"The effect of health reform on retirement. (Levy et al. 2015)","The effect of health reform on retirement. (Levy et al. 2015)","Study Type: Causal Impact Analysis","Levy, H., Buchmueller, T. C., & Nikpay, S. (2015). The effect of health reform on retirement (Research Paper 329). Ann Arbor, MI: Michigan Retirement Research Center.",,2015,https://deepblue.lib.umich.edu/handle/2027.42/116062,"September 2019","Older Workers",,"Low Causal Evidence","Regression Analysis",,"The study’s objective was to examine the impact of the expansion of Medicaid to low-income adults in 2014 on retirement rates of workers ages 55 to 64.
The authors used a nonexperimental analysis to compare those who lived in an expansion state to those who lived in other states. The study used data from the monthly Current Population Survey from January 2005 through July 2015.
The study found no statistically significant relationships between Medicaid expansion and retirement.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the expansion of Medicaid; other factors are likely to have contributed.",,"Following the passage of the Affordable Care Act in 2014, 30 states and the District of Columbia had expanded Medicaid eligibility to low-income, childless adults by July 2015. The authors used a nonexperimental analysis (regression model) to compare the retirement outcomes of older individuals (ages 55 to 64) in expansion and non-expansion states. The study used Current Population Survey data on more than 2,000,000 observations between 2005 and 2015.","Employment
The study found no statistically significant relationships between Medicaid expansion and retirement.","The study is a nonexperimental study that does not account for possible differences in pre-intervention measure of employment between the groups. Differences in the groups could explain the study’s findings.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Older worker, Low income","Health insurance, Older workers' programs","Employment-Low-No impacts",Employment,"Older Workers Review Protocol"
"New evidence on earnings and benefit claims following the changes in the retirement earnings test in 2000. (Song & Manchester 2006)","New evidence on earnings and benefit claims following the changes in the retirement earnings test in 2000. (Song & Manchester 2006)","Study Type: Causal Impact Analysis","Song, J. G., & Manchester, J. (2006). New evidence on earnings and benefit claims following the changes in the retirement earnings test in 2000. (ORES Working Paper 107). Washington, DC: Social Security Administration, Office of Policy, Office of Research, Evaluation, and Statistics. Also available in Journal of Public Economics, 91(3–4), 669–700 (2007).",,2006,,"October 2019","Older Workers",,"Moderate Causal Evidence","Comparison Group Design",,"The study examined the impact of the Senior Citizens Freedom to Work Act of 2000 on employment, earnings, and public benefit receipt outcomes of workers ages 65 to 69.
The study was a nonexperimental analysis and used Social Security Administrative data to estimate the impacts.
The study found that the Senior Citizens Freedom to Work Act of 2000 was associated with significant increases in the rate of Social Security claims for both sets of individuals affected by the law. The study found no statistically significant relationship between the Senior Citizens Freedom to Work Act of 2000 and earnings.
The quality of casual evidence presented in this report for Social Security claims and earnings outcomes is moderate because it was based on a well-implemented nonexperimental design. This means we are somewhat confident that the estimated effects are attributable to the Senior Citizens Freedom to Work Act of 2000, but other factors might also have contributed. The quality of causal evidence presented in this report on employment is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Senior Citizens Freedom to Work Act of 2000; other factors are likely to have contributed.","The Senior Citizens Freedom to Work Act of 2000, starting in 2000, eliminated the Social Security program’s earnings test for people older than the full retirement age (age 65) and allowed people ages 65 to 69 to defer benefits claiming and receive large benefits later. Before 2000, people eligible for Social Security retirement benefits who were ages 65 to 69 received decreased Social Security benefits if they earned more than the earnings test amount, and they were not permitted to defer benefits claiming. The benefits were reduced $1 for every $3 of income above the earnings test amount.","The study is a nonexperimental analysis that examined differences in the rate of Social Security claims, employment, and annual earnings from 2000 to 2003. For each year, the authors used statistical models (difference-in-difference approach) to compare two groups of people who were affected by the Senior Citizens Freedom to Work Act of 2000 (those who turned 65 in that year and those 65 to 69) with people who were not affected by the law (those 62 to 64 and 70 to 72). The study sample included 1,369,819 people ages 62 to 72 who were eligible for Social Security and were not receiving Social Security Disability Insurance or Social Security retirement benefits. The study data came from the Social Security Administration’s 1 percent (active) sample, which tracks a random subsample of individuals who report earnings to the Social Security Administration.","The study found that the Senior Citizens Freedom to Work Act of 2000 was associated with the following outcomes:
Employment
The authors found no clear evidence of the effect of the test’s removal on the overall rate of labor force participation.
Earnings
The authors found no statistically significant relationship between the Senior Citizens Freedom to Work Act of 2000 and earnings.
Public Benefit Receipt
In each of the years, the authors found that Social Security claims for those ages 65 to 69 were significantly higher than claims among those were not affected by the law. Benefit claims were also significantly higher for those who turned 65 in each year compared with those not affected by the law.","The study is a nonexperimental study that demonstrates the treatment and comparison groups had similar rates of Social Security claims and annual earnings before the intervention. There were, however, small pre-intervention trends between the treatment and comparison groups in their employment patterns.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Older worker","Older workers' programs, Other disparities or discrimination in employment and earnings","Earnings and wages-Mod/high-No impacts, Employment-Low-No impacts, Public benefits receipt-Mod/high-Unfavorable impacts","Earnings and wages, Employment, Public benefits receipt","Older Workers Review Protocol"
"Essays on incarceration and labor market outcomes. (Jung 2009)","Essays on incarceration and labor market outcomes. (Jung 2009)","Study Type: Causal Impact Analysis","Jung, H. (2009). Essays on incarceration and labor market outcomes. (Doctoral dissertation.) Retrieved from ProQuest. Accession No. 305059182.","Jung, H. (2014). Do prison work-release programs improve subsequent labor market outcomes? Evidence from the adult transition centers in Illinois. Journal of Offender Rehabilitation, 53, 384-402. doi: 10.1080/10509674.2014.922.158",2009,,"October 2019",Reentry,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of participating in the Adult Transition Centers (ATC) work-release program on employment and earnings outcomes for ex-prisoners in Cook County, Illinois. Prisoners in minimum-security prisons who are nearing their release date and meet certain conditions of criminal history are eligible to apply to transfer to ATCs.
The study used a nonexperimental design that compared changes in outcomes over time between the treatment group and a matched comparison group. The study used administrative data on prison admission and exit dates from the Illinois Department of Corrections and earnings data from the Illinois Department of Employment Security.
The study found no statistically significant relationships between the ATC work-release program and employment or earnings.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we would not be confident that any estimated effects would have been attributable to the ATC work-release program; other factors would likely have contributed, though the study did not find statistically significant effects.","The Illinois Department of Corrections’ ATCs offer eligible inmates an opportunity to live in a secure facility within a residential area to prepare for release on parole. While in the ATC, participants are responsible for performing daily in-house assignments and participating in outside employment, education, life skills, or community service. ATC staff will introduce participants to employers, or program participants can seek employment on their own. Participants meeting certain criteria (including, for example, engaging in 35 hours per week of employment, education, or public service at the first level) can advance to subsequent levels of programming that allow additional privileges (such as leave time to visit family). Eligibility for ATC work-release program services is determined by an application and the number of available beds at the ATC. To be eligible for transfer to ATC, prisoners must be in a minimum-security prison, have no less than two months remaining in their prison term, have not been charged with certain serious crimes including murder or arson, and have no documented involvement in organized crime activities or large-scale narcotics trafficking.","The study used a nonexperimental design to examine impacts of the ATC program on quarterly employment and earnings outcomes after release from prison. The author compared a treatment group of 6,056 eligible inmates who were selected for ATCs in Cook County, Illinois, with a comparison group of 6,136 inmates who were eligible but did not apply or were not selected for ATCs. These sample individuals were released from prison from 1995 to 2003. Administrative data on demographic characteristics and prison admission and exit dates came from the Illinois Department of Corrections and administrative data on quarterly earnings came from the Illinois Department of Employment Security. The author used statistical analysis (regression modeling) to compare the outcomes of treatment and control group members over time, accounting for differences because of demographic characteristics, calendar quarter and quarters relative to prison entry and exit, offense categories, seriousness of crime, and average individual outcomes (fixed effects). Outcomes were measured on average 21 quarters after release from prison.","Employment
The study found no statistically significant relationships between the ATCs and short-term (within two years of release) or long-term (more than two years after release) employment.
Earnings
The study found no statistically significant relationships between the ATCs and short-term (within two years of release) or long-term (more than two years after release) earnings.","The author did not account for existing differences between the groups in employment and earnings trends at least one year before the program. These existing differences between the groups—and not the ATC work-release program—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Justice-involved,Reentry,"Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Reentry Review Protocol"
"Prison-based education and reentry into the mainstream labor market (Tyler & Kling 2007)","Prison-based education and reentry into the mainstream labor market (Tyler & Kling 2007)","Study Type: Causal Impact Analysis","Tyler, J. H., & Kling, J. R. (2007). Prison-based education and reentry into the mainstream labor market. In D. Weiman & S. D. Bushway (Eds.), Barriers to reentry?: The labor market for released prisoners in post-industrial America (pp. 227-256). New York: Russell Sage Foundation.",,2007,https://www.nber.org/papers/w12114,"December 2019",Reentry,,"Low Causal Evidence",,,"The study’s objective was to examine the impact of obtaining a general education diploma (GED) while in prison from 1994 to 2000 in Florida on male prisoners’ post-incarceration employment, earnings, and recidivism.
The authors used a nonexperimental design to compare outcomes for those who obtained a prison-based GED (the treatment group) with those that did not (the comparison group). Drawing on administrative data from the state of Florida and Unemployment Insurance (UI) records, the authors estimated impacts on earnings and compared the means of the two groups for the employment and recidivism outcomes.
The study reported no statistically significant findings on earnings in the first and third years after release or on recidivism or employment outcomes. The study found that, in the second year after release, people who obtained a GED in prison earned $114 more per quarter than those who did not; this was a significant difference.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the program and did not eliminate concerns about self-selection into the prison-based GED program. This means we are not confident that the estimated effects are attributable to the prison-based GED program; other factors are likely to have contributed.","The intervention was the receipt of a GED through a Florida prison-based GED program for prisoners who entered without a high school diploma. The sample included 12,956 men who were newly incarcerated in a Florida state prison between 1994 and 2000 and who lacked a high school diploma when entering prison. During this period, 1,967 people obtained a GED in prison (the treatment group) and the remaining 10,989 people did not (the comparison group). Overall, the sample was 47 percent White, 44 percent African American, and 8 percent Hispanic. The average age was 28, and the average length of education was 9.5 years. In all, 31 percent of the sample were employed before their incarceration.
The authors examined the impact of receiving a prison-based GED on post-release earnings, employment, and recidivism outcomes using administrative data provided by the state of Florida and UI data. To estimate impacts on earnings, the authors used a statistical model to control for individual factors and background characteristics. For the employment and recidivism outcomes, the authors presented the mean outcomes for both groups without any statistical controls. The authors assessed the recidivism and earnings outcomes annually for three years after release, and the employment outcome for one year post-release.",,"Employment
The study did not find a statistically significant relationship between obtaining a GED in prison and employment one year after release.
Earnings
The study found that, in the second year after release, those who had obtained a GED in prison earned $114 more per quarter than those who had not obtained a GED. This was a statistically significant difference. The study did not find any statistically significant differences in earnings between the two groups in the first or third year after release.
Recidivism
The study found no statistically significant relationships between obtaining a GED in prison and recidivism, defined as a conviction resulting in a return to prison or probation, up to three years after release.","In this study, participants elected to receive the intervention (obtaining a GED) rather than being assigned to it, for example, by random assignment. In studies in which participants elect to receive an intervention, there is always a risk that those who do differ from those who don’t in important ways that influence study outcomes. For example, a prisoner that opts to obtain a GED while in prison could be more motivated than other prisoners, which might affect their later employment outcomes. In this study, the authors control for these possible mechanisms to a large extent in the earnings analysis using a statistical model, but there might still be bias from members electing into the intervention group. Moreover, the authors do not control for such mechanisms in the analyses of employment and recidivism outcomes.
In addition, the study noted large dissimilarities between the two groups on race and age. The treatment group had a significantly higher proportion of White people than the comparison group (60 percent and 45 percent, respectively) and a lower proportion of African Americans (32 percent and 46 percent, respectively). The treatment group was also significantly younger, with an average age of 25.6 years old compared with 28.8 years old for the comparison group. Although the authors accounted for race and age in their statistical models for the earnings outcomes, it is uncertain whether the analysis could fully account for existing differences this large between the groups. These large existing differences between the groups—and not the prison-based GED program—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Justice-involved,"Reentry, Other training and education","Earnings and wages-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Employment","Reentry Review Protocol"
"Evaluating recidivism and job quality outcomes for participants in the Hoosier Initiative for Reentry Employment (HIRE) program (Northcutt Bohmert et al. 2017)","Evaluating recidivism and job quality outcomes for participants in the Hoosier Initiative for Reentry Employment (HIRE) program (Northcutt Bohmert et al. 2017)","Study Type: Causal Impact Analysis","Northcutt Bohmert, M., Hood, B. J., & Meckes, J. (2017). Evaluating recidivism and job quality outcomes for participants in the Hoosier Initiative for Reentry Employment (HIRE) program. Corrections: Policy, Practice and Research, 2(2), 110-129.",,2017,https://doi.org/10.1080/23774657.2016.1277378,"December 2019",Reentry,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of job placement through the Hoosier Initiative for Reentry Employment (HIRE) program on recidivism among people released from Indiana prisons in 2014.
The authors used HIRE program data and Indiana Department of Corrections data on the prison population to compare the recidivism outcomes of previously incarcerated people who were placed in a job through HIRE with the outcomes of those who were not placed in a job by the HIRE program. The authors used a statistical model to compare rates of recidivism between these two groups.
The study found a statistically significant relationship between the intervention and recidivism. People obtaining a job placement were less likely to be reincarcerated between one to two years after release than nonparticipants who did not use intervention program services. The study also examined employment outcomes but these were not eligible for review.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to job placement through HIRE; other factors are likely to have contributed.","HIRE is an employment program in Indiana for felony offenders provided through the state employment assistance program called WorkOne and funded by the state and federal government. Initially, the program provides employment services over 15 weeks, focusing on soft skills and job readiness skills. HIRE also refers previously incarcerated people to community organizations for additional resources they might require. The program provides job search assistance and job placement. After placement, the program provides about one year of job coaching and other services to improve employment retention. Program records indicate that participants placed in employment in 2014 typically worked in production and manufacturing and earned $9.67 per hour.","The authors used a statistical model to compare recidivism outcomes of people placed into jobs by HIRE with the outcomes of people who were not placed into jobs by HIRE. The authors used HIRE program data and Indiana Department of Corrections data on all people released from a state prison in 2014. This included 328 people who were placed in a job through the HIRE program (the treatment group) and 17,901 other released prisoners who were not placed in a job through HIRE (the comparison group).
Across both groups, most people were male (85 percent), about 35 years old, and white (68 percent in the comparison group and 76 percent in the treatment group). The average length of prison stay was 27 months for the treatment group and 17 months for the comparison group. Fewer than half the people were high school graduates or had obtained a general education diploma.","Recidivism
The study found a statistically significant relationship between job placement through HIRE and recidivism: HIRE participants who were placed in a job were less likely to be reincarcerated, for any reason or for a technical violation, between one and two years after release than comparison group members who did not use HIRE services. The study did not find a statistically significant relationship between job placement through HIRE and reincarceration for a new offense.","The authors noted statistically significant existing differences between the treatment and comparison groups before the intervention. For example, the authors noted differences in the groups’ average length of incarceration and the types of crimes for which people were convicted. These existing differences between the groups and other potential unobserved differences—and not the HIRE program—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Justice-involved,"Other employment and reemployment, Reentry","Employment-Low-No impacts",Employment,"Reentry Review Protocol"
"Does trade adjustment assistance make a difference? [TAA versus no TAA] (Reynolds & Palatucci 2012)","Does trade adjustment assistance make a difference? [TAA versus no TAA] (Reynolds & Palatucci 2012)","Study Type: Causal Impact Analysis","Reynolds, K. M., & Palatucci, J. S. (2012). Does trade adjustment assistance make a difference? Contemporary Economic Policy, 30(1), 43-59. [TAA versus no TAA]","Reynolds, K., & Palatucci, J. (2008). Does trade adjustment assistance make a difference? Unpublished manuscript.",2012,http://faculty.smu.edu/Millimet/classes/eco7377/papers/reynolds%20palatucci%202…,"January 2020","Apprenticeship and Work-Based Training",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of Trade Adjustment Assistance (TAA) on the employment and earnings of displaced manufacturing workers. The authors investigated similar research questions in another study, examining the impact of TAA services including training relative to TAA services not including training, the profile of which is available [here].
The study uses a nonexperimental design to compare the outcomes of workers displaced from manufacturing jobs from 2003 to 2005 who received services through TAA with the outcomes of a comparison group of workers displaced during the same period who likely did not receive assistance. The authors collected data on the treatment and comparison groups from different sources. Data for program participants were based on U.S. Department of Labor Trade Act Participant reports. Data for the comparison group were based on the Displaced Worker Survey from the Current Population Survey.
The study did not find statistically significant relationships between receiving TAA and the employment or earnings of participants compared with those who did not receive assistance from the program.
The quality of causal evidence provided in this study is low because the study uses a nonexperimental design, and the authors did not demonstrate that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to TAA; other factors are likely to have contributed.","TAA is a federal program that provides occupational training and job search assistance for displaced workers. Participants also have access to supportive services including financial assistance, a health coverage tax credit, and relocation assistance to pursue employment in areas with greater employment opportunities. Participants must have completed or be enrolled in training or have received a waiver from training because they are nearing retirement, have poor health, or already have marketable skills to receive financial assistance (26 additional weeks of unemployment insurance). To be eligible for assistance, workers must have been employed by a company that was certified by the U.S. Department of Labor as having been negatively impacted by trade activity, including production losses because of increased imports.","The authors used statistical models to compare the employment and earnings outcomes of a treatment group that received TAA services with outcomes of a matched comparison group that likely did not. The authors selected the comparison group using a statistical procedure (propensity score matching) that matched comparison group members to treatment group members based on age, education, job tenure, state, year, and industry characteristics. People in both groups had been displaced from a manufacturing job from 2003 to 2005.
Data for program participants were drawn from U.S. Department of Labor Trade Act Participant reports. Data for the comparison group were drawn from the Displaced Worker Survey from the Current Population Survey. The sample includes 5,125 TAA participants and 469 nonparticipants. The treatment group is about 56 percent male with average age of about 47; members of the treatment group had an average of 9.5 years of tenure at the job from which they were displaced. The comparison group is about 63 percent male with average age of about 45; members of the comparison group had an average of 8.8 years of tenure at the job from which they were displaced.","Employment 
The study did not find a statistically significant relationship between receiving TAA and the employment of participants when compared with displaced manufacturing workers not receiving assistance from the program.
Earnings and wages
The study did not find a statistically significant relationship between receiving TAA and the earnings of participants when compared with displaced manufacturing workers not receiving assistance from the program.","The authors did not account for existing differences between the groups before participation in TAA. These existing differences between the groups—and not the intervention—could explain the observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Other barriers","Adult and Dislocated Worker programs, Job search assistance and supportive services, Work based and other occupational training","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Apprenticeship and Work-Based Training Review Protocol"
"Minnesota’s Affordable Homes Program: Evaluating the effects of a prison work program on recidivism, employment and cost avoidance (Northcutt Bohmert & Duwe 2012)","Minnesota’s Affordable Homes Program: Evaluating the effects of a prison work program on recidivism, employment and cost avoidance (Northcutt Bohmert & Duwe 2012)","Study Type: Causal Impact Analysis","Northcutt Bohmert, M., & Duwe, G. (2012). Minnesota’s Affordable Homes Program: Evaluating the effects of a prison work program on recidivism, employment and cost avoidance. Criminal Justice Policy Review, 23(3), 327-351. [Study 1, Contrast 1: Affordable Homes Program participants versus matched nonparticipants].",,2012,https://journals.sagepub.com/doi/pdf/10.1177/0887403411411911,"January 2020",Reentry,,"Low Causal Evidence","Comparison Group Design",,"The authors examine the impact of participation in Minnesota’s Affordable Homes Program (AHP) on post-release employment, earnings, and recidivism outcomes for people who were previously incarcerated. The authors also investigated whether completion of the program impacted similar outcomes, the profile of which is available here.
The authors used a nonexperimental design (propensity score matching) to create a comparison group of nonparticipants who were similar to program participants. The authors used data from the Minnesota Department of Employee and Economic Development, the Minnesota Bureau of Criminal Apprehension, and the Minnesota Department of Corrections to compare these groups’ post-release employment, earnings, and recidivism. • Participation in AHP was not associated with statistically significant effects on employment, earnings, or the recidivism as measured by rearrest, felony reconviction, or re-incarceration for a new crime.
The quality of causal evidence presented in this report for program participants compared with nonparticipants is low for post-release employment and earnings but moderate for outcomes related to recidivism. This means we are not confident in the report’s employment and earnings outcomes, but we are somewhat confident in its recidivism outcomes.","Minnesota’s AHP was intended to increase the number of affordable homes in the state and increase the likelihood of people obtaining employment in construction-related fields after release from incarceration. The eligibility criteria for AHP include current incarceration in a minimum security facility; having no discipline violations in the past six months of incarceration; and being physically capable of work. AHP participants received hands-on construction training and were placed in a work crew with a maximum of nine other people. The crews were housed in local minimum security correctional facilities—mostly county jails—throughout the state of Minnesota and worked four 10-hour days each week to build or remodel affordable housing units. Participants were paid $1.00 to $1.50 an hour for their work.","The authors used a nonexperimental design (propensity score matching) to create a comparison group of 224 previously incarcerated people in Minnesota who had not participated in the program but were otherwise similar to the 224 program participants. Participants were 73 percent White with an average age of 35. All study participants in both groups were released to the community from 1998 to 2008. The authors compared post-release employment and earnings of the two groups using data from the Minnesota Department of Employee and Economic Development. The authors used data from the Minnesota Bureau of Criminal Apprehension and the Minnesota Department of Corrections’ Correctional Operations Management System to compare rearrests, reconvictions, and re-incarcerations.","Employment
The study found no statistically significant relationships between participation in AHP and post-release employment.
Earnings
The study found no statistically significant relationships between participation in AHP and total wages earned post-release.
Recidivism
The study found no statistically significant relationships between participation in AHP recidivism, as measured by rearrest, felony reconviction, or re-incarceration for new crime.","Although the authors used a well-implemented nonexperimental design that produced moderate quality evidence on the relationship between participation in AHP and recidivism, the study does not adequately control for pre-program earnings or employment.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Justice-involved,Reentry,"Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Reentry Review Protocol"
"The impact of training on the frequency of internal promotion of employees and managers (West 2010)","The impact of training on the frequency of internal promotion of employees and managers (West 2010)","Study Type: Causal Impact Analysis","West, L. S. (2010). The impact of training on the frequency of internal promotion of employees and managers. (Doctoral dissertation). University of North Texas Digital Library. https://digital.library.unt.edu/ark:/67531/metadc30526/m2/1/high_res_d/dissertation.pdf [Manager sample]",,2010,https://digital.library.unt.edu/ark:/67531/metadc30526/m2/1/high_res_d/disserta…,"January 2020","Apprenticeship and Work-Based Training",,"Low Causal Evidence",,,"The study’s objective was to examine the relationship between formal training and promotion for managers. The author investigated similar research questions in another study, the profile of which is available [here]. The other study examined the relationship between formal training and promotion for employees who were not managers.
The author used statistical methods in a nonexperimental analysis to estimate impacts of hours of formal training, drawing on archival survey data from the 1996–1997 the National Organizations Survey.
The study found no statistically significant relationships between formal training and managers’ promotions.
The quality of causal evidence presented in this report is low because the author did not account for existing differences of managers in organizations that offered various hours of formal training. This means we would not be confident that the estimated effects would be attributable to formal training; other factors would have been likely to have contributed. The study did not, however, find statistically significant relationships.",,"Drawing on archival survey data from the 1996–1997 the National Organizations Survey, the study used statistical analyses to estimate the relationship between the number of hours of training offered at organizations to the frequency of promotions for managers.
The analysis included 118 organizations that reported offering formal training to their managers in the previous two years and that answered all the relevant questions in the survey. One individual in each organization completed the study. Hours of training represents the typical number of hours of formal training offered to managers at the organization in the previous two years; the average was 40.5 hours. Promotion was assessed by asking how often the organization filled manager vacancies with people already employed at the organization—never, rarely, often, or very often—but did not specify whether these different positions represented a promotion.","Employment
The study found no statistically significant relationships between formal training and promotion frequency for managers.","The author did not account for existing differences of managers in organizations that offered various hours of formal training. These existing differences—and not the formal training—could have explained observed differences in outcomes. The study did not, however, find statistically significant relationships.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Employed,"Other training and education","Employment-Low-No impacts",Employment,"Apprenticeship and Work-Based Training Review Protocol"
"Enhanced vocational rehabilitation for veterans with mild traumatic brain injury and mental illness: Pilot study (O'Connor et al. 2016)","Enhanced vocational rehabilitation for veterans with mild traumatic brain injury and mental illness: Pilot study (O'Connor et al. 2016)","Study Type: Causal Impact Analysis","O’Connor, M. K., Mueller, L., Kwon, E., Drebing, C. E., O’Connor, A. A., Semiatin, A., & Daley, R. (2016). Enhanced vocational rehabilitation for veterans with mild traumatic brain injury and mental illness: Pilot study. Journal of Rehabilitation Research & Development, 53(3), 307-319.",,2016,http://search.proquest.com/openview/ee479834dd8beedff02771b00bc95e0a/1?pq-origs…,"January 2020",Veterans,,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of an embedded cognitive rehabilitation intervention on employment and earnings.
The study was a randomized controlled trial with the authors using administrative data.
The study found no statistically significant relationships between an embedded cognitive rehabilitation intervention and employment and earnings after one year.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to an embedded cognitive rehabilitation intervention; other factors are likely to have contributed.","The embedded cognitive rehabilitation intervention provided enhanced vocational support to veterans with mild traumatic brain injury and mental health issues by adding an embedded cognitive rehabilitation intervention to vocational rehabilitation services. The cognitive rehabilitation intervention is a 12-session program designed to help these veterans obtain employment. During the sessions, veterans who were offered to participate in the intervention met with a cognitive rehabilitation specialist to (1) discuss strategies to help manage cognitive difficulties in a work environment, (2) build skills to identify and manage negative emotions and work behaviors, and (3) develop supportive relationships with employers and coworkers. Participating veterans also received a laptop containing software that provided work readiness and training and vocational rehabilitation modules, which were assigned as homework between sessions. In addition, they could meet jointly with the vocational rehabilitation specialist and the cognitive rehabilitation specialist.
The study took place at a U.S. Department of Veterans Affairs medical center. To be eligible for the intervention, veterans had to meet a number of criteria, including being age 18 or older, having a history of mild traumatic brain injury and a co-occurring mental illness, having impairment in cognitive functioning, not having a history of nontraumatic brain injury neuropsychological disorder that would result in being nonresponsive to the intervention, being enrolled in a vocational rehabilitation program, being unemployed or underemployed, speaking English, having more than 10 years of education, and having the potential to return to competitive employment within six months.","The authors conducted a randomized controlled trial that used administrative data to compare three key outcomes after one year: (1) employment rate, (2) number of weeks the individual worked at least one day, and (3) earnings. Veterans in the control group completed 12 sessions of therapy that did not focus on employment or cognitive rehabilitation. In all, 18 veterans were included in the final analysis sample: 10 in the intervention group and 8 in the control group. The authors also used a satisfaction interview to collect qualitative data on the veterans’ opinions of the intervention.
All study participants were men. The majority of the veterans were White, had a high school diploma, and served in the military after the Vietnam era. All study participants had at least one diagnosed psychiatric disorder, and most had at least one substance use disorder.","Employment
The study found no statistically significant relationships between the embedded cognitive rehabilitation intervention and employment. 
Earnings
The study found no statistically significant relationships between the embedded cognitive rehabilitation intervention and earnings.","This is a small study of 18 people that the authors used as a pilot study preceding a larger randomized controlled trial study.
In addition, the authors did not establish that the intervention and comparison groups were similar before the intervention. Specifically, the authors did not account for differences in terms of race, employment, and earnings between the groups. Existing differences between the groups—and not the intervention— could explain any observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Other barriers, Male, Veteran or military","Substance abuse recovery, Vocational rehabilitation","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Veterans Review Protocol"
"Participation in the transition assistance program and job placement outcomes of U.S. veterans (Silva 2011)","Participation in the transition assistance program and job placement outcomes of U.S. veterans (Silva 2011)","Study Type: Causal Impact Analysis","Silva, E. (2011). Participation in the transition assistance program and job placement outcomes of U.S. veterans. (Publication No. AAI1491543) [Master’s dissertation, University of Rhode Island]. ProQuest.",,2011,https://digitalcommons.uri.edu/dissertations/AAI1491543/,"April 2020",Veterans,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of the Transition Assistance Program (TAP) on employment success—an outcome that reflects both employment and the desired number of work hours.
The study involved a nonexperimental comparison group analysis that compared veterans who participated in TAP to those who did not, using data from the Current Population Survey and Veterans Supplement.
The study showed no statistically significant relationship between TAP and veterans’ employment success.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we could not be confident that any estimated effects would be attributable to TAP and not to other factors. However, the study did not show any statistically significant effects.","The National Defense Authorization Act for Fiscal Year 1991 created TAP. Administered by the U.S. Department of Defense, Department of Labor, and Department of Veterans Affairs, TAP is a voluntary program offered to people in the military and their spouses to ease the transition to civilian life by providing employment supports. TAP participants attend workshops, receive help finding jobs, write resumes, and participate in mock interviews.","The study involved a nonexperimental comparison group analysis to compare employment success for veterans who participated in TAP to those who did not. The sample consisted of 1,477 veterans, drawn from 10,766 respondents to the Veterans Supplement, who separated from the military between 1991 and August 2007 and self-identified as either participating or not participating in TAP. The study team defined employment as positive if a person had a job and worked at least the number of hours desired, somewhat positive if the person had a job but worked fewer hours than desired, and negative if the person was unemployed.
The sample was about 83 percent White, 11 percent African American, and 85 percent male. Roughly one-quarter had a high school diploma or less, and about 30 percent had some college education. Over one-fifth of the sample reported having a disability.","Employment
The study showed no statistically significant relationship between TAP and veterans’ employment success.","The authors did not account for any preexisting differences between the groups before program participation. Such differences—and not the program—could be responsible for any observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Job search assistance and supportive services, Veterans' reemployment","Employment-Low-No impacts",Employment,"Veterans Review Protocol"
"What factors contributed to changes in employment during and after the great recession? (Farooq & Krugler 2015)","What factors contributed to changes in employment during and after the great recession? (Farooq & Krugler 2015)","Study Type: Causal Impact Analysis","Farooq, A., & Kugler, A. D. (2015). What factors contributed to changes in employment during and after the great recession? IZA Journal of Labor Policy, 4(3), 1-28.",,2015,https://link.springer.com/article/10.1186/s40173-014-0029-y,"April 2020",Veterans,,"Low Causal Evidence","Comparison Group Design",,"This study examined the impact of the Work Opportunity Tax Credit (WOTC) on employment for veterans with disabilities and unemployed veterans.
The study used a nonexperimental design to examine the effects of the WOTC on employment, based on data from the Annual Social and Economic (ASEC) supplement of the Current Population Survey from 2003 to 2013.
The study showed no statistically significant relationships between the WOTC and employment for veterans with disabilities or for unemployed veterans.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that any estimated effects are attributable to the WOTC; other factors likely contributed to the outcomes. However, the study did not show any statistically significant effects.","The WOTC, established in 1996, is a set of tax credits designed to encourage employers to hire people from target groups that are typically less likely to obtain employment. Target groups include people who have received Temporary Assistance for Needy Families, veterans and 18- to 39-year-olds receiving benefits from the Supplemental Nutrition Assistance Program, ex-felons, recipients of Supplemental Security Income, and enterprise and renewal communities (distressed communities designated by the U.S. Department of Housing and Urban Development).
The target groups have changed since the WOTC was established. In 2007, the WOTC began extending tax credits through August 2011 to veterans with disabilities who were discharged from active duty within the past year as part of the U.S. Troops Readiness, Veteran’s Care, Katrina Recovery, and Iraq Accountability Act. Between 2009 and 2010, through the Recovery Act, the WOTC was expanded to include unemployed veterans discharged within the past five years who had collected unemployment benefits for four or more weeks in the past year. It was also extended to disconnected youth (ages 16 to 24) who had not worked regularly or attended school in the past six months. In 2011 and 2012, the VOW to Hire Heroes Act continued tax credits for unemployed veterans from the Recovery Act WOTC.","The study involved a nonexperimental analysis using a sample of 1,285,543 people from the ASEC supplement of the Current Population Survey from 2003 to 2013. The authors used statistical regression models to compare employment rates between (1) two treatment groups that were eligible for the WOTC and (2) one comparison group that was not. One treatment group consisted of disabled veterans discharged from active duty since 2001, and the other treatment group was made up of veterans unemployed for at least four weeks during the past year and discharged from active duty since 2001. The comparison group consisted of disabled nonveterans and unemployed nonveterans. The statistical models accounted for differences across the groups in age, race, gender, and educational background.","Employment
The study showed no statistically significant relationships between the WOTC and employment for veterans with disabilities or for unemployed veterans.","The authors did not account for preexisting differences between the groups before eligibility for the WOTC. Such preexisting differences—and not the WOTC—could be responsible for any observed differences in outcomes.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Disability, Veteran or military","Veterans' reemployment, Work Opportunity Tax Credit (WOTC)","Employment-Low-No impacts",,"Veterans Review Protocol"
"Effects of structured vocational services on job-search success in ex-offender veterans with mental illness: 3-month follow-up (LePage et al. 2011)","Effects of structured vocational services on job-search success in ex-offender veterans with mental illness: 3-month follow-up (LePage et al. 2011)","Study Type: Causal Impact Analysis","LePage, J. P., Washington, E. L., Lewis, A. A., Johnson, K. E., & Garcia-Rea, E. (2011). Effects of structured vocational services on job-search success in ex-offender veterans with mental illness: 3-month follow-up. Journal of Rehabilitation Research & Development, 48(3), 277-286. [Self-study vs. VERC]",,2011,https://www.rehab.research.va.gov/jour/11/483/pdf/page277.pdf,"April 2020","Reentry, Veterans",,"Low Causal Evidence","Comparison Group Design",,"The study examined the impact of self-studying the About Face (AF) manual on competitive employment.
In this nonexperimental study, the authors used statistical analyses to compare the employment outcomes of two groups of justice-involved veterans: (1) those who had access to the AF manual to study on their own (self-study program) and (2) those who only had access to the Veterans Employment Resource Center (VERC). The authors collected biweekly data from participants in person or over the phone.
The study showed no significant differences in competitive employment between the self-study condition (3 percent hired) and the VERC-only condition (0 percent hired).
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the AF manual; other factors likely contributed to the findings.","The AF manual includes 72 pages and covers topics such as identifying job skills, creating resumes, responding to challenging interview questions, and other interview strategies. One section of the manual covers the challenges of job searches for people with a history of incarceration and suggests strategies, such as connecting with a personal network, to find employment. The study team developed the manual using interviews with 200 justice-involved veterans; 22 justice-involved veterans provided detailed comments on the draft.","In this nonexperimental study, the authors compared the outcomes of two groups of justice-involved veterans: (1) those who had access to the manual to study on their own (self-study program) and (2) those who only had access to the VERC. Thirty-three justice-involved veterans had access to the AF manual, and 15 justice-involved veterans only had access to the VERC. The authors collected biweekly data from participants in person or over the phone.","Employment
The study showed no significant differences in competitive employment between the self-study condition (3 percent hired) and the VERC-only condition (0 percent hired).","The authors did not show that the two groups of justice-involved veterans—those who had access to the AF manual and those who did not—were similar in terms of gender or pre-intervention employment. Since gender and work history are related to the outcome (competitive employment), the differences between the two groups on these two variables could have biased the study results. The authors did, however, show that the two groups of justice-involved veterans did not differ significantly based on criminal history.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Job search assistance and supportive services, Veterans' reemployment","Employment-Low-No impacts",Employment,"Reentry Review Protocol"
"Effects of structured vocational services in ex-offender veterans with mental illness: 6-month follow-up (LePage et al. 2013)","Effects of structured vocational services in ex-offender veterans with mental illness: 6-month follow-up (LePage et al. 2013)","Study Type: Causal Impact Analysis","LePage, J. P., Lewis, A. A., Washington, E. L., Davis, B., & Glasgow, A. (2013). Effects of structured vocational services in ex-offender veterans with mental illness: 6-month follow-up. Journal of Rehabilitation Research & Development, 50(2), 183-191. doi:10.1682/JRRD.2011.09.0163 [Self-study vs. VERC]",,2013,https://www.rehab.research.va.gov/JOUR/2013/502/pdf/lepage502.pdf,"April 2020","Reentry, Veterans",,"Low Causal Evidence","Comparison Group Design",,"The study examined the impact of self-study of the About Face (AF) manual on competitive employment.
In this nonexperimental study, the authors used statistical analyses to compare the employment outcomes of two groups of justice-involved veterans: (1) those who had access to the AF manual to study on their own (self-study program) and (2) those who only had access to the Veterans Employment Resource Center (VERC). The authors collected monthly data from participants in person or over the phone.
The study showed no significant differences in competitive employment between the self-study condition (11.9 percent hired) and the VERC-only condition (16.7 percent hired). The study also showed no significant differences in average months employed (0.16 months for the self-study condition and 0.35 months for the VERC-only condition).
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the AF manual; other factors likely contributed to the findings.","The AF manual included 72 pages and covers topics such as identifying job skills, creating resumes, responding to challenging interview questions, and other interview strategies. One section of the manual covers the challenges of job searches for people with a history of incarceration and suggests strategies, such as connecting with a personal network, to find employment. The study team developed the manual using interviews with 200 justice-involved veterans; 22 justice-involved veterans provided detailed comments on the draft.","In this nonexperimental study, the authors compared the outcomes of two groups of justice-involved veterans: (1) those who had access to the AF manual to study on their own (self-study program) and (2) those who only had access to the VERC. Thirty-three justice-involved veterans received the AF manual, and 15 justice-involved veterans only had access to the VERC. The authors collected monthly data from participants in person or over the phone.","Employment
The study showed no significant differences in competitive employment between the self-study condition (11.9 percent hired) and the VERC-only condition (16.7 percent hired).
The study also showed no significant differences in average months employed (0.16 months for the self-study condition and 0.35 months for the VERC-only condition).","The authors did not show that the two groups of justice-involved veterans—those who received the AF manual and those who did not—were similar in terms of gender and pre-intervention employment. Since gender and work history are related to the outcome (competitive employment), the differences between the two groups on these two variables could have biased the study results. The authors did, however, show that the two groups of justice-involved veterans did not differ significantly based on criminal history.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Job search assistance and supportive services, Veterans' reemployment","Employment-Low-No impacts",Employment,"Reentry Review Protocol"
"STEM-Connect at the University of Vermont, College of Engineering and Mathematical Sciences: Program evaluation final report (Farrell et al 2017)","STEM-Connect at the University of Vermont, College of Engineering and Mathematical Sciences: Program evaluation final report (Farrell et al 2017)","Study Type: Causal Impact Analysis","Farrell, R., Harris, D., Meyers, H., & Ratmeyer, S. (2017). STEM-Connect at the University of Vermont, College of Engineering and Mathematical Sciences: Program evaluation final report. Swanton, VT: Core Research and Evaluation.",,2017,http://www.skillscommons.org/bitstream/handle/taaccct/13127/STEM-Connect%20Fina…,"April 2020","Community College",,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the effects of the University of Vermont’s (UVM) Science, Technology, Engineering and Math (STEM) Connect program on earnings and employment outcomes.
The study used a nonexperimental design to compare the outcomes of students who were enrolled in the UVM STEM-Connect program to a matched comparison group of students who took a STEM-Connect course but were not enrolled in the STEM-Connect program.
The study found no statistically significant relationships between STEM-Connect program participation and average quarterly wages or employment rates.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the UVM STEM-Connect program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The University of Vermont’s (UVM) College of Engineering and Mathematical Sciences department received a TAACCCT grant to design the STEM-Connect program with the goal of bolstering UVM’s ability to prepare students for the changing technology field and for employment in technology industries. The program was designed to target Trade Adjustment Assistance (TAA) eligible workers, veterans, dislocated workers, underemployed or unemployed adults, and those new to the STEM fields of study. Students in STEM-Connect certificate programs were able to earn credentials in Computer Software Certificates in Software Development, Web Development, Cybersecurity, Master’s Preparation and Self Design; Computer-Aided Engineering Technology; Complex Systems (master’s level); and Pre-Actuarial (Actuarial Science). STEM-Connect is administered in a traditional classroom setting as well as online, and includes mentoring and internship coordination. In addition to standard support services such as access to UVM's financial aid offices, counseling, and career development, STEM-Connect provides non-traditional students with enhanced recruitment, counseling, tutoring and academic advising opportunities.","The nonexperimental study took place at the UVM’s College of Engineering and Mathematical Sciences department in Burlington, Vermont. The treatment group consisted of 331 UVM students who participated in the TAACCCT-funded STEM-Connect program from 2014 to 2016, while the comparison group consisted of 2,088 UVM students who took a STEM-Connect course as part of their regular university program but did not participate in STEM-Connect. The authors matched STEM-Connect program participants to similar nonparticipants using demographic characteristics. Using data from UVM's student database and the Vermont Department of Labor (VDOL) database, the authors conducted chi-square analyses to compare outcomes between the treatment and comparison groups. Outcomes included average quarterly wages and employment rates.","Earnings and wages 
The study did not find a statistically significant relationship between STEM-Connect program participation and average quarterly wages.
Employment
The study did not find a statistically significant relationship between STEM-Connect program participation and employment rates.","The authors did not account for preexisting differences between the groups before program participation. Specifically, they did not account for differences in race/ethnicity when creating the matched comparison group which is required by the review protocol. These preexisting differences between the groups—and not the STEM-Connect program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Unemployed, Veteran or military","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) TAACCCT IV evaluation final report (Foster et al 2018)","Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) TAACCCT IV evaluation final report (Foster et al 2018)","Study Type: Causal Impact Analysis","Foster, L. R., Staklis, S., Ott, N. R., & Moyer, R. (2018). Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) TAACCCT IV evaluation final report. Raleigh, NC: RTI International.",,2018,https://www.skillscommons.org/bitstream/handle/taaccct/18334/KanTRAIN_TAACCCT_I…,"April 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) program on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare education, earnings, and employment outcomes of KanTRAIN participants to a matched comparison group.
The study found KanTRAIN participants were significantly more likely to earn more credits per semester, pass a higher percentage of the credits attempted, attain more credentials, and earn a nondegree credential or a college degree, and receive a wage increase than the comparison group.
The quality of causal evidence presented in this study is low because the authors used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the KanTRAIN program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Kansas Technical Re/training Among Industry-targeted Networks (KanTRAIN) program was funded by a Round IV TAACCCT grant and implemented by five community and technical colleges in Kansas. The main objectives of the KanTRAIN program were to expand institutional capacity and develop career pathway programs in health care and advanced manufacturing. The program sought to increase employer engagement, facilitate outreach to veterans, and upgrade technical skill training programs by aligning these pathways with statewide initiatives. To expand and strengthen existing career pathways, KanTRAIN incorporated several evidence-based strategies. These strategies included design and delivery of industry-focused and employer-driven workforce training models, integration of industry-recognized and stackable credentials into career pathway programs, strengthened case management and increased support for participants.","The study used a nonexperimental design to compare the outcomes of students who participated in the KanTRAIN program to students who did not participate. The authors matched KanTRAIN students to similar students using propensity scores developed from 21 demographic and academic variables. Across the five colleges, study participants included 4,617 students. The treatment group included 1,168 students who participated in the KanTRAIN program and were enrolled between fall of 2015 and spring of 2017. The comparison group was a historical cohort composed of 3,449 students that were enrolled between fall 2011 and summer 2015 in the same or similar programs (prior to KanTRAIN implementation). Using data from the KanTRAIN program and the Kansas Board of Regents, the authors conducted statistical models to examine differences in outcomes. Outcomes included number of credits earned, proportion of credits passed, nondegree credential attainment, college degree attainment, employment status, quarterly wages, and wage difference.","Education and skills gain 
The study found a significant relationship between program participation and general credits earned and passed, with KanTRAIN students earning 1.5 more general credits per semester, and passing a higher percentage than students in the comparison group (87% versus 83%).
The study found a significant relationship between program participation and technical credits earned and passed, with KanTRAIN students earning 2.1 more technical credits per semester, and passing a higher percentage than students in the comparison group (91% versus 88%).
The study found that KanTRAIN students were significantly more likely to earn a earn a nondegree credential (72%) and earn a college degree (11%) when compared to students in the comparison group (62% and 9%, respectively).
Earnings and wages
For study participants with pre-enrollment data, the study found that the likelihood of a wage increase was 6% higher for KanTRAIN students compared to students in the comparison group.
The study found no other statistically significant relationships between the KanTRAIN program and wages.
Employment 
The study found no statistically significant relationship between the KanTRAIN program and employment.","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the program. Also, the employment data used in the analyses were all self-reported and only available for participants employed or unemployed in Kansas or Missouri. Participants employed or unemployed outside of these states were not included in the employment outcome analysis of the study. Finally, the study did not account for changes in state or regional labor markets, thus, a broader economic context may have impacted the estimated effects of the KanTRAIN program on employment and earnings outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"Flint Hills Technical College in Emporia, Kansas
Garden City Community College in Garden City, Kansas
Washburn Institute of Technology in Topeka, Kansas
Washburn University Washburn University in Topeka, Kansas
Wichita State University Campus of Applied Sciences and Technology in Wichita, Kansas",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final evaluation Golden Triangle Modern Manufacturing Project (Harpole 2017)","Final evaluation Golden Triangle Modern Manufacturing Project (Harpole 2017)","Study Type: Causal Impact Analysis","Harpole, S. (2017). Final evaluation Golden Triangle Modern Manufacturing Project. SHH Consulting, LLC",,2017,https://www.skillscommons.org/handle/taaccct/15669,"April 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Golden Triangle Modern Manufacturing Project on education, earnings, and employment outcomes.
The author used a nonexperimental design to compare outcomes of Golden Triangle Modern Manufacturing Project participants to a matched comparison group.
The study found that program participation was significantly associated with increases in program retention, credential attainment, and earnings.
The quality of causal evidence presented in this study is low because the author used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to Golden Triangle Modern Manufacturing Project; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
East Mississippi Community College (EMCC) received a TAACCCT grant to implement the Golden Triangle Modern Manufacturing Project. Based on the model from the 2012 Missouri Manufacturing Workforce Innovation Networks TAACCCT project, the Golden Triangle Modern Manufacturing Project aimed to prepare adult students for careers in advanced manufacturing. Features included building programs that meet industry needs, enhancing career pathway options for learners and workers, accelerating and improving certification and employment attainment, and strengthening online and technology-enabled learning. The project added three credential- and craft-based technician programs, created a new career and technical education (CTE) developmental program, and engaged local manufacturers and workforce partners in these programs’ design and curriculum to concurrently meet industry demands and the needs of the target population, including veterans, TAA-eligible workers, and other unemployed or unskilled adults. These enhanced programs helped fill gaps in existing education and career training programs and relied on incorporating the National Association of Manufactures (NAM) endorsed credentials. More specifically, the project modernized EMCC's manufacturing-related CTE programs by stacking industry-recognized credentials into six CTE programs, having CTE instructors leveraging advanced technology, prioritizing soft skill development in their instruction delivery, and requiring all program participants to complete a Silver Career Readiness Certificate (CRC).","The study used a nonexperimental design to compare the outcomes of students who participated in the Golden Triangle Modern Manufacturing Project to those who did not. The comparison group included first-time students who were enrolled in comparable CTE programs offered at EMCC prior to October 2013. The author matched Golden Triangle Modern Manufacturing participants to similar nonparticipants using propensity scores developed from demographic, economic, and program-based characteristics. Study participants included 373 students in the treatment group and 548 in the comparison group. Student program performance and baseline data was provided and collected from October 2013 to September 2016 for the treatment group and from October 2009 to September 2013 for the comparison group. Using performance and baseline data, the author conducted statistical models to examine differences in outcomes. The outcomes included program credential completion and retention, enrollment in further education, employment of non-incumbent workers, job retention, and increase in wages for incumbent workers.","Education and skills gain 
The study found that participation in the Golden Triangle Modern Manufacturing Project was significantly associated with increased program retention, with higher rates of program retention among participating students (85%) compared to students in the comparison group (54%).
The study also found that Golden Triangle Modern Manufacturing students were significantly more likely to attain a credential (90%) when compared to students in the comparison group (46%).
The study did not find a significant relationship between participation in the Golden Triangle Modern Manufacturing Project and further education in a non-TAACCCT funded program of study.
Earnings and wages 
The study found that Golden Triangle Modern Manufacturing students were significantly more likely to have an earnings increase after program completion (21%) when compared to students in the comparison group (20%).
Employment
The study did not find a significant relationship between enrollment in the Golden Triangle Modern Manufacturing Project and the employment rate one quarter after program exit or the job retention rate two quarters after program exit.","Though baseline equivalency was achieved between the two research groups prior to the program intervention, there was a confounding factor present in this study. Given that the author used a cohort from previous enrollment years as the comparison group, differences in outcomes could be due to time-varying factors (such as overall changes at the community college or local labor market changes) and not the program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Unemployed, Low-skilled, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Alliance evaluation: Final evaluation report (Ho 2016)","Alliance evaluation: Final evaluation report (Ho 2016)","Study Type: Causal Impact Analysis","Ho, H. Y. (2016). Advanced Manufacturing Education (AME) Alliance evaluation: Final evaluation report (Grant Number TC-23753-12-60-A-27). Denver, CO: McREL International.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/10174/DOLAME_Y4_FinalEva…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"This study’s objective was to examine the impact of the Advanced Manufacturing Education (AME) Alliance on education, earnings, and employment outcomes.
The author used a nonexperimental design to compare education, employment, and earnings outcomes of AME participants to a matched comparison group.
The study found that AME participants were significantly more likely to complete a program of study relative to the comparison group.
The quality of casual evidence provided in this study is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the enhanced AME program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Advanced Manufacturing Education (AME) Alliance was formed with three community colleges and one state college in Minnesota upon receipt of a four-year TAACCCT grant. The AME redesigned advanced manufacturing degree programs at these colleges to better meet the needs of their students by developing a hybrid curriculum, providing additional support mechanisms to students, collaborating with employer and workforce partners, and utilizing technology across programs. This included purchasing technology for students, developing four hybrid courses and five modularized courses, hiring six educational and employment advisors, establishing a website providing general information about grant programs for students, and expanding employment partners.","The nonexperimental study was conducted across the AME alliance colleges in Minnesota (Central Lakes College, Pine Technical and Community College, Saint Cloud Technical and Community College, and Minnesota’s 360 Center for Manufacturing Excellence). The author matched participants enrolled in enhanced AME courses to nonparticipants, using propensity scores developed from baseline demographic information. Study participants included 180 participants enrolled in an enhanced AME program in the fall of 2014, as well as 180 comparison students who attended a non-enhanced AME or similar program from 2005-2010. Using data from student database records and Minnesota’s Department of Employment and Economic development, the author used statistical models to examine differences in outcomes between the groups. Outcomes included degree completion, and wage increases and employment six months after completion of an AME program.","Education and Skills Gain 
The study found that AME participation was significantly associated with program completion, with program participants being 1.6 times more likely to complete their program of study than the comparison group.
Earnings and Wages 
The study found no significant relationship between participation in the AME-enhanced program and earnings increase 6 months after completing an AME program.
Employment 
The study found no significant relationship between participation in the AME-enhanced program and employment 6 months after completing an AME program.","The author created a matched group of non-participating students to compare to students who were enrolled in an AME-enhanced program. However, the author did not appropriately control for other factors that could have affected the differences between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the enhanced AME program—could explain the observed differences in outcomes. Additionally, the author used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"RITA consortium final evaluation report (The Improve Group 2017)","RITA consortium final evaluation report (The Improve Group 2017)","Study Type: Causal Impact Analysis","The Improve Group. (2017). RITA consortium final evaluation report. Saint Paul, MN: The Improve Group.",,2017,http://www.skillscommons.org/bitstream/handle/taaccct/13172/RITA%20Final%20Summ…,"May 2020","Community College",,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact of enhancements to the Rural Information Technology Alliance (RITA) program on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare the outcomes of RITA participants to a matched comparison group at four community colleges.
The study found that RITA program students were significantly more likely to complete an associate’s degree at two of the four colleges; however, RITA students were significantly less likely to complete an associate’s degree at the other two colleges. RITA participation was also positively associated with completing any degree (associate, diploma, or certificate) at one school, and negatively associated with completing any degree at another school. These findings were significant.
The quality of causal evidence presented in this study is low because the authors used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the RITA program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Rural Information Technology Alliance (RITA) was comprised of one community college in Texas (North Central Texas College) and three community colleges in Minnesota (Pine City Community College, Ridgewater College, and Central Lakes College). RITA was funded during the third round of TAACCCT, receiving $18.3 million over four years. RITA allowed the colleges to enhance and redesign their information technology (IT) program and courses, including purchasing new equipment, software, labs, and faculty; hiring student advisors/coaches; offering online and hybrid courses; and better aligning their program, courses, and certificates with the needs of local businesses. Students were eligible for RITA if they enrolled in two or more IT courses between 2014 and 2017.","The nonexperimental study included community college students from the four RITA schools. The authors matched RITA participants to similar nonparticipants using demographic and education information. The treatment group included 1,895 students who participated in RITA and were enrolled between 2014 and 2017. The comparison group was a historical cohort of students who enrolled in one of the four colleges between 2009 and 2014 (the study authors do not report the number of comparison students). Both treatment and comparison group students took at least two IT courses; however, treatment students took enhanced courses within a redesigned program. Data sources included intake forms, school administrative data, and state employment data. The authors used statistical models to examine differences in outcomes between treatment and comparison groups. Outcomes included student completion rates, earnings, and employment outcomes.","Education and skills gains 
The study found that the RITA program was significantly associated with a higher likelihood of completing an Associate’s degree at North Central Texas College (NCTC) and Ridgewater College (RC). Conversely, RITA students were significantly less likely than comparison group students to complete an Associate’s degree at Central Lakes College (CLC) and Pine Technical and Community College (PTCC).
The study also found that RITA students at NCTC were significantly more likely than comparison group students to complete any degree (associate, diploma, or certificate). However, RITA students at PTCC were significantly less likely than comparison group students to complete any degree.
Earnings and wages 
The study did not find a significant relationship between RITA participation and earnings.
Employment
The study did not find a significant relationship between RITA participation and employment rates.","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in labor market conditions) and not the program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"Rural, United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Mixed impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Project IMPACT: Innovative multi-industry partnership and career training project final evaluation report. Gateway Community Technical College (Jensen et al. 2017)","Project IMPACT: Innovative multi-industry partnership and career training project final evaluation report. Gateway Community Technical College (Jensen et al. 2017)","Study Type: Causal Impact Analysis","Jensen, J., Horohov, J., & Waddington, J. (2017). Project IMPACT: Innovative multi-industry partnership and career training project final evaluation report. Gateway Community Technical College. Lexington, KY: University of Kentucky, College of Education, Educational Policy Studies & Evaluation.",,2017,https://www.skillscommons.org/bitstream/handle/taaccct/15674/IMPACT%20Final%20E…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of Project IMPACT on education, earnings, and employment outcomes.
Using academic records from the college as well as data obtained from the Kentucky Center for Workforce Statistics, the authors used a nonexperimental design to compare outcomes of Project IMPACT participants to outcomes of historical cohort students who were enrolled in the Project IMPACT programs of study prior to grant implementation.
The study found that participating in Project IMPACT was significantly associated with a higher likelihood to take courses, pass courses, earn credits, and receive credentials. The study also found that participating in Project IMPACT was significantly associated with higher earnings in the quarter following enrollment.
The quality of causal evidence presented in this study is low for the education and employment outcomes because the authors used a comparison group from previous enrollment years presenting a confounding factor and because the authors did not ensure the groups being compared were similar before the intervention, respectively. This means we are not confident that the estimated effects are attributable to Project IMPACT; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2013, Gateway Community and Technical College (GCTC) in Kentucky received a Trade Adjustment Assistance Community College Career Training (TAACCCT) grant to implement the Innovative Multi-Industry Partnership and Career Training (IMPACT) program. The purpose of the IMPACT project was to enhance career preparation in logistics, manufacturing, heating and cooling, and energy fields. The project provided a combination of academic and career coaching in cooperation with workforce development partners, community agencies, and industry employers. The program targeted eligible workers, veterans and underemployed adults who were pursuing careers in logistics, manufacturing, energy, and heating and cooling fields. Key components of Project IMPACT included academic and career coaching in partnership with workforce development partners, community agencies, and industry employers.","The study was conducted at Gateway Community Technical College in Florence, Kentucky. The authors compared outcomes of Project IMPACT participants to outcomes of historical cohort students who were enrolled in the Project IMPACT programs of study before implementation of the grant. The authors used a nonexperimental design with propensity score matching to ensure participants in both groups were similar on baseline characteristics. After the matching procedure, there were 96 students who had received the full Project IMPACT program treatment, and there were 93 students in the comparison group.","Education 
The study found that participating in Project IMPACT was significantly associated with taking and passing more courses. Project IMPACT students took an average of 20.4 courses and passed 18.2 of them; students in the comparison group took an average of 5.0 courses and passed 3.2.
The study found that participating in Project IMPACT was significantly associated with earning more credits, with higher proportions of Project IMPACT students earning credits (93%) than comparison students (73%).
The study found that participating in Project IMPACT was significantly associated with earning more credentials, with higher proportions of Project IMPACT students earning credentials (62%) than comparison students (15%).
Employment 
The study did not find a significant relationship between Project IMPACT and employment rates in the quarter after enrollment.
Earnings
The study found that participating in Project IMPACT was significantly associated with an increase in earnings in the quarter after enrollment. Project IMPACT students had nearly double the mean quarterly wages as the comparison students ($9,100 and $4,413, respectively).","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Additionally, the authors created a matched group of a historical cohort to compare to Project IMPACT students. However, for the employment outcomes, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as a pre-intervention measure of employment. These preexisting differences between the groups—and not Project IMPACT —could explain the observed differences in outcomes. Finally, the authors also note that employment data was only available for students one month after completion and therefore, students who were employed and also still enrolled in the program, were not included in the sample. They also note that the location of the program on the border of Kentucky and Ohio may have misrepresented the employment data as individuals who found work in Ohio would not have been captured within the Kentucky data (although most of the students did have Kentucky addresses).",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers, Veteran or military","Employer partnerships, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final evaluation report: Linn-Benton Community College (Kelley-Smith et al. 2017)","Final evaluation report: Linn-Benton Community College (Kelley-Smith et al. 2017)","Study Type: Causal Impact Analysis","Kelley-Smith, A., Schoeph, K., Hamai, T., & Bishop, M. (2017). Final evaluation report: Linn-Benton Community College. Albany, OR: LB iLearn Campus.",,2017,https://www.skillscommons.org/handle/taaccct/15675,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of degree and certificate programming offered at Linn-Benton Community College’s (LBCC) iLearn campus compared to traditional programming offered at the community college on various educational, employment, and earnings outcomes.
The study used a nonexperimental design to compare the outcomes of students enrolled in the Linn-Benton (LB) iLearn campus to students enrolled in traditional programming.
The study found that program participation was significantly associated with increased rates of program completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the LB iLearn campus program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Linn-Benton Community College (LBCC) in Oregon was awarded a TAACCCT grant in October 2013. LBCC used the TAACCCT funds to design and operate a virtual college called LB iLearn campus with the goal being to help dislocated workers, veterans, and other disadvantaged adults complete education and training so they can successfully obtain employment in Oregon's in-demand industries. These include healthcare, accounting, business and office administration, communications, and marketing. The virtual campus design included incorporating stacked and latticed credentials, using a competency-based framework that could accelerate learning, and offering extensive student support services through an online platform that used education modules for instruction. Specifically, LB iLearn campus staff offered customizable, online educational programming that was student-centric and allowed students to complete the program at their own pace. Additionally, this online platform provided academic and non-academic support to students through dedicated program personnel like Student Navigators, Content Experts, and Assessment Evaluators.","The authors used a nonexperimental design to compare the outcomes of students in the LB iLearn campus program to students attending the traditional programming offered at the LBCC campus. Study participants were comprised of students enrolled with their first credit during the grant period in a TAACCCT-funded LB iLearn campus program (i.e., treatment group) or in a traditional program comparable to a TAACCCT-funded LB iLearn campus program (i.e., comparison group). The grant period for the analysis included 15 consecutive quarters from Fall 2014 to Winter 2017. The authors matched program participants to similar nonparticipants using propensity scores developed from demographic and academic information. After matching. study participants included 193 students in the treatment group and 386 students in the comparison group. Using data from LBCC’s existing program review process, the National Student Clearinghouse, and the Oregon Employment Department, the authors used statistical models to examine differences in outcomes between the groups. Outcomes included program completion, credits earned, transfer to a 4-year institution, retention in the LB iLearn or traditional campus, employment placement, and wage upon completion of the program.","Education
The study found that LB iLearn campus students were 37.21 times more likely to complete the program than traditional comparison students and this result was statistically significant.
The study found that there was no statistically significant relationship between enrollment in the LB iLearn campus programs and number of credits earned, the likelihood of transferring to a four-year academic institution, or the likelihood of program retention.
Employment
The study found that there was no statistically significant relationship between enrollment in the LB iLearn campus programs and improved employment status upon program completion.
Earnings and Wages
The study found that there was no statistically significant relationship between enrollment in the LB iLearn campus programs and improved wages upon program completion.","The authors created a matched group of non-participating students to compare to LB iLearn campus students. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the LB iLearn campus program—could explain the observed differences in outcomes. Moreover, the authors noted that they could not find comparable comparison students for two programs (Computed Tomography and Social Media Specialist). Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other barriers, Dislocated or displaced worker, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Butler Community College TAACCCT final evaluation report (Kansas State University Office of Educational Innovation and Evaluation 2017)","Butler Community College TAACCCT final evaluation report (Kansas State University Office of Educational Innovation and Evaluation 2017)","Study Type: Causal Impact Analysis","Kansas State University Office of Educational Innovation and Evaluation. (2017). Butler Community College TAACCCT final evaluation report. Manhattan, KS: Kansas State University.",,2017,https://www.skillscommons.org/handle/taaccct/15664,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Butler Community College Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant on education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the outcomes of students who were in the Butler TAACCCT Information Technology (IT) program to a comparison group.
The study found that students in the Butler TAACCCT IT program were significantly more likely to complete their program, be retained in their program, and earn credentials than their counterparts in other selected Career and Technical Education (CTE) courses.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention or include sufficient controls. This means we are not confident that the estimated effects are attributable to the Butler TAACCCT IT program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Butler Community College in El Dorado, Kansas received a TAACCCT grant to increase the number of students with credentials in the Information Technology (IT) field. Butler created the Butler IT Institute and created a core set of courses for IT students to develop their skills and knowledge. The program was designed to target TAA-eligible, veterans, and other dislocated workers. The project also integrated academic and employment supportive services for students and developed IT-related degrees, credentials, and certifications in Cyber Security, Database Administration, Digital Media, Interactive & 3D Technologies, Internetworking Management, Software Development, Web Development, and Windows Administration.","The study used a nonexperimental design to determine the impact of the Butler Community College TAACCCT IT program on education, earnings, and employment outcomes. The treatment group consisted of 657 students taking TAACCCT IT program courses between 2013 and 2010, while the comparison group consisted of 88 students taking courses in the CTE degree tracks of Engineering Graphics Technology, Composite Engineering Technology, Marketing and Management, and Welding during the same time. Using data from Butler College's administrative records, the authors compared the differences in the proportion of students who achieved education, earnings, and employment outcomes between the treatment and comparison groups.","Education and skills gain 
The study found that students who participated in the Butler TAACCCT IT program were significantly more likely than comparison group students to complete a program of study (23.0% vs. 4.5%, respectively), be retained in a program of study (29.2% vs. 4.5%), and earn credentials (23.0% vs. 4.5%).
The study found no statistically significant relationships between participation in the Butler TAACCCT IT program and credit hour completion or the likelihood of pursuing further education.
Earnings and wages
The study found no statistically significant relationship between participation in the Butler TAACCCT IT program and a wage increase post-enrollment, for students who were employed at the beginning of the program.
Employment
The study found no statistically significant relationships between participation in the Butler TAACCCT IT program and the likelihood of being employed or retaining employment upon program completion.","The authors do not establish whether the treatment and comparison groups are significantly different at baseline. They provide information indicating there may be differences in characteristics such as average age, race, incumbent worker status, part-time status, and other demographic variables, but do not report whether these variations are statistically significant. Also, the authors did not control for any baseline differences as they used chi-square analyses. These preexisting differences between the groups—and not the TAACCCT IT program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Veteran or military","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Camden County College’s Trade Adjustment Assistance for Community College Career (Mabe 2016)","Camden County College’s Trade Adjustment Assistance for Community College Career (Mabe 2016)","Study Type: Causal Impact Analysis","Mabe, W. (2016). Camden County College’s Trade Adjustment Assistance for Community College Career Training grant: Training outcomes final evaluation report. New Brunswick, NJ: Edward J. Bloustein School of Planning and Public Policy at Rutgers University.","Cleary, J. M. (2015). An evaluation of Camden County College’s Trade Adjustment Assistance for Community College Career Training grant: Year 1 program implementation. New Brunswick, NJ: Edward J. Bloustein School of Planning and Public Policy at Rutgers University.",2016,https://strategies.workforcegps.org/resources/2017/06/07/15/35/Camden-County-Co…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Camden County College (CCC) workforce development program on earnings and employment outcomes.
Using the New Jersey Unemployment Insurance (UI) wage record data system., the author conducted a nonexperimental design to compare earnings and employment outcomes of students enrolled in the workforce development program to a matched comparison group.
The study found that participation in the workforce development program was associated with higher levels of employment and higher wages in the first two quarters after graduation. However, the study did not include tests of significance.
The quality of the causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the CCC workforce development program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2012, Camden County College (CCC) received a TAACCCT grant to develop and implement flexible workforce programs in advanced manufacturing, utilities, and transportation and logistics. The goal of CCC’s workforce development program was to meet job seeker and employer needs throughout the state of New Jersey. CCC used labor market information and discussions with employers to identify demand for jobs and common skill and credential requirements. Next, they developed a customized curriculum resulting in industry-recognized credentials, and used staff from the New Jersey Community College Consortium for Workforce and Economic Development to place graduates into jobs.","The study used a nonexperimental design to compare the outcomes of students who participated in the CCC workforce development program to students who completed comparable training programs at other institutions in New Jersey. The author matched workforce development students to similar students on one earnings variable and one employment variable. Study participants included 264 students in the treatment group and 152 students in the comparison group. Data were drawn from the New Jersey Unemployment Insurance (UI) wage record data system. The author examined employment status change three quarters before and three quarters after training. The author also compared the treatment and comparison groups on the differences in average earnings before and after the training programs. However, the study did not include tests of significance.","Earnings and wages 
The study found that program participation was associated with higher wages, with CCC workforce development program students having higher wages than students in the comparison group in the first two quarters after graduation. However, the two groups had similar wages in quarters 3 and 4.
Employment 
The study found that program participation was associated with higher employment rates, with CCC workforce development program students having higher employment rates than students in the comparison group in the first two quarters after graduation. However, the two groups had similar employment rates in quarters 3 and 4.","The author did not account for preexisting differences between the groups before the CCC workforce development program, such as race/ethnicity, age, or gender. These preexisting differences between the groups—and not the program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"Kansas City Kansas Community College: Training for Employment (T4E) program (Martin & Melzer 2016)","Kansas City Kansas Community College: Training for Employment (T4E) program (Martin & Melzer 2016)","Study Type: Causal Impact Analysis","Martin, G., & Melzer, B. A. (2016). Kansas City Kansas Community College: Training for Employment (T4E) program. Kansas City, MO: Evalytics, LLC.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Kansas City Kansas Community College’s Training for Employment (T4E) program on student education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the education, earnings, and employment outcomes of T4E students to a comparison group. Using data from the college’s student database and the American Job Link Alliance, the authors conducted statistical models to examine differences in outcomes between the groups.
The study found that participation in the T4E program was significantly associated with a higher likelihood of certificate completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the T4E program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Kansas City Kansas Community College’s (KCKCC) Training for Employment (T4E) program was developed with TAACCCT funds to increase student skills, knowledge, and employment outcomes in the construction and advanced manufacturing fields. The T4E program, piloted from 2014-2016, included six primary course pathways: 1) Building and Property Management Technology, 2) Construction Technology, 3) Electrical Technology, 4) Heating and Refrigeration, 5) Machine Technology, and 6) Welding Technology. In addition, students could enroll in a green construction program called Green-Up. The T4E program targeted TAA-eligible workers and other community college students in the advanced manufacturing and construction industries and strategies included creating stacked and latticed industry credential pathways that moved students toward employment and further education, alternative technology-based teaching and instruction, basic adult education and technical training, financial literacy, employability skills, and entrepreneurship training.","The study used a nonexperimental design to compare the outcomes of students who participated in the T4E program to students who did not participate. KCKCC’s main campus is located in Kansas City, Kansas, while its two extension campuses are located in Leavenworth, Kansas. The treatment group included 153 students enrolled in one of the six T4E treatment programs or the Green-Up program and exited in 2014. The comparison group included 117 students who took one of the six T4E programs (excluding Green-Up) and exited the program in fall 2011, spring 2012, or summer 2012. Outcomes included certificate completion, program completion, withdrawal or dropout status, employment retention, and earnings. Using data from KCKCC's Ellucian student database and the American Job Link Alliance databases, the authors used statistical tests to examine differences in the outcomes between treatment and comparison groups.","Education and skills gain 
The study found that T4E participation was significantly associated with higher rates of certificate completion (59%) than comparison group students (44%).
The study found no significant relationship between T4E participation and program completion or withdrawal/dropout rates.
Employment
The study found that T4E participation was associated with higher employment retention two quarters after exit than the comparison group (90% vs. 87%, respectively). However, no tests of significance were performed.
Earnings and wages
The study found that T4E participation was associated with lower average earnings over two quarters than the comparison group ($12,826 vs. $14,341, respectively). However, no tests of significance were performed.","When reporting employment and earnings outcomes, the authors noted that the comparison group had been in the workforce longer, so differences in average earnings may represent a difference in pay related to length of employment. The authors also did not perform tests of statistical significance on employment and earnings data. Additionally, the authors used a singular cohort from previous enrollment years as the comparison group and compared them to a three-cohort treatment group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy or course difficulty) and not the intervention. Lastly, the authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not T4E—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Basic skills, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Vincennes University Logistics Training and Education Center: Final evaluation report (Belville et al. 2016)","Vincennes University Logistics Training and Education Center: Final evaluation report (Belville et al. 2016)","Study Type: Causal Impact Analysis","Belville, J. Schoeph, K., Leger, R., Jenner, E., Maddox, D., Lass, K., & Beres, S. (2016). Vincennes University Logistics Training and Education Center: Final evaluation report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/9769/VU%20LTEC%20Final%20Evaluation%20Report%20TAACCCT%20R2_Sept%202016.pdf?sequence=1&isAllowed=y",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9679/VU%20LTEC%20Final%2…,"May 2020","Community College",,"Low Causal Evidence","Interrupted Time Series Design",,"The study’s objective was to examine the impact of the Vincennes University Logistics Training and Education Center (LTEC) initiative on employment and earnings outcomes. This summary focuses on the estimated outcomes of the Fork Lift Essentials (FLE) program.
The authors used an interrupted time series design to examine employment and earnings outcomes associated with the completion of LTEC's FLE program.
The study found FLE program completion was significantly associated with improved earnings in the first quarter immediately following program completion.
The quality of causal evidence presented in this report is low because the authors did not observe outcomes for multiple periods after the intervention. This means we are not confident that the estimated effects are attributable to the Logistics Training and Education Center initiative; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2012, Vincennes University (VU) was awarded a TAACCCT grant to establish the Logistics Training and Education Center (LTEC) initiative and purchase a facility to run the programs. The initiative was implemented to serve communities across nine counties in Indiana and prepare TAA-eligible workers and other participants for employment in the logistics industry. The main goal of the LTEC initiative was to increase the number of qualified candidates by providing them with the training and opportunity for employment. LTEC offered both academic and training programs. The academic programs included supply chain logistics management and tractor-trailer driver training. The training programs included global logistics associate, team lead essentials, and fork-lift essentials. The programs offered several support services to students which included academic coaching and advisement, distance education advisors for students enrolled in online courses, student meetings with employers during class time, tutoring and resume building assistance, job board, and soft skills incorporated into short-term training program courses. Additionally, a military education department for veterans was included in the initiative. Individuals could enter LTEC programs through referrals from the workforce system, partnerships with local companies, to word-of-mouth and walk-ins.","The study took place at the LTEC initiative facility in central Indiana. The authors used an interrupted time series with eight points of pre-intervention data and one point of post-intervention data to examine employment and earnings outcomes associated with the completion of LTEC's Fork-Lift Essentials (FLE) program. The study used administrative data collected by VU and economic data from the Indiana Department of Workforce Development (DWD). Study participants included 194 students enrolled in the FLE program between April 1, 2013 and September 30, 2015 with complete pre-program and outcome economic data. Statistical models were used to estimate the FLE program completion impact on quarterly wages and employment status.","Earnings and wages
The study found that FLE program completion was significantly associated with improved earnings in the first quarter immediately following program completion, with FLE completers demonstrating an increase of $473.47 in quarterly earnings.
Employment
The study did not find a significant association between the probability of employment and completion of the FLE program. 
 ","Although the authors observed the outcomes for multiple periods before the intervention, the study examined the effect of the program at only one point in time following the intervention. This makes it difficult to conclude that the observed changes in outcomes were driven by the intervention itself, instead of other concurrent changes (for example, changes in labor market conditions). Therefore, the study receives a low causal evidence rating.",,,,"Absence of conflict of interest",,"United States","Causal Impact Analysis","Adult, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"Vincennes University Logistics Training and Education: Final evaluation report (Belville et al. 2016)","Vincennes University Logistics Training and Education: Final evaluation report (Belville et al. 2016)","Study Type: Causal Impact Analysis","Belville, J., Schoeph, K., Leger, R., Jenner, E., Maddox, D., Lass, K., & Beres, S. (2016). Vincennes University Logistics Training and Education Center: Final evaluation report. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/9679/VU%20LTEC%20Final%20Evaluation%20Report%20TAACCCT%20R2_Sept%202016.pdf?sequence=1&isAllowed=y",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9679/VU%20LTEC%20Final%2…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Logistics Training and Education Center (LTEC) initiative on employment and earnings outcomes. This summary focuses on the estimated outcomes of the Tractor-Trailer Driver Training (TTDT) program. 
The authors used a nonexperimental design to compare the outcomes of TTDT program participants to a matched comparison group consisting of two student cohorts: a historical cohort of students from previous enrollment years and a concurrent cohort of students that attended a different college campus.
The study did not find significant relationships between completion of the TTDT program and employment status or quarterly earnings.
The quality of the causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the Logistics Training and Education Center initiative; other factors are likely to have contributed.
 ","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2012, Vincennes University (VU) was awarded a TAACCCT grant to establish the Logistics Training and Education Center (LTEC) initiative and purchase a facility to run the programs. The initiative was implemented to serve communities across nine counties in Indiana and prepare TAA-eligible workers and other participants for employment in the logistics industry. The main goal of the LTEC initiative was to increase the number of qualified candidates by providing them with the training and opportunity for employment. LTEC offered both academic and training programs. The academic programs included supply chain logistics management and tractor-trailer driver training. The training programs included global logistics associate, team lead essentials, and fork-lift essentials. The programs offered several support services to students which included academic coaching and advisement, distance education advisors for students enrolled in online courses, student meetings with employers during class time, tutoring and resume building assistance, job board, and soft skills incorporated into short-term training program courses. Additionally, a military education department for veterans was included in the initiative. Individuals could enter LTEC programs through referrals from the workforce system, partnerships with local companies, to word-of-mouth and walk-ins.
Several eligibility requirements had to be met before students could enroll in the Tractor-Trailer Driver Training (TTDT) program: 1) at least 18 years old, 2) pass a Department of Transportation (DOT) physical exam including a drug screen, 3) have a valid Indiana driver's license, 4) be able to read, write and speak English, 5) have a high school diploma, GED or be in the process of obtaining a GED, and 6) pass a minimum of three knowledge exams administered by the Bureau of Motor Vehicles.","The study took place at the LTEC initiative facility in Central Indiana. The authors compared the outcomes of TTDT program participants to non-participants. A two-stage procedure was used that pre-screened treatment students with a pooled group of similar students and then employed a statistical technique to weight the analytic sample. The treatment group consisted of 64 students that completed the LTEC’s TTDT program between January 1, 2015 and September 30, 2015 with complete pre-program and outcome economic data. The comparison group was a pooled sample comprised of two cohorts of 54 students. The first were students who completed the TTDT program at LTEC between January 1, 2014 and September 30, 2014 before the implementation of the TAACCCT funding enhancements. The second were students who completed the TTDT program between January 1, 2015 and September 30, 2015 but at a different campus in the VU system (VU Fort Branch). Using administrative data collected by VU and economic data from the Indiana Department of Workforce Development (DWD), the authors conducted statistical models to examine differences in quarterly wages and employment status between treatment and comparison groups.","Earnings and wages
The study did not find a significant association between completion of the TTDT program and quarterly wages.  
Employment
 The study did not find a significant association between completion of the TTDT program and employment status. 
 
 ","The authors did not account for preexisting differences between the groups before the LTEC initiative. These preexisting differences between the groups—and not the LTEC initiative—could explain the estimated outcomes. Moreover, the authors use a pooled comparison group of two different cohorts of students: a historical group from previous enrollment years and a concurrent cohort of students that attended a different college campus. Because the outcome data for the historical cohort was collected from participants at different times, the estimated outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. The college varying characteristics of the concurrent cohort comparison group also creates a confound. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest",,"United States","Causal Impact Analysis","Adult, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"Final evaluation of the ACED grant at Salt Lake Community College (Bragg et al. 2018)","Final evaluation of the ACED grant at Salt Lake Community College (Bragg et al. 2018)","Study Type: Causal Impact Analysis","Bragg, D., Cosgrove, J., Cosgrove, M. & Blume, G. (2018). Final evaluation of the ACED grant at Salt Lake Community College. Salt Lake City, UT: Bragg & Associates, Inc.",,2018,http://www.skillscommons.org/bitstream/handle/taaccct/18574/Final-SATTS-TAA-Rep…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to evaluate the impact of a competency-based education (CBE) program offered by Salt Lake Community College on educational, employment, and earnings outcomes.
The study used a nonexperimental design to compare outcomes among students enrolled in the CBE-based program and students who were not enrolled in CBE.
The study found CBE program participation was associated with a statistically significant increase in post-program wages.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to Salt Lake Community College’s CBE Program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories. Utah’s Salt Lake Community College (SLCC) received TAACCCT funding to convert programs of study offered by its School of Applied Technology and Technology Studies (SATTS) to competency-based education (CBE) to increase student retention and increase or accelerate program completion.
SLCC’s CBE program is self-paced and its programs of study use a hybrid delivery approach that includes online learning and face-to-face instruction. Moreover, the program measures learning by the demonstration and completion of competencies rather than time to completion (i.e. credits or clock hours). The program categorized 24 programs of study into six career clusters that align to industry sectors important to the Salt Lake City region. The Test for Adult Basic Education (TABE) was used to place students into a program of study for CBE. Students enrolled in the CBE curriculum received student services where administrators tracked their progression on competencies and gave them academic and career advice. Transcripts for the CBE program delineated competencies mastered and not course names so employers could more easily understand prospective employees’ capabilities and skillset. Another feature of the CBE program is that depending on the program of study, students can obtain three levels of awards after program completion: certificate of achievement, certificate of proficiency, and certification of completion.","The study used a nonexperimental design to compare the outcomes of students who were in the CBE program with students who were not. The authors matched CBE program participants to a retrospective sample of students enrolled in similar programs of study that were not enhanced by the CBE initiative using propensity scores developed from baseline demographic information. The final analytical sample was comprised of 1,448 study participants with 724 in the treatment and 724 in the comparison group. Data sources included intake forms, databases managed by various campus units at Salk Lake Community College, and the Utah Department of Workforce Services. The authors used statistical models with controls for demographic and programmatic characteristics to examine differences in outcomes between the groups. The outcomes of interest included program completion and post-program employment and wages.","Education and skills gain
The study found no significant relationship between CBE program enrollment and program completion.
Earnings and wages
The study found that CBE program enrollment was significantly associated with an average quarterly wage increase of $674.70 post-program.
The study also found that CBE program enrollment was significantly associated with a higher quarterly wage for any quarter two years after program completion.
Employment
The study found no significant relationship between CBE program enrollment and employment in the first quarter following program completion or any quarter two years after program completion.","The authors created a matched group of non-participating students to compare to CBE program participants. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not Salt Lake Community College’s CBE program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Third-party evaluation of the outcomes and impact of the National Information, Security & Geospatial Technologies Consortium (NISGTC) (Bridges et al. 2015)","Third-party evaluation of the outcomes and impact of the National Information, Security & Geospatial Technologies Consortium (NISGTC) (Bridges et al. 2015)","Study Type: Causal Impact Analysis","Bridges, K., Bishop, C., & Giani, M. (2015). Third-party evaluation of the outcomes and impact of the National Information, Security & Geospatial Technologies Consortium (NISGTC). Champaign, IL: Office of Community College Research and Leadership, University of Illinois at Urbana-Champaign.",,2015,https://occrl.illinois.edu/docs/librariesprovider4/nisgtc/nisgtc-impact.pdf,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to assess the impact of TAACCCT-funded programming offered at National Information Security & Geospatial Technology Consortium (NISGTC) colleges on educational and employment outcomes.
Using student-level data collected by the colleges’ Institutional Research offices and states’ Unemployment Insurance data systems, the study used a nonexperimental design to compare the outcomes of students enrolled in NISGTC programs with students enrolled in similar programs of study prior to the grant.
The study found that students in the NISGTC-affected programs were significantly more likely to attain credentials and have higher average quarterly earnings than students in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analysis for the education outcomes. This means we are not confident that the estimated effects are attributable to the NISGTC-affected programs; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The National Information Security & Geospatial Technology Consortium (NISGTC) was awarded a TAACCCT grant in 2011 to improve its colleges’ programs to better assist trade-impacted and low-skilled workers in completing post-secondary education and obtaining employment in the IT industry. NISGTC is comprised of seven community colleges and their affiliates located across the U.S. Federal funding from the TAACCCT grant was distributed across these sites to facilitate the implementation of enhanced education and training strategies to help prepare students for high-wage IT jobs. The objectives of the grant included working with national businesses to identify competencies to help students gain employment, align curriculum to cover these knowledge areas, provide virtual IT laboratory access and equipment to students, offer curriculum through online modules, and provide tailored tutoring, career coaching, mentoring, and placement services.","The authors used a nonexperimental design to compare the outcomes of students who participated in NISGTC-affected programs of study with students enrolled in earlier cohorts of similar programs prior to NISGTC implementation. Student demographic and educational outcomes data was provided by staff at the colleges’ Institutional Research offices and employment-based data was provided by states’ unemployment insurance (UI) data systems. Of the seven NISGTC co-grantee colleges, only Moraine Valley Community College in Palos Hills, Illinois and Collin College in Frisco, Texas were able to provide student-level data for a comparison cohort to use in the impact analysis. For Moraine Valley Community College, the treatment group was comprised of students enrolled in any Local Area Networking (LAN) course during the fall 2012 semester. For Collin College, the treatment group included students enrolled in its Cisco Certified Network Associate (CCNA) certificate program during the fall 2012 semester. The comparison groups for these two co-grantee colleges represented the same type of students, at each college respectively, but were enrolled in these programs during the fall 2009 semester (prior to NISGTC full implementation). The authors used several statistical models to assess NISGTC-affected programs’ impact on the following outcomes: credential attainment, employment retention, and average quarterly wages. For the first outcome, data from both community colleges were combined and analyzed and the total analytical sample included 244 students in the comparison group (i.e., retrospective students) and 377 in the treatment group (i.e., NISGTC students). For the earnings and employment outcomes, only data from Moraine Valley Community College was available, and its final analytical sample included 72 retrospective students and 210 NISGTC students.","Education and skills gain
The study found a statistically significant relationship between NISGTC participation and credential attainment, where NISGTC students were more likely to attain a credential than those in the comparison cohort (33 percentage point difference).
Earnings and wages
The study found a statistically significant relationship between average quarterly wage and program participation in which NISGTC students were estimated to earn $1,772.09 more than students in the comparison cohort on average.
Employment
The study did not find a statistically significant relationship between program participation and employment.","Across all outcomes, the authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. The authors used statistical methods to ensure students in the research groups were similar in terms of baseline demographic characteristics; however, for the education outcomes, they did not include a pre-intervention measure of education/training in their education models. These preexisting differences between the groups—and not the NISGTC program—could explain the observed differences in education outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Other, Low-skilled","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"New Mexico Junior College TAACCCT evaluation report (Dauphinee et al. 2017)","New Mexico Junior College TAACCCT evaluation report (Dauphinee et al. 2017)","Study Type: Causal Impact Analysis","Dauphinee, T., Davis, M., & Carr, S. (2017). New Mexico Junior College TAACCCT evaluation report. Albuquerque, NM: University of New Mexico, Center for Education Policy and Research.",,2017,http://www.skillscommons.org/bitstream/handle/taaccct/13866/2017%20NMJC%20Evalu…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of New Mexico Junior College’s (NMJC) TAACCCT program on employment and earnings outcomes.
The authors used a nonexperimental design to compare employment and earning outcomes of NMJC TAACCCT program participants to a matched comparison group using NMJC administrative data and data from New Mexico’s Department of Workforce Solutions.
The study did not find a significant relationship between program participation and employment or earnings outcomes.
The quality of causal evidence presented in this report is low because the authors did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to the NMJC TAACCCT program; other factors are likely to have contributed program.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The goal of New Mexico Junior College’s (NMJC) TAACCCT program was to address the lack of skilled workers for high demand energy industry professional technician occupations in the region. These industries included oil and gas production, commercial driving, instrumentation and controls, and electrical transmission technology. NMJC collaborated with local industry representatives to expand existing programs and create new innovative training modules to increase the number of professional technicians in high demand fields. The curriculum focused on industry needs, developed critical thinking skills, provided opportunities for practice and training, and included ongoing evaluation of training outcomes against industry needs and expectations.","The study used a nonexperimental design to compare the outcomes of students who participated in the NMJC TAACCCT program to students who did not participate. The comparison group students were selected from non-TAACCCT programs in Automotive Technology, Welding, Energy Technology/Technician, Ford Motor Asset Program, General Motors ASEP Program, and a CDL course. These programs were similar to the TAACCCT programs in terms of entry level of education, academic rigor, length of program, skill level, number of students enrolled, and shared student background characteristics. The authors matched NMJC TAACCCT program participants to similar nonparticipants using propensity scores developed from background characteristics. Data sources included student-level administrative data from NMJC as well as employment and earnings data from New Mexico’s Department of Workforce Solutions. The authors used statistical analyses to examine differences between the groups in employment and earnings outcomes; however, the authors do not state the analysis sample size for the treatment and comparison groups.","Earnings and wages
The study did not find a significant relationship between program participation and earnings outcomes.
Employment
The study did not find a significant relationship between program participation and employment outcomes.","Although the authors stated that they conducted propensity score matching, they did not present these findings in the report or baseline tables with the demographic information for both groups. Therefore, it is unknown whether the groups were balanced on baseline characteristics. For the earnings impact analyses, the authors controlled for baseline wages. However, the authors do not account for baseline employment outcomes, as required in the protocol. These preexisting differences between the groups—and not the NMJC TAACCCT program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"2016 PETC annual evaluation report, Eastern Iowa Community Colleges (de la Mora et al. 2017)","2016 PETC annual evaluation report, Eastern Iowa Community Colleges (de la Mora et al. 2017)","Study Type: Causal Impact Analysis","de la Mora, A., Callen, E., Kemis, M., & Abraham, W. (2017). 2016 PETC annual evaluation report, Eastern Iowa Community Colleges. Ames, IA: RISE.",,2017,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Pathways to Engineering Technology Careers (PETC) program on earnings and employment outcomes.
Using student surveys and wage data, the study used a nonexperimental design to compare the outcomes of students in the PETC program to a comparison group.
The study found that students in the PETC program were more likely to have higher wages than students in the comparison group.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the PETC program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Eastern Iowa Community Colleges was awarded a four-year TAACCCT grant to support education and training for jobs with high wages and high skills required. Eastern Iowa Community Colleges includes Clinton, Muscatine, and Scott. The Pathways to Engineering Technology Careers (PETC) program provided training in manufacturing. The program included services for students, regional marketing, and a goal to increase student enrollment in Engineering Technology courses. Using a hybrid model of online and face-to-face training, the program was flexible and self-paced for students.","Using student surveys and wage data from the state of Iowa, the authors used a nonexperimental design to compare the outcomes of students in the PETC program (treatment group) to students in the Engineering Technology programs prior to PETC in Fall 2011 and Spring 2012 (comparison group). While 483 participants were enrolled in the program for the four years of implementation (2012-2016), 85 completed the program with 221 still enrolled in the program as of June 30, 2016, the time of the evaluation. The final analysis group included 19 students in the treatment group, enrolled from Fall 2014 through Spring 2015; the comparison group also included 19 students. The authors conducted t-tests to test the differences between the treatment and comparison groups on the outcomes of earnings and employment.","Earnings and wages
The study found that PETC program participation was significantly associated with higher average wages, with students in the PETC program earning $4,500 more than students in the comparison group during the second full quarter after two semesters of enrollment.
Employment
The study found no significant relationship between PETC program participation and the number of jobs held post-intervention.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not the PETC program—could explain the observed differences in outcomes. The authors also used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. The authors also noted that they had small sample sizes, which limited the type of analyses that they were able to complete. They excluded participants from the analysis who had no wages during the time of the study, non-Iowa residents, and those who received an Associate degree. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"Iowa Western 2017 ITC2 annual  evaluation report: Information technology - Credentials to Careers (ITC2), Iowa Western Community College (de la Mora et al. 2017)","Iowa Western 2017 ITC2 annual  evaluation report: Information technology - Credentials to Careers (ITC2), Iowa Western Community College (de la Mora et al. 2017)","Study Type: Causal Impact Analysis","de la Mora, A., Callen, E., Kemis, M., & Abraham, W. (2017). Iowa Western 2017 ITC2 annual evaluation report: Information technology - Credentials to Careers (ITC2), Iowa Western Community College. Arnes, IA: Research Institute for Studies in Education.",,2017,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Information Technology – Credentials to Careers (ITC2) program on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare the outcomes of students who were enrolled in Computer Information Technology (CIT) programs after the initiation of ITC2 programming to those who were enrolled in CIT programs prior to the start of the ITC2 program.
The study found that ITC2 program participation was associated with a significant increase in certificate attainment.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention and did not include sufficient controls. This means we are not confident that the estimated effects are attributable to the ITC2 program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Information Technology – Credentials to Careers (ITC2)program began in 2014 and was implemented in Computer Information Technology (CIT) programs at Iowa Western Community College (IWCC). This included programs in computer science, application and web programming, cyber security, management information systems, network and system administration, programming, and graphic communications. The ITC2 program included curriculum development, increased course offerings, faculty/staff training, facilities improvement, and links to industry and 4-year institutions.","The study used a nonexperimental design to compare the outcomes of students who participated in the ITC2 program (starting in fall 2014) to students who were enrolled in CIT programs prior to the start of the ITC2 funding (fall 2011- spring 2013). Study participants included 52 students in the treatment group and 49 in the comparison group. Using data from Iowa Western Community College and Iowa Workforce Development, the authors conducted statistical tests to examine differences in outcomes between the groups. Outcomes included certificates awarded, number of Associate degrees earned, number of jobs, wages earned, and difference in wages earned over two time periods. The authors analyzed data from participants one quarter prior to enrollment at IWCC (time 1), and data from the second full quarter after two years of enrollment at IWCC (time 2; this is the time it takes to receive an Associate degree plus one quarter).","Education and skills gain
The study found that participation in the ITC2 program was significantly associated with certificate attainment, with ITC2 students earning 10% more certificates than students in the comparison group.
However, the study did not find a significant relationship between ITC2 program participation and the number of associate degrees earned.
Earnings and wages
The study did not find a significant relationship between ITC2 program participation and wages earned at time 2.
The study also did not find a significant relationship between ITC2 program participation and differences in wages from time 1 to time 2.
Employment
The study did not find a significant relationship between ITC2 program participation and the average number of jobs held at time 2.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not the ITC2 program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"ShaleNET Round 2 TAACCCT grant third-party evaluation final report (Dunham et al. 2016)","ShaleNET Round 2 TAACCCT grant third-party evaluation final report (Dunham et al. 2016)","Study Type: Causal Impact Analysis","Dunham, K., Hebbar, L., Kheman, D., Comeaux, A., Diaz, H., Folsom, L., & Kuang, S. (2016). ShaleNET Round 2 TAACCCT grant third-party evaluation final report. Oakland, CA: Social Policy Research Associates.",,2016,https://www.spra.com/wordpress2/wp-content/uploads/2016/12/ShaleNET-Full-Report…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the ShaleNET program on earnings and employment.
The study used a nonexperimental design to compare the outcomes of students who were in ShaleNET program to a matched comparison group of individuals receiving other federally funded employment services.
The study found no statistically significant relationships between ShaleNET program participation and employment or quarterly earnings.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the ShaleNET program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The ShaleNET initiative was created using TAACCCT funds to develop capacity for the workforce training system and for college students within the oil and gas industry. The program sought to enhance career pathways trainings, support students, and create partnerships with educational institutions, industry, and the public workforce system. The ShaleNET program added and improved training programs by enhancing curricula, learning opportunities, and equipment; provided academic, career, and employment support for students, enhanced recruiting and marketing efforts, and fostered collaboration with employment centers and other industry stakeholders.","The study used a nonexperimental design to compare the outcomes of students who participated in the ShaleNET program to students who did not participate across four colleges: Pennsylvania College of Technology (PCT), Westmoreland County Community College (WCCC) in Pennsylvania, Stark State College (Stark State) in Ohio, and Navarro College (Navarro) in Texas. The treatment group included 353 students who enrolled in ShaleNET noncredit programs between fall 2012 and fall 2015. The authors matched treatment students to a pool of 5,777 comparison students who received federally funded employment-related services other than training, and who entered and exited between the third quarter of 2013 and the fourth quarter of 2015. Outcomes included the likelihood of employment and quarterly earnings one quarter after program completion. Using data from the ShaleNET hub and state unemployment insurance (UI) wage record databases, the authors conducted statistical models to examine differences in the outcomes between the treatment and comparison groups.","Employment
The study found no statistically significant relationship between ShaleNET participation and the likelihood of employment.
Earnings and wages
The study found no statistically significant relationship between ShaleNET participation and quarterly earnings one quarter after program completion.","The treatment group was formed from students who underwent community college classes and other ShaleNET initiatives, while the comparison group was formed from those who accessed other federally funded workforce services. The authors do not provide information on the specific details of services accessed or note if the individuals accessing these alternate services were in college. Thus, treatment group participants who self-selected into the college ShaleNET program could differ in observable and unobservable ways from the comparison group, affecting the observed outcomes. Lastly, while the authors created a matched group of non-participating individuals to compare to ShaleNET students, they did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention levels of employment or earnings. These preexisting differences between the groups—and not the ShaleNET program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"Final evaluation report Trade adjustment Assistance Community  College and Career Training (TAACCCT) Grant: Round 2 CyberSecurity Career Pathways Program (Pacific Research and Evaluation 2016)","Final evaluation report Trade adjustment Assistance Community  College and Career Training (TAACCCT) Grant: Round 2 CyberSecurity Career Pathways Program (Pacific Research and Evaluation 2016)","Study Type: Causal Impact Analysis","Pacific Research and Evaluation. (2016). Final evaluation report Trade Adjustment Assistance Community College and Career Training (TAACCCT) Grant: Round 2 CyberSecurity Career Pathways Program. Portland, OR: Pacific Research & Evaluation, LLC.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the CyberSecurity Career Pathways Program (CSCPP) on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare the outcomes of students who were in the CSCPP to a historical comparison group.
When compared to a historical cohort, the study found a significant relationship between CSCPP participation and high rates of program completion and retention, and wage increases.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before program participation or include sufficient control variables. This means we are not confident that the estimated effects are attributable to CSCPP; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The CyberSecurity Career Pathways Program (CSCPP) was a TAACCCT-funded program at Utah Valley University in Orem, Utah, created to fill local industry and training gaps in the area. Efforts of CSCPP focused on developing and modifying courses and programs in information systems and technology from entry-level to post-baccalaureate certificates, and Associate (AS, AAS) and Bachelor (BS) degree programs. This included AS in Information Technology, AAS in Information Technology, BS in Information Technology, and BS in Information Systems. The CSCPP also enhanced student support services (e.g., advising, mentoring, tutoring, funding for industry certification exams, career guidance), and strengthened partnerships with local industry, including formation of an advisory board.","The nonexperimental study compared the outcomes of students who were in CSCPP to a historical comparison group. The treatment group included 288 students enrolled in the four CSCPP credential programs during the first year of the grant (2012-13 academic year). The comparison group included 291 students enrolled in one of the same four credentials in the 2008-09 academic year, prior to grant implementation. The authors examined program completion, program retention, credentials earned, wage increases, and employment attained and retained. Using institutional research data from Utah Valley University and data from the Utah Department of Workforce Services, the authors conducted chi-square analyses to examine differences between the groups, but did not control for differences in student characteristics.","Education and skills gain
The study found a significant relationship between CSCPP participation and program completion, with higher program completion rates in the treatment group (42%) than the comparison group (30%).
The study also found that CSCPP participation was significantly associated with retention rates, with higher retention rates in the treatment group (28%) than the comparison group (16%).
The study found no statistically significant relationship between participation in CSCPP and credentials earned.
Earnings and wages
The study found a significant relationship between CSCPP participation and wage increases for incumbent workers, with higher proportions of the treatment group receiving wage increases (79%) than the comparison group (70%).
Employment
The study found no significant relationships between participation in CSCPP and entering or retaining employment.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not CSCPP—could explain the observed differences in outcomes. The authors also used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the program. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final evaluation report Trade Adjustment Assistance Community College and Career Training (TAACCCT) Grant: Round 2 (Pacific Research and Evaluation 2016)","Final evaluation report Trade Adjustment Assistance Community College and Career Training (TAACCCT) Grant: Round 2 (Pacific Research and Evaluation 2016)","Study Type: Causal Impact Analysis","Pacific Research and Evaluation, LLC. (2016). Final evaluation report Trade Adjustment Assistance Community College and Career Training (TAACCCT) Grant: Round 2. Portland, OR: Rogue Community College.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9295/RCC%20TAACCCT%20Fin…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Pathways to Allied Health Professions (PATH) program on education, earnings, and employment outcomes.
Using institutional and student survey data, the authors conducted a nonexperimental study to compare the outcomes of PATH program participants to non-participants.
The study found no significant associations between PATH program participation and education, earnings, and employment outcomes.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the PATH program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2012, Rogue Community College (RCC) in Grants Pass, Oregon was awarded a TAACCCT grant to establish the Pathways to Allied Health Professions (PATH) program to address local healthcare worker shortages and provide the necessary skills and training to prepare individuals for employment in high-wage, high-skill occupations. The PATH program developed a series of courses and programs in the Allied Health field at RCC. The courses were designed to be completed in less than a year and offer employer/industry recognized credentials. The PATH program also created accelerated pathways for developmental education courses in reading, writing, and math. Additionally, several student services were provided to all program participants to assist with academic advisement and career guidance. Career guidance services included: The Job Council, work-based learning opportunities, informal guidance provided by instructors or a navigation coach.","The nonexperimental study compared the outcomes of PATH program participants to non-participants. The treatment group consisted of RCC students enrolled in nursing, physical therapy/OT, clinical lab, or medical assisting programs between Fall 2012 and Spring 2015. The comparison group included RCC students enrolled in emergency medical technical, nursing, and dental assisting programs between Fall 2008 and Spring 2011. The students in the comparison group were comparable on learning objectives, credential attainment, and certification outcomes. Study participants included 97 students in the treatment group and 113 in the comparison group. Using RCC institutional data and participant survey data, the authors compared mean differences between the groups on program completion rates, retention rates, credential earned, continued enrollment, wage increases for incumbent workers, entered employment, and retained employment for non-incumbent workers. Due to data limitations and small sample sizes, the authors did not present findings for retention rates.","Education and skills gain
The study did not find significant associations between participating in the PATH program and program completion, credentials earned, or continued enrollment.
Employment
The study did not find a significant association between participation in the PATH program and entering employment or retaining employment.
Earnings and wages
The study did not find a significant association between participation in the PATH program and wage increases for incumbent workers.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. These preexisting differences between the groups—and not the PATH program—could explain the observed differences in outcomes. The authors also used cohorts from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the PATH program. Lastly, the authors reported the findings of the analyses as statistically significant. However, the authors do not define the statistical significance threshold. Without further details about the specific p-value or the threshold used to establish significance, we could not be sure that the findings reported were significant at the 5 percent level, so we have reported that there were no statistically significant impacts.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Basic skills, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final evaluation report: Trade Adjustment Assistance Community – round 4: Nevada Community College Consortium (Pacific Research and Evaluation 2018)","Final evaluation report: Trade Adjustment Assistance Community – round 4: Nevada Community College Consortium (Pacific Research and Evaluation 2018)","Study Type: Causal Impact Analysis","Pacific Research and Evaluation. (2018). Final evaluation report: Trade Adjustment Assistance Community – round 4: Nevada Community College Consortium. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/18580/NCCC%20TAACCCT%20Final%20Evaluation%20Report-Pacific%20Research%20and%20Evaluation.pdf?sequence=1&isAllowed=y",,2018,https://www.skillscommons.org/bitstream/handle/taaccct/18580/NCCC%20TAACCCT%20F…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Nevada Community College Consortium (NCCC) Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program on education, earnings, and employment.
Using community college data, the authors used a nonexperimental design to compare the outcomes of students in the program to a comparison group comprised of both concurrent and historical cohorts of students.
The study found no statistically significant relationships between NCCC and education, earnings, and employment outcomes.
The quality of causal evidence presented in this report is low because the authors did not include sufficient control variables. This means we are not confident that the estimated effects are attributable to NCCC; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Nevada Community College Consortium (NCCC) received a four-year, $9.9 million TAACCT grant. The grant was administered by Western Nevada College (WNC), as the lead college and partnered with Great Basin College (GBC) and Truckee Meadows Community College (TMCC). The NCCC program was designed to create or enhance courses in four industries that included healthcare, transportation, advanced manufacturing, and Heating, Ventilation, and Air Conditioning (HVAC). Program components as part of the grant included accelerated coursework, hands-on learning with equipment, and work-based experiences such as internships. Additionally, the program included a student support component to provide academic and career guidance. The program also partnered with local industries in curriculum development, financial assistance, supplies, recruitment, and job/internship placement.","Using data from each of the three colleges, the authors used a nonexperimental design with a comparison group of historical and concurrent students to assess differences between the groups on academic, earnings, and employment outcomes. The comparison group varied for each of the eight programs across the three schools. Two of the programs had a concurrent comparison group, while four included a historical comparison group. Two of the programs did not have a comparison group available and were not included. The sample sizes for the impact study varied across schools: GBC had 323 in the treatment and 131 in the comparison group; TMCC had 292 in the treatment and 33 in the comparison; and WNC had 339 in the treatment and 105 in the comparison group.","Education and skills gain
Using community college data, in a separate analysis for each of the three community colleges as part of the NCCC, the study found no statistically significant relationships between NCCC and completion rates, credits earned, and certificates earned.
Earnings and wages
Using data from WNC, the study found no statistically significant relationships between NCCC and earnings increases.
Employment
Using data from WNC, the study found no statistically significant relationships between NCCC and employment or employment retention.","The authors did not account for other factors that could have affected the difference between the treatment and comparison groups such as age, gender, race/ethnicity, pre-intervention measures of financial disadvantage or pre-intervention measures on the outcomes. These preexisting differences between the groups—and not the NCCC program—could explain the observed differences in outcomes. The authors also used a cohort from previous enrollment years as the comparison group in some of their analyses. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Evaluation of the Alabama/Florida Technical Employment Network TAACCCT program (PTB & Associates 2016)","Evaluation of the Alabama/Florida Technical Employment Network TAACCCT program (PTB & Associates 2016)","Study Type: Causal Impact Analysis","PTB & Associates. (2016). Evaluation of the Alabama/Florida Technical Employment Network TAACCCT program. Bethesda, MD: PTB & Associates.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/15590/George%20C.%20Wall…,"May 2020","Community College",,"Low Causal Evidence",,,"The study’s objective was to examine the impact of the Alabama-Florida Technical Employment Network (AF-TEN) on student education and employment outcomes.
The study used a nonexperimental design to compare the education and employment outcomes of students who were in AF-TEN-enhanced welding programs to a matched comparison group of program students from prior years.
The study found that participation in the AF-TEN programs was significantly associated with fewer completed credit hours.
The quality of causal evidence presented in this report is low because the authors used a comparison group from previous enrollment years, which presents a confound. This means we are not confident that the estimated effects are attributable to the AF-TEN intervention; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Alabama-Florida Technical Employment Network (AF-TEN) was created using TAACCCT funds to address regional workforce needs in the welding industry. AF-TEN targeted dislocated workers and sought to create educational pathways to credentialing, build relevant and career-focused programs, implement online and hybrid courses, technology, and learning modules, develop employer and stakeholder partnerships, provide employment placement services, and create agreements with four-year colleges. At two Network colleges in Alabama, which had existing welding programs, AF-TEN added new equipment and programs that focused on developing multiple career pathways into the welding industry for students of all experience levels.","The study used a nonexperimental design to compare the outcomes of students who participated in the AF-TEN program to a comparison group of students from prior years. The study was conducted at Wallace Community College in Dothan, Alabama and Lurleen B. Wallace Community College in Andalusia, Alabama. The treatment group included a cohort of 461 students at the two schools who newly enrolled in an AF-TEN-enhanced welding program between 2014 and 2016. The authors compared treatment students to 393 matched comparison group students in the welding program during the 2012-2013 academic year, before AF-TEN improvement. Using data from college administrative records, state employment agencies, and state one-stop centers, the author matched treatment and comparison groups using propensity score inverse probability weighting. Outcomes included total credit hours, credentials, and employed during/after enrollment. The authors used statistical tests to examine differences in the outcomes between treatment and comparison groups.","Education and skills gain
The study found that participation in the AF-TEN was significantly associated with the completion of 19.9 fewer credit hours than the comparison group.
The study did not find a significant relationship between participation in the AF-TEN and credential completion during and after enrollment.
Employment
The study did not find a significant relationship between participation in the AF-TEN and employment rates during and after enrollment.","The authors used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-Unfavorable impacts, Employment-Low-No impacts","Education and skills gains, Employment","Community College Review Protocol"
"Evaluation of the AMP PLUS program: Final report (Public Policy Associates 2016)","Evaluation of the AMP PLUS program: Final report (Public Policy Associates 2016)","Study Type: Causal Impact Analysis","Public Policy Associates. (2016). Evaluation of the AMP PLUS program: Final report. Lansing, MI: Public Policy Associates, Incorporated.",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/9571/FVTC%20TAACCCT%20Ro…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Advanced Manufacturing Pathways (AMP) PLUS program on education and employment outcomes.
The authors used a nonexperimental design to compare the outcomes of students who participated in the AMP PLUS program to a historic comparison group using the college’s administrative data.
The study found no statistically significant relationships between participating in the AMP PLUS program and student retention, degree completion, credit hours completed, or employment.
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the AMP PLUS program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
Fox Valley Technical College in Appleton, Wisconsin was awarded a TAACCCT grant to implement the Advanced Manufacturing Pathways (AMP) PLUS program. The AMP PLUS program was designed to increase student retention and completion, as well as accelerate the time to receive industry-recognized credentials. The program began in early 2013 and focused on three technical areas of need in the region: Welding, Automation/Electronics, and Machine Tool Technology. The AMP PLUS program created technology-enhanced apps to ease scheduling and shorten the time it took for students to begin career pathways, provided enhanced wraparound support services for career navigation, and developed stackable credits and additional employer-recognized non-degree certifications, in addition to providing other flexible learning opportunities (e.g., flexible scheduling).","The nonexperimental study took place at Fox Valley Technical College and compared the outcomes of students participating in the AMP PLUS program to students who did not. The treatment group consisted of 371 students who first enrolled in Welding, Automation/Electronics, or Machine Tool Technology programs between January 1, 2013 and January 1, 2015. The comparison group included 566 students who were enrolled in the same programs from 2010-2012 and left the college or graduated before the grant was implemented. Using the college’s administrative records, the authors conducted statistical models to examine differences in outcomes between the groups. Outcomes included student retention, degree completion, credit hours completed, and post-program employment.","Education and skills gain
The study found no statistically significant relationships between the AMP PLUS program and student retention, degree completion, or credit hours completed.
Employment
The study found no statistically significant relationship between the AMP PLUS program and employment.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. Specifically, the authors did not account for differences in gender or baseline education outcomes. These preexisting differences between the groups—and not the AMP PLUS program—could explain the observed differences in outcomes. In addition, the authors used a comparison group from previous enrollment years. Because the outcome data on the two groups were collected from participants at different times, observed outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-No impacts, Employment-Low-No impacts","Education and skills gains, Employment","Community College Review Protocol"
"An analysis of the Veterans Employment Initiative (Moses-Stanton 2018)","An analysis of the Veterans Employment Initiative (Moses-Stanton 2018)","Study Type: Causal Impact Analysis","Moses-Stanton, S. (2018). An analysis of the Veterans Employment Initiative. (Doctoral dissertation). Retrieved from ProQuest. Accession No. 10745075.",,2018,https://search.proquest.com/openview/c2ddd5b8502115f80bf2a7352fe1e281/1?cbl=187…,"May 2020",Veterans,,"Low Causal Evidence","Comparison Group Design",,"The study’s objective was to examine the impact Presidential Executive Order 13518 (EO13518) had on the employment of veterans who served in the military starting in September 2001 (Gulf War II-era veterans).
The author used graphical analysis to examine employment outcomes for veterans and non-veterans after the implementation of EO13518. The study used Integrated Public Use Microdata Series (IPUMS USA) data for the years 2008, 2009, 2014, and 2015.
The study did not show evidence of a relationship between the implementation of EO13518 and employment of Gulf War II-era veterans. The study did not statistically compare the employment of the veterans and non-veterans following implementation of the Executive Order.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we would not be confident that any estimated effects would be attributable to the implementation of EO13518; other factors would be likely to have contributed to the findings. However, the study did not find statistically significant effects.","To improve employment of veterans, President Obama signed EO13518, Employment of Veterans in the Federal Government, into law in 2009. This order aimed to both increase the employment of veterans within the federal government and set an example to private employers. EO13518 laid out several requirements for federal agencies, including developing plans, a program office, and human resources training for promoting veterans’ employment opportunities within the agencies.","The author examined employment outcomes for Gulf War II-era veterans and non-veterans after the implementation of EO13518. Specifically, she examined Gulf War II-era veterans (who served starting in September 2001) and non-veterans ages 18 to 54 in the years 2008, 2009, 2014, and 2015. The study used data from the IPUMS USA, which includes data from the U.S. Census and the American Community Survey. These data include information on veteran status, employment status and type, age, educational achievement, and service-connected disability.
The author used graphical analysis to examine changes in employment outcomes for veterans and non-veterans but did not compare outcomes statistically across the groups.","Employment
The study did not show evidence of a relationship between the implementation of EO13518 and employment of Gulf War II-era veterans. The study did not statistically compare the employment of the veterans and non-veterans following implementation of the Executive Order.","The author did not account for preexisting differences between the groups before the implementation of EO13518. These preexisting differences between the groups—and not the Executive Order—could explain the observed differences in outcomes. However, the study did not find statistically significant differences.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Veteran or military","Job search assistance and supportive services, Veterans' reemployment","Employment-Low-No impacts",Employment,"Veterans Review Protocol"
"Information technology career preparation for rural areas (Habs-Vaughn et al. 2019)","Information technology career preparation for rural areas (Habs-Vaughn et al. 2019)","Study Type: Causal Impact Analysis","Hahs-Vaughn, D. L., Swan, B., & Clark, M. H. (2019). Information technology career preparation for rural areas. Community College Journal of Research and Practice, 1-13.","Swan, B., Hahs‐Vaughn, D., Fidanzi, A., Serpa, A., DeStefano, C., & Clark, M. H. (2017). Florida XCEL‐IT: Information Technology Careers for Rural Areas final evaluation report. (Rep. No. CFTAACCCT2013.Y4F). Orlando, FL: University of Central Florida, Program Evaluation and Educational Research Group (PEER).",2019,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Florida Information Technology Careers for Rural Areas (XCEL-IT) program on education, employment, and earnings outcomes.
The authors used a nonexperimental design to compare the differences in outcomes between XCEL-IT program participants and a comparison group.
The study found that XCEL-IT program participation was significantly related to program completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the XCEL-IT program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Florida Information Technology Careers for Rural Areas (XCEL-IT) grant was funded through the TAACCCT program. The award was to a single-state consortium led by the College of Central Florida to include seven colleges. The community colleges represent 22 of the state's 67 counties with 15 counties located in critical economic rural areas. The program provided training in IT for manufacturing, logistics and supply chain management, and cyber security; and it also focused on entrepreneurship. The program targeted adults who were TAA eligible, or over age 24 and were unemployed or underemployed, including Veterans. Additionally, the program refined its career pathways for non-credit courses in the Bachelor of Applied Science degree in IT, cybersecurity, logistics, manufacturing, entrepreneurship, and developmental education. The program also created an Employment Results Scorecard to offer information on employment for students; and improved techniques for conducting outreach to eligible workers in rural areas.","The study used a nonexperimental design to assess the differences between the treatment group and a matched comparison group. The treatment group included students who enrolled in XCEL-IT courses during year 2 of the project (October 2014-September 2015). The comparison group was selected by each institution based on a ""best-match"" to the XCEL-IT programs. The comparison program could be a similar program but not one that was funded with other grant monies that occurred in former years or concurrently with the grant. Of the seven colleges in the consortium, one did not have a comparable program and did not provide a comparison group of students. The authors matched XCEL-IT program participants to similar nonparticipants using propensity scores developed from demographic information. The treatment group and the comparison group both had 394 students each in the analysis. Each institution provided student records with educational outcomes data and student demographic information. Each institution also provided linked unemployment wage data provided by the local workforce investment board or workforce partner. The authors tested program impacts with statistical tests on the following outcomes: program completion, continued enrollment, employment, and wage increase.","Education and skills gain
The study found that participants in the XCEL-IT program were eight times more likely to complete the program than students the comparison group, and the results were statistically significant.
There was no statistically significant association between the XCEL-IT program and the likelihood to continue enrollment in other education.
Earnings and wages
The study did not find a statistically significant relationship between the XCEL-IT program and the likelihood to receive a wage increase.
Employment
The study did not find a statistically significant relationship between the XCEL-IT program and the likelihood to enter employment for those who were not employed prior to the program.","Although the authors accounted for baseline demographic characteristics, they did not control for a preintervention measure of education in their analytic model which is required by the review protocol. The preexisting differences between the groups on this variable—and not the XCEL-IT program—could explain the observed differences in outcomes. Also, students were not matched within their colleges; the comparison group used for matching was pooled across the schools due to data issues and missing data from one college. Finally, according to the authors, due to the variety of programs offered through the XCEL-IT program (18 programs), colleges were not provided with the types of programs that would be comparable to the XCEL-IT program. These inherent differences in the comparison programs present a confound. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"College of Central Florida in Ocala, Florida Eastern
Florida State College in Cocoa, Florida
Florida Southwestern State College in Fort Myers, Florida
North Florida Community College in Madison, Florida
Palm Beach State College in Palm Beach County, Florida
South Florida State College in Avon Park, Florida
St. John’s River State College in Palatka, Florida",,"Absence of conflict of interest.",,"Rural, United States","Causal Impact Analysis","Adult, Unemployed, Veteran or military","Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Developing pathways for careers in health: The Los Angeles Healthcare Competencies  to Careers Consortium (Tan & Moore 2017)","Developing pathways for careers in health: The Los Angeles Healthcare Competencies  to Careers Consortium (Tan & Moore 2017)","Study Type: Causal Impact Analysis","Tan, C., & Moore, C. (2017). Developing pathways for careers in health: The Los Angeles Healthcare Competencies to Careers Consortium. Sacramento, CA: Education Insights Center, California State University, Sacramento.",,2017,,,"Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Los Angeles Healthcare Competencies to Careers Consortium (LAH3C) on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare outcomes between students in the LAH3C with a matched comparison group.
The study found that LAH3C program participation was significantly related to improvements in retention, credit completion, and program completion.
The quality of causal evidence presented in this study is low because the authors used a comparison group from previous enrollment years presenting a confounding factor. This means we are not confident that the estimated effects are attributable to the LAH3C; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Los Angeles Healthcare Competencies to Careers Consortium (LAH3C) received a TAACCCT grant to better structure pathways to health careers and to promote student progress, completion, and employment. The grant was implemented by a consortium of eight colleges in the Los Angeles Community College District (LACCD). LAH3C students were guided through a pathway toward college completion as they advanced through the four tiers of the Health Science Pathways for Academic, Career and Transfer Success (H-PACTS) framework. In Tier 1, students were assessed for college readiness, preexisting knowledge of the healthcare industry, and strengths and areas for improvement. Tier 2 involved Health Occupation courses designed to build foundational and core competencies in healthcare. These courses also covered concepts related to academic and career readiness, such as teamwork, cultural awareness, and professionalism. Tier 3 included pursuit of a degree in one of 11 programs of study. Students who advanced to Tier 4 completed an Associate's degree and/or transferred to a four-year institution. Support services were available to encourage and monitor students as they progress. The majority of the programs were short-term certificates, but they ranged from eight weeks to two-year degree programs.","The study used a nonexperimental design to compare the outcomes of students who participated in LAH3C to students who did not participate. The treatment group was comprised of LAH3C students across the eight colleges in the LACCD (California). The comparison group was a historical cohort of LACCD students majoring in healthcare in the 2011-2012 academic year. The authors matched LAH3C students to similar students using propensity scores developed from a set of sociodemographic and educational background variables. After propensity score matching, the treatment group included 6,056 students and the comparison group included 13,190 students. Data sources included student surveys, a student database maintained by staff, the district’s Office of Institutional Research student information system data, and employment records held by California's Employment Development Department. The authors conducted t-tests and chi-square analyses to examine differences in outcomes between the groups. For the retention and completion outcomes, they also conducted statistical models with controls for demographic variables, education, income, and study site.","Education and skills gains
The study found a significant relationship between LAH3C participation and retention rates, with LAH3C participants approximately twice as likely to be retained to the second term than students in the comparison group.
The study also found a significant relationship between LAH3C participation and program completion, with LAH3C participants seven times as likely to complete their program than students in the comparison group
The study found that LAH3C participation was significantly related to higher credit completion rates and lower rates of course failure. When compared to non-participants, LAH3C participants had higher credit completion rates (22 percentage point difference) and lower failure rates (8 percentage point difference).
However, the study found that LAH3C participation was significantly related to higher rates of courses dropped than the comparison group (4 percentage point difference).
Earnings and wages
The study found no significant relationship between LAH3C participation and earnings.
Employment
The study found no significant relationship between LAH3C participation and employment.","The comparison group was comprised of a historical cohort and included students enrolled in any healthcare major. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes at the community college) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,"East Los Angeles College (ELAC)
Los Angeles City College (LACC)
Los Angeles Harbor College (LAHC)
Los Angeles Mission College (LAMC)
Los Angeles Pierce College (LAPC)
Los Angeles Southwest College (LASW)
Los Angeles Trade Technical College (LATTC)
West Los Angeles College (WLAC)",,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Mixed impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Final evaluation of Southwest Virginia  Community College’s PluggedInVA TAACCCT Grant initiative (Topper et al. 2016)","Final evaluation of Southwest Virginia  Community College’s PluggedInVA TAACCCT Grant initiative (Topper et al. 2016)","Study Type: Causal Impact Analysis","Topper, A., Munn, W., Bor, E., Clery, S., & Smith, B. (2016). Final evaluation of Southwest Virginia Community College’s PluggedInVA TAACCCT Grant initiative. Retrieved from: https://www.skillscommons.org//handle/taaccct/15650",,2016,https://www.skillscommons.org//handle/taaccct/15650,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the PluggedInVA (PIVA) program on education and employment outcomes.
Using college administrative and state workforce data, the authors conducted a nonexperimental study to compare outcomes between students in the PIVA program with a comparison group of non-participating students.
The study found no statistically significant relationships between participation in the PIVA program and education or employment outcomes.
The quality of causal evidence presented in this report is low because the authors did not ensure the groups being compared were similar before the intervention or include sufficient control variables. This means we are not confident that the estimated effects are attributable to PluggedInVA; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2012, Southwest Virginia Community College (SWCC) in Richlands, Virginia received a TAACCCT grant to fund the PluggedInVA (PIVA) program. SWCC served the Appalachian region of Virginia with a higher poverty rate than the rest of the state. The PIVA program provided training in the high growth areas of Pharmacy Technician, Phlebotomy Technician, Paraoptometric, and Crime Scene Technician. The program aimed to serve local TAA-eligible and unemployed adults. To be eligible for the program, students needed to not have a college degree, not be a prior felon, and be accepted to the program. The programs were each six-months long and provided GED training and college credit, which was offered as stacked credentials and was transferable. Some of the classes were online or hybrid and the grant integrated more technology into the curricula. Students could receive support services, such as tutoring or referrals for other services.","The study used a nonexperimental design to compare the outcomes of students who were enrolled in the PIVA program to non-participating students who were enrolled in the same academic programs during the same time period. Study participants included 92 students in the treatment group and 51 in the comparison group. Data sources included college administrative data, data from the National Student Clearinghouse, and data from the Virginia Employment Commission. The authors compared the differences in the proportion of students who achieved education and employment outcomes between the treatment and comparison groups.","Education and skills gain
The study did not find statistically significant relationships between participation in the PIVA program and program persistence, capstone course completion, program completion, or enrollment in additional postsecondary education.
Employment
The study found no statistically significant relationship between participation in the PIVA program and employment after graduation.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. The authors note that the composition of the treatment and comparison groups varied by gender and Pell grant status and that they controlled for these differences. However, the authors do not state how the differences were controlled as they used Fisher’s Exact Test for their analyses because they could not conduct regression analyses due to small sample sizes. Moreover, the groups differed by high school completion status and prior number of credit hours accumulated and this was not controlled for in the analyses. These preexisting differences between the groups—and not the PIVA program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Unemployed, Other","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-No impacts, Employment-Low-No impacts","Education and skills gains, Employment","Community College Review Protocol"
"Oregon Credentials, Acceleration and Support for Employment (CASE) evaluation report: Results, key issues and implications for policy, practice and systems (Watrus & Fercho 2015)","Oregon Credentials, Acceleration and Support for Employment (CASE) evaluation report: Results, key issues and implications for policy, practice and systems (Watrus & Fercho 2015)","Study Type: Causal Impact Analysis","Watrus, B., & Fercho, H. (2015). Oregon Credentials, Acceleration and Support for Employment (CASE) evaluation report: Results, key issues and implications for policy, practice and systems. Retrieved from https://www.roguecc.edu/Programs/CareerPathways/pdf/CASE%20final%20evaluation%20report_Final.pdf",,2020,https://www.roguecc.edu/Programs/CareerPathways/pdf/CASE%20final%20evaluation%2…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Credentials, Acceleration and Support for Employment (CASE) program on education and employment outcomes.
The authors used a nonexperimental design to compare the outcomes of students who were in the CASE program to a matched comparison group.
The study found a positive relationship between participation in the CASE program and program completion and employment; however, the study did not include tests of statistical significance.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar or use sufficient. This means we are not confident that the estimated effects are attributable to the CASE program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2011, a TAACCCT grant was awarded to fund the Credentials, Acceleration and Support for Employment (CASE) program. The CASE program was a collaborative effort between 17 community colleges in Oregon, a number of state agencies, employers, and community partners. Its main objective was to improve educational and employment outcomes for TAA eligible, unemployed, and underemployed individuals using three strategies: career pathways, career coaching, and credit for prior learning. Participants were eligible to receive CASE program components such as start-to-finish career coaching and enhanced curriculum.","The study was conducted across 17 community colleges in Oregon that took part in the CASE program. The authors examined differences in educational and employment outcomes between students participating in the CASE program and a matched comparison group of students. The groups were matched on age, gender, adult basic skills (ABS), and the community college attended. The treatment group was constructed based on programs of study aggregated up to career areas and consisted of participants who entered training in 2011-12. The comparison group included participants who entered training in 2009-10 that were enrolled in key courses tied to the programs of study in which the treatment participants were enrolled. Sample sizes for both groups ranged from 76-296 depending on the program of study. Three different data sources were used in the analysis: I-TRAC was used for the CASE group educational outcomes, OCCURS was used for the comparison group educational outcomes, and Oregon Employment Department unemployed insurance (UI) wage data was used for the employment outcomes for both groups. The authors used statistical models to evaluate differences between the two groups on completion rates, credential earned, credits completed, degree, entered employment rate, and employment retention rate. However, the study did not include tests of statistical significance.","Education and Skills Gain
The study found a relationship between participation in the CASE program and program completion rates, where CASE participants were more likely to complete the program than comparison group students across all five programs of study examined (Business Management, Food & Natural Resources, Health Services, Human Resources, and Industrial & Engineering Systems).
The study found a relationship between participation in the CASE program and earned credentials, where CASE participants were more likely to complete the program than comparison group students across all five programs of study examined.
The study found a relationship between participation in the CASE program and earned degrees, where CASE participants were more likely to complete the program than comparison group students across all five programs of study examined.
Employment 
The study found a relationship between participation in the CASE program and entered employment, where CASE participants had higher entered employment rates than comparison group students for all fields except Food & Natural Resources.
The study found a relationship between participation in the CASE program and retained employment, where CASE participants had higher retained employment rates than comparison group students for Business Management, Food & Natural Resources, and Human Resources.","The authors created a matched group of non-participating students to compare to the CASE students. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention degree of financial disadvantage. These preexisting differences between the groups—and not the CASE program—could explain the observed differences in outcomes. The study also examined outcomes across 17 community colleges, with the authors noting that not all the treatment students accessed the same CASE program services. Therefore, services varied across the Oregon community college system and these differences in implementation could have also impacted the observed outcomes. Lastly, the author used a cohort from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the CASE program.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Unemployed, Other","Capacity building programs, Community college education and other classroom training","Education and skills gains-Low-No impacts, Employment-Low-No impacts","Education and skills gains, Employment","Community College Review Protocol"
"Evaluation of the Illinois Network for Advanced Manufacturing: Final Report (Westat 2016)","Evaluation of the Illinois Network for Advanced Manufacturing: Final Report (Westat 2016)","Study Type: Causal Impact Analysis","Westat. (2016). Evaluation of the Illinois Network for Advanced Manufacturing: Final Report. Rockville, MD: Westat & GEM Software Development, Inc.",,2016,https://www.researchgate.net/publication/315664265_Evaluation_of_the_Illinois_N…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Illinois Network for Advanced Manufacturing (INAM) program on education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the outcomes of students who were in the INAM program to a comparison group of students who took similar courses before the creation of INAM. Using data from college databases and the Illinois Department of Employment Security, the authors conducted statistical models to examine differences in outcomes between the groups.
The study found that participation in the INAM program was significantly associated with improved rates of certificate or degree completion.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the INAM program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories
The Illinois Network for Advanced Manufacturing (INAM) was developed using TAACCCT funds and consisted of 21 colleges throughout the state. INAM aimed to prepare participants for advanced manufacturing careers in six areas: certified production technicians, mechatronics, precision machining (CNC or computer numerical control), industrial maintenance, welding, and metalworking. The programs targeted TAA workers, veterans, incumbent workers, and the unemployed, as well as other students who applied. To achieve their goal, INAM: (1) developed student educational plans; (2) improved curricula to facilitate attainment of industry credentials; (3) obtained industry-relevant equipment; (4) developed online and technology-based learning; (5) recognized prior student academic achievements to accelerate program completion; (6) created internships and on-the-job training opportunities; (7) instituted career placement services; and (8) created articulation agreements with higher learning institutions.","The nonexperimental study was conducted at the colleges that formed INAM, although several colleges were excluded due to the strategies used to recruit treatment and comparison group students. The treatment group for educational outcomes consisted of 2,645 students taking INAM courses who were recruited into the INAM program by education planners from summer 2013 through 2015. The planners intentionally excluded some students who did not fit the program intent, were not intending on taking further courses, had poor grades, or had already taken other INAM courses. For employment and wage outcomes, this sample was restricted to only students in one of four chosen colleges who completed a certificate. The comparison group consisted of students enrolled in similar courses before the creation on INAM, including those in INAM courses but not in the INAM program. Furthermore, the comparison group for employment outcomes was limited to 549 students from four colleges where wage data was available.
Outcomes included completion rates, incumbent workers with wage increases, incumbent workers with no wage increases, wage declines after enrollment, non-incumbent workers with increased wages, non-incumbent workers with zero wages before and after college entrance, predicted wages in the twelfth quarter, and employment. Using data from college databases and the Illinois Department of Employment Security, the authors conducted statistical models to examine differences in outcomes between treatment and comparison groups.","Education and skills gain
The study found that participation in the INAM program was significantly associated with improved rates of completion; students in the treatment group were 1.4 times as likely to complete a certificate or degree than students in the comparison group.
Earnings and wages
The study found no significant relationship between participation in the INAM program and predicted wages in the twelfth quarter.
Employment
The study found no significant relationship between participation in the INAM program and employment.","When creating the treatment and comparison groups, education planners selectively approached students in INAM courses with information on enrolling in the program. Potentially inconsistent selection of students, as well as student self-selection into the INAM program, represents a confound. Additionally, the authors used students from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Lastly, the authors did not account for preexisting differences between the groups before program participation or include sufficient control variables. The authors did not provide demographic information about the comparison group. These preexisting differences between the groups—and not the INAM program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Dislocated or displaced worker, Unemployed, Other, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"East Los Angeles College: Technology & Logistics program TAACCCT final evaluation report (Wiima 2016)","East Los Angeles College: Technology & Logistics program TAACCCT final evaluation report (Wiima 2016)","Study Type: Causal Impact Analysis","Wijma, C. (2016). East Los Angeles College: Technology & Logistics program TAACCCT final evaluation report. San Francisco, CA: WestEd.",,2016,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the enhanced East Los Angeles College’s (ELAC) Technology and Logistics program on student’s education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the education, earnings, and employment outcomes of students in the ELAC Technology and Logistics program to a matched comparison group of students in the Automotive Technology program.
The study found that participation in the ELAC Technology and Logistics program was significantly associated with improved course and degree completion.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the enhanced ELAC Technology and Logistics program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The East Los Angeles College’s (ELAC) Technology and Logistics program used TAACCCT funds to fulfill employer needs in the in-demand logistics industry. To do this, they improved their Logistics program by creating and improving courses, implementing an Associate of Science degree in Technology and Logistics, and providing supports for students in the Logistics program, including online student coaching; career/job coaching; life/academic coaching with career guidance counselor assistants; and academic counseling.","The study used a nonexperimental design to compare the outcomes of students who participated in the ELAC Technology and Logistics program to students who did not participate. The treatment group included 504 students who enrolled in the ELAC Technology and Logistics program during the 2012, 2013, and 2014 school years. The comparison group were students from the Automotive Technology (AutoTech) program enrolled during the same time period. Treatment students were matched with comparison students using age group, ethnicity, education level, educational goal attainment, and financial aid variables. Outcomes included course and certificate/degree completion rates, employment rates, median and mean quarterly wages, and mean weekly wage. Using data from ELAC's student database, departmental student data, the State of California’s Employment Development Department, and student exit surveys, the author used statistical models to examine differences in the outcomes between treatment and comparison groups. However, the study did not include tests of statistical significance for the employment and earnings outcomes.","Education and skills gain
The study found that participation in the ELAC Technology and Logistics program was significantly associated with improved course (51.1% vs 25.5%) and degree (37.1% vs 12.2%) completion than participation in the comparison program.
Employment
The study found that participation in the ELAC Technology and Logistics program was associated with lower rates of employment than participation in the comparison program (58.8% vs. 63.7%). However, the study did not include tests of statistical significance.
Earnings and wages
The study found that participation in the ELAC Technology and Logistics program was associated with higher wages than participation in the comparison program: median quarterly wages were $6,540 for the treatment vs. $4,332 for the comparison group; mean quarterly wages were $7,512 for the treatment vs. $5,658 for the comparison group; and, average weekly wages were $343 for the treatment vs. $277 for the comparison group. However, the study did not include tests of statistical significance.","Although the author reports differences in employment rates and wages between the treatment and comparison groups, no tests of significance were performed on these outcomes. Furthermore, the author did not control for gender in the matching analysis as required by the protocol. Lastly, the author did not report the final number of AutoTech students included in the analysis sample for the comparison group, nor information about their demographics. Because the author did not ensure groups were similar at baseline, these differences—and not the ELAC Technology and Logistics program—could explain the observed differences in the outcome. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training, Science, Technology, Engineering, and Math (STEM) programs","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Oklahoma City Community College TAACCCT: Commercial Food Equipment Service Technician (CFEST) program third-party evaluation final report (WorkED Consulting 2018)","Oklahoma City Community College TAACCCT: Commercial Food Equipment Service Technician (CFEST) program third-party evaluation final report (WorkED Consulting 2018)","Study Type: Causal Impact Analysis","WorkED Consulting. (2018). Oklahoma City Community College TAACCCT: Commercial Food Equipment Service Technician (CFEST) program third-party evaluation final report. Burke, VA: WorkED Consulting, LLC.",,2018,,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the Commercial Food Equipment Service Technician (CFEST) program on education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the outcomes of CFEST program participants to a matched comparison group at the same community college.
The study did not find a significant association between participation in the CFEST program and earned credentials, program completion, attainment of employment, or hourly wage.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the CFEST program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
In 2014, Oklahoma City Community College (OCCC) was awarded a TAACCCT grant to implement the Commercial Food Equipment Service Technician (CFEST) program, designed to train a workforce in commercial food equipment maintenance and repair with skills in hydraulics/pneumatics, refrigeration, electrical, gas, and steam equipment. The program used a competence-based training model that included in-person instruction and applied learning. Industry-recognized credentialing and credit for prior learning were key components of the program. CFEST launched in July 2015 as a four- to six-week program. CFEST incorporated feedback from employers and students, and content areas were enhanced and modified for the program over time. By the fall of 2017, the program had been expanded to a 12 to 14-week program. In addition to college certificates and industry-recognized credentials, social support services (e.g., counseling, advising, and referrals) were offered to participants.","The study used a nonexperimental design to compare the outcomes of students who participated in the CFEST program to a comparison group of students in another training program at the same community college. All study participants self-enrolled into their training programs. Propensity score matching was used to form a comparison group similar to the students in the CFEST program. The treatment group included 117 participants and the comparison group included 18 participants from a pharmacy technology program. Using student surveys and administrative data, the authors compared earned credentials and program completion rates between groups, and earnings and employment outcomes of participants in both groups before and after program participation.","Education and skills gain
The study did not find significant relationships between CFEST program participation and earned credentials or program completion.
Earnings and wages
The study did not find a significant relationship between CFEST program participation and hourly wages.
 Employment
The study did not find a significant relationship between CFEST program participation and employment.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables, including controlling for the age of the participants. These preexisting differences between the groups—and not the CFEST program—could explain the observed outcomes. The authors also used participants from a different program of study as the comparison group. The study’s estimated effects could be attributed to program varying factors (e.g., academic design, program structure, expectations) and not the CFEST program. Finally, self-reported wage data were from 11 CFEST participants and 9 comparison group participants, not the full sample, making sample sizes insufficient for analysis. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Trade Adjustment Assistance Community College and Career Training II Grant  Program (TAACCCT II): Summative Evaluation of the University of District of Columbia, Community College’s Transportation Academy (Yoder & Ryland 2016)","Trade Adjustment Assistance Community College and Career Training II Grant  Program (TAACCCT II): Summative Evaluation of the University of District of Columbia, Community College’s Transportation Academy (Yoder & Ryland 2016)","Study Type: Causal Impact Analysis","Yoder, B., & Ryland, A. (2016). Trade Adjustment Assistance Community College and Career Training II Grant Program (TAACCCT II): Summative Evaluation of the University of District of Columbia, Community College’s Transportation Academy. Retrieved from https://www.skillscommons.org/bitstream/handle/taaccct/15654/University%20of%20the%20District%20of%20Columbia%20-%20Final%20Evaluation%20-%202017.01.05.pdf?sequence=1&isAllowed=y",,2016,https://www.skillscommons.org/bitstream/handle/taaccct/15654/University%20of%20…,"May 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the TAACCCT funded Transportation Academy program on education, earnings, and employment outcomes.
The study used a nonexperimental design to compare education, earnings, and employment outcomes of Transportation Academy participants to a matched comparison group from three different institutional data sources.
The study did not find a significant association between participation in the Transportation Academy program and education, earnings, or employment outcomes.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before intervention. This means we are not confident that the estimated effects are attributable to the Transportation Academy program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The University of the District of Columbia, Community College (UDC-CC) was awarded a TAACCCT grant to create and implement the Transportation Academy, a three-year program designed to train unemployed and underemployed Washington DC residents to work in transportation related fields. The Transportation Academy focused on building programs in three key areas of transportation needs: automotive technology, electrical, and heating, ventilation, and air conditioning (HVAC). There were five core elements implemented with the Transportation Academy: evidence-based design, industry engagement to identify credentials, plans to stack and lattice credentials, online and technology-enabled learning, and strategic alignment.","The study used a nonexperimental design to compare the outcomes of students who participated in the Transportation Academy program to a comparison group of non TAACCCT participants in a similarly structured credentialing program, the Construction Academy. Propensity score matching was used to form a comparison group similar to the students that received the program using demographic information. Study participants included 38 students in the treatment group and 38 in the comparison group. Three different data sources were used in the analyses: AspirePath data system and the National Student Clearinghouse for educational outcomes and Jacob France Institute data for employment and earnings outcomes. The authors conducted a series of chi-square tests to evaluate differences between the groups on the outcomes of continued enrollment, employment post-program completion, retained employment, and wage increase.","Education and skills gain
The study did not find significant relationships between Transportation Academy program participation and further enrollment in educational courses and completion rates.
Earnings and wages
The study did not find a significant relationship between Transportation Academy program participation and wage increases.
Employment
The study did not find significant relationships between Transportation Academy program participation and employment or retained employment.","The authors did not account for preexisting differences between the groups before program participation or include sufficient control variables, including a pre-intervention measure of financial disadvantage or a pre-intervention measure on education and employment. These preexisting differences between the groups—and not the Transportation Academy program—could explain the observed outcomes. The authors also used cohorts from different programs of study as the comparison group. The study’s estimated effects could be attributed to program varying factors and not the Transportation Academy. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Unemployed, Other","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"North Dakota State College of Science TAACCCT Project: North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI) final evaluation report (WorkED Consulting, 2017)","North Dakota State College of Science TAACCCT Project: North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI) final evaluation report (WorkED Consulting, 2017)","Study Type: Causal Impact Analysis","WorkED Consulting. (2017). North Dakota State College of Science TAACCCT Project: North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI) final evaluation report. Burke, VA: WorkED Consulting",,2017,,"August 2020","Community College",,"Low Causal Evidence","Interrupted Time Series Design",,"The study’s objective was to examine the impact of the North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI) on earnings and employment outcomes.
The authors used an interrupted time series design to examine employment and earnings outcomes associated with the completion of the NDAMSTI program.
The study found that participation in the NDAMSTI program was associated with higher hourly wages, on average. However, the study did not include tests of statistical significance.
The quality of causal evidence presented in this report is low because the authors did not account for trends in outcomes before the intervention. This means we are not confident that the estimated effects are attributable to the NDAMSTI program; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
North Dakota State College of Science (NDSCS) was awarded a TAACCCT grant to establish the North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI). The main goal of the NDAMSTI program was to upgrade skills for trade impacted workers, other dislocated workers, and veterans to pursue jobs and careers in the high-demanding fields of welding, manufacturing, and mechatronics. Features of the program included developing and expanding online learning options, developing new curriculum tailored to adult learners, building stackable programs and credentials, expanding hours and locations for classes and access to programming, promoting degree completion options, and enhancing transfer and articulation to four-year universities.","The study took place at North Dakota State College of Science (NDSCS) in Wahpeton, North Dakota. The authors used an interrupted time series to evaluate differences in outcomes before and after participation in the NDAMSTI program. Study participants included 80 students participating in the NDAMSTI program who completed both the baseline survey (administered at the beginning of the program) and exit survey (administered the semester that the student completed the program). Using survey, NDSCS administrative, and state employment and education data, the authors conducted statistical tests to estimate the NDAMSTI program completion impact on earnings and wages, and employment status. However, the study did not include tests of statistical significance.","Earnings and wages
The study found that NDASMTI program completion was associated with improved wages, where participants who were employed at completion had a higher hourly wage on average ($13.82) compared to baseline wages ($10.79).
Employment
The study did not find an association between employment status and completion of the NDASMTI program.","The authors compared the employment and earnings outcomes of participants measured once before and once after they participated in the NDASMTI program. CLEAR’s guidelines require that the authors observe outcomes for multiple periods before the intervention to rule out the possibility that participants had increasing or decreasing trends in the outcomes examined before enrollment in the program. Without knowing the trends before program enrollment, we cannot rule this out. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis",Adult,"Basic skills, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"North Dakota State College of Science TAACCCT Project: North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI) final evaluation report (WorkED Consulting, 2017)","North Dakota State College of Science TAACCCT Project: North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI) final evaluation report (WorkED Consulting, 2017)","Study Type: Causal Impact Analysis","WorkED Consulting. (2017). North Dakota State College of Science TAACCCT Project: North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI) final evaluation report. Burke, VA: WorkED Consulting.",,2017,,"August 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI) on education, earnings, and employment outcomes.
The authors used a nonexperimental design to compare the outcomes of the NDAMSTI program participants to a historic cohort of students enrolled in the same welding program five years prior to the grant funding.
The study found that NDAMSTI participation was positively related to program completion and negatively related to rates of continuing education, employment, and wage increases. However, the study did not include tests of statistical significance.
The quality of causal evidence presented in this report is low because the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the NDAMSTI program; other factors are likely to have contributed.","The U.S. Department of Labor’s (DOL) Trade Adjustment Assistance for Community Colleges and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
North Dakota State College of Science (NDSCS) was awarded a TAACCCT grant to establish the North Dakota Advanced Manufacturing Skills Training Initiative (NDAMSTI). The main goal of the NDAMSTI program was to upgrade skills for trade impacted workers, other dislocated workers, and veterans to pursue jobs and careers in the high-demanding fields of welding, manufacturing, and mechatronics. Features of the program included developing and expanding online learning options, developing new curriculum tailored to adult learners, building stackable programs and credentials, expanding hours and locations for classes and access to programming, promoting degree completion options, and enhancing transfer and articulation to four-year universities.","The nonexperimental study was conducted at North Dakota State College of Science (NDSCS) in Wahpeton, North Dakota and compared students who participated in the NDAMSTI program to students who did not participate. The authors matched NDAMSTI participants to similar non-participants using demographic information. The treatment group included 300 students who participated in the NDAMSTI program and were enrolled in the NDSCS welding program from Fall 2016 to Spring 2017. The comparison group was a historical cohort composed of 325 students that were enrolled in the same NDSCS welding program from Fall 2009 to Spring 2013, five years prior to the implementation of the NDAMSTI program. Data sources included baseline and endline surveys, NDSCS administrative data, and state employment and education data. The authors examined differences in program completion, continuing education, wages, and employment status between treatment and comparison groups. However, the study did not include tests of statistical significance.","Education and skills gains
The study found that NDAMSTI participation was related to program completion, with program participants having higher completion rates than comparison students (81% vs. 65%).
However, the study found that a higher percentage of students in the comparison group continued their education compared to students in the treatment group (53% vs. 47%).
Earnings and wages
The study found a negative association between program participation and improved wages, with higher rates of wage increases in the comparison group (84%) than the treatment group (78%).
Employment
The study found that NDAMSTI participation was related to lower rates of employment, where 64% of treatment participants were employed after program completion compared to 66% of participants in the comparison group.","The authors created a matched group of non-participants to compare to NDAMSTI participants. However, the authors did not account for other factors that could have affected the difference between the treatment and comparison groups, such as pre-intervention education/training or degree of financial disadvantage as required by the protocol. These preexisting differences between the groups—and not the NDAMSTI program—could explain the observed differences in outcomes. Additionally, the authors used a cohort of NDSCS students from previous enrollment years as the comparison group. Because the outcome data on the two groups were collected from participants at different times, differences in outcomes could be due to time-varying factors (such as overall changes in the economy) and not the intervention. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest. ",,"United States","Causal Impact Analysis",Adult,"Basic skills, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Southeastern Economic and Education Leadership Consortium (SEELC) final report (Takyi-Laryea et al. 2017)","Southeastern Economic and Education Leadership Consortium (SEELC) final report (Takyi-Laryea et al. 2017)","Study Type: Causal Impact Analysis","Takyi-Laryea, A., Gall, A., Chamberlin, M., Naughton, L., & Spychala, B. G. (2017). Southeastern Economic and Education Leadership Consortium (SEELC) final report. Fairfax, VA: ICF.",,2017,http://www.skillscommons.org/handle/taaccct/13986,"August 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the WorkKeys® assessment, a component of the TAACCCT-funded programming at Southeastern Economic and Education Leadership Consortium (SEELC) colleges, on education, earnings outcomes, and employment.
The authors used a nonexperimental design to compare the education, earnings outcomes, and employment of SEELC program participants that took the WorkKeys® assessment to a matched comparison group of SEELC participants enrolled at the same colleges who did not take the assessment.
The study found a significant relationship between taking the WorkKeys® assessment and earnings, with SEELC participants who took the assessment receiving higher post-program hourly wages than the comparison group of students who did not take the assessment.
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention nor include sufficient control variables. This means we are not confident that the estimated effects are attributable to the SEELC consortium WorkKeys® assessment; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Southeastern Education and Economic Leadership Consortium (SEELC) received TAACCCT funds to administer the program across six community and state colleges (Northeast State Community College, Palm Beach State College, Pellissippi State Community College, Polk State College, Randolph Community College, and Vance-Granville Community College) in three states: Florida, North Carolina, and Tennessee. The program goal was to improve education and training opportunities for trade adjustment assistance (TAA) eligible workers, veterans, and other adults who have been unemployed by facilitating a change in its approach to serving employers, workers, and the community at large. The SEELC program focused on three target industries: welding, computer-integrated machining, and advanced manufacturing technology. The program implemented three evidence-based strategies: 1) creating career pathways, accredited testing facilities, and applied credit options; 2) providing student support services; and 3) promoting employer and industry engagement. The SEELC program also used the WorkKeys® assessment, developed by ACT, to identify education and training gaps that needed to be addressed for each student. The assessment provided students with a score that could then be matched to employers and job opportunities.","The nonexperimental study was conducted at three of the six colleges in the consortium (Vance-Granville, Northeast, and Palm Beach) and compared the education, employment, and earnings outcomes of students who took the WorkKeys® assessment to students who did not. The study sample included 917 students enrolled in the grant funded programs between Fall 2014 and Spring 2017. The treatment group consisted of 115 students who completed the WorkKeys® assessment as part of the intervention. The comparison group consisted of students from the same three colleges enrolled in the grant funded programs who did not complete the WorkKeys® assessment. Treatment students were matched with comparison students using demographic characteristics. Data sources included administrative data records, surveys, and program documentation. The study examined the impact of the WorkKeys® assessment on credential completion, credential attainment, earnings and wages, and employment by comparing differences between the treatment and comparison groups.","Education and skills gains
The study did not find significant relationships between taking the WorkKeys® assessment and completion of any credential, attaining a certificate, or attaining a degree.
Earnings and wages
The study found a significant relationship between taking the WorkKeys® assessment and earnings for incumbent workers. On average, post-program hourly wages were $2.36 higher for incumbent working students who took the assessment compared to students who did not take the assessment.
Employment
The study did not find a significant relationship between taking the WorkKeys® assessment and employment in the first quarter.","The authors used propensity score matching to create a comparison group; however, they did not account for preexisting differences between the groups in baseline education and earnings outcomes or include sufficient control variables as outlined in the protocol. These preexisting differences between the groups—and not the SEELC consortium WorkKeys® assessment—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.",,"United States","Causal Impact Analysis","Adult, Veteran or military","Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Hendricks et al. 2018)","University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Hendricks et al. 2018)","Study Type: Causal Impact Analysis","Hendricks, A., Mitran, A., & Ferroggiaro, E. (2018). University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Contract #: PO-GF-2015-C-0134-DJ). Fairfax, VA: ICF.",,2018,,"August 2020","Community College",,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of the University of the District of Columbia Community College (UDC-CC) grant-funded healthcare and hospitality programs on education, earnings, and employment outcomes. This summary focuses on the healthcare program. 
The authors used a randomized controlled trial to compare the education, employment, and earnings outcomes of participants enrolled in healthcare pathway courses and received the contextualized learning program, compared to participants enrolled in the same program, who received the traditional curriculum.
The study found that program participation was significantly related to increased credential attainment.
The quality of causal evidence presented in this report is low because it was based on a randomized controlled trial with unknown attrition and the authors did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to the UDC-CC grant-funded healthcare program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The University of the District of Columbia Community College (UDC-CC) developed a program using TAACCCT grant funds to implement training pathways within the framework of the healthcare and hospitality industries. The goal was to develop a program in these high-growth, high-demand industries that addressed the needs of District employers. UDC-CC had previously received a TAACCCT grant that provided improved training pathways, focusing on low-skilled adults, both incumbent and new workers. The currently funded program expanded on the hospitality and healthcare programs by promoting lifelong learning and attainment of stackable credentials to assist participants in pursuing promising career pathways. Enhancements included contextualized and work-based learning, accelerated classes, technology-enabled classes, improved computer labs, staff and instructor development initiatives, and an improved student intake process.","The study took place on the campus of the UDC-CC in Washington, District of Columbia. The authors used a randomized controlled trial to compare the outcomes of the treatment group to the control group. Participants included students enrolled between Summer 2016 and Spring 2018 in one of the seven courses in the healthcare pathway (Electronic Health Records, First Aid & CPR, Introduction to Healthcare, Medical Billing & Coding, Medical Office Administrative Assistant, Medical Terminology & Anatomy, and Nursing Assistant). All participants had the same recruitment and intake process, including the Comprehensive Adult Participant Assessment Systems exam, document check, and discussions with career advisors. The treatment group consisted of 291 students that were enrolled in the healthcare pathway program courses and received contextualized learning. The control group consisted of 188 students who were enrolled in the same healthcare course but received the traditional curriculum. The study used UDC-CC administrative records, follow-up surveys, and unemployment insurance data to examine the impact of the grant-funded healthcare program on education, earnings, and employment outcomes.","Education and skills gains
The study found a significant relationship between grant-funded healthcare program participation and credential attainment, with higher proportions of students in the treatment group earning a certificate in a healthcare program (66%) than students in the control group (55%).
Earnings and wages
The study did not find a significant relationship between grant-funded healthcare program participation and quarterly wages.
Employment
The study did not find any statistically significant relationships between grant-funded healthcare program participation and full- or part-time employment.","The study was a randomized controlled trial with unknown attrition in each group. In cases of high or unknown attrition, a study can receive a moderate causal evidence rating if the analysis controls for possible differences in background characteristics of the treatment and control groups. However, the authors in this study did not control for any such differences between the groups due to the empirical approach used. These preexisting differences between the groups—and not the grant-funded healthcare program—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating. Finally, the authors recognize that some contamination occurred between the treatment and control groups, leading to some control participants receiving elements of the treatment curriculum and thus potentially affecting the observed differences between the groups.",,,,"Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.",,"United States","Causal Impact Analysis","Adult, Low-skilled","Basic skills, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Hendricks et al. 2018)","University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Hendricks et al. 2018)","Study Type: Causal Impact Analysis","Hendricks, A., Mitran, A., & Ferroggiaro, E. (2018). University of the District of Columbia - Final annual evaluation report TAACCCT grant program, round 4 (Contract #: PO-GF-2015-C-0134-DJ). Fairfax, VA: ICF.",,2018,,"August 2020","Community College",,"Low Causal Evidence",Nonexperimental,,"The study’s objective was to examine the impact of the University of the District of Columbia Community College (UDC-CC) grant-funded healthcare and hospitality programs on earnings and employment outcomes. This summary focuses on the hospitality program. 
The authors used a nonexperimental design to compare the earnings and employment outcomes of participants before and after they participated in the grant-funded hospitality program.
The study found that program participation was significantly related to increased earnings.
The quality of causal evidence presented in this study is low because the authors did not observe outcomes for multiple time points before the intervention nor account for selection into the intervention. This means we are not confident that the estimated effects are attributable to the UDC-CC grant-funded hospitality program; other factors are likely to have contributed.","The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The University of the District of Columbia Community College (UDC-CC) developed a program using TAACCCT grant funds to implement training pathways within the framework of the healthcare and hospitality industries. The goal was to develop a program in these high-growth, high-demand industries that addressed the needs of District employers. UDC-CC had previously received a TAACCCT grant that provided improved training pathways, focusing on low-skilled adults, both incumbent and new workers. The currently funded program expanded on the hospitality and healthcare programs by promoting lifelong learning and attainment of stackable credentials to assist participants in pursuing promising career pathways. Enhancements included contextualized and work-based learning, accelerated classes, technology-enabled classes, improved computer labs, staff and instructor development initiatives, and an improved student intake process.","The study took place on the campus of the UDC-CC in Washington, District of Columbia. The authors compared the outcomes of participants before and after they participated in the hospitality program. Participants included 108 students that were enrolled in hospitality pathway program courses between Summer 2016 and Fall 2017. The four hospitality courses included (1) Guest Services—Hospitality, (2) Front Desk Representative, (3) Customer Service—Retail, and (4) Bring Your ""A"" Game. All participants completed the Comprehensive Adult Participant Assessment Systems exam prior to enrolling in program courses. The study used UDC-CC administrative records, follow-up surveys, and unemployment insurance data to examine the impact of the grant-funded hospitality training program on earnings and employment outcomes.","Earnings and wages
The study found that hospitality program completion was significantly associated with increased earnings, where average quarterly wages increased from $985 to $2,197 three months after exiting the program.
Employment
The study found no statistically significant relationships between hospitality program participation and full- or part-time employment.","The authors compared the outcomes of participants once before and once after they participated in the UDC-CC grant-funded hospitality program. CLEAR’s guidelines require that the authors observe outcomes for multiple periods before the intervention to rule out the possibility that participants had increasing or decreasing trends in the outcomes examined before enrollment in the program. Without knowing the trends before program enrollment, we cannot rule this out. Further, CLEAR’s guidelines require that the authors demonstrate that participants would not have selected into the intervention based on pre-intervention trends in the outcomes and/or their own characteristics. Because the hospitality program participants comprise only those students who applied for and were admitted to the program, they are likely different from eligible students who were not admitted or did not apply. It is possible that the characteristics of the participants led to the results, not the program itself. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest",,"United States","Causal Impact Analysis","Adult, Low-skilled","Basic skills, Capacity building programs, Community college education and other classroom training","Earnings and wages-Low-Favorable impacts, Employment-Low-No impacts","Earnings and wages, Employment","Community College Review Protocol"
"CogSMART compensatory cognitive training for traumatic brain injury: Effects over 1 year (Twamley et al. 2014)","CogSMART compensatory cognitive training for traumatic brain injury: Effects over 1 year (Twamley et al. 2014)","Study Type: Causal Impact Analysis","Twamley, E. W., Thomas, K. R., Gregory, A. M., Jak, A. J., Bondi, M. W., Delias, D. C., & Lohr, J. B. (2014). CogSMART compensatory cognitive training for traumatic brain injury: Effects over 1 year. Journal of Head Trauma Rehabilitation, 30(6), 391-401.",,2014,https://doi.org/10.1097/HTR.0000000000000076,"May 2020",Veterans,,"Low Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of Cognitive Symptom Management and Rehabilitation Therapy (CogSMART) and enhanced supported employment (ESE) on employment, earnings, and health outcomes.
The authors conducted a randomized controlled trial that randomly assigned participants to the intervention and comparison groups. Employment data came from weekly reports on job attainment, hours worked, and wages earned.
The study did not find any statistically significant relationships between CogSMART plus ESE and employment, earnings, or health outcomes.
The quality of causal evidence presented in this report is low because the study had a confounding factor. Specifically, one interventionist delivered CogSMART plus ESE and another interventionist implemented services to the comparison group, which makes it impossible to separate the effect of CogSMART plus ESE from the effect of the individual interventionists. This means we are not confident that the estimated effects are attributable to CogSMART plus ESE; other factors are likely to have contributed to the findings.","CogSMART is a cognitive symptom and rehabilitative therapy approach designed to improve post concussive symptoms, prospective memory, attention, learning and memory, and executive functioning in people with traumatic brain injury (TBI). The intervention provides instruction on strategies and habits related to handling post-concussive symptoms and enhancing attention, learning, memory, and executive functioning. Examples of strategies and habits include headache management, daily calendar use, name-learning strategies, self-talk, and self-monitoring. People in the intervention group received CogSMART instruction for two hours per week for one year.
ESE provides work rehabilitation that focuses on rapid searching for competitive employment in the community, tailored to individual interests and preferences. People in the intervention group engaged with ESE for two hours a week for one year.","The authors used a randomized controlled trial design and statistical models to examine the impact of CogSMART plus ESE on employment and health outcomes among veterans with mild to moderate TBI.
The study included 50 veterans in the VA San Diego Healthcare System: 25 veterans were randomly assigned to be in the CogSMART plus ESE group and 25 veterans were randomly assigned to be in the ESE group. People were referred to the study from the VA San Diego Healthcare System Wellness and Vocational Enrichment Clinic, TBI Cognitive Rehabilitation Clinic, Polytrauma Clinic, and Neuropsychological Assessment Unit. Eligible participants included veterans of Operation Iraqi Freedom or Operation Enduring Freedom who have a history of mild to moderate TBI; who have an impairment in attention, processing speed, working memory, learning, memory, or executive functioning; who are unemployed but willing to work; who are abstinent from drugs and alcohol; and who did not participate in any other intervention studies.
People in the study were, on average, 32 years old. The majority were male and about one-third were White. They had 13 to 14 years of education.
The authors used statistical models and data from multiple measures to compare the intervention and comparison groups on employment, earnings, and health outcomes. The authors did not include control variables in their statistical models to assess the effects of the intervention on competitive employment. They did include, however, controls for age and education in their statistical models to assess the effects of the intervention on the length of employment outcome as well as for the earnings and health outcomes.","Employment
The study found no statistically significant relationships between CogSMART plus ESE and employment.
Earnings
The study found no statistically significant relationships between CogSMART plus ESE and earnings.",,,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Disability, Veteran or military","Health, Job search assistance and supportive services, Supported employment or other employment supports, Veterans' reemployment","Earnings and wages-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Employment","Veterans Review Protocol"
"Family Independence Initiative (FII): Follow-up study final report (Tao & Alamprese 2003)","Family Independence Initiative (FII): Follow-up study final report (Tao & Alamprese 2003)","Study Type: Causal Impact Analysis","Tao, F., & Alamprese, J. A. (2003). Family Independence Initiative (FII): Follow-up study final report. Retrieved from http://abtassociates.com/reports/2003/family-independence-initiative-%28fii%29-follow-up-stu.aspx.",,2003,http://abtassociates.com/reports/2003/family-independence-initiative-%28fii%29-…,"October 2020","Literacy, Low-Income Adults",,"Low Causal Evidence",Pre/Post,,"The study’s objective was to examine the impact of the Family Independence Initiative (FII) on education, earnings, employment, and public benefit receipt outcomes of low-income adults.
The authors used a pre-post study design to compare outcomes before and after participating in a work-focused family literacy program.
The study found that program participation was associated with increased employment, earnings, and receipt of a high school diploma or GED, and a reduction in government cash assistance. However, the study did not include tests of statistical significance.
The quality of causal evidence presented in this report is low because the authors did not observe outcomes for multiple periods before and after the intervention. This means we are not confident that the estimated effects are attributable to the Family Independence Initiative; other factors are likely to have contributed.","The Family Independence Initiative (FII), developed by the National Center for Family Literacy (NCFL), was a work-focused family literacy program designed to assist welfare recipients in meeting the requirements of welfare reform. FII included work preparation and work experience activities in addition to family literacy activities. The goal was to improve basic literacy skills while learning strategies for obtaining and retaining employment. To be eligible for the program, participants had to be low-income families that collected public assistance. The program was offered at 15 sites located in six cities in the United States.: Akron, Ohio (2 sites), Boulder Colorado (3 sites), Canton, Ohio (4 sites), Charlotte, North Carolina (2 sites), Long Beach, California (2 sites), and Philadelphia, Pennsylvania (2 sites).","The study used a pre-post design to compare outcomes of welfare recipients before and one-year after they participated in the FII program. Study participants included two cohorts. Cohort 1 had 70 families who enrolled in the FII program in 1999-2000 and cohort 2 had 37 families who enrolled in the FII program in 2000-2001 (107 families total). To be eligible for the study, families must have complete pre-and post-test data for the year in which they were enrolled in the program.
Data sources included individual in-person interviews and the completion of one of four standardized tests of adult basic skills: Tests of Adult Basic Skills (TABE), Comprehensive Adult Student Assessment System (CASAS), Basic English Skills Test (BEST), or Adult Language Assessment Scales (A-LAS). The authors compared differences in outcomes before and after program participation. Outcomes included changes in education, earnings, employment, and public benefit receipt. However, no tests of statistical significance were provided.","Education and skills gains 
The study found that 27 percent of participants who did not have a high school diploma at program enrollment obtained a high school diploma or GED by the one-year follow-up.
Earnings and wages
At the one-year follow-up, the study found that 67 percent of participants reported an increase in household income, 13 percent reported a decrease in household income, and 30 percent reported no change in household income.
At the time of the one-year follow-up, participants were earning an average of $7.85 per hour (above the minimum wage), and wages ranged from slightly below minimum wage to $15.00 per hour.
Employment
The study found that the employment rate of study participants improved one year after program enrollment, with 36 percent of participants employed at follow-up compared to 29 percent at program enrollment.
The study also found that for individuals who were unemployed during the FII enrollment year, 34 percent obtained work by the one-year follow-up.
Public benefit receipt
The study found that for participants who reported government assistance as their primary source of their income at FII enrollment (57% of participants), only 25 percent of these participants were receiving government cash assistance at follow-up.
At the one-year follow-up, the study found that 49 percent of participants reported no government assistance, 22 percent reported receiving one or two sources of government assistance, and 32 percent reported receiving three or more sources of government assistance.","The authors compared the outcome of participants measured once before and once after they participated in the Family Independence Initiative. CLEAR’s guidelines require that the authors observe outcomes for multiple periods before the intervention to rule out the possibility that participants had increasing or decreasing trends in the outcomes examined before enrollment in the program. Without knowing the trends before program enrollment, we cannot rule this out. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Low income","Basic skills, Other training and education","Earnings and wages-Low-No impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts, Public benefits receipt-Low-No impacts","Earnings and wages, Education and skills gains, Employment, Public benefits receipt","Literacy Review Protocol"
"Follow-up on North Carolina community college cooperative education graduates: Additional education and salary gains (Johnson 2000)","Follow-up on North Carolina community college cooperative education graduates: Additional education and salary gains (Johnson 2000)","Study Type: Causal Impact Analysis","Johnson, L. R. (2000). Follow-up on North Carolina community college cooperative education graduates: Additional education and salary gains (Unpublished doctoral dissertation.). North Carolina State University.",,2000,https://archive.org/details/ERIC_ED458904,"January 2021","Community College",,"Low Causal Evidence",Nonexperimental,,"The study's objective was to examine the impact of cooperative education on education, earnings, and employment outcomes.
The study used a nonexperimental design to compare the outcomes of graduates of a cooperative education program, graduates who did not participate in a cooperative education program that was offered at their college, and graduates who attended a school in which cooperative education was not offered. Using survey data, the author conducted statistical tests to examine the differences between the three groups 10 years after they graduated.
The study found no statistically significant relationship between graduating from a cooperative education program and pursuing additional education, employment status, or earning a higher salary.
The quality of causal evidence presented in this report is low because the author did not ensure that the groups being compared were similar before the intervention. This means we are not confident that the estimated effects are attributable to cooperative education; other factors are likely to have contributed.","Cooperative education was started by Herman Schneider in the early 1900s to blend classroom instruction with on-the-job experience in a related field. The author described the components of cooperative education programs in general, but did not describe the programs offered through the North Carolina Community College System (NCCCS). The NCCCS is comprised of 58 colleges, 22 of which had a cooperative education program.","The study used a nonexperimental design to compare the outcomes of graduates who participated in cooperative education to graduates who did not. The initial study on which this one was based started with 1986-1987 graduates randomly selected from 11 colleges with a cooperative education program and 11 colleges without such a program. The colleges were initially matched on size, program offerings, location, urban or rural location, and county-level employment rates and weekly wages. The focus was on students earning an associate degree in applied science, omitting nursing and other health-related degrees including radiology, phlebotomy, and medical technology. Graduates were randomly selected after three groups were created: 1) graduates of a cooperative education program, 2) graduates who did not participate in a cooperative education program but could have because it was offered at their college, and 3) graduates who attended a school in which cooperative education was not offered. Graduates in group three served as the comparison group.
The sample for this study included 1,323 graduates from the NCCCS who last completed a telephone follow-up survey in 1995. All graduates were mailed a card to confirm their address. Surveys were then mailed to 674 graduates based on confirmed addresses. Of these, 259 were returned (38.4%) and 241 were analyzed (18 surveys were unusable according to the author). The final analytic sample included 58 percent women and was 87 percent white; almost half (49%) were 31 to 50 years of age. Of the 241 analyzed surveys, 26 percent were in group one, 37 percent were in group two, and 37 percent were in group three. The author conducted chi-square analyses to examine differences between the groups on four outcomes (enrollment in additional education, completion of a bachelor’s degree, earnings, and employment status) 10 years after they graduated.","Education and skills gain
The study did not find a statistically significant relationship between participation in cooperative education and completion of a bachelor's degree.
The study did not find a statistically significant relationship between participation in cooperative education and current enrollment in additional education.
Earnings and wages
The study did not find a statistically significant relationship between participation in cooperative education and earning a higher salary.
Employment
The study did not find a statistically significant relationship between participation in cooperative education and employment status.","The author did not account for preexisting differences between the treatment and comparison groups before program participation, such as age, race/ethnicity, gender, a pre-intervention measure of financial disadvantage, or a pre-intervention measure of earnings and employment. These preexisting differences between the groups—and not participation in cooperative education—could explain the observed outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.",,,,"Absence of conflict of interest.",,"United States","Causal Impact Analysis","Adult, Low-skilled","Employment and Training Services, Training and Education, Community college education and other classroom training","Earnings and wages-Low-No impacts, Education and skills gains-Low-No impacts, Employment-Low-No impacts","Earnings and wages, Education and skills gains, Employment","Community College Review Protocol"
"Liquidity affects job choice: Evidence from Teach For America (Coffman et al. 2019)","Liquidity affects job choice: Evidence from Teach For America (Coffman et al. 2019)","Study Type: Causal Impact Analysis","Coffman, L. C., Conlon, J. J., Featherstone, C. R., & Kessler, J. B. (2019). Liquidity affects job choice: Evidence from Teach For America. The Quarterly Journal of Economics, 134(4), 2203–2236. doi:10.1093/qje/qj018  [Control group v. Treatment group (additional grant award)]",,2019,https://www.povertyactionlab.org/sites/default/files/research-paper/Liquidity_J…,"May 2021",,,"High Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of additional grant funds—beyond what is typically received through Teach For America’s (TFA) Transitional Grants and Loans program—on whether TFA candidates accepted employment as teachers. The authors investigated similar research questions for another contrast, the profile of which is available here. 
	
	
	This study used a randomized controlled trial design. The findings are based on TFA’s Transitional Grants and Loans program data. The authors used a regression analysis that controlled for demographics, such as age and gender, to explore the relationship between being offered a larger grant and employment.  
	

	The study found no relationship between being offered a larger grant amount and employment.  
	
	
	The quality of causal evidence presented in this report is high. This means we would be confident that any estimated effects would be attributable to TFA’s Transitional Grants and Loans program and not to other factors. However, the study did not find statistically significant effects. ","TFA is a selective program that recruits individuals, many from highly ranked U.S. colleges and universities, to teaching positions. TFA’s Transitional Grants and Loans program provides TFA teachers with funds to support the transition to teaching through a grant or loan. Recipients can use the funds to cover transition costs such as travel, moving, and certification fees; however, there are no restrictions on use.  

To apply for the Transitional Grants and Loans program, TFA candidates complete an application that requests information about their debts and savings, salary, dependents, and portions of their federal tax returns and Free Application for Federal Student Aid. Because the application is extensive, TFA candidates typically apply after they have accepted a TFA position.  

The amount of funds provided through the Transitional Grants and Loans program can range from a $500 loan to a grant of more than $8,000. The amount is the TFA candidate’s expected expense minus their expected contributions. Candidates’ expected expenses are calculated based on their training and employment locations and whether they need to relocate. Their expected contribution is based on measures such as their savings amount, amount of parental support, and number of dependents.  

In this study, some teachers received additional grants or loans beyond the typical amount. ","This study used a randomized controlled trial design. Individuals accepted to be TFA teachers were randomly assigned into study groups. In the study’s first year and first half of the second year, individuals were randomly assigned to one of three groups: the control group, a treatment group that received an additional $600 grant, or a treatment group that received an additional $600 loan. In the second half of the second year, individuals were randomly assigned to one of four groups: either one of the three groups previously described or a fourth treatment group in which individuals received a $1,200 grant. In the third year, individuals who had the highest financial need based on their expected contribution calculation were randomly assigned to one of four groups as previously described. Individuals who were in the top 80 percent were randomly assigned to a control group, to a treatment group that received an additional $1,800 grant, or to a treatment group that received an additional $1,800 loan.  

This profile focuses on the treatment group that was offered a grant of $600, $1,200, or $1,800 compared to the control group that was offered only the grants or loans typically provided by TFA. The study also examined the effects of receiving a larger loan than a teacher would typically receive, which is reported in a separate profile. The authors used a regression analysis that controlled for demographic characteristics of sample members to explore the relationship between the additional grant amount and whether candidates accepted a teaching position, as measured by whether they attended their first day of employment. The findings are based on TFA’s Transitional Grants and Loans program data. The study included 7,295 individuals from across the United States who were accepted into TFA and applied to the Transitional Grants and Loans program. The majority of the sample was female (76 percent), a third of the sample was White (34 percent), and the average age was 26 years.   ","Employment. There was no relationship between being offered an additional grant amount beyond the control award and employment. In other words, results suggest that receiving an additional $600, $1,200, or $1,800 of grant funds did not influence TFA candidates to begin employment in a teaching position through TFA. ","Probabilities of assignment to each group varied over time and by decile of expected contribution. The authors account for these changes in their analysis. ",,,,"Absence of conflict of interest.",,,"Causal Impact Analysis","Adult, Education professional","Supported employment or other employment supports","Employment-Low-No impacts",Employment,"Review Protocol"
"Liquidity affects job choice: Evidence from Teach For America (Coffman et al. 2019)","Liquidity affects job choice: Evidence from Teach For America (Coffman et al. 2019)","Study Type: Causal Impact Analysis","Coffman, L. C., Conlon, J. J., Featherstone, C. R., & Kessler, J. B. (2019). Liquidity affects job choice: Evidence from Teach For America. The Quarterly Journal of Economics, 134(4), 2203–2236. doi:10.1093/qje/qj018  [Control group v. Treatment group (additional loan)]",,2019,https://www.povertyactionlab.org/sites/default/files/research-paper/Liquidity_J…,"May 2021",,,"High Causal Evidence","Randomized Control Trial (RCT)",,"The study’s objective was to examine the impact of additional funds through a loan—beyond what is typically received through Teach For America’s) Transitional Grants and Loans program—on whether TFA candidates accepted employment as teachers. The authors investigated similar research questions for another contrast, the profile of which is available here. 
	
	
	This study used a randomized controlled trial design. The findings are based on TFA’s Transitional Grants and Loans program data. The authors used a regression analysis that controlled for demographics, such as age and gender, to explore the relationship between being offered additional funds through a loan and employment.  
	

	The study found no relationship between being offered an additional loan—beyond what is typically received—and employment.  
	
	
	This study receives a high evidence rating.  This means we would be confident that any estimated effects would be attributable to TFA’s Transitional Grants and Loans program and not to other factors. However, the study did not find statistically significant effects. ","TFA is a selective program that recruits individuals, many from highly ranked U.S. colleges and universities, to teaching positions. TFA’s Transitional Grants and Loans program provides TFA teachers with funds to support the transition to teaching through a grant or loan. Recipients can use the funds to cover transition costs such as travel, moving, and certification fees; however, there are no restrictions on use.  

To apply for the Transitional Grants and Loans program, TFA candidates complete an application that requests information about their debts and savings, salary, dependents, and portions of their federal tax returns and Free Application for Federal Student Aid. Because the application is extensive, TFA candidates typically apply after they have accepted a TFA position.  

The amount of funds provided through the Transitional Grants and Loans program can range from a $500 loan to a grant of more than $8,000. The amount is the TFA candidate’s expected expense minus their expected contributions. Candidates’ expected expenses are calculated based on their training and employment locations and whether they need to relocate. Their expected contribution is based on measures such as their savings amount, amount of parental support, and number of dependents.  

In this study, some teachers received additional grants or loans beyond the typical amount. ","This study used a randomized controlled trial design. Individuals accepted to be TFA teachers were randomly assigned into study groups. In the study’s first year and first half of the second year, individuals were randomly assigned to one of three groups: the control group, a treatment group that received an additional $600 grant, or a treatment group that received an additional $600 loan. In the second half of the second year, individuals were randomly assigned to one of four groups: either one of the three groups previously described or a fourth treatment group in which individuals received a $1,200 grant. In the third year, individuals who were in the highest 20 percent of financial need based on their expected contribution calculation were randomly assigned to one of four groups as previously described. The remaining 80 percent of individuals were randomly assigned to a control group, to a treatment group that received an additional $1,800 grant, or to a treatment group that received an additional $1,800 loan.  

This profile focuses on the treatment groups that received a loan of $600 or $1,800 compared to the control group that was offered only the grants or loans typically provided by TFA. The study also examined the effects of receiving a larger grant than a teacher would typically receive, which is reported in a separate profile. The authors used a regression analysis that controlled for demographic characteristics of sample members to explore the relationship between being offered an additional loan and whether TFA candidates accepted a teaching position, as measured by whether they attended their first day of employment. The findings are based on TFA’s Transitional Grants and Loans program data. The study included 7,295 individuals from across the United States who were accepted into TFA and applied to the Transitional Grants and Loans program. The majority of the sample was female (76 percent), a third of the sample was White (34 percent), and the average age was 26 years.   ","Employment. There was no relationship between being offered additional funds through a loan beyond the control award and employment. In other words, results suggest that receiving an additional $600 or $1,800 in loans did not influence TFA candidates to begin employment in a teaching position through TFA. ","Probabilities of assignment to each group varied over time and by decile of expected contribution. The authors account for these changes in their analysis. ",,,,"Absence of conflict of interest.",,,"Causal Impact Analysis","Adult, Education professional","Supported employment or other employment supports","Employment-Low-No impacts",Employment,"Review Protocol"