Absence of conflict of interest. This study was conducted by staff from MEF Associates, which administers CLEAR. Therefore, the review of this study was conducted by independent consultants trained in applying the CLEAR causal evidence guidelines.
Citation
Highlights
-
The study's objective was to examine the impact of two approaches to subsidized employment for Temporary Assistance for Needy Families (TANF) recipients: Paid Work Experience (PWE) in the nonprofit or public sector, and On-the-Job Training (OJT) in for-profit companies. Many outcomes were evaluated in the domains of employment, earnings and wages, education and training, public benefit receipt, and health and safety. This profile examines comparisons between the PWE group and a control group. The authors investigated similar research questions for two other comparisons, the profiles of which can be found here:
-
The study used a randomized controlled trial design. The impact analysis uses administrative data and survey data. The authors compared the outcomes of those receiving OJT to those in the control group.
-
The study found OJT participants were more likely to be employed, earned more, had smaller TANF payments, and were more likely to have left TANF than control group participants in the first year of follow-up, but none of these effects were significant for the last year of follow-up. However, OJT participants were less likely to report experiencing serious psychologocial distress in the month prior to the 30-month survey than control group members.
-
The quality of causal evidence presented in this report is high because it was based on a well-implemented randomized controlled trial.
Intervention Examined
On-the-Job Training (OJT)
Features of the Intervention
Following the 2007-2009 economic recession, subsidized employment interventions gained substantial attention. In 2010, the U.S. Departments of Health and Human Services and Labor made substantial investments in studying 8 subsidized employment interventions aimed at disadvantaged populations, including both Paid Work Experience (PWE) and On-the-Job Training (OJT) approaches for TANF recipients. This study focuses on programs in Los Angeles County, which has a large TANF program and a diverse context.
This review focuses on the OJT intervention. OJT offers wage subsidies to for-profit employers who agree to place employees onto their payrolls after an initial two-month trial period, during which participants are paid by the South Bay Workforce Investment Board. If participants are shifted onto the employer's payroll, the employers receive a partial wage subsidy for each participant for up to an additional four months. This approach aims to simulate a "real world" work environment and hopes to move participants into long-term unsubsidized work with participating employers. The program serves TANF recipients who participated in Los Angeles County's welfare to-work program, but were unable to find unsubsidized jobs during a four-week job club.
Features of the Study
The study used a randomized controlled trial design. The researchers used a lottery-like process to randomly assign 2,622 individuals to one of three groups: the PWE program group (n=874), an OJT program group (n=877), or the control group (n=871). All participants were TANF recipients in Los Angeles County who participated in the county’s welfare-to-work program, but were unable to find unsubsidized jobs during a four-week job club. To be included, participants had to be receiving TANF benefits and have five or more months left of TANF eligibility, must not have participated in Transitional Subsidized Employment in the preceding 12 months, and must have been able to work the required hours and have no major identified barriers that would prevent them from working.
Of the full sample, most (85.5%) of study participants were female, and more than 80% were non-White (55% Hispanic/Latino; 32% Black, non-Hispanic; 3% Asian, non-Hispanic). 39% of participants lacked a high school diploma or an equivalent credential. On average, participants were 31.7 years old.
Participants assigned to the OJT group were referred to a Worksource Center to be placed into the corresponding type of subsidized job. Like those in the control group, they could still continue to receive TANF benefits, as well as other TANF welfare-to-work services and community services. Of those randomized to the OJT group, 42 percent were placed in subsidized employment. On average, OJT placements were made in 33 days and lasted about 84 days. Individuals in the control group were not placed into the subsidized job, but they could still receive TANF benefits, other TANF welfare-to-work services and community services.
The impact analysis uses administrative data and survey data. Administrative data included quarterly wage data from the National Directory of New Hires and TANF and food stamp payment records from the Los Angeles Department of Public Social Services. Survey data were collected from the full study sample at approximately 30 months after random assignment, including information about employment, education, training, and personal and economic well-being. The authors used a statistical model to compare the outcomes of those in OJT to those in the control group, accounting for initial characteristics.
Findings
Employment
-
OJT participants were more likely to have been employed during the first year of follow-up than control group participants (18.3 percentage points according to administrative data, 14.5 percentage points according to survey data), but no more likely to be employed during the last year of follow-up.
-
OJT participants were 5.4 percentage points more likely to be currently employed than control group members at the time of the 12-month survey, but were no more likely to be currently employed at the time of the 30-month survey.
-
OJT participants were employed for 0.6 more quarters, on average, than control group participants during the first year of follow-up, but there were no such differences in the last year of follow-up.
Earnings and wages
-
Participants in the OJT program earned $1,269 more than those in the control group during the first year following enrollment, but did not earn significantly more in the last year of follow-up or when examining the total 30-month follow-up period.
-
OJT participants had a total income $1,084 more in the first year of follow-up than control group participants, but their total income was no greater in the last year of follow-up.
-
OJT participants were no more likely to have received income from working in the month prior to the 30-month survey, or to be earning greater than $12.00 or greater than $15.00 per hour at that time.
Education and skills gains
-
Compared to those in the control group, participants receiving OJT were no more likely to be participating in postsecondary education at the time of the 30-month survey.
-
OJT participants were similar to the control group in terms of highest level of school completed.
Health and safety
-
OJT participants were no more likely to have health insurance (in general or through an employer) than control group participants.
-
There were no significant differences between OJT participants and control participants in terms of the percent reporting they were in "good, very good, or excellent health", scores on the social support scale, or ratings of overall happiness. However, those in the OJT group were 3.2 percentage points less likely to report experiencing serious psychological distress in the past month.
Public benefits receipt
-
OJT participants received $181 less in TANF payments in the first year of follow-up compared to control group members, but received similar amounts from TANF during the last year of follow-up and for the total follow-up period.
-
Similarly, OJT participants were 5.0 percentage points more likely to have left TANF in the year following enrollment, but effects were no longer significant for the 30-month follow-up period.
-
There were no significant differences between the OJT and control groups in terms of income received from food stamps or unemployment insurance, or the percent receiving the following: Supplemental Security Income, public assistance or welfare s, child support, childcare subsidies or vouchers, benefits from Section 8 or other housing assistance, food stamps, or Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) benefits.
Training
-
Participants in the PWE program were no more likely than control group participants to be participating in vocational training or to have earned a professional license or certification at the time of the 30-month survey.
Considerations for Interpreting the Findings
This profile describes multiple related impacts on outcomes related to: Employment; Earnings and wages; Education and skills gains; Training; Public benefit receipt; and Health and safety. Performing multiple statistical tests on related outcomes makes it more likely that some impacts will be found statistically significant purely by chance and not because they reflect program effectiveness. The findings presented do not include statistical adjustments to account for the multiple tests, so the number of statistically significant findings in these domains is likely to be overstated.
The authors also point out at that placement rates for the OJT program differed across participating Worksource Centers. However, the authors were unable to study effects of the interventions at the individual center-level. For that reason, readers should recognize that the impacts of the OJT program may vary significantly depending on the operational context in which it is delivered.
Authors also note that the composite income measure was unlikely to have captured all income received by individuals and did not include other household members' income. For those reasons, it likely underestimated the true total incomes of participants.
Causal Evidence Rating
The quality of causal evidence presented in this report is high because it was based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects are attributable to the On-the-Job Training program and not to other factors.