Citation
Hendra, R., Dillman, K.-N., Hamilton, G., Lundquist, E., Martinson, K., Wavelet, M., Hill, A., & Williams, S. (2010). The Employment Retention and Advancement project: How effective are different approaches aiming to increase employment retention and advancement? Final impacts for twelve models. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families. [PASS - Riverside]
Highlights
- The study’s objective was to estimate the impact of post-employment job supports and supportive services on recent welfare recipients’ employment and benefits receipt outcomes after four years. The Riverside Post-Assistance Self-Sufficiency (PASS) site was one of two in Riverside, California, that participated in the Employment Retention and Advancement (ERA) project.
- The authors randomly assigned 2,770 employed single parents who had recently left the Temporary Assistance for Needy Families (TANF) program to either a treatment group that received ERA services or a control group that could participate in other employment-related programs in the community. The authors analyzed data from Unemployment Insurance (UI), TANF, and Food Stamps administrative records and surveys administered 12 and 42 months after random assignment.
- The study found that individuals assigned to the Riverside PASS ERA program had significantly higher earnings in the first and fourth years after random assignment and were more likely to be employed in every quarter of the fourth year, compared with those in the control group.
- The quality of causal evidence provided in this study is high for most outcomes because it was based on a well-conducted randomized controlled trial. This means we are confident that the estimated effects are attributable to the Riverside PASS ERA program and not to other factors. However, the study receives a moderate causal evidence rating for outcomes from the 12-month survey because that data source had high attrition, yet the analysis included adequate statistical controls to correct for resulting imbalance.
Intervention Examined
The Employment Retention and Advancement Project, Riverside Post-Assistance Self-Sufficiency (PASS) Program
Features of the Intervention
The ERA project was introduced in 1999 as a nationwide exploration of factors that help welfare recipients not only find employment but retain their positions and advance in their careers. Riverside PASS was one of 16 sites across the United States to receive funding from the U.S. Department of Health and Human Services to implement a program intended to improve welfare recipients’ employment outcomes.
Riverside’s PASS program aimed to improve employment retention and career advancement among employed individuals who had recently left the TANF program. Local PASS service providers contacted clients randomly assigned to the treatment group and then provided post-employment services and supportive services based on clients’ needs. Available services included case management; counseling and mentoring; reemployment activities such as assistance with job search and resume preparation; life-skills workshops; referrals to training and education opportunities; supportive services such as child care, transportation, books, and equipment; and referrals to social services programs.
Features of the Study
From July 2002 to June 2003, about 2,770 employed single parents who had recently exited TANF were randomly assigned to either a treatment group, which received ERA services, or a control group, which could participate in other programs offered in the community. Most participants (90 percent) were female, about half were Hispanic, and the average age was 31 years.
The authors estimated employment and earnings impacts by comparing regression-adjusted UI wage records of treatment and control group members over the four years after random assignment. The authors also calculated program impacts for benefits receipt measures, including whether participants received TANF or Food Stamps, through four years after random assignment using TANF and Food Stamps administrative records. Finally, the authors estimated impacts on self-reported employment and earnings using data from 12- and 42-month surveys administered to a subsample of study participants.
Findings
- The study found that those in the PASS program group were 3.7 percentage points more likely than their control group counterparts to be employed in every quarter during the fourth year after random assignment. They also earned, on average, $967 more in the fourth year than control group members. These differences were statistically significant at the 5 percent level.
- PASS program group members earned, on average, $852 more in the first year after random assignment than those in the control group, significant at the 1 percent level.
Considerations for Interpreting the Findings
Many sample members selected for the 12-month survey did not respond, introducing the potential for sample imbalance among survey respondents. Imbalance in the respondent sample could, in turn, bias the estimated program effects. Because the authors statistically accounted for possible differences in the background characteristics of treatment and control respondents to the 12-month survey, the earnings and employment outcomes from this survey receive a moderate rating. Because this review is based on a previous review conducted by the Employment Strategies for Low-Income Adults Evidence Review which did not report outcomes with a low evidence rating, CLEAR did not review two public assistance outcomes from the 12-month survey. These outcomes received a low evidence rating because the authors neither demonstrated that the groups were similar before the program began nor accounted for any baseline differences that might have existed between the study groups.
Across data sources, the study authors estimated multiple related impacts on outcomes related to employment, earnings, and benefits receipt. Performing multiple statistical tests on related outcomes makes it more likely that some impacts will be found statistically significant purely by chance and not because they reflect program effectiveness. The authors did not report that they performed statistical adjustments to account for the multiple tests, so the number of statistically significant findings in these domains is likely to be overstated.
Causal Evidence Rating
The quality of causal evidence provided in this study is high for most outcomes because it was based on a well-conducted randomized controlled trial. This means we are confident that the estimated effects are attributable to the Riverside PASS ERA program and not to other factors. The study receives a moderate causal evidence rating for outcomes from the 12-month survey because it was a randomized controlled trial with high attrition that included adequate statistical controls to correct for resulting imbalance. This means we would be somewhat confident that any estimated effects would be attributable to the Riverside PASS ERA program, but other factors might also have contributed. However, the study did not find statistically significant effects on the outcomes from the 12-month survey.