Citation
Butler, D., Alson, J., Bloom, D., Deitch, V., Hill, A., Hsueh, J., Jacobs, E., Kim, S., McRoberts, R., & Redcross, C. (2012). What strategies work for the hard-to-employ? Final results of the hard-to-employ demonstration and evaluation project and selected sites from the Employment Retention and Advancement project. (OPRE Report 2012-08.) Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. [NYC SACM]
Highlights
- The study’s objective was to examine the impact of the Substance Abuse Case Management (SACM) program on employment, earnings, and public assistance receipt after two years.
- The authors attempted to implement a randomized controlled trial, but they used a process that resulted in nonrandom assignment. The study estimated impacts by comparing the regression-adjusted outcomes of the treatment and control groups, using data collected from New York City public assistance records and Unemployment Insurance wage records for New York State.
- The study found no statistically significant relationships between SACM and employment, earnings, or public assistance receipt.
- The quality of causal evidence presented in this report is low because the assignment mechanism was nonrandom and the authors did not demonstrate that the treatment and control groups were similar. This means we would not be confident that any estimated effects would be attributable to the Substance Abuse Case Management program; other factors are likely to have contributed. However, the study did not find statistically siginificant effects.
Intervention Examined
The Substance Abuse Case Management (SACM) Program
Features of the Intervention
The goal of the SACM program was to identify public assistance clients with substance abuse problems and provide them with supportive services so that they could find and maintain employment and self-sufficiency. The New York City-based program screened Temporary Assistance for Needy Families (TANF) recipients for substance abuse problems. Clinically oriented program staff such as social workers then conducted a two- to three-hour substance abuse assessments to determine the type of substance abuse treatment needed and the client’s readiness to participate in employment-related services. Case managers referred clients to treatment and supportive services, monitored clients’ progress, and referred clients to welfare-to-work activities when treatment was complete. Participation in the in-depth assessment and substance abuse treatment was mandatory; failure to attend could result in cancellation of benefits.
Features of the Study
The authors designed the study as a randomized controlled trial; however, the referral process resulted in nonrandom assignment. More than 8,800 TANF and Safety Net recipients whose application questionnaires indicated possible substance abuse were referred via an electronic scheduling system to New York City’s Human Resources Administration for more detailed assessments. When the SACM program had available slots, the automated scheduling system referred applicants to the program. Otherwise, applicants were referred to services-as-usual treatment. The full sample was 70 percent male, with 80 percent ages 31 or older. Fewer than one-third had been employed in the previous year, and most had received cash assistance (50 percent) or food stamps (62 percent) over that period.
The authors estimated the impact of SACM using regression analysis. They analyzed data collected from New York City public assistance records and Unemployment Insurance wage records for New York State.
Findings
- The study found no statistically significant relationships between the SACM program and employment, earnings, or public assistance receipt after two years.
Considerations for Interpreting the Findings
The referral process presented a problem for randomization because new SACM slots always became available in the morning. Thus, in instances of systematic differences between applicants who applied for appointments in the morning and those who applied in the afternoon, the assignment mechanism would not generate equivalent groups. For example, those eager to engage in treatment and return to work could be more likely to appear for scheduling early in the morning. In addition, staff might have known that the probability of assignment to the SACM group was higher in the morning and thus encouraged participants, who in their opinion needed extra help, to show up earlier rather than later. The study authors argued that such probabilities were remote but presented evidence of differences between the groups and acknowledged that the assignment process was not entirely random. Because the integrity of the random assignment process was compromised, the study was reviewed as a quasi-experimental comparison group design.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the assignment mechanism was nonrandom and the authors did not demonstrate that the groups were similar before the intervention began. This means we would not be confident that any estimated effects would be attributable to the SACM program; other factors are likely to have contributed. However, the study did not find statistically significant effects.
Additional Sources
Martinez, J., Azurdia, G., Bloom, D., & Miller, C. (2009). The Employment Retention and Advancement project: Results from the Substance Abuse Case Management program in New York City. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.