Absence of conflict of interest.
Citation
Amin, S., Chojnacki, G., Congdon, B., Davis, S., Langan, A., Deutsch, J., Welch, E., Spitzer, A., & Johnson, A. (2022). Behavioral interventions to improve work search among UI claimants: Results from North Carolina and Washington. US Department of Labor. [North Carolina]
Highlights
- The study's objective was to examine the impact of a behavioral intervention designed to improve work search compliance among Unemployment Insurance (UI) claimants in North Carolina. The study authors also investigated a related behavior intervention implemented in Washington State, the profile of which can be found here.
- The study was a randomized control trial (RCT) that used state administrative data to compare the outcomes of North Carolina UI claimants in the treatment group to a control group. The authors also used a quasi-experimental design (QED) to conduct a hazard analysis that used administrative data and compared outcomes for North Carolina UI claimants who submitted claims before and after the statewide implementation of the behavioral intervention.
- The study did not find any statistically significant relationships between North Carolina’s behavioral intervention to improve work search compliance among UI claimants and claimant knowledge of work search requirements, the rate of work search overpayments, or the acceptability of work search contacts.
- The quality of causal evidence presented in this report is moderate. The RCT analysis received a moderate causal evidence rating because it was a randomized controlled trial with high attrition, but the authors ensured that the groups being compared were similar before the intervention. The QED analysis also received a moderate causal evidence rating because it was based on a well-implemented non-experimental design. This means we would be somewhat confident that any estimated effects would be attributable to North Carolina’s behavioral intervention to improve work search compliance among UI claimants, but other factors might also have contributed. However, the study did not find statistically significant effects.
Intervention Examined
North Carolina’s behavioral intervention to improve work search compliance among UI claimants
Features of the Intervention
The U.S. Department of Labor Behavioral Intervention (DOLBI) project supports efforts to improve labor outcomes using behavioral interventions. In 2019, the DOLBI project partnered with the U.S. Department of Labor Office of Unemployment Insurance (OUI) to support work with two states – North Carolina and Washington – interested in experimenting with behavioral interventions designed to improve work search compliance and reduce improper payment rates. In North Carolina, the behavioral intervention introduced a weekly pop-up alert within the online claims filing system. This alert included a hyperlink to a tip sheet detailing valid employer contacts and adequate proof for different modes of contact. Additionally, a weekly email with the same information was sent out to UI claimants. North Carolina’s behavioral intervention was implemented in July 2019 for a randomly selected subset of UI claimants and rolled out to all UI claimants starting in October 2019.
Features of the Study
The study used both an RCT and a QED to examine the impact of North Carolina’ behavioral intervention designed to improve work search compliance among UI claimants. The RCT randomly assigned 69,761 UI claimants in North Carolina who filed a claim in July and August of 2019 – prior to the full statewide rollout of the behavioral intervention in October 2019 – to either a treatment or a control group. The RCT analysis sample included 24,416 claimants, with 12,154 claimants randomly assigned to the treatment group and 12,262 to the control group. The treatment group received the behavioral intervention. The control group did not receive behavioral intervention. The RCT analysis examined whether claimants self-attested work search noncompliance, which the authors interpreted as an indicator of whether the claimant understood North Carolina’s work search requirements. The authors used a statistical model to compare self-attested work search noncompliance rates for UI claimants in the treatment group to the control group.
The study also included two quasi-experimental analyses that compare outcomes for North Carolina UI claimants who submitted claims before and after the statewide implementation of the behavioral intervention in October 2019. The first analysis examined the likelihood that a claimant had an overpayment within the first five weeks of their UI claim using a sample of 183,724 claimants who filed UI claims between November 2018 and February 2020. The second analysis examined the proportion of a UI claimant’s work search contacts deemed acceptable during a benefit accuracy measurement (BAM) audit, using BAM microdata data on a sample of 461 paid claims (and a total of 1,455 associated work search contacts) filed between November 2018 and February 2020.
Findings
Public Benefits Receipt
- The study did not find a statistically significant relationship between North Carolina’s behavioral intervention to improve work search compliance among UI claimants and claimant knowledge of work search requirements.
- The study did not find a statistically significant relationship between North Carolina’s behavioral intervention to improve work search compliance among UI claimants and the share of claimants receiving an overpayment within the first five weeks of a UI claim.
- The study did not find t a statistically significant relationship between North Carolina’s behavioral intervention to improve work search compliance among UI claimants and the likelihood that a work search contact was deemed acceptable.
Considerations for Interpreting the Findings
Although the RCT randomly assigned 69,761 UI claimants to the treatment or control groups, only 24,416 UI claimants were included in the analysis sample. Because of the high level of attrition, the RCT is not eligible for a moderate causal evidence rating, the highest rating available for experimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this report is moderate. The RCT analysis received a moderate causal evidence rating because it was a randomized controlled trial with high attrition, but the authors ensured that the groups being compared were similar before the intervention. The QED analysis also received a moderate causal evidence rating because it was based on a well-implemented non-experimental design. This means we would be somewhat confident that any estimated effects would be attributable to the behavioral intervention, but other factors might also have contributed. However, the study did not find any statistically significant effects.