This study was conducted in part by staff from MEF Associates, which is a subcontractor on the CLEAR team. Therefore, the review of this study was conducted by Abt Associates staff trained in applying the CLEAR causal evidence guidelines.
Citation
Highlights
-
The study’s objective was to examine the impact of the Minnesota Subsidized and Transitional Employment Demonstration (MSTED) on participants’ employment, earnings, receipt of public benefits, and educational attainment.
-
The study used a randomized control trial that assigned Minnesota Family Investment Program (MFIP) participants receiving employment services either to receive subsidized employment (the MSTED treatment group) or to continue receiving MFIP services as usual (the control group). Data sources used in the study include staff and participant interviews, surveys, case reviews, and administrative data.
-
The study found that after one year, participants who had access to MSTED services were significantly more likely to be employed than the control group and significantly more likely to be employed in all quarters.
-
Overall, the quality of causal evidence associated with the administrative data in this report is high because it comes from a well-implemented randomized controlled trial. This means we are confident that these estimated effects are attributable to MSTED, and not to other factors.
-
However, the quality of causal evidence from the survey data is low because the authors did not account for other factors that could have affected the difference between the treatment and comparison groups. This means we are not confident that the estimated effects are attributable to MSTED; other factors are likely to have contributed.
Intervention Examined
Minnesota Subsidized and Transitional Employment Demonstration
Features of the Intervention
Minnesota's Temporary Assistance for Needy Families program, MFIP, provides families with financial assistance and employment services. To improve low employment rates among MFIP participants, the state developed the Minnesota Subsidized and Transitional Employment Demonstration (MSTED). MSTED provided subsidized employment to MFIP recipients who were unable to find employment after six months. Three nonprofit organizations in Minnesota provided MSTED services. These services included subsidized employment as well as training in job readiness skills such as resume writing and interviewing.
Features of the Study
The study is a randomized control trial that assigned 799 participants receiving MFIP employment services to the MSTED treatment group (n=403) or to a control group receiving MFIP services as usual (n=396). At the start of the study in November 2014, participants had to (1) have received MFIP benefits for at least six months, (2) have earned $1,200 or less in that time, (3) be currently unemployed, (4) not be minors who were parents, (5) not be parents between ages 18 and 24 who were in a full-time education program, and (6) not be exempt from MFIP work requirements. In mid-2015, due to an improving economy and fewer than expected referrals, eligibility for MSTED was expanded to any unemployed individual who was receiving MFIP benefits, was not pursuing a full-time education program, and was not exempt from work activity requirements. On average, study participants were approximately 30 years old, female (82%), and never married (75%). MSTED was implemented by three nonprofit organizations in three counties in Minnesota including Ramsey County, Dakota County, and Hennepin County. Each county contracted with one or two MSTED providers.
Data sources used in the study include staff and participant interviews, surveys, case reviews, and administrative data. Administrative data sources include Workforce One and MAXIS, the Minnesota Department of Health and Human Services data systems; payment records from MSTED providers; the National Directory of New Hires; and state administrative data. Participants also provided survey data in the form of a baseline information questionnaire and a 12-month follow-up survey. The authors use a statistical model to compare the outcomes of treatment and control group members.
Findings
Administrative data on employment
-
After one year, participants who had access to MSTED services were significantly more likely to be employed (86.9%) compared to participants in the control group (80.2%) and were significantly more likely to be employed in all quarters (36.1%) compared to participants in the control group (29.3%).
-
During the first quarter of the second year, the study found no statistically significant relationship between access to MSTED services and employment.
Survey data on employment
-
The study found no statistically significant relationship between access to MSTED services and self-reported permanent employment, temporary employment, or working more than 34 hours per week in a current job.
Administrative data on earnings
-
The study found no statistically significant relationship between access to MSTED services and total earnings after one year.
Survey data on earnings
-
The study found no statistically significant relationship between access to MSTED services and earning more than $15 per hour in a current job.
Administrative data on public benefits receipt
-
The study found no statistically significant relationship between access to MSTED services and number of months receiving MFIP payments, amount of MFIP payments, or leaving MFIP during the first year.
-
The study found no statistically significant relationship between access to MSTED services and number of months receiving SNAP payments, amount of SNAP payments, or leaving SNAP during the first year.
Survey data on public benefits receipt
-
The study found no statistically significant relationship between access to MSTED services and receiving subsidized child care in the past month, receiving food stamps in the past month, or living in emergency or temporary housing in the past month.
Survey data on education and skills gains
-
The study found no statistically significant relationship between access to MSTED services and receiving a high school diploma or equivalent certificate or earning a professional license or certification.
Considerations for Interpreting the Findings
The authors conducted a randomized control trial; however, the outcomes based on survey data had high levels of attrition. Additionally, although the authors controlled for some demographic characteristics of participants, they did not specify whether they accounted for pre-intervention differences on measures related to the outcome domains. The review protocol requires that analyses of employment-related outcomes must include controls for previous employment or earnings from at least one year before program participation, that education-related outcomes should control for baseline education level and socioeconomic status, that earnings-related outcomes control for previous earnings at least one year before program participation, and that public benefits receipt-related outcomes control for public benefit receipt or socioeconomic status. The authors did not note if they included these control variables, so preexisting differences between the groups—and not MSTED— could explain the observed differences in outcomes. Therefore, the survey outcomes are not eligible for a high causal evidence rating, the highest rating available for experimental designs.
The study authors estimated multiple related impacts on outcomes related to employment, earnings, education and skill gains, and public benefit receipt. Performing multiple statistical tests on related outcomes makes it more likely that some impacts will be found statistically significant purely by chance and not because they reflect program effectiveness. The authors did not perform statistical adjustments to account for the multiple tests, so the number of statistically significant findings in these domains is likely to be overstated.
Causal Evidence Rating
Overall, the quality of causal evidence presented in this report is high because the administrative data came from a well-implemented randomized controlled trial. This means we are confident that the estimated effects are attributable to MSTED, and not to other factors.
However, the quality of causal evidence for the survey findings is low because the authors did not account for other factors that could have affected the difference between the treatment and comparison groups. This means we are not confident that the estimated effects are attributable to MSTED; other factors are likely to have contributed.