This study was conducted by staff from Abt Associates, which administers CLEAR. Therefore, the review of this study was conducted by independent consultants trained in applying the CLEAR causal evidence guidelines.
Citation
Highlights
- The study’s objective was to examine the impact of Health Profession Opportunity Grants (HPOG) on employment, education, earnings, and public benefit receipt three years after participants were randomly assigned.
- The study is a randomized controlled trial that assigned participants to the treatment group that could access HPOG services or to the control group that could not. The primary data sources are employment and earnings data from the National Directory of New Hires (NDNH) and a follow-up survey initiated three years after random assignment. The authors used a statistical model to compare the outcomes of treatment and control group members.
- The study found that the treatment group did not have higher average quarterly earnings or higher employment rates than the control group three years after program enrollment.
- This study receives a high evidence rating for the earnings and employment outcomes because it was based on a randomized controlled trial with low attrition. This means we would be confident that any estimated effects would be attributable to HPOG, and not to other factors. However, the study did not find statistically significant effects.
Intervention Examined
Health Profession Opportunity Grants (HPOG)
Features of the Intervention
HPOG began in 2010, providing education and training for jobs in healthcare to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. HPOG programs offer education and occupational training courses combined with support services to build skills and to support career pathways into the healthcare industry. Individual HPOG grantees have flexibility in how they design and implement their programs, including which healthcare occupations their training targets. HPOG 1.0 programs focused on a range of occupations, from entry level (e.g., Nurses’ Aide or Phlebotomist) to more advanced (e.g., Registered Nurse), with corresponding differences in the length of training. HPOG programs also offered a varying range of support services including case management, tutoring, assistance with transportation or childcare costs, and financial assistance for training.
Features of the Study
The study is a randomized controlled trial that assigned 13,802 eligible individuals to either a treatment group that could access HPOG training and services or to a control group that could not. Most HPOG grantees used a 2:1 assignment ratio so that two were assigned to the treatment group for each person assigned to the control group. Three out of 23 grantees used an equal 1:1 assignment ratio. The study randomized 8,729 participants to the treatment group and 5,103 to the control group.
All HPOG grantees served TANF recipients and other low-income individuals. Programs varied in how they defined “low income” with most setting the threshold between 150 and 250 percent of the federal poverty level. In selecting participants for randomization, grantees also considered prior education, basic skills, and likelihood of succeeding in training. Treatment group members had access to the HPOG programs, which generally included more financial assistance and support services than were available to the control group. Control group members generally had less access to financial assistance and support services but had access to training courses that were similar in type, amount, and quality to those available to the treatment group.
The primary data sources are a follow-up survey initiated three years after randomization and employment and earnings data from the National Directory of New Hires (NDNH). NDNH data are available quarterly for the full sample and span from two years prior to random assignment through three years after (i.e.,12-13 quarters following random assignment). The three-year follow-up survey had a 73 percent response rate and was completed an average of 40 months after randomization.
The authors used a statistical model to compare the outcomes of treatment and control group members. The model controlled for baseline characteristics including average quarterly earnings in the year prior to the study, educational attainment, race/ethnicity, and participation in the Supplemental Nutrition Assistance Program (SNAP).
At baseline, participants were 32 years old on average. Eighty-nine (89) percent of participants were female, 84 percent were not married, and 63 percent had dependent children. About a quarter of study participants were enrolled in school or training at baseline and 43 percent were employed.
Study Sites
Twenty-three HPOG grantees participated in this study. These grantees implemented programs in 92 locations across 19 states. Participating grantees included workforce development agencies, nonprofits, and postsecondary educational institutions.
Findings
Earnings and wages
- The study found no statistically significant difference in average quarterly earnings between the treatment and control groups three years after random assignment. This result was consistent across subgroups based on participants’ race/ethnicity, age, educational attainment, parental status, employment status, and number of barriers to school or work, all measured at baseline.
Employment
- The study reported no statistically significant difference in the share of treatment and control group members who were employed three years after random assignment. This finding was consistent for most eligible subgroups. The one exception was for individuals with no barriers to school or work at baseline. For this subgroup, a positive effect of HPOG on employment three years after random assignment was found, although this difference could be due to chance given the number of statistical tests included.
Education and skill gains
- The study suggested that a higher share of the treatment group than of the control group obtained a certificate or credential or completed a degree (described as “training completion”) three years after random assignment. The result was consistent across subgroups.
Training
- The study suggested that treatment group members were enrolled full-time in education or training for more months, on average, than control group members. Results were in the same direction across subgroups, but not all the differences were statistically significant.
Health and safety
- The study suggested that treatment group members had lower levels of perceived stress and more positive self-evaluations than control group members three years after random assignment.
Public benefits receipt
- The study suggested that there was no difference in the share of treatment and control group members who received TANF, SNAP, or Medicaid three years after random assignment. Similarly, there were no significant differences in this outcome for most of the eligible subgroups examined. However, the study suggested that, among participants with a high school education or less, a higher share of the treatment group than the control group received TANF, SNAP, or Medicaid three years after random assignment.
Employer benefit receipt
- The study suggested that a higher share of the treatment group than of the control group was employed in a job that offered a supportive working environment three years after random assignment.
- The study suggested that for a higher share of the treatment group than the control group, their most recent job offered health insurance.
- Subgroup results were not examined for the supportive working environment outcome. For the outcome measuring whether participants’ most recent job offered health insurance, subgroup differences between the treatment and control groups were not statistically significant.
Attitudes
- The study suggested that treatment group members had more confidence in their career knowledge than control group members three years after random assignment.
- The study suggested that a higher share of the treatment group than of the control group believed they were marking progress toward their long-term educational goals.
- Subgroup results were not examined for the second outcome, perception of educational progress. For the first outcome, the study suggested a statistically significant positive relationship between HPOG and confidence in career knowledge for individuals who were unemployed at baseline, who had some postsecondary education at baseline, or who had one or more barriers to school or work at baseline.
Considerations for Interpreting the Findings
Although the study design was a randomized controlled trial, the study had high attrition for all outcomes derived from the three-year follow-up survey. In addition, the authors did not account for key differences between study groups included in the analysis at baseline. Specifically, the study authors did not control for age or gender in their statistical model or demonstrate baseline equivalence among treatment and control group members on these characteristics, as required by CLEAR protocols to meet the standards for moderate evidence. Therefore, this study receives a low causal rating for these outcomes.
The study authors performed one-sided hypothesis tests for the confirmatory and secondary outcomes, which the authors justify based on HPOG’s theory of change predicting that the offer of program services would have a favorable and directional impact on the specified outcomes. However, one-sided tests increase the study’s chance of finding a statistically significant impact relative to the standard two-sided hypothesis tests.
Additionally, for some outcomes, the study reports a less stringent statistical significance level, considering p-values of less than 0.10 to be significant, though it is standard practice to consider statistical significance if the p-value is less than 0.05. Only results that demonstrate a p-value of less than 0.05 are considered statistically significant in this profile.
The study authors also note that the HPOG evaluation assesses the average impact of the HPOG model across many different implementation contexts, which may affect the results. Finally, the authors note that the follow-up interval of three years may not be enough time to see earnings gains.
Causal Evidence Rating
The quality of causal evidence presented in this report is high for the earnings and employment outcomes, because they were based on a well implemented randomized controlled trial. This means we would be confident that any estimated effects are attributable to HPOG and not to other factors. However, the study did not find statistically significant effects for these outcomes. For other outcomes, including education and skill gains, employer benefit receipt, attitudes, and health and safety, the quality of causal evidence presented in this report is low because these outcomes had high attrition and the authors did not use sufficient controls in their analysis. This means we are not confident that the estimated effects are attributable to HPOG; other factors are likely to have contributed.
Additional Sources
Litwok, D., Walton, D., Jackson, R., & Peck, L. R. (2019). Health Profession Opportunity Grants (HPOG 1.0) impact study: Three-year impacts report appendices. OPRE Report 2019-114. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Litwok, D., Walton, D., Peck, L. R., & Harvill, E. (2018). Health Profession Opportunity Grants (HPOG) impact study’s three-year follow-up analysis plan. OPRE Report 2018-124, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.