Skip to main content

Health Profession Opportunity Grants (HPOG 1.0) impact study interim report: Program implementation and short-term impacts. (Report No. 2018-16a) (Peck et al. 2018)

Review Guidelines

Absence of conflict of interest.

Citation

Peck, L. R., Werner, A., Harvill, E., Litwok, D., Moulton, S., Fountain, A. R., & Locke, G. (2018). Health Profession Opportunity Grants (HPOG 1.0) impact study interim report: Program implementation and short-term impacts. (Report No. 2018-16a). Washington, DC: Office of Planning. Research, and Evaluation, Administration for Children and Families, US Department of Health and Human Services.

Highlights

  • The study’s objective was to examine the impact of the Health Profession Opportunity Grant (HPOG) program on educational progress, earnings, and employment.
  • The authors used a randomized controlled trial to analyze the effects of the program between a treatment group, an enhanced treatment group, and a control group.
  • The study found that HPOG participants had significantly higher rates of educational progress, significantly higher earnings in the fifth quarter, and a significantly higher likelihood to be employed in healthcare than the control group.
  • The quality of causal evidence presented in the study is high for the employment and earnings outcomes because they are based on a well-implemented randomized controlled trial with low attrition. However, the quality of causal evidence is moderate for the educational progress and employment in healthcare outcomes because sample attrition for these outcomes was high, but the authors controlled for key differences between the treatment and control groups at baseline. This means we are somewhat confident that estimated effects on these outcomes would be attributable to HPOG, but other factors might have also contributed.

Intervention Examined

The Health Profession Opportunity Grant (HPOG)

Features of the Intervention

Administered by the U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation, the Health Profession Opportunity Grants (HPOG) program provides education and training in the healthcare field to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. The programs were delivered by institutions of higher education (21 HPOG programs), workforce agencies (12 HPOG programs), or government or nonprofit agencies (9 HPOG programs). The standard treatment included pre-training activities, training courses, and support services; the individual programs had discretion in how they designed and implemented their programs as they related to the local healthcare labor market opportunities. The enhanced treatment also included emergency assistance, non-cash incentives, or facilitated peer support groups. The types of programs varied from entry level to more skilled to the length of the training program. Eligibility into the HPOG program varied across the programs. TANF recipients were considered eligible across all of the programs, but for income eligibility, the programs varied on how they measured income ranging from a federal poverty threshold to Supplemental Nutrition Assistance Program (SNAP) eligibility, or a program specific income threshold.

Features of the Study

Participants were randomized at the program level. In 23 HPOG programs, applicants were assigned to the standard HPOG group (treatment) or the control group. In 19 HPOG programs, applicants were assigned to one of three conditions: 1) standard HPOG; 2) enhanced HPOG; or 3) control group. The total analysis sample was 13,802, which included 8,729 participants in the treatment group (7,231 in standard HPOG and 1,468 in enhanced HPOG) and 5,103 participants in the control group. The authors assessed the impacts of the HPOG program by comparing the pooled treatment group (standard + enhanced) and the control group as well as impacts between the standard and enhanced groups. The authors estimated impacts using multi-level regression models, controlling for demographic characteristics and outcomes at baseline. The impact analyses used administrative data for five quarters post-randomization, including data for employment and earnings from the National Directory of New Hires (NDNH). The authors also used data from a 15-month follow-up survey for educational progress (completed training or currently enrolled in training) and if the participant’s current or most recent job was in healthcare.

Findings

Education and skills gain

  • The study found that the standard HPOG program increased completion of training as the treatment group was seven percentage points more likely than the control group to complete training, and the results were statistically significant.
  • The study found no statistically significant differences between the enhanced treatment group and the control group on educational progress.

Earnings and wages

  • The study found that the HPOG program increased earnings by $137 in Q5 for HPOG participants as opposed to those in the control group, and the result was statistically significant.
  • The study found no statistically significant differences between the enhanced treatment group and the control group on earnings.

Employment

  • The study also found that there was no association between standard and enhanced HPOG and the likelihood to be employed in Q5.
  • The study found that HPOG participants had significantly higher rates of employment in the healthcare field than the control group (by 11 percentage points).
  • The study found no statistically significant differences between the enhanced treatment group and the control group on employment.

Considerations for Interpreting the Findings

Although the study design was a randomized controlled trial, the study had high attrition for the educational progress and employment in healthcare outcomes. However, the authors demonstrated that the study groups included in the analysis had equivalent baseline characteristics. Therefore, the study receives a moderate causal evidence rating for those outcomes. Additionally, the study combines information from multiple programs: 21 were institutions of higher education which were community and technical colleges; 12 were workforce agencies (WIBs and one-stop centers); and 9 were government or non-profit agencies. The authors also note that the program design and operation varied based on the local area and there was not a single model used across the country. Finally, the p-values reported in the study are based on one-tailed t-tests, which likely overstates the number of statistically significant findings based on a more appropriate two-tailed test.

Causal Evidence Rating

The quality of causal evidence presented in the study is high for the employment and earnings outcomes because they are based on a well-implemented randomized controlled trial with low attrition. This means we are confident that the estimated effects on these outcomes are attributable to HPOG and not to other factors. However, the quality of causal evidence is moderate for the educational progress and employment in healthcare outcomes because sample attrition for these outcomes was high, but the authors controlled for key differences between the treatment and control groups at baseline. This means we are somewhat confident that estimated effects on these outcomes would be attributable to HPOG, but other factors might have also contributed.

Additional Sources

Harvill, E., Litwok, D., Moulton, S., Fountain, A. R., & Peck, L. R. (2018). Technical supplement to the Health Profession Opportunity Grants (HPOG) impact study interim report: Report appendices (Report No. 2018-16b). Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Reviewed by CLEAR

May 2020

Topic Area