This study was conducted by staff from Abt Associates, which co-administers CLEAR. The review of this study was conducted by ICF, which co-administers CLEAR and is trained in applying the CLEAR causal evidence guidelines.
Citation
Juras, R., & Buron, L. (2021). Summary and Insights from the Ten PACE and HPOG 1.0 Job Training Evaluations: Three-Year Cross-Site Report (Report No. 2021-155). Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Highlights
- The study's objective was to examine the impact of the Pathways for Advancing Careers and Education (PACE) and Health Profession Opportunity Grant (HPOG) programs on college credential receipt and earnings.
- The study was a randomized controlled trial that assigned eligible participants to the treatment or control groups. Data sources included surveys, national administrative data, and local college records. The authors used statistical models to compare the outcomes of treatment and control group members at the three-year follow-up.
- When compared to control group participants, the study found that a significantly higher proportion of PACE participants at two sites earned a college credential and PACE participants at one site earned significantly more in quarters 12 and 13.
- This study receives a high evidence rating. This means we are confident that the estimated effects are attributable to PACE and HPOG 1.0 programs, and not to other factors.
Intervention Examined
Health Profession Opportunity Grants (HPOG)
Pathways for Advancing Careers and Education (PACE)
Features of the Intervention
Administered by the U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation, the Pathways for Advancing Careers and Education (PACE) and Health Profession Opportunity Grants (HPOG) programs provided education and training to Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. The PACE programs focused on postsecondary training and education in high-demand employment sectors, while the HPOG programs focused on the healthcare field. Both programs incorporated a career pathways framework that allowed participants to advance through career pathway steps, leading to higher credentials and employment opportunities. The programs were offered at various sites across the United States. While the components and intensity of the programs varied by site, they typically included basic skills instruction, occupational training, academic and non-academic support, financial assistance, work experience during training, and employment services after training.
Features of the Study
The study was a randomized controlled trial that assigned eligible PACE participants and HPOG 1.0 participants to the treatment or control group, at each affiliated program site. Members of the treatment group received program services, while all other supports and services were accessible in the community for both the treatment and control group members. Eligibility requirements for participation in the evaluations varied across program sites. Common criteria included being at least 18 years old, having a low income, encountering barriers to education or employment, and showing interest in specific degrees or career pathways as determined by the respective sites. The sample was selected from PACE and HPOG 1.0 evaluation sites across the U.S. Of the 9,254 participants at PACE sites, 5,028 individuals were randomly assigned to the treatment group and 4,226 were randomly assigned to the control group. Of the 13,802 participants at HPOG 1.0 sites, 8,699 individuals were assigned to the treatment group and 5,103 were assigned to the control group.
The data sources included baseline surveys to gather demographic and background information, three-year follow-up surveys to collect data on outcome measures, the National Student Clearinghouse (NSC) for college enrollment data, the National Directory of New Hires (NDNH) for employment and earnings data, and local college records for data on education and training enrollment and credentials receipt. The main education outcome was credential receipt, defined as obtaining a degree or certificate for completing regular college courses requiring at least a full year of credit. This outcome was examined at five PACE sites (CES, I-BEST, PTH, PCPP, and VIDA). The main earnings outcome was average quarterly earnings in quarters 12 and 13 (Q12-Q13) after random assignment, examined at all study sites. The authors used statistical models to assess whether there were differences in outcomes between the treatment and control group members across program sites.
Study Sites
There were 10 evaluation sites in the study:
- PACE evaluation sites:
- Bridge to Employment in the Healthcare Industry (BTH): San Diego, CA
- Carreras en Salud (CES): Chicago, IL
- Health Careers for All (HCA): Seattle, WA
- Integrated Basic Education and Skills Training (I-BEST): three colleges in Washington
- Pathways to Healthcare (PTH): Tucson, AZ
- Patient Care Pathway Program (PCPP): Madison, WI
- Valley Initiative for Development and Advancement (VIDA): Lower Rio Grande Valley, TX
- Workforce Training Academy Connect (WTAC), Des Moines, IA
- Year Up: Atlanta, Bay Area, Boston, Chicago, National Capital Region, New York City, Providence, Greater Seattle
- The HPOG 1.0 evaluation pooled data from 42 programs in 19 states across the U.S.
Findings
Education and skills gains
- The study found that a significantly higher proportion of participants at the PTH (18.4%) and VIDA (55.7%) sites earned a credential that typically takes a year or longer to earn compared to control group participants (11.3% and 45%, respectively).
- The study found no significant differences in credential receipt between the treatment and control groups at other PACE sites (CES, I-BEST, and PCPP).
Earnings and wages
- The study found that participants at the Year Up sites had significantly higher average quarterly earnings in Q12-Q13 ($6,782) relative to the control group participants ($4,925).
- The study found no significant differences in average quarterly earnings in Q12-Q13 between the treatment and control groups at all other PACE sites and the HPOG 1.0 sites.
Considerations for Interpreting the Findings
Although the study was a randomized controlled trial, there are several factors to consider when interpreting the findings. First, there was high attrition at the CES site for the college credential receipt outcome at the three-year follow-up, but the authors ensured that the groups were similar before participation in the programs. Also, the control groups at the PACE and HPOG 1.0 sites had access to some of the same supports and services as the treatment group members which could dilute the estimates of program impacts. Lastly, the study reports a less stringent statistical significance level, considering p-values of less than 0.10 to be significant, though it is standard practice to consider statistical significance if the p-value is less than 0.05. Only results that demonstrate a p-value of less than 0.05 are considered statistically significant in this profile.
Causal Evidence Rating
The quality of causal evidence presented in this report is high for the credential receipt outcome for the I-BEST, PTH, PCPP, and VIDA sites and for the earnings outcome at all sites because it was based on a randomized controlled trial with low attrition. This means we are confident that the estimated effects are attributable to the PACE and HPOG 1.0 programs, and not to other factors. The quality of causal evidence is moderate for the credential receipt outcome for the CES site because the outcome was based on a randomized controlled trial with high attrition, but the authors accounted for existing differences between the treatment and control groups. This means we are somewhat confident that the estimated effects would be attributable to the PACE program at this site, but other factors might also have contributed. However, the study did not find statistically significant effects.