Citation
Millenky, M., Bloom, D., & Dillon, C. (2010). Making the transition: Interim results of the National Guard Youth ChalleNGe program evaluation. New York: MDRC.
Highlights
- This report presents results from a 21-month follow-up study of the National Guard Youth ChalleNGe program, a disciplinary and educational intervention for unemployed youth ages 16 to 18 who are not in school. Related reports examine outcomes at 9 months and 3 years post-intervention.
- The program’s effectiveness was evaluated using a randomized controlled trial conducted in 10 states. This report examined the effects of the program on participants’ educational attainment, earnings, employment, and other outcomes measured through a survey conducted about 21 months after participants entered the study.
- The study found statistically significant, positive impacts of the ChalleNGe program on earnings, education, and training outcomes measured after 21 months.
- The quality of causal evidence presented in this study is high because it is a well-implemented randomized controlled trial. This means we are confident that the effects estimated in this study are attributable to the National Guard Youth ChalleNGe program, and not to other factors.
Intervention Examined
The National Guard Youth ChalleNGe Program
Features of the Intervention
The National Guard Youth ChalleNGe program began in the early 1990s with a mission to target at-risk youth and equip them with the skills and training to have successful adult lives. To be eligible, youth must be 16 to 18 years of age, have dropped out of or been expelled from school, be unemployed, not be drug users, and not be heavily involved in the criminal justice system.
The 17-month program consists of a two-week Pre-Challenge phase, a 20-week Residential phase, and a one-year Post-Residential phase. Participants live in barracks-style housing (sometimes on a military base) in a very disciplined environment during the first two phases. They wear their hair short, are referred to as cadets, and wear military uniforms. In the Pre-Challenge phase, participants are oriented to the program’s rules and begin physical training. During the Residential phase, they participate in a number of different activities addressing eight core pillars: leadership/followership, responsible citizenship, service to community, life-coping skills, physical fitness, health and hygiene, job skills, and academic excellence. Most of their time is spent in an educational component that is usually geared toward receiving a General Educational Development (GED) certificate. During the Post-Residential phase of the program, after participants are placed in employment, education, or military service, they receive structured mentoring.
Features of the Study
The study was a randomized controlled trial conducted at 10 program sites. In 2005 and 2006, about 3,000 eligible applicants were randomly assigned to either receive an offer to participate in the program (the treatment group) or to not receive an offer for the program in the current or future application cycles (the control group). For the 21-month survey, the authors attempted to survey a subsample of those originally randomly assigned (916 treatment and 592 control youth) and ultimately surveyed 736 treatment and 460 control youth. To estimate the effectiveness of the ChalleNGe program, the authors compared the outcomes of the treatment group with outcomes of the control group.
This study examined the effects of the ChalleNGe program on education, employment, and training outcomes, relative to the effects of other employment and training programs used by control group youth, measured about 21 months after random assignment. The study also examined outcomes on military enlistment, delinquency and criminal activity, health, sexual activity, and substance use.
Study Sites
- California
- Florida
- Georgia
- Illinois
- Michigan
- Mississippi
- New Mexico
- North Carolina
- Texas
- Wisconsin
Findings
- The study found statistically significant, positive impacts of the ChalleNGe program on employment, education, and training outcomes measured after 21 months.
- Twenty-one months after randomization, youth in the treatment group were earning more per week than their control counterparts ($209 compared to $169).
- Compared to youth in the control group, youth in the treatment group were significantly more likely to have earned a GED certificate (48 percent versus 22 percent), a high school diploma (22 percent versus 16 percent), or college credits (25 percent versus 10 percent).
- Thirty percent of youth in the treatment group had ever received vocational training compared to 23 percent of youth in the control group, a statistically significant difference.
Considerations for Interpreting the Findings
Because this study only included youth who chose to apply for the program, these results cannot be assumed to apply to any youth who could be forced to attend the program. Youth who applied to the program may differ from other youth in a meaningful way. For example, youth who applied to ChalleNGe may have higher motivation than the “average” youth.
Not all youth who were randomly assigned to receive an offer to participate in the program did ultimately participate. However, youth were included in the treatment group regardless of whether they successfully completed the program. This could have reduced the estimated impact of the program. Additionally, sites were not randomly selected for participation in the study. Therefore, casual conclusions can only be drawn for the subset of sites that participated in this study and not the ChalleNGe program nationally.
The authors estimated multiple related impacts on outcomes related to education, employment, and earnings. Performing multiple statistical tests on related outcomes makes it more likely that some impacts will be found statistically significant purely by chance and not because they reflect program effectiveness. The authors did not perform statistical adjustment to account for the multiple tests, so the number of statistically significant findings in these domains is likely to be overstated.
Causal Evidence Rating
The quality of causal evidence presented in this report is high because it was based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects are attributable to the National Guard Youth ChalleNGe program, and not to other factors.