Skip to main content
UNITED STATES DEPARTMENT OF LABOR

Adapting to local context: Findings from the YouthBuild evaluation implementation study (Weigand et al., 2015)

  • Findings

    See findings section of this profile.

    Evidence Rating

    Not Rated

Review Guidelines

Implementation Guidelines

Absence of conflict of interest. 

Citation

Weigand, A., Manno, M., Leshnick, S., Trekson, L., Geckeler, C., Lewis-Charp, H., Sinicrope, C., Clark, M. & Nicholson, B. (2015) Adapting to local context: Findings from the YouthBuild evaluation implementation study. MDRC.

Highlights

  • The study objective was to determine the design, local context, and participant characteristics of YouthBuild programs. 
  • The authors conducted an implementation and fidelity evaluation using program and participant data, site visits, and interviews. Six program components were examined: program operations, academic services, postsecondary education, youth leadership and community service, vocational-training services, supportive services, and transition services. 
  • YouthBuild programs maintained relatively high fidelity across many services, although there was variation in fidelity scores among sites and program elements. 
  • The study was fairly comprehensive in that it examined a very broad range of services and design elements in a large number of YouthBuild programs. However, the evaluation would be difficult to replicate given the lack of information on respondent selection and qualitative analysis. 
  • The embedded impact study was reviewed by CLEAR in August 2022. 

Intervention Examined

Youthbuild

Features of the Intervention

The US Department of Labor and AmeriCorps fund 270 YouthBuild programs which are operated by a variety of non-profit organizations, local government agencies, community colleges, and school districts. YouthBuild is designed to give a "second chance" to high school dropouts who are unemployed by providing them with supportive services and educational and vocational opportunities. 

YouthBuild applicants go through a rigorous screening process called the "Mental Toughness Orientation". Core YouthBuild services for selected participants include youth leadership and community service activities, programs to increase academic, employment, social, and life skills, work-readiness training, vocational training (particularly in construction), case management, and counseling. Programs offer a stipend or wage for both work activities and academic participation, and participants are rewarded for good performance. After program graduation, YouthBuild offers support to ensure continued success. 

Program implementation at each YouthBuild site is expected to vary because each one has a different local context, including diverse leadership, staff, funding, and services. The program guidelines set by YouthBuild are intentionally broad, allowing each site to adapt to the needs of the local population. 

Features of the Study

The study examined the fidelity of implementation in a sample of YouthBuild programs, taking into account variation in program operation and capabilities, community characteristics, and participant characteristics across sites. The programs studies were implemented in 75 sites located in 31 states, the US Virgin Islands, and Washington, DC. 

The authors examined participant characteristics through participant readiness, risk factors, prior academic and work experience, and supportive-service needs. Contextual factors were evaluated through data on the local economy, neighborhood conditions, and availability of outside services. Finally, to examine implementation and participant experiences, the authors conducted site visits during which they interviewed participants, staff, and partners about the structure, quality, and content of educational activities, vocational training, youth development opportunities, and supportive and transitional services. 

During each site visit the researchers completed a fidelity assessment based on YouthBuild USA’s 2007 Program Design Standards. A total of 61 standards constituted 6 assessment categories: general program operations, academic services, postsecondary education, vocational and construction training, supportive services, and leadership and community service. For each standard, site visitors awarded 100 points for “meets standard,” 50 points for "partially meets standard," 0 points for "does not meet standard," or chose “does not apply” / “unable to observe.” Participant data and program documents such as grant applications, training plans, and partnership agreements were also used in the fidelity assessment. Lastly, sites completed a survey to provide information on program history and structure. 

Findings

Intervention activities/services:

  • YouthBuild programs generally succeeded in targeting disadvantaged young people while screening for those most motivated to “make a change” and succeed in the program. 
  • All sites studied offered community services opportunities, and the majority required it. Youth Policy Councils were implemented in 72% of programs, and academic and work leadership roles were offered at 91% percent of programs, but leadership classes were only available at 56%. 
  • All sites studied offered classroom or independent educational services oriented to gaining a GED or high school diploma, and most offered some form of post-secondary services. Almost all sites met the minimum student-instructor ratio of 1:28, and the average class size was around 12. 
  • Almost all sites offered construction training, workplace safety training, and related skills. A fifth of these offered training in other fields (healthcare, culinary arts, etc.) in addition to construction. 
  • Programs were generally able to supplement education and vocational services with supportive services to address individual barriers to successful employment. 

Fidelity: 

  • Since the YouthBuild design and performance standards are purposefully broad, programs implemented services differently and still received high fidelity scores. 
  • Over half of all sites received a fidelity score of 80 or above out of 100. This means that most of their program components were implemented in such a way that they "met the standards" of YouthBuild. 
  • Certain program components had higher fidelity scores than others. Those with average scores above 80 included vocational and construction training, academic services, supportive services, and general program operations. 
  • Fidelity to YouthBuild's post-secondary standards and values of youth development and leadership varied most greatly. 

Implementation Challenges and Solutions: 

  • Identifying work sites and employment opportunities during and after construction training was affected by a downturn in the housing market during the study period. 
  • Transportation for community service and other events was a common challenge. 
  • Programs with smaller budgets had more staffing challenges, which affected Youth Policy Councils and youth leadership standards, academic standards, and the quality of case management. 
  • Staff and participants found that vocational training was optimized when students were informed on how their work benefited the community, and when they were given hands-on training and tasks suited to their individual interests and experience. 

Considerations for Interpreting the Findings

The main strength of the study was that it examined a broad range of services and design elements in a large number of YouthBuild programs. However, the process of evaluation would be difficult to replicate with the information given. Specifically, the authors did not discuss quality assurance measures or the process for respondent selection for project director, case manager, or participant interviews. The study has limited information on qualitative analysis but included a wide range of data which allowed diverse perspectives to be described. 

Reviewed by CLEAR

July 2023

Topic Area