Absence of conflict of interest. This study was conducted by staff from ICF, which administers CLEAR. Therefore, the review of this study was conducted by an independent consultant trained in applying the CLEAR causal evidence guidelines.
Citation
Takyi-Laryea, A., Gall, A., Chamberlin, M., Naughton, L., & Spychala, B. G. (2017). Southeastern Economic and Education Leadership Consortium (SEELC) final report. Fairfax, VA: ICF.
Highlights
- The study’s objective was to examine the impact of the WorkKeys® assessment, a component of the TAACCCT-funded programming at Southeastern Economic and Education Leadership Consortium (SEELC) colleges, on education, earnings outcomes, and employment.
- The authors used a nonexperimental design to compare the education, earnings outcomes, and employment of SEELC program participants that took the WorkKeys® assessment to a matched comparison group of SEELC participants enrolled at the same colleges who did not take the assessment.
- The study found a significant relationship between taking the WorkKeys® assessment and earnings, with SEELC participants who took the assessment receiving higher post-program hourly wages than the comparison group of students who did not take the assessment.
- The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar before the intervention nor include sufficient control variables. This means we are not confident that the estimated effects are attributable to the SEELC consortium WorkKeys® assessment; other factors are likely to have contributed.
Intervention Examined
The Southeastern Economic and Education Leadership Consortium (SEELC)
Features of the Intervention
The U.S. Department of Labor's (DOL) Trade Adjustment Assistance Community College and Career Training (TAACCCT) program provided $1.9 billion in grants to community colleges to improve skills and support employment in high-demand industries, notably manufacturing, health care, information technology, energy, and transportation. Through four rounds of funding, DOL awarded 256 TAACCCT grants to approximately 800 educational institutions across the United States and its territories.
The Southeastern Education and Economic Leadership Consortium (SEELC) received TAACCCT funds to administer the program across six community and state colleges (Northeast State Community College, Palm Beach State College, Pellissippi State Community College, Polk State College, Randolph Community College, and Vance-Granville Community College) in three states: Florida, North Carolina, and Tennessee. The program goal was to improve education and training opportunities for trade adjustment assistance (TAA) eligible workers, veterans, and other adults who have been unemployed by facilitating a change in its approach to serving employers, workers, and the community at large. The SEELC program focused on three target industries: welding, computer-integrated machining, and advanced manufacturing technology. The program implemented three evidence-based strategies: 1) creating career pathways, accredited testing facilities, and applied credit options; 2) providing student support services; and 3) promoting employer and industry engagement. The SEELC program also used the WorkKeys® assessment, developed by ACT, to identify education and training gaps that needed to be addressed for each student. The assessment provided students with a score that could then be matched to employers and job opportunities.
Features of the Study
The nonexperimental study was conducted at three of the six colleges in the consortium (Vance-Granville, Northeast, and Palm Beach) and compared the education, employment, and earnings outcomes of students who took the WorkKeys® assessment to students who did not. The study sample included 917 students enrolled in the grant funded programs between Fall 2014 and Spring 2017. The treatment group consisted of 115 students who completed the WorkKeys® assessment as part of the intervention. The comparison group consisted of students from the same three colleges enrolled in the grant funded programs who did not complete the WorkKeys® assessment. Treatment students were matched with comparison students using demographic characteristics. Data sources included administrative data records, surveys, and program documentation. The study examined the impact of the WorkKeys® assessment on credential completion, credential attainment, earnings and wages, and employment by comparing differences between the treatment and comparison groups.
Findings
Education and skills gains
- The study did not find significant relationships between taking the WorkKeys® assessment and completion of any credential, attaining a certificate, or attaining a degree.
Earnings and wages
- The study found a significant relationship between taking the WorkKeys® assessment and earnings for incumbent workers. On average, post-program hourly wages were $2.36 higher for incumbent working students who took the assessment compared to students who did not take the assessment.
Employment
- The study did not find a significant relationship between taking the WorkKeys® assessment and employment in the first quarter.
Considerations for Interpreting the Findings
The authors used propensity score matching to create a comparison group; however, they did not account for preexisting differences between the groups in baseline education and earnings outcomes or include sufficient control variables as outlined in the protocol. These preexisting differences between the groups—and not the SEELC consortium WorkKeys® assessment—could explain the observed differences in outcomes. Therefore, the study is not eligible for a moderate causal evidence rating, the highest rating available for nonexperimental designs.
Causal Evidence Rating
The quality of causal evidence presented in this study is low because the authors did not ensure that the groups being compared were similar nor include sufficient control variables. This means we are not confident that the estimated effects are attributable to the SEELC consortium WorkKeys® assessment; other factors are likely to have contributed.