Citation
McConnell, S., Stuart, E., Fortson, K., Decker, P., Perez-Johnson, I., Harris, B., & Salzman, J. (2006). Managing customers’ training choices: Findings from the Individual Training Account experiment. Final report, Series: ETAOP 2007-01. Washington, DC: Employment and Training Administration, U.S. Department of Labor.
Highlights
- The study’s objective was to compare the impacts of individual training account (ITA) approaches on customers’ training completion, employment, earnings, and benefits receipt outcomes after 15 months.
- The authors randomly assigned about 8,000 adults and dislocated workers from eight workforce investment agencies across the United States to three ITA approaches: structured customer choice, maximum customer choice, or guided customer choice. The authors analyzed data from Unemployment Insurance (UI) records and from a survey conducted on a random subsample of study participants.
- The study found the customers in the structured customer choice approach received significantly higher amounts of public assistance benefits than did customers in the guided customer choice group. The study also found that the structured customer choice group had higher earnings and lower education and training compared with the maximum customer choice group, whereas the maximum customer choice group had lower earnings and higher public assistance benefits compared with the guided customer choice group.
- The quality of causal evidence provided in this study is high because it is based on a well-implemented randomized controlled trial. This means we are confident that the estimated effects are attributable to differences in the ITA approaches, and not to other factors.
Intervention Examined
The Individual Training Accounts (ITA) Approaches
Features of the Intervention
The study examined dislocated workers and other adults who were new customers of the participating Workforce Investment Area (WIA) agencies who were eligible for ITAs. WIAs have flexibility in how they implement ITAs, which customers use to fund training programs of their choice from a wide selection of state-approved providers. The ITA experiment tested three alternative ways to manage customers’ choices of training programs: structured customer choice, guided customer choice, and maximum customer choice.
Dislocated workers and adults (18 years of age and older) eligible for WIA training assigned to the structured customer choice group received intensive, mandatory weekly counseling. Counselors directed clients to training selections that maximized return on investment and could reject clients’ choices that did not do so. Workers in this condition could also use an ITA to cover the full cost of training (up to $8,000). Those assigned to the guided customer choice group received less intensive but still mandatory weekly counseling that guided them to appropriate training selections. Counselors could not reject clients’ choices under this condition. Those in the guided customer choice group also received a fixed ITA established for their area ($3,000 to $5,000). Those assigned to the maximum customer choice group did not automatically receive, but could request, counseling to guide them to appropriate training selections and received a fixed ITA established for their area ($3,000 to $5,000). Under each condition, regional workforce boards and human services departments provided program services.
Features of the Study
To be eligible, customers had to be determined as in need of training, have the skills and qualifications to complete training, have received at least one core and one intensive service, be unable to obtain funding assistance to pay for training from other sources, and consent to participate in the study. About 8,000 dislocated workers and adults eligible for WIA-funded training were randomly assigned to one of three tested ITA approaches. Randomization occurred individually by study staff on a rolling basis as site staff submitted the names of eligible customers. Intake occurred during an approximately 18-month period from December 2001 to August 2002, depending on the site. Follow-up continued for 15 months after random assignment. The authors estimated site-specific impacts using regression models that included site, group assignment dummy variables interacted with site, and baseline characteristics.
Study Sites
- Phoenix, Arizona
- Maricopa County, Arizona
- Bridgeport, Connecticut
- Jacksonville, Florida
- Atlanta, Georgia
- Northeast Georgia
- North Cook County, Illinois
- Charlotte, North Carolina
Findings
- The study found that customers in the structured customer choice approach reported receiving significantly more Food Stamps, receiving an average of $78 more during the follow-up period than did customers in the guided customer choice approach ($374 versus $296). This finding was significant at the 5 percent level.
- Customers in the structured customer choice approach reported significantly higher earnings, with an average of $1,571 more during the follow-up period than customers in the maximum customer choice approach ($16,685 versus $15,514). This finding was significant at the 5 percent level.
- Customers in the structured customer choice approach were significantly less likely to complete a training program during the 15-month follow-up period compared with customers in the maximum customer choice approach (39 versus 43 percent). This finding was significant at the 5 percent level.
- Based on administrative records, customers in the maximum customer choice approach had significantly lower earnings, with an average of $1,206 less during the follow-up period than customers in the guided customer choice approach ($14,225 versus $15,431). This finding was significant at the 5 percent level.
- Based on the administrative records, customers in the maximum customer choice approach received significantly more UI benefits, with an average of $217 more during the 15-month follow-up period than guided customer choice customers ($3,483 versus $3,266). This finding was significant at the 5 percent level.
- Customers in the maximum customer choice approach reported receiving significantly more Food Stamps, with an average of $95 more during the follow-up period than customers in the guided customer choice approach ($391 versus $296). This finding was significant at the 5 percent level.
Considerations for Interpreting the Findings
The study authors estimated multiple related impacts on outcomes related to short-term benefit receipt. Performing multiple statistical tests on related outcomes makes it more likely that some impacts will be found statistically significant purely by chance and not because they reflect program effectiveness. The authors did not report that they performed statistical adjustments to account for the multiple tests, so the number of statistically significant findings in these domains is likely to be overstated.
The authors also noted that the structured customer choice approach was generally not implemented as planned. Specifically, counselors were rarely directive with customers, vetoed unauthorized training choices, and did not push customers toward low-cost, high-return training. This could have weakened the contrast between the services received by the treatment and control groups.
Causal Evidence Rating
The quality of causal evidence provided in this study is high because it was based on a well-implemented randomized controlled trial. This means we are confident that any estimated effects are attributable to the differences in the ITA approaches and not to other factors.
Additional Sources
McConnell, S., Stuart, E., Fortson, K., Decker, P., Perez-Johnson, I., Harris, B., & Salzman, J. (2006). Managing customers’ training choices: Findings from the Individual Training Account experiment. Final report appendices. Washington, DC: Employment and Training Administration, U.S. Department of Labor.