Citation
Demetry, C., Hubelbank, J., Blaisdell, S., Sontgerath, S., Nicholson, M.E., Rosenthal, E., & Quinn, P. (2009). Supporting young women to enter engineering: Long-term effects of a middle school engineering outreach program for girls. Journal of Women and Minorities in Science and Engineering 15, 119-142.
Highlights
- The study’s objective was to assess the long-term impact of Camp Reach, a summer engineering enrichment program for middle school girls, on enrollment in STEM courses in high school and college.
- Admission to Camp Reach was determined through a lottery of girls who applied to the program and met eligibility requirements. Six to seven years later, the authors administered a survey to 109 girls who had been admitted to the program and 107 girls who had not. Many girls in the control group attended other, similar programs.
- The study found that members of the control group who had attended other summer science or engineering programs at Worcester Polytechnic Institute (WPI) were significantly more likely to have taken high school calculus than were members of the Camp Reach Partial group. There were no statistically significant differences on taking physics, computer science, other science and engineering courses, or planning to major in engineering in high school.
- The quality of causal evidence presented in this report is low because the authors adjusted the research sample after random assignment and did not include adequate controls to ensure that the resulting groups were similar on all relevant attributes. This means we are not confident that the estimated effects were attributable to Camp Reach; other factors are likely to have contributed.
Intervention Examined
Camp Reach
Features of the Intervention
Camp Reach was a two-week summer engineering program for rising 7th-grade girls run through WPI. The program aimed to encourage girls’ interest in and pursuit of engineering and other STEM fields through design workshops and mentoring. It emphasized collaborative problem-solving through a collaborative public service engineering project, which showed how engineering can be a helping profession. The camp promoted continued engagement through alumnae newsletters and reunions.
Features of the Study
Camp Reach made 30 places available to interested applicants each year; those applicants meeting minimum eligibility criteria were entered in a lottery through which camp organizers selected the summer’s participants, who formed the treatment group. The group of selected participants was then adjusted to reflect the demographic profile of Worcester, Massachusetts, where the program was held. Those who were not invited to participate formed the control group.
The authors contacted 252 girls who had applied to Camp Reach from 1997 to 2001 in 2004 through 2007, six to seven years after they had applied, to complete a survey. Of these, 176 responded. The authors compared the responses of the treatment and control group members on survey questions about their knowledge of engineering, the science and engineering courses they took in high school, and plans to major in science or engineering in college.
To estimate impacts, the authors organized the treatment and control groups into subgroups. First, they distinguished between girls in the treatment group who had attended the summer session of Camp Reach (Camp Reach Partial) and those who had participated in alumnae reunions and the initial summer session (Camp Reach Full). Because further program participation was nonexperimental, this review focuses on treatment survey respondents who attended only the Camp Reach summer session. In addition, the authors organized the control group into those who had or had not attended other science and engineering programs, because many survey respondents who were not selected for Camp Reach ultimately participated in similar programs offered by WPI.
Findings
- The study found that members of the control group who had attended other summer science or engineering programs at WPI were significantly more likely to have taken high school calculus than were members of the Camp Reach Partial group. There were no statistically significant differences on taking physics, computer science, other science and engineering courses, or planning to major in engineering in high school.
- There were no statistically significant differences between members of the Camp Reach Partial group and members of the control group who did not attend other science or engineering camps at WPI.
Considerations for Interpreting the Findings
The authors acknowledged that the randomly selected attendee groups were modified after random selection to replicate Worcester’s demographic profile. This adjustment means that the assignment was no longer random, and therefore this study cannot receive a high causal evidence rating. The adjustment of the research sample after random assignment could have induced differences between the treatment and control groups on characteristics that also influenced the outcomes of interest. Because the techniques the authors used to estimate the program’s effect did not adequately control for these potential differences, they cannot distinguish the effects of Camp Reach from the effects of differences in relevant demographic characteristics.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors adjusted the research sample after random assignment and did not include adequate controls to ensure that the resulting groups were similar on all relevant attributes. This means we are not confident that the estimated effects were attributable to Camp Reach; other factors are likely to have contributed.