Skip to main content

More than robots: An evaluation of the FIRST robotics competition participant and institutional impacts (Melchior et al. 2005)

Review Guidelines

Citation

Melchior, A., Cohen, F., Cutter, T., & Leavitt, T. (2005). More than robots: An evaluation of the FIRST robotics competition participant and institutional impacts. Waltham, MA: Brandeis University Center for Youth and Communities Heller School for Social Policy and Management.

Highlights

  • The study’s objective was to examine the impact of the FIRST Robotics Competition (FRC) on participants’ education and career outcomes. The FRC was a six-week competition in which teams of high school male and female students (with the help of mentors) used a set of standard parts to build a robot. Students showcased these robots by competing against teams from other high schools.
  • The authors used propensity-score matching to create treatment and comparison groups and conducted an analysis of retrospective survey data to measure education and career outcomes of FRC participants who graduated from the program from 1999 to 2003. They administered a researcher-designed survey to students in the treatment group in 2004 and collected data from a national survey—the Beginning Postsecondary Student Survey (BPSS)—to measure outcomes for the comparison group, who began college in 1995-1996.
  • The study found that a higher percentage of female FRC participants majored in engineering compared with females in the comparison group.
  • The quality of causal evidence presented in this report is low because the authors did not demonstrate equivalency of the treatment and comparison groups on all required measures. This means we are not confident that the estimated effects are attributable to the FRC; other factors are likely to have contributed.

Intervention Examined

FIRST Robotics Competition

Features of the Intervention

The FRC was a six-week competition in which teams of high school students (with the help of teachers at their school and community mentors) used a set of standard parts to build a robot. Students showcased these robots by competing against teams from other high schools in a variety of challenges at regional and national events. During the program, students were required to attend team meetings, raise funds, build and program the robot, develop a strategy for the competition, and present the robot to various audiences. Typically, teams divided these tasks among the team members so specific work varied by student. Schools across the country were able to form their own FRC team and students self-selected themselves into the teams. The goal of the program was to increase students’ STEM interests and abilities, while also developing teamwork, leadership, and project management skills.

Features of the Study

The primary goal of the study was to retrospectively determine the impact of participation in the FRC on a variety of outcomes for all students, but particularly low-income and minority students. To do so, the authors first invited all of the high schools in three large metropolitan areas (New York City, Detroit/Pontiac metropolitan area, and San Jose/San Francisco Bay area) that had been running the FRC program for at least two years to participate in the study. Of the 30 schools invited to participate, 14 had the interest and necessary data to do so. These 14 schools identified 360 former program participants to form the treatment group, of which 173 ultimately filled out the study-administered survey (a 48 percent response rate) and were included in the analysis.

The researchers compared the study-administered survey data collected from the treatment group with the BPSS survey data to create a comparison group using propensity-score matching. Using 147 exact matched pairs—matching on participants’ demographics, parental education, economic status of the high school, and various measures of high school STEM engagement—the authors compared the two groups on a variety of education, career, and developmental outcomes by calculating the difference in average outcome between the treatment and comparison groups and determining the statistical significance.

Findings

  • The study found that a higher percentage of female FRC participants majored in engineering compared with the comparison group (32.6 versus 8.7 percent).
  • The authors also found a marginally statistically significant difference in the proportion of females majoring in STEM-related fields more generally (in addition to engineering) (41.3 percent of the treatment group compared with 21.7 percent of the comparison group).

Considerations for Interpreting the Findings

This study has several issues that should be considered when interpreting the findings. First, the treatment and comparison groups might differ substantially on both observed and unobserved characteristics. The authors did not match students on age as required for this review, and students could self-select into the intervention, which could bias the results. Second, data for the treatment group came from study-administered surveys of participants who were several years removed from program participation, such that some survey results might be subject to recall bias. Third, the treatment sample might suffer from nonresponse bias because only 14 of 30 treatment schools participated and only 173 of 360 former students responded to the study-administered survey. Fourth, the treatment and comparison groups did not attend college at the same time, and the authors used different data sources to measure their outcomes, both of which could confound the results.

Causal Evidence Rating

The quality of causal evidence presented in this report is low because the authors did not demonstrate equivalency of the treatment and comparison groups on all required measures. This means we are not confident that the estimated effects are attributable to the FRC; other factors are likely to have contributed.

Reviewed by CLEAR

January 2016