Citation
Smith, J., Lewis, K., Hawthorne, L., & Hodges, S. (2013). When trying hard isn’t natural: Women’s belonging with and motivation for male-dominated STEM fields as a function of effort expenditure concerns. Personality & Social Psychology Bulletin, 39(2), 131-143. [Study 3]
Highlights
- The study examined the effect of providing information about effort requirements to succeed in a fictitious male-dominated eco-psychology master’s program on female students’ attitudes toward their ability to succeed in this field.
- The authors randomly assigned 48 female undergraduates to receive one of four verbal messages from an advisor about their ability to succeed in a fictitious master’s program in eco-psychology; the women also received a printed brochure distributed to all study groups. The messages varied in terms of emphasis placed on students’ ability versus level of effort. Specifically, three of the message groups conveyed that student success depended on (1) their natural ability (natural ability group), (2) putting in more effort than peers (more effort group), or (3) putting in the same high level of effort as peers (normal effort group). The advisors did not discuss effort requirements with the fourth group that served as a control group (the no information group). Students then immediately completed a questionnaire measuring self-doubt, perceived confidence, future interest in eco-psychology, and how well they thought they would fit into the program.
- The study found that students in the normal effort group exhibited a significantly higher sense of academic belonging to and future interest in the eco-psychology program than the other three groups. They also had higher perceived competence compared with the more effort group and exhibited lower self-doubt than the no information and more effort groups.
- The quality of causal evidence presented in this report is low because the authors did not include sufficient controls in their analysis to account for differences between each group, which is required of randomized controlled trials (RCTs) with high attrition. This means we are not confident that the estimated effects are attributable to the type of message the study participants received; other factors are likely to have contributed.
Features of the Study
The authors used an RCT design, recruiting 48 upper-level female ecology and psychology students at a northwestern U.S. university to participate in a student feedback and pre-enrollment session on a new, fictitious graduate degree program to be offered in eco-psychology. All participants received the same brochure that presented the program as male-dominated, using mainly photographs of men and male faculty names, and were asked to complete a career inventory to share with a program advisor. All students were then told they were good candidates and would likely succeed, but were randomly assigned to receive one of four supplemental messages: (1) they would have to rely on their natural ability (natural ability group); (2) they would have to put in a lot more effort than other students (more effort group); (3) they would have to put in a lot of effort, just like everyone else (normal effort group); or (4) no additional information was provided (no information group). After receiving the message, students immediately completed a questionnaire measuring self-doubt, perceived confidence, future interest in eco-psychology, and how well they thought they would fit into the program. The authors compared responses between each of the four groups using separate one-way analyses of variance for each outcome measure.
Findings
- Students in the normal effort group exhibited a significantly higher sense of academic belonging (ranging from 0.64 to 0.86 higher on a 5-point scale) and future interest in the eco-psychology program (from 22.59 to 30.00 percentage points) over any other group. They also had higher perceived competence (1.36 points on a 7-point scale) than the more effort group and lower self-doubt (0.84 and 0.71 points, respectively, on a 6-point scale) than the no information and more effort groups.
Considerations for Interpreting the Findings
Information contained in the study suggests that not all randomly assigned students were included in all analyses, but we were not able to reach the authors to determine from which groups students were dropped. Due to small sample sizes and without knowing from which groups students were dropped, CLEAR must assume high differential attrition and consider the study quasi-experimental. In a quasi-experimental study, the authors must account for baseline differences between the treatment and comparison groups on measureable characteristics that could affect the outcomes of interest. The authors in this study did not control for any such differences between the groups due to the empirical approach used.
Given the multiple study groups, a large number of comparisons were made regarding attitudes toward STEM. Performing multiple statistical tests on related outcomes makes it more likely that some impacts will be found statistically significant purely by chance and not because they reflect actual group differences. The authors did not perform statistical adjustments to account for the multiple tests conducted, so the number of statistically significant findings in these domains is likely to be overstated.
Causal Evidence Rating
The quality of causal evidence presented in this report is low because the authors did not include sufficient controls in their analysis to account for differences between each group, as required for RCTs with high attrition. This means we are not confident that the estimated effects are attributable to the type of message the study participants received; other factors are likely to have contributed.