Donald P. Green
Institution for Social and Policy Studies Yale University
Draft: October 19, 2006
Previous Mind Lab experiments have demonstrated the program’s effectiveness in imparting game-playing skills and have strongly suggested the program’s effectiveness in improving math and verbal test-taking performance as well. The experiment described in this report extends this line of research by assessing the impact of a curriculum enhancement program, typically provided in the form of an after-school activity, that was designed to improve the academic performance of low-achieving students.
Research Design
In the fall of 2005, 123 fourth and fifth graders in ten schools participated in a randomized experiment. In eight of the ten sites, the participants were identified by the school as children who were having academic difficulties. In each site, we randomly assigned approximately half of the eligible children to the treatment group; the remainder were assigned to the control group. Children in the treatment group received a Mind Lab curriculum supplement program lasting 10 weeks. The program was taught by experienced Mind Lab teachers and consisted of 10sessions covering topics ranging from general Problem solving ability to problem solving in math and verbal comprehension.
Hypothesis
The central hypothesis of the study is that students randomly assigned to the Mind Lab treatment group will finish the academic year with higher verbal and math grades than their counterparts in the control group.
Outcome Measures
The two main outcome measures are year-end verbal and math grades. These grades were administered using the teacher’s own testing methods and standards; for this reason, we focus our statistical analysis on pair-wise comparisons between treatment and control groups within each site. Each grade is reported on a scale of 0 to 100, with 100 representing a perfect score.
In order to guard against Hawthorne effects, teachers were not informed about which students participated in the curricular enhancement program. Discussions with teachers after the study suggested that they were unaware of which children were participating in the program and accompanying research.
Statistical Model
In keeping with past evaluations of Mind Lab’s effect, outcomes are analyzed using a linear regression model in which outcomes (math grades and verbal grades) are regressed on a variable scored 1 if the subject was assigned to the treatment group and 0 otherwise. The model also includes as a covariate the pretest grades of each participant. Although not strictly necessary for unbiased estimation of the treatment effect, the inclusion of a covariate improves the statistical precision with which the treatment effect is estimated. Thus, the model for math grades is
Mathposttest = α + βTreatment + γMathpretest + Umath ,
where the dependent variable is the math grade recorded at the end of the school year, the regressor of interest is the treatment variable, and the covariate is the math grade recorded at the end of the term prior to the start of the study. The disturbance term (U) represents unmeasured causes of math grades. Thanks to the random assignment of the treatment, the experiment provides unbiased estimates of the key parameter β.
The corresponding model for verbal grades is
Verbalposttest = α + βTreatment + γVerbalpretest + Uverbal .
The verbal and math regressions are not independent, because math and verbal grades are correlated (r = .60) at the start of the experiment, changes in math scores over the time period covered by the study are correlated .36 with changes in verbal scores.
Nevertheless, the two dependent variables are sufficiently distinct to make each interesting in its own right.
Because individuals (not classrooms or grades) were assigned to treatment and control groups, it is unnecessary to correct for clustered assignment when calculating the standard errors of the estimated treatment effect.
Each of these two regressions was computed for each of the ten schools, providing ten effect estimates for math scores and ten estimates for verbal scores. Tests for homogeneity was performed using the META command in Stata 8/SE, revealing no evidence of heterogeneous treatment effects (p = .31, .57) for math scores or verbal scores, respectively. We therefore report a precision weighted average of the ten estimated effects.
results
The results suggest that the program produced small but noticeable gains for both math and verbal outcomes. The estimated effect of the treatment on math grades was a gain of
1.80 points. The one-tailed p-value of this estimate is .048, with is allows us to reject the null hypothesis of no positive effect at the conventional .05 level of significance. The
estimated treatment effect is a gain of 1.48 points in verbal grades. This p-value associated with this estimate is .067, which falls just short of the .05 level of significance. Taken together, the results suggest that the treatment program produced gains in school performance.
Conclusion
The Mind Lab program provides a useful supplement to standard curricular activity when directed at children experiencing difficulty in school. The relatively brief programs studied here generated improved grades of between 1 and 2 points, relative to a control group that received no intervention. Although the magnitude of this treatment effect may seem small, it is important to interpret the treatment effects in relation to the unexplained variation in post-test scores within the control group, whose scores reflect the variation one would expect to find in the absence of any intervention. The treatment effect for math represents 1.80/9.08 = 19.8% of the standard deviation in test scores that remains after one controls for pre-test scores and for each school. Similarly, the treatment effect for verbal grades represents 1.48/7.30 = 20.3% of the residual standard deviation in post- test scores in the control group. Thus, while neither intervention produces a dramatic transformation in the academic achievement of the treatment group, the gains achieved over a short period of time are nonetheless substantial.
This study has focused on academic outcomes. Not measured here are social and emotional outcomes, such as students’ willingness and capacity to interact with each
other in a constructive manner, attentiveness to instructions, and positive affect during and toward school. It should be noted that, during the course of the experiment, Mind Lab instructors repeatedly commented on improvements in the participants’ classroom behavior and general outlook. This aspect of the Mind Lab curriculum warrants special attention in future studies.
A final question is the extent to which the intervention’s impact could have been enhanced through more direct collaboration with teachers. Our aim in the current study was to assess the effects of a stand-alone program, but other research suggests that academic gains may be enhanced by integrating the Mind Lab curriculum more directly in coursework. This hypothesis, too, warrants further investigation and indeed inspired the follow-up study conducted in 2005-2006.
