The overall model results indicated that MathForward participation was associated with significantly higher gains in mathematics achievement, when compared to students in the
district not in the program. Grade-by-grade analyses suggest that gains associated with being assigned to the program were primarily in Grades 7 and Grade 8. Furthermore, except for Hispanics in 9th grade, there was no evidence from this study that achievement gaps were closing for students in the program, relative to students not in the program. In grade 7, implementation rates were higher than in the other grades, suggesting one possible explanation for this pattern of results. Teachers made more frequent use of the most powerful TI-Navigator tools in this grade level, and 7th grade teachers also were =more likely to adjust their instruction on the basis of formative assessment data collected using TI-Navigator than were teachers in other grades. The fact that the treatment effect was strongest in this grade suggests that the achievement gains are at least partly attributable to the program. These results are suggestive of the promise of the intervention, but the models tested here do not permit us to conclude that we have unbiased estimates of program impact. There were significant differences between the two groups with respect to both student background and prior achievement, and propensity score matching did not yield groups with enough overlap to create a matched comparison group. The use of gain scores can mitigate potential effects of differences, but the fact that program students had more room
to grow may have affected the results. Thus, we cannot conclude that the significant gains observed in this study were caused by the program. Because of the threats to internal validity, there are limits to both generalizability and potential significance for policymakers beyond RISD. We know little about how the achievement gains of low-performing students compared to those of students with similar profiles as program students, since so many students in the district participated that a matched comparison group was impossible to construct. Comparison groups from outside the district may yield better estimates of impact in future years, but these students may
not share enough of the same policy and district context to yield valid results.
The limitations of the study do not prohibit either TI or RISD from drawing lessons from the study. The relationship between implementation and gains suggests the promise of the
program for high-implementing classrooms; it also suggests the need to understand how to support such implementation in the future. For RISD, the gains made by students are
confirmation of its policy and approach: MathForward students are making significant gains, at least in 7th and 8th grade. The poor results in 9th grade suggest, furthermore, that the district take a closer look to uncover why these results were not as strong as for the other two grades.