Research Findings

What Are the Research Findings on the Impact of Math Pathways & Pitfalls?

Two national randomized trial studies have been conducted on the efficacy of Math Pathways & Pitfalls. In addition, four qualitative/mixed methods studies were conducted to help describe and explain the effects. Summaries and detailed reports of the efficacy studies are available below:

The Impact of Math Pathways & Pitfalls on Students’ Mathematics Achievement and Mathematical Language Development: A Study Conducted in Schools with High Concentrations of Latino/a Students and English Learners, 2005–2010
U.S. Department of Education’s Institute of Education Sciences Study (R305K050050)
The Impact of Math Pathways and Pitfalls on Students’ Mathematics Achievement, 2000–2005
National Science Foundation Experimental Study (ESI-9911374)

The Impact of Math Pathways & Pitfalls on Students’ Mathematics Achievement and Mathematical Language Development: A Study Conducted in Schools with High Concentrations of Latino/a Students and English Learners, 2005–2010
U.S. Department of Education’s Institute of Education Sciences Study (R305K050050)

Download the Executive Summary (PDF) | Download the detailed findings (PDF)

The purpose of this five-year study was to evaluate the efficacy of Math Pathways & Pitfalls, a supplementary K–8 curriculum for students, with professional development for teachers. This discussion-based curriculum has a dual focus on building mathematical concepts and developing mathematical language, with an overall goal of effective and equitable learning. The Math Pathways & Pitfalls intervention takes the unique approach of not only fostering correct ways to represent and reason about mathematical concepts, but also explicitly calling students’ attention to common pitfalls and misconceptions. Math Pathways & Pitfalls provides a model for teaching and learning mathematical concepts that can be applied to mathematics lessons in any adopted curriculum.

WestEd researchers used a cluster-randomized experimental design to investigate the efficacy of using these instructional materials and procedures in place of 15 hours of regular mathematics lessons during each of two academic years (grades 4 and 5). Nearly 70% of the participating students were Latino/a, 55% were classified as English learners, and 75% were eligible for free or reduced-price meals. This study examined the impact of Math Pathways & Pitfalls on mathematics achievement and mathematical language development, with special interest in the effects for Latino/a students and English learners.

The research study, funded by the U.S. Department of Education’s Institute of Education Sciences, was conducted with 126 teachers and over 3,300 of their consenting students in Arizona, California, and Illinois. Nearly 70% of the participating students were Latino/a, 55% were classified as English learners, and 75% were eligible for free or reduced-price meals.

Research Questions

The major aim of this study was to test the effectiveness of the Math Pathways & Pitfalls instructional materials on improving student performance in mathematics and academic language development. The following research questions were addressed:

Do students who use Math Pathways & Pitfalls lessons in place of 15 hours of their regular mathematics curriculum exhibit greater increases in mathematics achievement, mathematical language development, and English language proficiency than their counterparts exposed only to regular lessons?

What are the cumulative or generative impacts of Math Pathways & Pitfalls? That is, do students exposed to Math Pathways & Pitfalls lessons during two academic years (in place of 30 hours of their regular mathematics curriculum) exhibit greater annual improvements in mathematics achievement, mathematical language development, and English language proficiency than their counterparts (a) exposed only to regular lessons, and (b) exposed to Math Pathways & Pitfalls lessons during only one academic year (15 hours)?

Methodology

The study used a cluster-randomized experimental design in which classrooms were randomly assigned to either a treatment group or a delayed-treatment control group.

During the first implementation year, treatment-group teachers participated in Math Pathways & Pitfalls professional development and replaced approximately 15 hours of their regular mathematics lessons with Math Pathways & Pitfalls lessons and participated in after-school meetings to support implementation of the lessons. Delayed-treatment control-group teachers used only their regular mathematics lessons and participated in after-school meetings unrelated to Math Pathways & Pitfalls.

During the second implementation year, teachers in the delayed-treatment control group participated in Math Pathways & Pitfalls professional development. Both treatment- and delayed-treatment control-group teachers replaced approximately 15 hours of their regular mathematics lessons with Math Pathways & Pitfalls lessons. In addition, all participating teachers were asked to attend 8 hours of after-school meetings. Students in participating teachers’ classrooms were tracked over two years to examine short-term and generative Math Pathways & Pitfalls impacts.

Mixed-modeling procedures were used to estimate Math Pathways & Pitfalls impacts on students’ mathematics and language achievement as measured by standardized tests and project-designed tests related to the content of each instructional unit. Additional qualitative studies were conducted to assess the fidelity of implementation, provide formative feedback on the implementation process, and to help explain the results.

Mixed-modeling procedures were used to estimate Math Pathways & Pitfalls impacts on students’ mathematics and language achievement. Study outcome measures included standardized test scores for mathematics and English language proficiency, as well as project-administered mathematics and mathematical language assessments. Three supplemental data sets were also used: teacher surveys, classroom observation reports, and teacher interview reports.

Key Findings

Results from this efficacy study indicate that exposure to Math Pathways & Pitfalls over two years raised project-administered, as well as standardized mathematics test scores for the full sample of students and for the sub-samples of Latino/a students, English learners, and English-proficient students. When comparing two years of Math Pathways & Pitfalls experiences to one year, effect sizes were as high as .40 for English learners and .29 for Latinos on standardized mathematics tests. These findings suggest that Math Pathways & Pitfalls adds considerable value to the regular mathematics curriculum in promoting mathematics achievement. The primary conclusions of this study, stated below, are discussed in relationship to the goals and design of the Math Pathways & Pitfalls curriculum.

Math Pathways & Pitfalls increases mathematics achievement. Central to each Math Pathways & Pitfalls lesson is the unique approach of inviting students to analyze correct and flawed (“pitfall”) solution processes. This practice has been shown to lead students to use more correct procedures and remember more correct concepts than only analyzing correct examples (Durkin & Rittle-Johnson, under review). This practice may play an important role in increasing mathematics achievement for Math Pathways & Pitfalls students.

Also, Math Pathways & Pitfalls aims to develop student’s metacognition (self-monitoring or awareness of one’s own thinking). The metacognitive behavior prompted by Math Pathways & Pitfalls may help students work though mathematical misconceptions and avoid or detect errors. We hypothesize that students transfer metacognitive behavior learned through Math Pathways & Pitfalls to their regular mathematics lessons. This would help explain the significant impact of Math Pathways & Pitfalls on not only the Number Sense & Operations portion of standardized tests, but also the significant impact on the Total Mathematics portion of standardized tests.

Math Pathways & Pitfalls has a positive impact on mathematical language development. All Math Pathways & Pitfalls lessons incorporate support for developing mathematical language. For example, lessons engage students in expressing and comprehending mathematical ideas symbolically and verbally, as well as orally and in writing. Teaching guides inform teachers about language issues, key mathematical terms are introduced in context to set the stage for discussion, and a Discussion Builders poster scaffolds discussions. Significant increases on the project-designed mathematical language assessment indicate that Math Pathways & Pitfalls plays a role in developing mathematical language, which may in turn facilitate mathematics learning.

Math Pathways & Pitfalls raises standardized test scores for both English learners and English-proficient students, as well as Latino/a students. Researchers investigating the implementation of Math Pathways & Pitfalls lessons (Khisty & Radosavljević, 2010; Razfar & Leavitt, 2010) found several elements of effective instruction for Latino/a students, English learners, and bilingual students. They include, for example, directing students to respond to others’ ideas; scaffolding discussion; emphasis on analyzing ways to think about a problem; enabling students to have what they say valued by others; varied participant structures; and introduction of key mathematical terms for use in discussion. Few studies have reported successful interventions for raising the mathematics achievement of English learner or Latino/a student groups, so these results are especially notable.

The results of this research, combined with findings from an earlier efficacy study in grades 2, 4, and 6 (Heller, Curtis, Rabe-Hesketh, & Verboncoeur, 2007), funded by the National Science Foundation, confirm the positive effects of Math Pathways & Pitfalls. These studies provide considerable evidence that Math Pathways & Pitfalls increases mathematics learning for students with a variety of economic, geographic, ethnic, and language backgrounds. Future research is needed to understand how Math Pathways & Pitfalls influences teaching and supports the adoption of new practices. Ultimately this information can be valuable in designing effective curricula and instruction for students, and professional development for teachers.

The Impact of Math Pathways and Pitfalls on Students’ Mathematics Achievement, 2000–2005
National Science Foundation Experimental Study (ESI-9911374)

Download the Executive Summary (PDF) | Download the detailed findings (PDF)

WestEd researchers implemented a cluster-randomized experimental design in five school districts across the nation over a two-year period. In the first year of the study, second-, fourth-, and sixth-grade teachers were randomly assigned within their school districts in roughly equal numbers to either an experimental group or a control group. The experimental group teachers were taught how to implement Math Pathways & Pitfalls during a six-hour professional development (PD) session in the summer. Project consultants, trained by project staff, conducted the PD. In the first year of the study, experimental group teachers substituted Math Pathways & Pitfalls for a portion of their regular mathematics curriculum. The control group teachers used their regular mathematics curriculum, and received whatever professional development they normally were provided during that year. A total of 99 teachers and 1,971 students participated in the first year of the study.

In the second year of the study, control group teachers received four days of professional development and then substituted Math Pathways & Pitfalls for a portion of their regular mathematics curriculum. The experimental group teachers from the first year of the study had the option of teaching Math Pathways & Pitfalls again in the second year. These teachers were not obligated to do so, however, and because many elected not to participate in the second year of the study, results from the first year only are presented in this report.

Research Questions

This study addressed two sets of questions. First, the evaluation was designed to measure the impact of Math Pathways & Pitfalls on the mathematics that second-, fourth-, and sixth-grade students learn. The research questions addressed are:

What is the impact of Math Pathways & Pitfalls on students’ knowledge of the mathematics topics addressed, compared to that of students using the regular math curriculum?

How equitable is the impact of Math Pathways & Pitfalls on students’ mathematics knowledge across levels of English language proficiency and entering mathematics ability?

To contribute to the interpretation of the results, the research also examined the fidelity of lesson implementation as enacted within Math Pathways & Pitfalls classrooms, compared to the structure and processes that were intended by the curriculum designers. Research questions addressed are:

How closely does Math Pathways & Pitfalls as enacted follow the structure, content, and discourse processes that were intended by the curriculum designers?

How does Math Pathways & Pitfalls as enacted in classrooms that had greater student math score gains compare with Math Pathways & Pitfalls in classrooms with lower student gains?

Methodology

The project-designed Pitfalls Quiz, which was given at the beginning and end of the school year, was the primary instrument used to measure students’ mathematical knowledge. A separate Pitfalls Quiz was developed for each grade level, each of which was aligned with Math Pathways & Pitfalls lessons used for that grade level. Items on the Pitfalls Quizzes assess concepts and procedures that are known to cause difficulty for students as identified from the research literature and prominent assessments such as the National Assessment of Educational Progress (NAEP) and Trends in International Mathematics and Science Study (TIMSS). The tests contain one or more items that relate directly to the content of each lesson, and a few additional items that were indirectly related to the lesson content. Most items on these tests were multiple-choice format, with one correct answer. At least one of the choices for items in the multiple-choice format contained a common misconception that students have with regard to the concept being assessed. In addition to the Pitfalls Quizzes, standardized mathematics achievement test score data were collected.

Key Findings

This cluster randomized experimental study found significant positive effects (ESS = 0.43 to 0.66) for Math Pathways & Pitfalls students as measured by a project-designed mathematics achievement test. Study participants included geographically, linguistically, and ethnically diverse students in grades 2, 4, and 6.

Findings from this large-scale experimental study, conducted in 2000–2005, demonstrated that teaching using only 15 hours of Math Pathways & Pitfalls lessons in place of regular mathematics lessons boosts student achievement during the course of one school year.

The graphs below show students’ adjusted posttest mean scores on project-designed mathematics quizzes. These quizzes assess key standards using items similar to those found on state and national assessments, such as the National Assessment of Educational Progress (NAEP). Statistically significant positive effects were found (effect size statistics = 0.43 to 0.66) for Math Pathways & Pitfalls students in all grades. Study participants included 2,816 geographically and ethnically diverse students in grades 2, 4, and 6.

This graphic shows a bar chart of the adjusted posttest mean scores percent correct for Math Pathways & Pitfall students and non-Math Pathways & Pitfalls students, in grades 2, 4, and 6.
On the Pitfalls Quiz posttest in grade 2, MPP students averaged 71% correct and Non-MPP students averaged 61% correct.
On the Pitfalls Quiz posttest in grade 4, MPP students averaged 45% correct and Non-MPP students averaged 31% correct.
On the Pitfalls Quiz posttest in grade 6, MPP students averaged 56% correct and Non-MPP students averaged 50% correct.

This graphic shows a bar chart of the adjusted posttest mean scores percent correct for Math Pathways & Pitfall students and non-Math Pathways & Pitfalls students, in grades 2, 4, and 6. The chart also depicts differences between English learners and non-English learners.

On the Pitfalls Quiz posttest in grade 2, MPP students who were also English learners students averaged 70% correct and Non- MPP students who were also English learners students averaged 59% correct. On the Pitfalls Quiz posttest in grade 2, MPP students who were not English learners students averaged 71% correct and Non- MPP students who were not English learners students averaged 61% correct.

On the Pitfalls Quiz posttest in grade 4, MPP students who were also English learners students averaged 43% correct and Non- MPP students who were also English learners students averaged 30% correct. On the Pitfalls Quiz posttest in grade 4, MPP students who were not English learners students averaged 45% correct and Non- MPP students who were not English learners students averaged 32% correct.

On the Pitfalls Quiz posttest in grade 6, MPP students who were also English learners students averaged 58% correct and Non- MPP students who were also English learners students averaged 41% correct. On the Pitfalls Quiz posttest in grade 6, MPP students who were not English learners students averaged 56% correct and Non- MPP students who were not English learners students averaged 50% correct.

* Heller et al., 2000.

Math Pathways & Pitfalls Classroom Observation Transfer Study, 2006–2007

Download a summary of the analysis and findings (PDF)

This qualitative study was conducted by Heller Research Associates and funded by the U.S. Department of Education’s Institute of Education Sciences (R305K050050). It was designed to describe ways in which teachers may transfer practices from the Math Pathways & Pitfalls curricula to their regular, non-Math Pathways & Pitfalls lessons in math or other subjects. The pedagogical practices of interest in this study focused on three dimensions: mathematics, language and discourse, and equity. The study documents examples of ways in which teachers borrow practices from the Math Pathways & Pitfalls lessons, and reasons for those choices. Descriptive data on transfer can be drawn on in future Math Pathways & Pitfalls professional development as a source of ideas that teachers might find useful.

Research Questions

The purpose of the Transfer Study is to describe ways in which the research-based teaching and learning practices introduced through Math Pathways & Pitfalls may transfer to regular, non-MPP lessons in math. The classroom practices to be examined are related to three dimensions described in the Math Pathways & Pitfalls design framework: mathematics, language and discourse, and equity.

The following research questions will be addressed:

What, if any, aspects of the Math Pathways & Pitfalls program content, structure, or process do treatment teachers transfer to non-Math Pathways & Pitfalls lessons?

Which, if any, of the following strategies appear in non-Math Pathways & Pitfalls lessons by the end of the Math Pathways & Pitfalls implementation year?

Teacher asks students more open-ended kinds of questions that closely resemble those in Math Pathways & Pitfalls lessons;

Teacher checks for understanding;

Teacher responds to student errors by exploring the reasoning underlying them;

Students have more opportunities to talk with one another to make sense of the mathematics or other content in the lesson;

There are whole class discussions during which there is sustained dialogue about the mathematics or other content in the lesson;

Students give explanations and justifications for their answers;

Students ask questions of one another or incorporate input from their peers as they formulate an idea or solution;

Alternative ideas, solutions, and representations are presented and examined;

Students discuss pitfalls or identify faulty reasoning.

What reasons do teachers give for these modifications?

Methodology

Study participants included ten 4th-grade teachers from the San Francisco Bay Area who volunteered to replace approximately 15 hours of their regular adopted curriculum with Math Pathways & Pitfalls lessons; to attend 4 days of professional development; and to be observed and audiotaped.

Two non-Math Pathways & Pitfalls lessons were observed and audiotaped in each participating classroom. The beginning-of-year observations were conducted in October 2006, and end-of-year observations took place in April–May 2007. Observations were conducted using a protocol that was developed in the Fall of 2006.

After each observation, the researcher interviewed the teachers by telephone to obtain additional information about the lesson that had been observed.

Key Findings
Observations

Over the six-month period between observations, rubric scores increased, indicating that teachers’ instruction generally developed in all three areas (mathematics, language, and equity) after implementation. Average scores show classrooms in the beginning-developing range during the fall observation, and in the developing-advanced range during the spring observation. All teachers improved in at least one area, and no teacher scored lower during the second observation than he/she did on the first. These results indicate that teaching practices during regular, non-Math Pathways & Pitfalls math lessons developed in the direction of general and specific aspects of the Math Pathways & Pitfalls approach.

Mean total scores on the 9-point rubric for the post-Math Pathways & Pitfalls implementation data were significantly greater than mean total scores before Math Pathways & Pitfalls was implemented (Wilcoxon T+ = 36, p < .005, n = 8). Math Pathways & Pitfalls practices appear to transfer from Math Pathways & Pitfalls lessons to non-Math Pathways & Pitfalls lessons, but without a comparison group we cannot tell for sure.

Interviews

Teachers were better able to articulate the causes and effects of increased participation during the second interview, and were generally more specific about effective strategies and the strengths and weaknesses of their students. During the second interview, the interviewer also asked specifically how the Math Pathways & Pitfalls course was affecting their non-Math Pathways & Pitfalls lessons, and teachers cited a wide range of Math Pathways & Pitfalls teaching strategies and practices. In addition, teachers were generally very positive about their Math Pathways & Pitfalls experience.

Conclusions

In short, these results indicate that teaching practices during regular, non-Math Pathways & Pitfalls math lessons developed in the direction of general and specific aspects of the Math Pathways & Pitfalls approach that was implemented over the intervening six-month period. This is an important finding because it raises the possibility that a relatively small “dose” of Math Pathways & Pitfalls lessons (about 15 hours during the school year) may have a powerful effect on mathematical learning. If so, studies of this phenomenon may lead to understandings about which practices are more likely to transfer, and provide clues about how Math Pathways & Pitfalls supports the adoption of new practices. Ultimately this information will be valuable in the design of new instructional materials and professional development for teachers.

A Descriptive Analysis of Math Pathways and Pitfalls in a Latina/o Bilingual Classroom

Download the research findings (PDF)

This qualitative study was conducted by external researchers Lena Licón Khisty and Alex Radosavljević of the University of Illinois at Chicago. The paper was presented at the 2010 Annual Meeting of the American Education Research Association in Denver, Colorado. The study was funded by the U.S. Department of Education’s Institute of Education Sciences (R305K050050).

Abstract

This paper describes findings of a qualitative study of the implementation of Math Pathways & Pitfalls in an urban Bilingual classroom that is predominantly Latina/o. Analysis of videotaped Math Pathways & Pitfalls lessons both confirms earlier findings of the positive effects of Math Pathways & Pitfalls with bilingual learners and provides insights of characteristics of mathematics learning environments that support Latina/o students. The discussion links the positive effects of Math Pathways & Pitfalls to principles of effective instruction with bilingual learners, and thus, demonstrates how mathematics teaching can incorporate these principles.

Building Mathematics Discussions in Elementary Classrooms with Latino/a English Learners

This qualitative/mixed-methods study was conducted by external researchers Aria Razfar and Della Leavitt of the University of Illinois at Chicago. The paper was presented at the 2010 Annual Meeting of the American Education Research Association in Denver, Colorado. The study was funded by the U.S. Department of Education’s Institute of Education Sciences (R305K050050).

Abstract

Drawing on Cultural Historical Activity Theory (CHAT) and Gee’s (1996) notions of primary and secondary discourses, this paper examines the use of Math Pathways & Pitfalls discussion builder stems to mediate mathematical learning and language development. Through analyses of the social organization of learning and classroom discourse conducted in the later stages of an urban classroom of Latino/a English language learners and several African American students, we examine how the solving of mathematical problems in the domain of fractions and decimals is mediated by the cultural artifacts provided by Math Pathways & Pitfalls. In particular, we focus on discussion builder stems, the pitfall narrative, and other elements of the Math Pathways & Pitfalls curriculum. We further aim to show how the participants display a multiplicity of functions, purposes, and uses as they draw on the Math Pathways & Pitfalls register to engage in mathematical problem solving as well as participate in the broader classroom culture. Finally, we explore how Math Pathways & Pitfalls in conjunction with other primary and secondary discourses can lead to the development of meta-level discourses as exemplified by the pitfall metadiscourse.

Examining Academic Language in Mathematics Test Items for English Language Learners