An Evaluation of Calculus Reform:
A Preliminary Report of a National Study

This is a preliminary report of a study at NSF dealing with what NSF has looked for, what it has found, and directions for future study. One such direction will be to shift from "teaching" to "student learning" and the learning environment.

Background and Purpose

In 1986, the National Science Board (NSB) published a report from their Task Committee on Undergraduate Science and Engineering Education, whose charge was to consider the role of the National Science Foundation (NSF) in undergraduate education. The Neal Report, as it is commonly known, outlined the problems in undergraduate mathematics, engineering, and science education that had developed in the decade prior to 1986. Specifically, the Task Committee discussed three major areas of the undergraduate environment that required the highest level of attention: (1) laboratory instruction; (2) faculty development; and, (3) courses and curricula. Their primary recommendation to the NSB was the development of a plan for "new and innovative program approaches that will elicit creative proposals from universities and colleges...as part of the National Science Foundation" ([5], p. v). As a response to this and other reports (e.g., [2], [10]), NSF published their first program announcement [6] for calculus in 1987, with the first awards implemented in 1988. This program has served as a driving force for the national effort in the mathematics community known as the calculus reform movement.

Institutions nationwide have implemented programs as part of the calculus reform movement, many of which represent fundamental changes in the content and presentation of the course. For example, more than half of the projects funded by NSF use computer laboratory experiences, discovery learning, or technical writing as a major component of the calculus course, ideas rarely used prior to 1986 [3]. The content of many reform courses focuses on applications of calculus and conceptual understanding as important complements to the computational skills that were the primary element of calculus in the past. It is believed by many that such change is necessary for students who will live and work in an increasingly technical and competitive society.

A critical part of this process of change is the evaluation of these programs and their impact on the learning environment. As early as 1991, NSF began receiving pressure from the academic community and Congress to place more emphasis on evaluating the impact of these developments on student learning and the environment in undergraduate institutions (see [7]). Although this pressure has resulted in a heightened awareness of the need for evaluation and financial support for a few such studies, the area of evaluation research in undergraduate reform is still in its infant stages, with much of the work done by graduate students in the form of unpublished doctoral dissertations and master's theses. Only through additional studies can the mathematics community continue to develop the calculus course in ways that are most conducive to the needs of their students, the profession, and society.

A number of reports that present programmatic information and indicators of success in the efforts to incorporate technology and sound pedagogical methods in calculus courses have indeed been written (e.g., [8], [9], [12]). Reform has received mixed reviews, with students seemingly faring better on some measures, while lagging behind students in traditional courses on others. However, these reports present only limited information on student learning in reform courses, primarily because the collection of reliable data is an enormous and complicated task and concrete guidelines on how to implement meaningful evaluations of reform efforts simply do not exist ([12]). The need for studies that determine the impact of these efforts, in combination with the increase in workload brought on by reform, is creating an environment of uncertainty. Funding agencies, institutions, and faculty require the results of such studies to make informed decisions about whether to support or withdraw from reform activities.

This study is being conducted as a part of a larger effort by NSF to evaluate the impact of reform in science, mathematics, engineering, and technology (SMET) education at the undergraduate level. This study has been designed to investigate what is currently known about the effect of calculus reform on (1) student learning, attitudes, and retention; (2)use of mechanisms that historically have been shown to improve the learning environment, i.e., faculty development activities, student-centered learning, and alternative methods of delivery and assessment of knowledge; and, (3) the general educational environment. Preliminary results from this project will be reported here, including information from NSF projects, insights from the mathematics community, and anticipated implications for future efforts in calculus.

Method

The calculus reform initiative that NSF encouraged through awards from 1988 to 1994 set the direction for much of the undergraduate reform that has followed. Therefore, it is especially important that a thorough study of this pioneering effort be conducted in order to make informed decisions not only about the future of calculus, but also regarding all reform efforts in undergraduate SMET education. To this end, the project was designed to synthesize what is currently known about the impact of calculus reform on the learning environment, including student learning, attitudes, and retention. The information to be presented is not intended to be a definitive end to the evaluation process, but rather a progress report of what is known to date. It is expected that this information will generate discussion within the academic community, resulting in additional evaluation studies.

For the purpose of this evaluation study, information was gathered using the following methods:

1. A search of literature was conducted, including journal articles, conference proceedings, dissertations, and other relevant publications, to identify evaluations of calculus reform that are available. The information was compiled and synthesized to determine what is currently known about calculus reform from these evaluations.

2. Folders containing all information that has been submitted to NSF for each NSF-funded calculus project were searched for any proposed evaluation of the project and corresponding findings, as well as dissemination information. This evaluation information has been summarized in a qualitative database and cross-analyzed between projects. A framework for the information to be documented in this database was developed in consultation with NSF program officers prior to the development of the database. Precise definitions to be used in determining the existence of various evaluation, dissemination, and reform activities for each project were also developed and used to guide the data entry.

3. A letter was developed and sent to approximately 600 individuals who have participated in the reform efforts. The letter requested assistance in the compilation of existing evaluation studies in calculus reform. The mailing list included not only principal investigators from NSF calculus projects, but also individuals from non-funded efforts and others who have been involved in the evaluation of calculus projects. The names of those who have not been affiliated with an NSF project were obtained through the literature search. The letter was sent via email to a select group of mathematicians and mathematics educators for feedback and comments prior to the mailing.

Preliminary Findings

1. Analysis of NSF Projects

NSF folders have been obtained for the 127 projects awarded to 110 institutions as part of the calculus initiative (1988-94). Each folder was reviewed and analyzed as discussed above, yielding the following information:

• NSF funding for individual projects ranged from $1,500 per year to $570,283 per year, with a mean annual award of $186,458; the duration of the awards was usually two years or less;
• computer use/laboratory experience, applications, and conceptual understanding are the objectives of reform deemed most important in the projects;
• pedagogical techniques most often cited as part of the projects are technical writing, discovery learning, use of multiple representations of one concept, and cooperative learning;
• the most popular methods for distributing results to the community were conference presentations, journal articles, organization of workshops, and invited presentations at college colloquia;
• evaluations conducted as part of the curriculum dev-elopment projects mostly concluded that students in reform courses had better conceptual understanding, higher retention rates, higher confidence levels, and greater levels of continued involvement in mathematics than those in traditional courses; however, scores on common traditional exams yielded mixed results, making it unclear whether there was any significant loss of traditional skills in reform students.

2. Information obtained from the Community

Almost 200 evaluation documents have been obtained from the academic community, including published papers and curricular materials, dissertations, printed conference presentations and proceedings, letters describing results from projects, and internal reports submitted within various colleges and universities. All information is being summarized in qualitative data files that will be discussed in conjunction with the data on the NSF projects in the full report. A preliminary review of the information submitted has revealed the following trends:

• the reform effort has motivated many (often controversial) conversations among faculty about the way in which calculus is taught; these conversations are widespread and continuous and they have resulted in a renewed sense of importance about undergraduate mathematics education;
• in general, regardless of the reform method used, the attitudes of students and faculty seem to be negative in the first year of implementation, with steady improvement in subsequent years while making continuous revisions based on feedback;
• most faculty believe that the "old" way of teaching calculus was ineffective; however, there is much debate about whether reform is moving in the right direction;
• there does not seem to be any consistency in the characteristics of faculty who are for/against reform; however, the students who seem to respond most positively to reform are those with little or no experience in calculus prior to entering the course, with the students who excel in the traditional environment having the most negative reactions to reform methods;
• the requirements of standardized tests seem to have the greatest effect on the adoption of reform practices by secondary AP calculus teachers; i.e., the recently implemented requirement of graphing calculators on the AP calculus exam has ensured their use in virtually every secondary calculus classroom, even though many AP teachers are opposed to such a requirement;
• faculty universally agree upon the importance of evaluating the effect of the reform efforts on student learning, faculty and student attitudes, and curriculum development; they believe that this information is needed to justify their work within academic departments, to understand the impact of various methods, and to give them motivation to continue in the struggle.

Implications of Findings for the Reform Environment

The existence of common elements in many of the calculus reform projects with varying levels of success implies that the impact of reform is perhaps not so dependent upon what is implemented, but rather the educational environment that is created in which to implement it. This environment is determined in part by the level of departmental commitment and the amount of faculty involvement in the reform efforts. Specifically, departments in which the reform courses are developed and supported by only a small portion of the faculty inevitably confront difficulties, whether those difficulties be the inconsistency between their courses or simply the exhaustion of those involved as they work to keep the program going, often at the expense of a positive relationship with their colleagues.

However, even departments that are very committed to reform will confront problems. It is the anticipation of these problems as well as the construction of methods for handling them that seems most critical to the continuing success of a program. For example, the expectations of a reform calculus course are often ones with which students (and their instructors) have had very little experience. How can students be engaged in productive group learning situations? What are the most effective and appropriate uses of computers? These and the multitude of other questions that reform implies about the educational experience in calculus have made apparent the need for additional support from a variety of sources as both students and faculty adjust their styles in the classroom. These necessary resources include, for example, technical and educational support for faculty and special study sessions and computer assistance for students.

Directions for Further Study

A detailed report of the results from this evaluation project will be published and distributed as appropriate to the broader community. This report will provide information that can be used not only by NSF, but also by the mathematics community at large to inform educational improvements at the undergraduate level. In addition, the report will recommend a plan for the continuing evaluation of calculus reform. The need for studies that employ more in-depth, rigorous data collection, including studies of the long-term impact of the reform efforts, is clearly an area for further research.

Specifically, projects that contribute to the sparse literature addressing the effects of undergraduate reform on student learning, attitudes, and retention are much needed. Although there has been considerable work in the area of student learning in calculus (e.g., [1], [4], and [11]), most of these investigations have looked at how students learn. It is imperative to understand not only how students learn, but also the actual impact of different environments created by the calculus reform movement on their ability to learn. This information can be used to inform allocation of resources and to help the faculty involved in reform to continue developing their ideas in ways that will be most productive for them and their students.

In addition, this study has made evident the need to investigate further the impact of reform on faculty, departments, and institutions, including methods for developing the teaching styles of current and future instructors in ways that are conducive to reform. Such work and the implementation of the resulting findings could help faculty to better understand their students and the pedagogical methods that best suit their needs at various times in their undergraduate experience. It will also help departments and institutions to provide the most effective educational experience for their students.

This research was supported by a grant from the American Educational Research Association which receives funds for its "AERA Grants Program" from the National Science Foundation and the National Center for Education Statistics (U.S. Department of Education) under NSF Grant #RED-9452861. Opinions reflect those of the author and do not necessarily reflect those of the granting agencies. The author would like to thank John Luczak, Patty Monzon, Natalie Poon, and Joan Ruskus of SRI International for their assistance in compiling the information for this report.

Please send all correspondence regarding this article to:
     Dr. Susan L. Ganter
     Director, Program for the Promotion of Institutional Change
     American Association for Higher Education
     One Dupont Circle, Suite 360
     Washington, DC 20036
     sganter@aahe.org
     202-293-6440, ext. 32

References

[1] Davidson, N.A. Cooperative Learning in Mathematics: A Handbook for Teachers, Addison-Wesley Publishing Company, Inc., Menlo Park, CA, 1990.

[2] Douglas, R. Toward a Lean and Lively Calculus, Report of the Tulane Conference on Calculus, Mathematical Association of America, Washington DC, 1987.

[3] Ganter, S.L. "Ten Years of Calculus Reform and its Impact on Student Learning and Attitudes," Association for Women in Science Magazine, 26(6), Association for Women in Science, Washington, DC, 1997.

[4] Heid, M.K. "Resequencing Skills and Concepts in Applied Calculus using the Computer as a Tool," Journal for Research in Mathematics Education, 19 (1), National Council of Teachers of Mathematics, Reston, VA, 1988, pp. 3-25.

[5] National Science Board. Undergraduate Science, Mathematics and Engineering Education: Role for the National Science Foundation and Recommendations for Action by Other Sectors to Strengthen Collegiate Education and Pursue Excellence in the Next Generation of U.S. Leadership in Science and Technology, Report of the Task Committee on Undergraduate Science and Engineering Education, Neal, H., Chair, Washington DC, 1986.

[6] National Science Foundation. Undergraduate Curriculum Development in Mathematics: Calculus. Program announcement, Division of Mathematical Sciences, Washington, DC, 1987.

[7] National Science Foundation. Undergraduate Curriculum Development: Calculus, Report of the Committee of Visitors, Treisman, P. Chair, Washington, DC, 1991.

[8] Roberts, A.W., ed. Calculus: The Dynamics of Change, Mathematical Association of America, Washington, DC, 1996.

[9] Solow, A., ed. Preparing for a New Calculus, Conference Proceedings, Mathematical Association of America, Washington, DC, 1994.

[10] Steen, L.A., ed. Calculus for a New Century, Mathematical Association of America, Washington, DC, 1987.

[11] Tall, D. "Inconsistencies in the Learning of Calculus and Analysis," Focus on Learning Problems in Mathematics, 12 (3 and 4), Center for Teaching/Learning of Mathematics, Framingham, MA, 1990, pp. 49-63.

[12] Tucker, A.C. and Leitzel, J.R.C., eds. Assessing Calculus Reform Efforts: A Report to the Community, Mathematical Association of America, Washington, DC, 1995.