Evaluating the Effects of Reform

West Point turned an entire department around. Using an in-depth assessment study with careful attention to the needs of client disciplines, the department created a brand new curriculum, and continues to study it with the "Fullan model" which the author investigated in his dissertation.

Background and Purpose

In 1990, the US Military Academy at West Point changed to a bold new core mathematics curriculum that addressed seven topics in four semester-long courses. The needs for change were both internal and national in scope. Internally, math, science and engineering faculty were disappointed in the math abilities of the junior and senior students. Externally, the national reform movement was providing support in the form of interest, initiative and discussion. Initially, my research evaluated this curriculum change from three perspectives: how the new curriculum fit the national recommendations for reform, how the change was implemented, and what the effects were on student achievement and attitudes toward mathematics. This paper will report on the resulting longitudinal comparison of two cohorts of about 1000 in size, on the "steady-state" assessment of subsequent cohorts, and on the changes as a result of these assessments. This study informs the undergraduate mathematics and mathematics education community about the effects of mathematics reform on student performance, about the implementation and value of department-level reform and evaluation, and the implications and prospects of research of this type.

a) Description of Students: Admission to West Point is extremely competitive. Current levels of admission are greater than ten applicants for each acceptance. The goal of the admissions process is to accept students who are as "well-rounded" as possible, including both physical and leadership aspects. Average Math SAT scores are around 650. Education at West Point is tuition-free. In general, West Point cadets are very good students from across the nation with diverse cultural backgrounds.

The comparison cohort entered West Point in July 1989 and began the old core mathematics curriculum in August 1989. There were approximately 1000 students who finished the four core courses together. Most of these students graduated in May 1993. The reform cohort entered West Point in July 1990 and was the first group to take the new curriculum, starting in August 1990. There were approximately 1000 students who finished the four core courses together. Most of these students graduated in May 1994.

b) Descriptions of Old and New Curricula: West Point's core curriculum comprises 31 out of the 40 courses required for graduation. Throughout the first two years, all students follow the same curriculum of five academic courses each semester in a two-semester year. Approximately 85% of any freshman or sophomore class are studying the same syllabus on the same day. Over the first two years, every student must take four math courses as well as year-long courses in chemistry and physics. Approximately 85% choose a major toward the end of the third semester. During their last two years, all students take one of seven five-course engineering sequences. Thus, the core mathematics program provides the basis for much of the student's education, whether he or she becomes an English, philosophy, or math, science, and engineering major.

The old core math curriculum was traditional in context and had no multivariable calculus, linear algebra, or discrete math. The courses were Calculus I, Calculus II, Differential Equations, and Probability and Statistics. For those who majored in most engineering fields there was another required course called Engineering Mathematics that covered some multivariable calculus, some linear algebra, and some systems of differential equations.

The current core curriculum (initiated in August 1990) covers seven topics in four semesters over the first two years. The topics are discrete mathematics, linear algebra, differential, integral and multivariable calculus, differential equations, and probability and statistics. A discrete mathematics course focused on dynamical systems (or difference equations) and on the transition to calculus (or continuous mathematics) starts the two-year sequence. The mathematics needed for this course is new to the majority of high school graduates but is also intuitive and practical. Linear algebra is embedded in a significant way, as systems of difference equations are covered in depth. So, two of the seven-into-four topics are addressed in this first course. Further, this course by design provides a means to facilitate the accomplishment of many other reform goals and the goals of the curriculum change (see appendix), such as integrating technology, transitioning from high school to collegiate mathematics, and modeling "lively" application problems.

The current Calculus I course finishes differential calculus and covers integral calculus and differential equations through systems, thus addressing three of the seven-into-four topics. Calculus II is a multivariable calculus course. In addition to the above, these new calculus courses differ from their predecessors by integrating more technology, utilizing more interactive instruction, and including more group projects that require mathematical modeling, writing for synthesis, and peer-group interaction. The probability and statistics course, the last of the four core courses, has been taught for over thirty years to all second-year students and is gaining in importance to our engineering curricula.

The use of technology and the integration of the content in this curriculum into one program provide the opportunities to fit the topics from seven courses into the four semesters. In short, the gains are coverage of linear algebra, discrete math, and multivariable calculus, while totally integrating modeling and technology use. The losses are relatively minor: reduced emphasis on analytic geometry, series, and integration techniques and movement of Laplace transforms to an Engineering Mathematics elective specifically for those majoring in engineering.

c) Framework for Evaluation: Utilizing the three perspectives of (1) reform, (2) implementation process, and (3) comparison of two student cohorts, I conducted three different analyses of the curriculum change. Although this change appears to have come from needs internal to West Point, during the time period much was being said nationally about mathematics education at all levels from kindergarten through college. While the NCTM Standards [3] showed the way for K-12, the colleges have had their own voices for reform particularly the Committee on the Undergraduate Program in Mathematics (CUPM) Recommendations [5] in 1981 and the whole Calculus Reform movement which appears to have its beginnings around 1986. In my studies the recommendations of this national reform movement were best synthesized in Reshaping College Mathematics [4]. I used these recommendations and others to analyze whether West Point's curriculum reform had core characteristics similar to those of the national reform movement.

Large-scale educational change is difficult to initiate, to implement and to maintain. There are many obstacles to overcome in starting up and maintaining a new curriculum, not the least of which is a resistance to change itself. Consequently, the documentation of a major curriculum reform is extremely valuable to all those who wish to attempt such an innovation. One model whose designer addresses the procedures and factors that make up a successful educational change is posed by Michael Fullan in [1]. Fullan says that educational change has two main aspects: what to change and how to change. The national reform movement has provided a consensus of what to change and Fullan provides a theoretical model to compare the West Point innovation against. According to Fullan educational change has three phases: initiation, implementation, and continuation, all leading to outcomes. Each of these phases interacts with its sequential neighbor. In this study, Fullan's factors for each of these phases were analyzed for relevance and impact to the change process. In short, the Fullan model provides a cogent framework to evaluate the change process.

The outcomes for this evaluation were student achievement and attitudes. Mathematics reform at the college level, as with most other educational reforms, seeks to improve student learning and attitudes in the hope that this improvement will in turn motivate students to further study and application of the mathematics they have learned. Further, these outcomes are addressed to determine if the goals for the curricular reform are being accomplished.

Method

My focus was to evaluate the impact of reform on student performance and attitudes toward mathematics by comparing the achievement and attitudes of the two student cohorts described above. To conduct the evaluation, I formulated twelve guiding questions, nine of which describe the context of my study and provide input for the reform and implementation perspectives. The remaining three questions focus on the comparison of the two cohorts.

Data for the two contextual perspectives were obtained through an extensive literature search of recent reform and educational change references, review of historical documents, and interviews of students and faculty. Data for the comparison of the two cohorts were obtained from quizzes, exams, questionnaires, interviews and grades. Except for the interviews, most data were collected and stored for later analysis. There was almost no a priori experimental design.

Findings

The details of this analysis are contained in my dissertation [6], which was completed while I was an associate professor in mathematics at West Point. In addressing the three perspectives I have outlined above, I found for the reform perspective that the revised curriculum at West Point used for the core mathematics curriculum was consistent in most ways with the national call for reform in mathematics curriculum at the college level. For the implementation perspective, I reported that the processes for implementation of the curriculum change involve many factors, but that the change studied was successful in accomplishing its articulated goals (see appendix). The informed and empowering leadership of the department head and the involvement and consensus-building style of the Department of Mathematical Sciences senior faculty in the change process were the key factors that motivated the implementation of the revised curriculum. Over the two years prior to the August 1990 implementation, the senior faculty planned and built institution-wide consensus for the initiation of the revised core mathematics curriculum. The planning and implementation continued through January 1992, when the last of the four courses began. Improvements to this original revised curriculum have continued through the present. Articulated goals for this curriculum for the most part appear to have been accomplished, and having continued with the curriculum for seven years, the change appears to be institutionalized.

Finally, for the comparison perspective I evaluated the effects of the change in curriculum in terms of student mathematics achievement and attitudes toward mathematics. Students in the reform cohort under the current core curriculum were compared with students under a traditional curriculum. The comparison of the two cohorts in terms of student achievement and attitudes was difficult. My planned data collection included results of math tests, common quizzes, questionnaires, and interviews. I found these data very informative about a certain cohort. Yet, a direct comparison was like comparing apples and oranges, or the results were inconclusive. The lesson learned is that experimental design is needed before-the-fact for these instruments to be compared. At the same time, this needed before-the-fact planning may not be feasible.

In contrast, my analysis of grades was intended purely for informational purposes, but produced the most compelling results. The comparison of grades in follow-on courses such as physics and engineering science, which pride themselves in standardization from one year to the next, showed significant improvements between cohorts. The tables below from my dissertation show the results of the comparison of grades for the two-semester physics sequence. Similarly, I looked at eight engineering science courses taken by a total of 85% of each of the cohorts. Four of these eight courses showed significant results (p-value < 0.05) and a similar shift in grades to the physics courses. The reform group performed better in these courses.

I was able to respond to all my guiding questions except the comparison of attitudes of the two groups. Attitude data for the comparison group were either not available, of such a small scale, or not of similar form to make a reasonable comparison feasible. I found myself comparing different questions and having to draw conclusions from retrospective interviews of students and faculty. If questionnaires are to be used, some prior planning is needed to standardize questions. However, the data from the student and faculty interviews indicate some improvement by the reform group in the areas desired to be affected by the revised curriculum. Attitudes had not been measured until 1992, after the revised curriculum had been implemented.

Use of Findings

While I concluded that the reform curriculum was successfully implemented, the change process in the Mathematical Sciences Department at West Point is still ongoing. This current year saw the adoption of a new calculus text. Further, for varying reasons of dissatisfaction, availability, cost, and adjusting to other changes, four different differential equations texts have been used over the six years of the new curriculum. In addition, the text for the discrete dynamical systems course will change this fall. All courses are using interdisciplinary small group projects designed with other departments and disciplines. The student growth model (more below) developed in 1991 and used to shape the four-course program is being updated continuously. At the same time, the faculty development program that supports improvements to the curriculum and the way we teach has been significantly enhanced in the last two years. Finally, assessment instruments are currently under great scrutiny to ensure that they mirror the goals of the student growth throughout the four-course program and attitude questionnaires have been administered each semester since spring 1992.

The most significant framework for change over the past eight years has been the department's focus on improving student growth over time. The senior faculty started by establishing program goals in the spring of 1990 (see appendix). Basically, these goals are difficult, as they are geared to developing aggressive and confident problem solvers. Their intent is that students are required to build mathematical models to solve the unstructured problems that they will face in the real world. The senior faculty operationalized these goals by establishing five educational threads that were integrated throughout the four courses. Still in effect, these five threads are scientific computing, history of mathematics, communications, mathematical reasoning and mathematical modeling. Each of the courses wrote objectives to address each of these specific ideas toward accomplishing over time the goals we established for the program. These threads and the course objectives are called the "student growth model." As a result, all assessment instruments are designed to address these objectives and thereby measure student growth.

The department turns over about one third of its faculty each year. As a result, over time the student growth model becomes unclear and open to interpretation. As students and faculty become less familiar with the student growth objectives they become unfocused. Therefore, approximately two years ago the senior faculty found it necessary to articulate the student growth model in terms of content threads. The result was nine: vectors, limits, approximation, visualization, models (discrete and continuous, linear and nonlinear, single- and multi-variable), functions, rates of change, accumulation, and representations of solutions (numerical, graphical, symbolic, descriptive). The intent was that both students and faculty could more readily identify growth if it was articulated in mathematical terms. Further, these content threads provide avenues for better streamlining of the curriculum to enhance depth on those topics essential to growth in the program. Finally, they facilitate the design of assessment instruments by having objectives that are content-specific. Further articulation of the goals and objectives for these content threads are forthcoming as well as a requisite assessment scheme.

Evaluation of the program continues. As a result of this initial study, since 1992 I have created a database for each cohort of grades of all mathematics-based core courses. Further, I have used common attitude questions on entry and at the end of each mathematics course. Study of these data are ongoing and are used to inform senior faculty about specific mid-program and mid-course corrections. These instruments actually tell more about the growth of the cohort over time rather than serve to compare one group to the other. However, looking at the same course over time does inform about trends in that course.

A recent additional evaluation tool uses portfolios to measure conceptual growth over time. We have been using student portfolios since 1993, mostly as self-evaluation instruments. This past year we instituted five common questions following the themes of our five educational threads that all students must respond to in each of their core mathematics courses. Each is supposed to be answered with a paragraph up to a half-page typewritten and included in the portfolio for each of the four courses. A couple of examples from this year are: (1) Define a function. Give an example of a function from this course and explain its use. (2) Discuss how a math modeling process is used in this course. Describe the impact of "assumptions" and how one can "validate" their model. Further, each subsequent portfolio must contain the responses to the questions from the previous course(s). This gives the student and faculty an example of an individual student's growth over the span of the four courses. Since we just started this year, we do not know how this will work. But we have hopes that this snapshot will be valuable to both the student and the faculty.

Success Factors

The references below proved excellent in shaping a consensus interpretation of the national reform movement. Further, the West Point Math Sciences Department was very good about articulating goals for the new curriculum (see appendix). The fact that they were written down allowed me to understand very quickly the stated focus of what I was trying to evaluate. It further allowed me to conclude that their stated goals had been accomplished. At the same time, the Fullan model provided a cogent model for evaluating educational change at the undergraduate level.

I understand that West Point is not the typical college, and for my dissertation [6] I devoted an entire appendix to the issue of generalizability. In this appendix I enclosed letters from prominent faculty familiar with our curricula from large research universities to small liberal arts colleges that supported the generalizability of the results. I believe that most of what I have related here is generalizable to other schools and other programs. The evaluation I have conducted is an example of the use of the model in [2] The bottom line is that the MAA model works and the assessment process can be very useful in informing senior faculty who must make curricular decisions.

In closing, assessment at the department level is a process that can involve the entire faculty, build consensus, inform decisions about improving curricular programs, and evaluate student learning over time. My experience with the evaluation of the curriculum reform at West Point is an example of this. I hope that the ideas posed here will encourage others to proactively design assessment programs with the goal of improving student learning.

References

[1] Fullan, M.G. The New Meaning of Educational Change (2nd ed.), Teachers College Press, New York, 1991.

[2] Mathematical Association of America. "Assessment of Student Learning for Improving the Undergraduate Major in Mathematics," Focus, 15 (3), 1995, pp. 24-28.

[3] National Council of Teachers of Mathematics. Curriculum and Evaluation Standards for School Mathematics, NCTM, Reston, VA, 1989.

[4] Steen, L.A., ed. Reshaping College Mathematics, MAA Notes Number 13, Mathematical Association of America, Washington DC, 1989.

[5] Tucker, A. ed. Recommendations for a General Mathematical Sciences Program: A Report of the Committee on the Undergraduate Program in Mathematics, Mathematical Association of America Washington, DC, 1981.

[6] West, R.D. Evaluating the Effects of Changing an Undergraduate Mathematics Core Curriculum which Supports Mathematics-Based Programs, UMI, Ann Arbor, MI, 1996.

1. Learn to use mathematics as a medium of communication that integrates numeric, graphic, and symbolic representations, structures ideas, and facilitates synthesis.

2. Understand the deductive character of mathematics, where a few principles are internalized and most notions are deduced therewith.

3. Learn that curiosity and an experimental disposition are essential, and that universal truths are established through proof.

4. Understand that learning mathematics is an individual responsibility, and that texts and instructors facilitate the process, but that concepts are stable and skills are transient and pertain only to particular applications.

5. Learn that mathematics is useful.

6. Encourage aggressive problem solving skills by providing ample opportunities throughout the core curriculum to solve meaningful practical problems requiring the integration of fundamental ideas encompassing one or more blocks of lessons.

7. Develop the ability to think mathematically through the introduction of the fundamental thought processes of discrete, continuous, and probabilistic mathematics.

8. Develop good scholarly habits promoting student independence and life-long learning ability.

9. Provide an orderly transition from the environment of the high school curriculum to the environment of an upper divisional college classroom.

10. Integrate computer technology throughout the four-semester curriculum.

11. Integrate mathematical modeling throughout the curriculum to access the rich application problems.