and (2) Find the maximum and minimum values of
y = sinxcosx on [0, p]
| Jack Bookman | Charles P. Friedman |
| Duke University | University of Pittsburgh |
This in-depth study analyzes a Calculus reform program. It looks not only at analytical gains in student understanding but affective gains as well.
Background and Purpose
As part of the National Science Foundation's Calculus Initiative, Lawrence Moore and David Smith developed a new calculus course at Duke University. The course, called Project CALC, differs from the traditional Calculus course in several fundamental ways. The traditional course emphasizes acquisition of computational skills whereas the key features of Project CALC are real-world problems, activities and explorations, writing, teamwork, and use of technology. The explicitly stated goals for Project CALC are that students should
Project CALC classes meet for three 50-minute periods with an additional two-hour lab each week. Classroom projects are worked in teams. For approximately five of the labs or classroom projects each semester, teams of two students submit a written report. In the traditional Calculus class taught at Duke, classes (maximum size of 35) meet three times per week for 50 minutes. The lectures usually closely follow a textbook like Thomas and Finney's Calculus and Analytic Geometry. Though there is, naturally, some overlap, the two courses do not have the same mathematical content.
Method
Paralleling Moore and Smith's development of the course, the authors of this paper designed and implemented an evaluation of Project CALC. The evaluation had two phases. During the first years of the project, the emphasis was on formative evaluation, during which the evaluator
During the later years, the emphasis changed to comparing traditionally taught students with experimentally taught students on a set of outcomes. The outcome based phase had three main components
Findings
Formative Evaluation. During the 90-91 and 91-92 academic years, PC was taught to about one-third of the students enrolled in beginning calculus. The primary focus of the evaluation during the second year of the project's development was on formative evaluation and on observing and describing qualitatively the differences between PC and TR. By the end of the first semester certain strengths and problems became apparent [1, 5, 6]. Students were upset and unhappy about Lab Calculus I. They complained about the unfamiliar course content, pedagogy and style. They felt that the course required an unreasonable amount of time outside of class and they had difficulty learning how to read mathematics. The student response to traditional Calculus I was better but not glowing. The TR students felt that the course was too hard and went too fast. They also felt that the course was not useful and presented "too much theory."
The attitudes of the students in Project CALC improved remarkably from the fall to the spring. In the spring semester, though they still complained that the material was too vague and complicated, they also stated that they understood the material rather than just having memorized it and that it was interesting to see the connections to the real world. In the second semester, the responses of the TR students were much harsher. Their most positive comments were that the course "forces students to learn lots of material" and the "basics of calculus were taught." One difference was remarkably clear from the classroom observations: The level of attention and concentration of the PC students was much higher. In every TR class observed, at least some of the students fell asleep and most of them started packing their books before the lecture was finished. These behaviors were not observed in PC classes. In fact, often the class ran five minutes overtime without the students even noticing. In summary, PC students were engaged and relatively interested in their work. The TR students were largely passive in class and alienated. On the other hand, TR students knew what to expect and what was expected of them, whereas the PC students probably expended a lot of energy figuring out what the course was about.
Problem Solving Test. In year two of the study, a five-question test of problem solving was administered to Calculus II students (both PC and TR). An effort was made to make the testing and grading conditions for the students as uniform as possible. The test consisted of five items novel to both groups of students and that reflected topics that were taught in both courses. The items were selected with contexts from various fields: biology, chemistry, economics as well as mathematics. On both versions of the test, PC students outperformed the TR students, with highly statistically significant differences, indicating that Project CALC made some progress towards meeting its goals. A more detailed discussion of both the method and the findings can be found in [4].
Retention Study. Also during the third year of the project, in the spring of 1992, the five-part test developed during the first year was administered to a group of sophomores and juniors half of whom had a year of PC in their freshman year and half of whom had a year of TR in their freshman year. Approximately one third of randomly selected students agreed to participate in the study. The five sections of the test addressed writing, basic skills, problem-solving, concepts, and attitudes. The test items are briefly described below.
Attitudes. Each student was asked to indicate the extent to which he or she agreed or disagreed with 40 items such as "I've applied what I've learned in calculus to my work in non-mathematics courses" and "When I was in calculus, I could do some of the problems, but I had no clue as to why I was doing them."
Skills. Ten items such as: (1) Compute
and (2) Find the maximum and minimum values of
y = sinxcosx on [0, p]
Writing. A short essay addressing the question, "Illustrate the concept of derivative by describing the difference between the average velocity and instantaneous velocity."
Problem solving. Two non-routine word problems.
Conceptual understanding. Ten items such as, "The graphs of three functions appear in the figure below [omitted here]. Identify which is f, which is f´, and which is f´´."
The results showed a highly statistically significant difference in attitude favoring PC students. This was somewhat surprising in light of the fact that the same students who had often complained, sometimes bitterly, about PC exhibited, a year and two years later, much more positive attitudes about calculus and its usefulness than their traditionally taught peers. A more complete discussion of attitudes is given in [5]. Not surprisingly, the regular students outperformed PC students on the skills calculations, though not statistically significantly. This finding that TR students were better at these kind of skills was verified in much of the interview data collected during the five year study. PC students out-performed TR students on the writing test but not statistically significantly. The results of the problem solving were somewhat consistent with the problem solving test described above (see [3]). The scores on the conceptual understanding test were poor for both groups. It is highly possible that fatigue could have been a factor in their poor performance, since, during the previous two hours, the subjects had taken four other subtests. In retrospect, the subjects were probably asked to do too much in one sitting. The most useful information was gotten from the attitude and skills subtests. The results of the attitude test were powerful and dramatic. The results of the skills test, though not quite statistically significant were indicative of a weakness in the experimental course already perceived by students and faculty.
Follow-up Study. The evaluation in the fourth and fifth years of the study focused on the question, do experimental students do better in and take more subsequent courses that require calculus? The grades of two groups of students were examined. in a set of math related courses. The detailed results of this study are available elsewhere ([3, 6]). To better understand these results seven pairs of these students — matched by major and SAT scores, with one subject from the experimental course and one from the traditional course — were interviewed.
It was found that on average, the TR students did better by 0.2 out of a 4-point GPA. This is a small but statistically significant difference. It seems that though there were few differences between the grades of the two groups in the most mathematical courses, the significant differences overall may be explained partially by differences in the performance in other classes. In particular, there were significant differences in the grades of TR and PC students in Introductory Biology with TR students outperforming others. On average, PC students took about one more math, physics, engineering, and computer science course than the TR students and PC students took significantly more math courses. It is not clear why we got these results. One possible explanation is that during the first two weeks of classes, when students were allowed to add and drop courses, the most grade oriented and risk aversive students, who are often pre-medical students, may have switched from the PC to TR course. Though the students who were the subjects of this study were randomly assigned to PC and TR classes, it was not possible to force them to remain in their assigned sections.
In the interview data, most of the students felt that they were adequately prepared for future courses. In general, the PC students were less confident in their pencil and paper computational skills, but more confident that they understood how calculus and mathematics were used to solve real world problems [5].
Project CALC has made some modest headway in keeping more students in the mathematical "pipeline," that is increasing continuation rates in mathematics and science courses. As mentioned above, PC students took more of the most mathematically related courses. In addition, retention rates from Calculus I to Calculus II have improved slightly.
Use of Findings
The evidence, from both the qualitative and quantitative studies, indicates several strengths and weaknesses of the project. The observations and interviews, as well as the problem solving tests and attitude instrument, indicate that the students in PC are learning more about how math is used than their counterparts in the TR course. On the other hand, as reported in interviews, students and faculty in the early version of the PC course did not feel that enough emphasis was placed on computational skill, whereas this has been the main emphasis of the TR course. This view is supported by the results of the skills test. Although, there is a danger in overemphasizing computational skills at the expense of other things, PC students should leave the course with better abilities in this area. The course has been revised to address these problems and in more recent versions of the course, there is considerably more practice with basic skills such as differentiation, integration and solving differential equations. (See [2].)
Students in PC became better problem solvers in the sense that they were better able to formulate mathematical interpretations of verbal problems and solve and interpret the results of some verbal problems that required calculus in their solution. Although PC appears to violate students' deeply held beliefs about what mathematics is and asks them to give up or adapt their coping strategies for dealing with mathematics courses, their attitudes gradually change. When surveyed one and two years after the course, PC students felt, significantly more so than the TR students, that they better understood how math was used and that they had been required to understand math rather than memorize formulas. Observations of students in class showed that PC students are much more actively engaged than were TR students. The evidence gathered indicates some improvements in continuation rates from Calculus I to Calculus II and from there into more advanced mathematics classes.
Success Factors
The heart of program evaluation is assessment of student learning, but they are not the same thing. The purpose of program evaluation is to judge the worth of a program and, in education, that requires that assessment of student learning take place. But program evaluation may include other things as well, for example, problems associated with implementation, costs of the program, communication with interested parties. Assessment of student learning is a means to achieve program evaluation, but assessment of student learning is also a means of understanding how students learn and a means of evaluating the performance of individual students [8].
Educational research in a controlled laboratory setting is critical for the development of our understanding of how students learn which in turn is critical for learning how to assess student learning. Program evaluation must rely on this research base and may use some of the same methods but it must take place in a messier environment. In this evaluation for example, while it was possible to randomly assign students to classes, it was not possible to keep them all there. The calculus program at a university is a much more complex setting than would be ideal for conducting basic educational research. On the other hand, program evaluation can contribute to an understanding of what actually happens in real, complex learning situations.
This study used both qualitative and quantitative methods. For example, the interview data collected during the formative evaluation pointed out concerns about student's level of computational skills. The quantitative data collected the next year corroborated this. In addition, the interviews conducted with students two years after they took the course, helped provide insight into the quantitative attitude instrument given in the previous year. Traditionally, these two methodologies were seen as diametrically opposed but this attitude has changed as more evaluators are combining these approaches [7].
There is not always a clear distinction between formative and summative evaluation though Robert Stake summed up the difference nicely when he stated that: "When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative." [9] The summative aspects of this evaluation were used to inform changes in the course and became, therefore, a formative evaluation as well [2].
Inside evaluators (evaluators who work in the program being evaluated) are subject to bias and influence. On the other hand, outside evaluators are less familiar with the complexities of the situation and can rarely spend as much time on an evaluation as an insider can. To overcome these problems, this evaluation used both an inside and outside evaluator. The insider conducted most of the evaluation; the outsider served as both a consultant and auditor, whose job, like that of certified public accountants, was to periodically inspect the "books" to certify that the evaluation was being carefully, fairly and accurately executed.
Finally we offer some advice for those readers who are planning on conducting program evaluations at their institutions. It is both necessary and difficult to communicate the value and difficulties of program evaluation. Beware of administrators who say: "We're going to apply for a continuation of the grant, so we need to do an evaluation." Be prepared for results that are unexpected or that you don't like. Always look for sources of bias and alternative explanations. Develop uniform, replicable grading schemes and where possible use multiple graders. Be clear and honest about limitations of the study, without being paralyzed by the limitations. Get an outsider to review your plans and results.
References
[1] Bookman, J. Report for the Math Department Project CALC Advisory Committee — A Description of Project CALC 1990-91, unpublished manuscript, 1991.
[2] Bookman, J. and Blake, L.D. Seven Years of Project CALC at Duke University — Approaching a Steady State?, PRIMUS, September 1996, pp. 221-234.
[3] Bookman, J. and Friedman, C.P. Final report: Evaluation of Project CALC 1989-1993, unpublished manuscript, 1994.
[4] Bookman, J. and Friedman, C.P. "A comparison of the problem solving performance of students in lab based and traditional calculus" in Dubinsky, E., Schoenfeld, A.H., Kaput, J., eds., Research in Collegiate Mathematics Education I, American Mathematical Society, Providence, RI, 1994, pp. 101-116.
[5] Bookman, J. and Friedman, C.P. Student Attitudes and Calculus Reform, submitted for publication to School Science and Mathematics.
[6] Bookman, J. and Friedman, C.P. The Evaluation of Project CALC at Duke University — Summary of Results, unpublished manuscript, 1997.
[7] Herman, J.L., Morris, L.L., Fitz-Gibbon, C.T. Evaluators Handbook, Sage Publications, Newbury Park, CA. 1987.
[8] Schoenfeld, A.H. et al. Student Assessment in Calculus. MAA Notes Number 43, Mathematical Association of America, Washington, DC, 1997.
[9] Stevens, F. et al. User-Friendly Handbook for Project Evaluation, National Science Foundation, Washington D.C., 1996.
[10] Smith, D.A. and Moore, L.C. "Project CALC," in Tucker, T.W., ed., Priming the Calculus Pump: Innovations and Resources, MAA Notes Number 17, Mathematical Association of America, Washington, DC, 1990, pp. 51-74.
[11] Smith, D.A. and Moore, L.C. (1991). "Project CALC: An integrated laboratory course," in Leinbach, L.C., et al, eds., The Laboratory Approach to Teaching Calculus, MAA Notes Number 20, Mathematical Association of America, Washington, DC, 1991, pp. 81-92.
[12] Smith, D.A. and Moore, L.C.(1992). "Project CALC:
Calculus as a laboratory course," in Computer
Assisted Learning, Lecture Notes in Computer Science 602,
pp. 16-20.
 |
||
 |
 |
 |
 |