Portfolio Assessment of the Major

A Midwestern, comprehensive university which has five different programs in the mathematical sciences — General, Applied Mathematics, Computational Mathematics, Probability and Statistics, and Mathematics Education — requires students to maintain assessment portfolios in courses which are common to all five of the emphases. The portfolios of those who have graduated during the year are examined by a department assessment committee shortly after the close of the spring semester.

Background and Purpose

Evaluating its degree programs is not a new activity for the Department of Mathematical Sciences at Northern Illinois University. The Illinois Board of Higher Education mandates periodic program reviews, while the Illinois Board of Education, NCATE, and the North Central Association also require periodic reviews. The University regularly surveys graduates concerning their employment, their satisfaction with the major program completed, and their view of the impact the University's general education program had on their learning. The Department also polls graduates concerning aspects of the undergraduate degree program. However, in 1992 in response to an upcoming accreditation review by the North Central Association the Department developed a new assessment scheme for its undergraduate major program—one which is longitudinal in character and focuses on how the program's learning goals are being met.

The B.S. in Mathematical Sciences at Northern Illinois University is one of 61 bachelor's degree programs the University offers. As a comprehensive public institution, the University serves nearly 24,000 students of which about 17,000 are undergraduates who come geographically mostly from the top third of the State of Illinois. Located 65 miles straight west of Chicago's loop, Northern Illinois University accepts many transfer students from area community colleges. For admission to NIU as a native student an applicant is expected to have three years of college preparatory high school mathematics/computer science. Annually about 50 students obtain the B.S. in Mathematical Sciences of which about half are native students. Each major completes at least one of five possible emphases: General, Applied Mathematics, Computational Mathematics, Probability and Statistics, and Mathematics Education. Ordinarily, about one-half of the majors are in Mathematics Education, about one-quarter are in Probability and Statistics, and the remaining quarter is divided pretty evenly among the other three emphases.

Regardless of the track a student major may choose, the Department's baccalaureate program seeks to develop at least five capabilities in the student. During the Spring of 1992 our weekly meeting of the College Teaching Seminar (a brown-bag luncheon discussion group) hammered out a formulation of these capabilities which were subsequently adopted by the Department's Undergraduate Studies Committee as the basis for the new assessment program. These capabilities are:

1. To engage effectively and efficiently in problem solving;
2. To reason rigorously in mathematical arguments;
3. To understand and appreciate connections among different areas of mathematics and with other disciplines;
4. To communicate mathematics clearly in ways appropriate to career goals;
5. To think creatively at a level commensurate with career goals.

A discussion of a mechanism which could be used to gauge student progress towards the acquisition of these goals led to the requirement of an Assessment Portfolio for each student. The Portfolio would contain work of the student from various points in the program and would be examined at the time of the student's graduation.

Method

Common to all of the emphases for the major are the lower division courses in the calculus sequence, a linear algebra course, and a computer programming course. At the junior level, each emphasis requires a course in probability and statistics and one in model building in applied mathematics, while at the senior level each student must take an advanced calculus course (which is an introduction to real analysis). Each emphasis requires additional courses involving the construction of rigorous mathematical proofs and at least one sequence of courses at the upper division intended to accomplish depth.

The choices made for material for the assessment portfolio were:

1. the student's final examination from the required lower division linear algebra course (for information related to goals 1 and 2);
2. the best two projects completed by the student in the model building course—the course requires 5 projects for each of which a report must be written up like a technical report (for information related to goals 1, 3, and 4)
3. two homework assignments (graded by the instructor), one from about midsemester and one from late in the semester from the advanced calculus course (for information related to goals 1, 2, 4, and 5)
4. the best two homework assignments from two other advanced courses determined according to the student's emphasis, for example, those in the probability and statistics emphasis are to present assignments from a theoretical statistical inference course and a methods course, while those in the general emphasis present work from the advanced calculus II course and from a second senior-level algebra course (for information related to all five goals);
5. a 250-300 word typed essay discussing the student's experience in the major emphasizing the connections of mathematics with other disciplines (for information related to goals 3 and 4).

Items a)-d) are all collected by instructors in the individual courses involved and placed in files held in a Department Office by office personnel. The student is responsible for the essay in e) and will not be cleared for graduation until the essay is written and collected by the Department's Director of Undergraduate Studies. Thus, all items to be in the portfolio are determined by the faculty, but the student's essay may offer explanation for the portfolio items. The student's graduation is contingent upon having the portfolio essay submitted, but otherwise the portfolio has no "control" in the graduation process. Students are told that the collection of the items for the portfolio is for the purpose of evaluating the degree program.

After the close of the spring semester each year, a committee of senior faculty examines the assessment portfolios of the students who have graduated in the just completed year and is expected to rate each capability as seen in the portfolio in one of three categories—strong, acceptable, and weak. After all portfolios have been examined, the committee may determine strengths and weaknesses as seen across the portfolios and make recommendations for future monitoring of aspects of the program, or for changes in the program, or for changes in the assessment procedures. These are carried to the Department's Undergraduate Studies Committee for consideration and action.

Findings

The implementation of this new assessment scheme has resulted in the third review of portfolios being conducted in May-June 1997. Given the time lag inherent in the collection of the portfolios, the initial review committees have not been able to work with complete portfolios. However, these year-end committees made some useful observations about the program and the process. They discovered:

1. that the linear algebra course appeared to be uneven as it was taught across the department — on the department-wide final examinations, student papers showed concentrated attention in inconsistent ways to certain aspects of the examination, rather than a broad response to the complete examination;
2. that some students who showed good capability in mathematical reasoning on the linear algebra final examination experienced a decline in performance on homework assignments completed in the upper division courses emphasizing rigorous mathematical reasoning;
3. that evaluating capabilities was difficult without having a description of those capabilities in terms of characteristics of student work;
4. that student essays were well written (a pleasant surprise!), but the understanding expressed concerning connections between mathematics and other disciplines was meager;
5. that there appeared to be a positive influence on course-work performance when courses were sequenced in certain orders (e.g., senior-level abstract algebra taken a semester before the advanced calculus).

Use of Findings

In response to the discoveries made by the year-end committees, the Department took several actions.

First, to address the concerns related to the linear algebra course, the coordinators for the course were asked to clarify for instructors that the course was intended not only to enable students to acquire some computational facility with concepts in elementary linear algebra, but also to provide students with a means to proceed with the gradual development of their capacity in rigorous mathematical reasoning. This meant that some faculty needed to teach the course with less concentration on involvement with numerical linear algebra aspects of the course and greater concentration on involvement with students' use of precise language and construction of "baby" proofs. Further, a lower division course on mathematical reasoning was introduced as an elective course. This elective course which enables students to study in mathematical settings logical statements, logical arguments, and strategies of proof can be recommended to those students who show a weak performance in the elementary linear algebra course or who have difficulty with the first upper division course they take which emphasizeds proof construction. It can, of course, also be taken by others who simply wished to take such a course.

A second area of response taken by the Department was to ease the execution of the assessment process by introducing a set of checklists. For each capability a checklist of characteristics of student work which that capability involved were defined. For instance, under problem solving is listed: student understands the problem (with minor errors? gross errors?), approach is reasonable or appropriate, approach is successful, varied strategies and decision making is evident, informal or intuitive knowledge is applied.

Regarding the need for more connections to disciplines outside mathematics, as yet no satisfactory plan has emerged. While students in the applied emphasis must choose an approved minor area of study in the University, there is reluctance to require all majors to have such a minor and a lack of appropriate "single" courses offered in the University which easily convey the mathematical connections. For now the agreed upon strategy is to rely on connections expressed in the current set of required courses (including the model building course) and in programs offered by the Math Club/MAA Student Chapter.

The observation concerning the performance of students when courses were taken in particular sequential patterns was not new to those who were seasoned advisers in the Department. But it did trigger the reaction that new advisers should be especially alerted to this fact.

Success Factors

While the work of the first review committee of faculty was lengthy and cautious because of the newness of the process, the next review committees seemed to move right along with greater knowledge of what to do and how to do it. The portfolio material enables the growth of a student to be traced through the program, and the decision of what to include makes it possible to have complete portfolios for most transfer students as well as for the other students. Further even incomplete portfolios help in the discernment of patterns of student performance. However, it does take considerable time to read through the complete folder for an individual student.

So far the University has been willing to provide extra monetary compensation for the additional faculty time involved in evaluating the portfolios. Should this no longer be the case, an existing Department committee could examine a subset of portfolios each year, or reduce consideration of all the objectives to tracing one objective in one year and a different objective in the next.

The program has not been difficult to administer since most of the collection of materials is done through the individual course instructors. The Department's Director of Undergraduate Studies needs to be sure these faculty are aware of what needs to be gathered in the courses and to see that the appropriate duplication of materials is done for inclusion in the folders. In addition, he/she must collect the student essays.

The process has the added benefit that involvement of many faculty in collecting the materials makes more faculty think about the degree program in its entirety, rather than merely having a focus on the individual course taught. Finally, the mechanism of the portfolios being collected when they are, and evaluated AFTER the student graduates, along with the qualitative nature of the evaluation process, insures that the assessment IS of the program rather than of individual students, faculty, or courses.