A Joint Written Comprehensive Examination to Assess Mathematical Processes and Lifetime Metaskills

A rather unique approach to giving a comprehensive exam to seniors is described in this article by a faculty member at a small, private co-ed college in the Midwest. The exam is taken by seniors in their fall semester and lasts one week. It is a written group exam which is taken by teams of three to five students. Currently, the exam is written and graded by a faculty colleague from outside the college. As part of a college-wide assessment program, the Department of Mathematical Sciences developed a departmental student learning plan, detailing the goals and objectives which students majoring in mathematics or computing should achieve by the time of graduation. For mathematics, there were three major goals. The first goal related to an understanding of fundamental concepts and algorithms and their relationships, applications, and historical development. The second centered on the process of development of new mathematical knowledge through experimentation, conjecture, and proof. The third focused on those skills which are necessary to adapt to new challenges and situations and to continue to learn throughout a lifetime. These skills, vital to mathematicians and non-mathematicians alike, include oral and written communication skills, the ability to work collaboratively, and facility with the use of technology and information resources.

The focus of efforts to assess students' attainment of these goals on a department-wide basis is a college-mandated senior comprehensive practicum, the format of which is left to the discretion of individual departments. In mathematics, the senior comprehensive is a component of the two credit hour senior seminar course, which is taught in the fall (to accommodate mathematics education majors who student teach in the spring). The seminar is designed as a capstone course, with its content slightly flavored by the interests of the faculty member who teaches it, but generally focusing on mathematics history and a revisiting of key concepts from Background and Purpose

Franklin College is a private baccalaureate liberal arts college with approximately 900 students. It is located in Franklin, Indiana, a town of 20,000 situated 20 miles south of Indianapolis. About 40% of our students are first-generation college students, and about 90% come from Indiana, many from small towns.

The Department of Mathematical Sciences offers majors in applied mathematics, "pure" mathematics, mathematics education, computer science, and information systems, and graduates between 8 and 15 students each year. The department has actively pursued innovative teaching strategies to improve student learning and has achieved regional and national recognition for its efforts, including multiple grants from the Lilly Endowment, Inc., and the National Science Foundation, and was named in 1991 by EDUCOM as one of the "101 Success Stories of Information Technology in Higher Education." The department seeks to promote active learning in the classroom through the implementation of cooperative learning and discovery learning techniques and the incorporation of technology. Members of the department have been unanimous in their support of the department's initiatives, and have participated on their own in a variety of programs at the national, regional, and local levels to improve teaching and learning. The department also maintains close ties with local public and private school systems, and has worked with some of those schools to aid in their faculty development efforts.

a variety of courses and relying largely on student presentations. The latter provide some information for departmental assessment of student achievement in some of the areas of the three major goals. Other assessment methods, including departmental exit interviews and assessment efforts in individual classes, provide useful information but lack either the quantitative components or the global emphasis needed for a complete picture of student achievement.

Previously our senior comprehensive had consisted of a one-hour individual oral examination, with three professors asking questions designed to assess the student's mastery of concepts from his/her entire mathematics program. The emphasis of the oral exam was to draw together threads which wound through several different courses. Those professors evaluated the quality of the responses and each assigned a letter grade; those grades were then averaged to produce the student's grade on the examination. Students were given some practice questions beforehand, and the exam usually started with one or more of those questions and then branched into other, often related topics.

When additional money was budgeted for assessment efforts, a decision was made to add the Major Field Achievement Test (MFAT) from Educational Testing Service as an additional component of the senior comprehensive to complement the oral examination. However, that still left two of the three goals almost untouched, and we could find no assessment instruments available that came close to meeting our needs.

So we decided to come up with our own instrument. We wanted to address essentially the second and third goals from above, consisting of the experimentation-conjecture-proof process and the lifetime metaskills. We also wanted to include the modeling process, which is part of the first goal dealing with concepts and their application and development but is not covered in either the oral examination or the MFAT. We did not see any need to address the oral communication aspect of the lifetime metaskills in the third goal, which we felt was sufficiently covered in the oral examination. The result was the joint written comprehensive examination, which we have used since 1993.

Method

The joint written comprehensive examination is given around the fifth or sixth week of the senior seminar class. Students are informed at the beginning of the semester about the purpose and format of each of the components of the senior comprehensive practicum, but no specific preparation is provided for the joint written exam other than the general review included in the senior seminar course. Students in the class are divided into teams of 3-5. The class determines how the teams are to be selected if there are more than five students in the class. They are given a week to complete the test, and the class does not meet that week. Tests consist of four to five open-ended questions, and involve modeling, developing conjectures and writing proofs, and use of library and electronic information resources. Any available mathematical software is allowed. The team distributes the workload in any manner it deems appropriate, and is responsible for submitting one answer to each question.

Since the senior oral examination is evaluated by departmental faculty, we have a colleague from another college or university who is familiar with our environment write and grade the joint written comprehensive exam for a small stipend. The first two years we were fortunate to have a colleague who formerly taught at Franklin and now teaches at a slightly larger university in Indianapolis develop the exam, and her efforts helped smooth out many potential rough spots. The last two years we have employed colleagues at other similar institutions in Indiana, which has provided some variety in the questions and therefore allowed us to obtain more useful assessment information.

The following are a few excerpts from questions from the tests. Each of the four tests given thus far is represented with one question. (If taken in their entirety, the four questions together are a little more than the length of a typical exam.)

1. To facilitate a presentation for the annual Franklin College Math Day, it is necessary to construct a temporary computer communication link from the Computer Center to the Chapel. This link is to be strung by hanging cable from the tops of a series of poles. Given that poles can be no more than z feet apart, the heights of the poles are h feet, cable costs $c per foot and poles cost $p each, determine the minimum cost of the materials needed to complete the project. (Be sure to include a diagram and state all the assumptions that you make.)
2. Find and prove a simple formula for the sum

   

3. Geographers and navigators use the latitude-longitude system to measure locations on the globe. Suppose that the earth is a sphere with center at the origin, that the positive z-axis passes through the North Pole, and that the positive x-axis passes through the prime meridian. By convention, longitudes are specified in degrees east or west of the prime meridian and latitudes in degrees north or south of the equator. Assume that the earth has a radius of 3960 miles. Here are some locations:

   St. Paul, Minnesota (longitude 93.1° W, latitude 45° N)
   Turin, Italy (longitude 7.4° E, latitude 45° N)
   Cape Town, South Africa (longitude 18.4° E, 33.9° S)
   Franklin College (longitude 86.1° W, latitude 39.4° N)
   Calcutta, India (longitude 88.2° E, latitude 22.3° N)
   ...

   (b) Find the great-circle distance to the nearest mile from St. Paul, Minnesota, to Turin, Italy.

   (c) What is the distance along the 45° parallel between St. Paul and Turin?

   ...

   (f) Research the "traveling salesman problem." Write a well-prepared summary of this problem. In your writing indicate how such a problem might be modeled in order to find a solution.

   (g) Starting and ending at Franklin College indicate a solution to the traveling salesman problem which minimizes the distance traveled. (Assume there is a spaceship which can fly directly between cities.)

4. One of the major ideas of all mathematics study is the concept of an Abstract Mathematical System (AMS for short). Consider the AMS which is defined by:

   A nonempty set S = {a, b, c, …} with a binary operation * on the elements of S satisfying the following three assumptions:

   A1: If a is in S and b is in S then (a*b) is in S.

   A2: If a*b = c then b*c = a.

   A3: There exists a special element e in S such that a*e = a for each a in S.

   Which of the following are theorems in the above abstract mathematical system? Prove (justify) your answers.

   (a) e is a left identity, that is, e*a = a for all a in S.

   (b) a*a = e for all a in S.

   ...

   (d) * is a commutative operation.

   ...

Findings

For individual students, results from each of the components of the senior comprehensive practicum (the oral examination, the joint written examination, and the MFAT) are converted into a grade. These grades are assigned a numerical value and averaged on a weighted basis for a grade on the entire practicum, which appears on the transcript (with no impact on the GPA) and also is part of the grade for the senior seminar course. The oral examination is weighted slightly more than the other two due to its breadth and depth, and has remained unchanged since the tests complement each other so well. For the department, the results of all three components are evaluated to determine whether any modifications to departmental requirements or individual courses are indicated.

In analyzing the results of the joint written comprehensive exam, what has struck us first and most strongly has been the need for more focus on modeling. Students have had difficulty in approaching the problems as well as in communicating their assumptions and solutions. We have also noted room for improvement in the use of information resources. Real strengths have been the use of technology, particularly in the experimentation-conjecture-proof process, and working collaboratively, although the latter seems stronger in the development of solutions than in putting results in writing.

The remainder of the senior comprehensive, the oral exam and the MFAT, have confirmed the impressions of our students which faculty members have developed by observing and interacting with them over four years, albeit with an occasional surprise. The difference in the tests' formats has allowed almost all students to showcase their abilities and accomplishments. In addition, our students have compiled a strong record of success both globally and individually on the MFAT, with consistently high departmental averages and very few students falling below the national median.

Use of Findings

Two major changes have resulted from these findings. First, a modeling requirement (either a course in simulation and modeling or a course in operations research, both part of our computing curriculum) has been added to the related field requirement for mathematics education and "pure" mathematics majors, solely based on the outcomes of the joint written comprehensive exam. We have also tried to be more conscious of the process of developing and analyzing models in the applied mathematics curriculum. Second, as a result of trying to answer the question of how to develop the competencies we are testing in the senior comprehensive exams, our department has also moved beyond curriculum reform of single courses to programmatic reform, in which we develop goals and objectives for individual courses and course sequences under the framework of the departmental goals and objectives. This has led to the identification of developmental strands, in which each objective (such as the development of oral communication skills) is specifically addressed in three or four courses with an emphasis on building on previous accomplishments rather than each course standing alone. These strands begin with the freshman calculus courses and continue through the entire four-year program. Sophomore-level courses in multivariable calculus and linear algebra have been particularly focused as a result, whereas in the past we would fluctuate in which goals were emphasized and to what degree.

A more subtle effect has been in faculty's awareness of what goes on in other courses. Not only are we more in touch with what content students have seen before they arrive in our classes, we also know (or can determine) how much exposure they have had to applications, written and oral communication skills, and other components of our goals and objectives. Although our department has had a long- standing tradition of effective working relationships and good communications, we have been surprised at how helpful this process has been.

The influences of the other two components of the senior comprehensive have been less pronounced. The oral exam has emphasized to faculty the importance of tying concepts together from course to course. The success of our students on the MFAT has been a helpful tool in recruiting and in the college's publicity efforts as well as in demonstrating the quality of the department's educational efforts to the college administration.

Success Factors

The joint written comprehensive examination fits well into Franklin's liberal arts philosophy since students are accustomed to being asked to draw together threads from a variety of courses and topics, both within and across disciplines. The fact that our department unanimously endorses the importance of all three components of the senior comprehensive results in an emphasis on similar themes in a variety of courses and encourages students to take the exams seriously. The willingness of our test authors to take on such a unique challenge has played a major role in the progress we've achieved. We have talked about the possibility of arranging for a consortium of similar colleges and universities to work together on a common joint written examination, with the result that the workload could be spread around and additional data and ideas generated.

Probably the biggest practical drawbacks to the implementation of a joint written comprehensive examination in the manner we have are logistical, including funding to pay the external reviewer and identifying willing colleagues. It would also be difficult for those departments which do not have a specific course or time frame in which such an instrument can be administered, since the investment of student time is quite large. It is also vital to get departmental consensus about what students should be able to do and how to use the information acquired, although that will probably be less of an issue as accrediting agencies move strongly toward requiring assessment plans to achieve accreditation.