Undergraduate Mathematics Program Assessment – A Case Study

Introduction

            American University is a private liberal arts college located in Washington, D. C.  It has approximately 10,000 students, 60% of whom are undergraduates.  The Department of Mathematics and Statistics has 15 full-time faculty and 8 part-time instructors who support the university’s mathematics competency requirement.  Their programs include a bachelors and masters in both mathematics and statistics.  The department currently has 15 undergraduate majors, mostly in the mathematics program.  Recent efforts to revitalize the major are showing promise in increasing interest in the programs.  The department hopes that focusing on assessment of learning outcomes will contribute to their ability to improve the programs and expand their numbers.

Background

            In summers of 2001 and 2002, a small group of AU faculty and staff were sent to the American Association of Higher Education’s Assessment conference to gather information on implementing an assessment process as a routine part of the university’s annual review.  When the group returned and reported their findings to the Provost, they decided to start a process of assessment by focusing on undergraduate major programs.  This effort was initiated in the fall of 2001 by charging departments with writing their program goals and learning outcomes using an internally designed learning goals and objectives format.   The Department of Mathematics and Statistics received the form (See Figure 1) and proceeded to have discussions on how to complete the form for its undergraduate programs in Mathematics, Applied Mathematics, and Statistics.  By Spring of 2002, the department submitted their Learning Goals for the B. S. in Mathematics.

Learning Goal I:

Students are expected to acquire ability and skills in Calculus of one and several variables, vector analysis, basic linear algebra and elements of the theory of vector spaces.  Students are to develop appreciation for mathematical reasoning and acquire skills in logical deduction.  Ability to formulate definitions, to apply the methods of direct proof and indirect proof to solve problems is expected on a basic level.  Emphasis is given to developing ability to communicate effectively in explaining the overall processes and the particular steps in the solving of a mathematical problem.

Learning Goal II:

The mathematics major is expected to develop a fundamental understanding of several major realms of mathematics.  Students are expected to understand the different methods in real analysis and modern algebra, and be able to apply the methods in a rigorous manner.  Further understanding of the span of mathematics is expected in the curriculum of a major.  Students should be able to demonstrate understanding of the basic methods of inquiry in at least three specific areas in mathematics. Areas of expertise of faculty members include history of mathematics, mathematical logic, set theory, complex analysis, differential equations, geometry, number theory, topology, harmonic analysis, numerical analysis, probability and statistics.  Ability to communicate mathematical ideas clearly and logically is given continued emphasis.

The scope and depth of the program provide students with the ability to continue their study in a graduate program, or to teach in classrooms, or to enter the industrial world.

Objective/Outcome 1: 

Ability and skills in Calculus of one and several variables, vector analysis, basic linear algebra and elements of the theory of vector spaces.  Appreciation for mathematical reasoning and acquire skills in logical deduction.  Ability to formulate definitions, to apply the methods of direct proof and indirect proof to solve problems is expected on a basic level.

Ability to communicate effectively in explaining the overall processes and the particular steps in the solving of a mathematical problem.

Methods of Assessment:
               Evaluation of student’s grades in the following courses:
               1.  MATH-221 (Calculus I)
               2.  MATH-222 (Calculus II)
               3.  MATH-223 (Calculus III)
               4.  MATH-310 (Linear Algebra)
               5.  MATH-322 (Advanced Calculus)
Standards for Success:
               Each student receives a grade of C or better in each of these five course

Objective/Outcome 2:
          Understanding of the different methods in real analysis and modern algebra, and be able to apply the methods in a rigorous manner.

Methods of Assessment 1:
               Evaluation of student’s grade in the following courses:
               MATH-512 (Intro to Modern Algebra I)
               MATH-513 (Intro to Modern Algebra II)
               MATH-520 (Intro to Analysis I)
               MATH-521 (Intro to Analysis II)
Standards for Success:
     Each student receives a grade of C or better in each of these four courses.
Methods of Assessment 2:
    Supplemental questions specifically addressing this objective on student Course Evaluations for MATH-513 and MATH-521.
Standards for Success:
     Majority of students “strongly agree”or “agree” to each of the supplemental questions.

(See Figure 2 for Institutional Form Version)

Unfortunately, there was very little information on how to complete the forms and while the department considered very carefully what it valued and taught, their submission of learning outcomes reflected the traditional gauge of assessing programs through grades in courses, teaching evaluations, and vague assessments of outcomes.  Departments across the university completed the forms in a similar fashion and then turned their attention to pressing issues raised by the President’s Fifteen Points Plan which included shifting the university’s focus from graduate to undergraduate education, restructuring the faculty senate, and reducing the overall size of the university.  Thus, the Mathematics and Statistics faculty were immersed in an intensive Ph.D. program review that would ultimately lead to the elimination of their doctoral programs. 

By spring 2003, the Ph.D. program review was completed and the new streamlined faculty senate was formed.  A substantial portion of the Middle States self-study was underway, which led to the creation of a faculty and staff Project Team on Learning Outcomes and Assessment charged with examining the progress of the earlier learning outcomes submissions.  Upon reviewing the assessment plans, the team found that additional action was needed to help departments generate assessment cycles that truly informed programmatic change.  The Project Team issued recommendations for improving assessment plans that included encouraging departments to write learning objectives that focus on results instead of process and to identify three to six learning outcomes that are critical to the program and are observable and explicit.  The Project Team also recommended listing learning objectives separately, especially if they would require separate and different assessment strategies.  The Team urged departments to visit their professional web sites and go to any discipline-specific assessment conferences.

The Department’s Course of Action

After attending the Project Team’s workshops and reading their materials, the department decided to send a team to the Mathematical Association of America’s Supporting Assessment in Undergraduate Mathematics (SAUM) series of three workshops over a three-year period.  During the initial workshop in Highpoint, North Carolina, the team drafted a sample revised program to present to the department.  The goal of the team was to present the department with a concrete example of a learning outcome and possible ways to assess the outcome.

Program Goals:

Our goals include teaching the essential skills of mathematical literacy and proficiency. Literacy and proficiency in mathematics include not only the ability to comprehend mathematical reasoning but also the ability to express oneself mathematically: to formulate an argument as well as follow it. Our students will be able to understand and apply mathematics as a model for finding solutions to real-life problems.

Objective/Outcome 1: 
           Student will be able to apply the methods of direct proof and indirect proof to solve problems. 

Methods of Assessment:

• Final examinations from Advanced Calculus, Linear Algebra, Analysis I, Analysis II, Modern Algebra I and Modern Algebra II will each have a department selected problem that requires skill in use of direct or indirect proofs.
• Give an exit exam that contains solving problems using direct and indirect proofs. (For assessment of program)
• Exit interview to include question on how to solve a problem using a direct proof or indirect proof.
• Folder on each major’s finals
• Portfolio

Standard for Success:
Faculty will develop a rubric that describes the expected characteristics of solving a problem using direct or indirect proof. 

Objective/Outcome 2: 
          Students will be able to develop a mathematical model from a real life application.

Methods of Assessment:
Give an exit exam that contains at least one posed application problem.
Standard for Success:
 Faculty will develop a rubric that describes the expected characteristics of modeling.

(See Figure 3 for Institutional Form Version)

            The team members purposefully included multiple forms of assessment to serve as a springboard for discussion when they presented the revised goal to the department.  When they met with the department, they compared the original goals of the department with the revised simplified (illustrative) goal.  The team suggested that the department build their learning objectives from a program goal which the team drafted from the department’s mission statement.  The discussion was spirited and sometimes negative.  Faculty were not convinced that this process was useful and they were skeptical that there would be meaningful results.  Several faculty members opined that information for improving the programs could be obtained more simply by reflecting on their students’ performances and discussing their observations about students among one another.  One faculty member was insistent that since our program was so small, we knew our students in terms of what their strengths and weaknesses were.   

Members of the team countered that while informal discussions are helpful, they do not pinpoint precisely the skills that are lacking (or are prevalent) in our students.  The team emphasized the importance of instituting a process that produces observable evidence of student learning which in turn informs program improvement.  The team suggested that if the department reviewed students’ attempts at proofs, that there would be variability in the degree of rigor and in the soundness of presentation.  Setting up a situation that would allow the department to observe students’ work would be relatively straight-forward as long as there was a specific targeted outcome, such as appropriate use of notation and terminology in the conduct of a proof.  The team convinced faculty that assessment does not have to be drawn out or complicated and – in fact – can be conducted with processes that already exist.  In short, what do we have to lose by taking a focused look at our students’ behaviors when asked to perform a task that reflects skills gained through pursuing our program? 

A turning point in the discussion came when one of the team members asked the mathematics faculty, “When our students graduate, what activity that is observable do you think our majors should be able to do?”   A faculty member responded that he would like students to be able to pick up a mathematics book and teach themselves some new mathematics using the skills they have gained in the program.  Other faculty members agreed and then the discussion shifted to developing a rubric that would describe the level of performance expected from a mathematics major.  Ultimately, the department assigned a mathematician and a statistician to write the first learning outcome and assessment for their respective programs.  The new outcomes are given below:

Mathematics, B. S. Program Goals:

Our goals include teaching the essential skills of mathematical literacy and proficiency. Literacy and proficiency in mathematics include not only the ability to comprehend mathematical reasoning but also the ability to express oneself mathematically: to formulate an argument as well as follow it. Our students will be able to understand and apply mathematics as a model for finding solutions to real-life problems.

Mathematics Objective/Outcome 1: 

Student will be able to orally explain a concept in mathematics from an advanced level mathematics text (the concept should be one that immediately or closely follows the last concept discussed in one of the last courses in their program).
Methods of Assessment:
Seniors will give an oral/chalkboard presentation to three faculty members.
Standards for Success:
Faculty will develop a rubric that describes the expected characteristics of self-teaching a concept including proper use of notation and procedures for interpreting and explaining a mathematical concept.

Statistics, B. S. Program Goals:

Our goals include teaching the essential skills of statistical literacy and proficiency. Literacy and proficiency in statistics include not only the ability to comprehend statistical reasoning but also the ability to use and interpret data effectively. Our students will be able to understand and apply statistics as a model for finding solutions to real-life problems.

Statistics Objective/Outcome 1: 
The student will be able to summarize and describe data, conduct graphical analyses, carry out basic formal statistical procedures and effectively write up the analysis.

Methods of Assessment:
In each of the statistical methods courses (STT515, STT516, STT521, STT522, STT424) professors will assign at least one project in which students use the methods learned in that class to explore and analyze a complex data set and write up the analysis.

Standards for Success:
At the end of each academic year a team of three statistics faculty will evaluate the project according to an agreed upon rubric.

Professors will keep a copy of the projects of the Statistics majors for the purposes of assessing the outcome stated above.

Evaluation of the projects:  At the end of each academic year a team of three statistics faculty will evaluate the project according to an agreed upon rubric.

Rubric: 

1. Concise but clear description of problem.
2. Description of method used for analysis, including a discussion of advantages, disadvantages and necessary assumptions.
   
3. Discussion of results.
4. Conclusion including a discussion of limitations of analysis.

Standards:

1. Advanced:  Easy to read, concise correct with all important pieces of information included.  Appropriate use, display and description of graphs.
   
2. Proficient:  Correct statements with all important pieces of information included.  Appropriate use of graphs.
   
3. Basic:  Correct statements, but some important aspects of the problem omitted. 
   
4. Unacceptable:  Incorrect statements.  Inappropriate use of graphs.  Unintelligible sentences.

Expected Standard for the program:  A majority of students will score at least proficient on all four pieces of the rubric.

(See Figure 4 for Institutional Form Version)

What’s Next

            The faculty plan to meet to develop at least two or three more learning outcomes to be included in an on-going assessment cycle.  Rubrics need to be constructed for the mathematics self-teaching experience.  The assessments for each of the above learning outcomes will begin fall semester 2004.  The results will be analyzed in May of 2005.  At that time, faculty will review the report on the analyses and then make decisions on what programmatic changes (if any) need to be made to respond to the results.  In the meantime, the cycle for assessing other outcomes will begin in fall semester 2005 along with reassessing the first set of outcomes.

Bibliography

(2004). Undergraduate programs and courses in the mathematical sciences : CUPM curriculum guide 2004. Washington, DC, Mathematical Association of America.

Middle States Association of Colleges and Schools. Commission on Higher Education. (2002). Characteristics of excellence in higher education : eligibility requirements and standards for accreditation. Philadelphia, Pa. (3624 Market St., Philadelphia 19104), Commission on Higher Education, Middle States Association of Colleges and Schools.

 Figure 1

Figure 2

The 2001 Submission of Learning Outcomes for the B. S. in Mathematics