 |
Additional Online Case Studies & Appendices | |
|
Assessment of Bachelors’ Degree ProgramsJames R. Fulmer and Thomas
C. McMillan Department of Mathematics
& Statistics University of Arkansas at
Little Rock Abstract. In this case study, we will examine the process that we currently use for assessing the baccalaureate degree programs in the Department of Mathematics and Statistics at the University of Arkansas at Little Rock (UALR). Over the last year, the authors of this study have participated in the MAA workshop Supporting Assessment of Undergraduate Mathematics (SAUM). We will include details of how insights gained at this workshop have been incorporated into our assessment of bachelor’s degree programs. Assessment of Mathematics at UALR. The assessment process at UALR contains several components. Each degree program has an assessment plan, which describes how that program is assessed each year. The assessment cycle covers the calendar year, January 1 through December 31. During this time, various assessment activities are conducted to collect the data prescribed by the assessment plan. In January, each program prepares an Assessment Progress Report, covering the previous year. The report should focus on 1) the use of assessment for program building and improvement, 2) the faculty and stakeholder involvement, and 3) the methods defined by the assessment plan. These reports are evaluated by the College Assessment Committee, using a rating scale of 0 through 4, on the basis of the three items previously listed. The College Assessment Committee compiles a College Summary Report and submits it to the Dean of the college in March. All assessment reports are due in the Provost’s Office by April 1. The chairs of the college assessment committees form the Provost’s Advisory Assessment Group. This committee meets monthly and establishes overall policies and guidance for program assessment on campus. The Department of Mathematics & Statistics at the University of Arkansas at Little Rock has three ongoing assessment programs: core assessment, undergraduate degree assessment, and graduate assessment. This study deals only with the undergraduate assessment program. At the time we entered the SAUM workshop, our department had already designed and implemented an assessment process. Our experience with the process identified its shortcomings and our participation in SAUM gave us insights into how the process could be improved. What has resulted is not so much a new assessment process, but a logical restructuring of the existing process so that more meaningful data can be collected and that data can be interpreted more easily. Since most of the instruments that we use to collect data were in use before we implemented the changes, in this paper we will concentrate on the new logical structure of our assessment process and give only a brief description of the problems and shortcomings that we identified in the earlier assessment program. The main problem with the process used to assess undergraduate degree programs was that the data being collected were only loosely related to departmental goals and student learning objectives. Our department has established a mission statement, goals and student learning objectives. However, the data collected from student portfolios, student presentations, alumni and employer surveys, and the exit examination were not easily interpreted in a way that measured our relative success in achieving these goals and objectives. Another problem we encountered is the low return rate for alumni and employer surveys. Finally, we found that, although we seemed to have a massive amount of assessment data, there would be so few data points relating to a particular student learning objective as to be statistically insignificant. The result of the assessment process was an annual report that beautifully summarized the data we collected, but did not clearly suggest trends. The difficulty in interpreting the data presented an impediment to the successful completion of the most important part of the assessment cycle: using the result of assessment to improve the degree programs. New Directions in Assessment at UALR. Assessment in the Department of Mathematics and Statistics continues to be driven by the goal statement that is published in the university catalog: “The objectives of the department are to prepare students to enter graduate school, to teach at the elementary and secondary levels, to understand and use mathematics in other fields of knowledge with basic mathematical skills for everyday living, and to be employed and to act in a consulting capacity on matters concerning mathematics.” (Emphasis added to identify items in the department’s mission statement that are relevant to baccalaureate degree assessment.) Using insights we gained in SAUM, we have given our assessment process a logical structure that should make interpretation of the data more natural. We have redesigned the logical structure of assessment using a “top-down” approach. The department has identified several student learning objectives that are solid evidence of our students’ meeting the department’s established goals. For each of these student learning objectives, we established “assessment criteria”, which, if satisfied by the students, are strong evidence that the objective has been attained. Finally, for each assessment criterion, we established one or more “assessment methods” for gathering evidence that the students have satisfied the criterion. The top-down approach to assessment that we developed over the year of our participation in SAUM is summarized in Exhibit A. This top-down approach has two significant advantages. First, since each assessment method is explicitly related to a set of assessment criteria, assessment instruments can be designed to collect the best possible data for measuring student achievement on that criterion. Here is an example. One of our assessment criteria is that students should be able to demonstrate at least one relationship between two different branches of mathematics. We have looked for evidence for this criterion in student portfolios, where it may or may not have been found. Under our new scheme, since we anticipate that student portfolios will be used to evaluate this criterion, the process for completing portfolios has been redesigned to guarantee that portfolios contain assignments in which students attempt to demonstrate a relationship between two different branches of mathematics. Students in the differential equations course, for example, can be given a portfolio project that draws on their knowledge of linear algebra. Students in advanced calculus may be asked to draw on their knowledge of geometry or topology. The second advantage of this top-down approach is that sufficient data will be collected relative to each assessment criterion. The assessment process involves the independent evaluation of student work (portfolios, written and oral presentations) by members of the department’s assessment committee. Each committee member is guided in his or her evaluation by a rubric in which each question has been specifically designed to collect data relating to an assessment criterion. The design of all assessment instruments (including surveys, rubrics and interviews) is guided by the assessment criterion they will measure. The explicit connection between assessment method and assessment criterion will facilitate the interpretation of the data. Although it may not be clear in the first few assessment cycles whether the data suggest a modification of the assessment method or an improvement in the degree program, it is evident that convergence to a meaningful assessment program, which provides useful feedback, will not occur if this explicit connection between assessment criterion and assessment method is not made. Mathematics and Statistics faculty are responsible for collecting and interpreting assessment data. The department coordinates its assessment activities with the college and university. The next to the last step in the assessment process at UALR is the preparation of an assessment progress report that is evaluated by our colleagues in the College of Science and Mathematics. The assessment progress report is made available to all interested faculty at the annual College Assessment Poster Session. Every assessed program is represented at this spring event with a poster that summarizes the results included in the report. The critical final step of our new process will be a departmental assessment event at which faculty members give careful consideration to the report prepared by the Departmental Assessment Committee and the evaluation from the College Assessment Committee. This most important step is the “closing of the feedback loop.” All mathematics faculty will examine the assessment data for evidence that suggests appropriate changes to the degree program. Schedule of assessment activities. Collection of data for assessment at UALR covers the calendar year, January through December. Assessment activities cover a four semester cycle: spring, summer, fall, and a follow-up spring semester. The following schedule describes these assessment activities. Early in the spring semester, the department assessment committee identifies about five or six courses as “portfolio courses” for assessment purposes during the calendar year. The instructor of a “portfolio course” is responsible for making assignments for students that will gather information pertaining to the student learning objectives in our assessment plan. The instructor collects these “portfolio assignments” at the end of the semester and places them in the students’ portfolios. Here are some examples of portfolio assignments:
A second assessment activity is Mathematics Senior Seminar/Capstone course in which students enroll during the spring of their senior year. One of the requirements of the course is the ETS-Major Field Test, which is required of all majors in the baccalaureate degree program. We also strongly urge students in the baccalaureate mathematics degree programs to take the ETS-Major Field test during their junior year. Thus, we can accumulate data on how students improve between their junior and senior year with regard to scores on the ETS-MFT mathematics test. On the advice of ETS, we have not established a cut-off or passing score that mathematics majors must make in order to graduate or pass the senior seminar course. We, of course, want our students to give their best efforts on the ETS-MFT. One incentive is a departmental award for the student(s) who score highest on the examination. We also appeal to students sense of citizenship in the department (“Your best effort will help us improve the program and will benefit students who follow you.”) Finally, students are aware that their scores on the MFT are a part of their record within the department and will be one factor in how professors remember them. A third assessment activity is an oral presentation made by each student to peers and mathematics faculty during the Senior Seminar/Capstone course. This presentation is based on a project that the student has developed during the senior seminar course. The oral presentation is to be supported by a written handout report describing its details. The oral presentation and written reports are evaluated by faculty using rubrics that have been designed to collect data for measuring the assessment criteria. A fourth assessment activity during the senior seminar/capstone course for each major is an exit survey, administered near the end of the course. The survey includes both subjective and objective response questions. During the summer semester, the department assessment committee evaluates the portfolios, which now contain the spring portfolio assignments of each mathematics major, using a portfolio rubric that was developed by the department faculty. Instructors of “portfolio courses” that had been designated early in the spring semester, continue to make and collect certain “portfolio assignments” that provide data for measuring the student learning objectives. During the fall semester, instructors of portfolio courses continue making and collecting certain portfolio assignments. A second activity is administering the alumni and employer surveys. Both surveys are sent by mail to each alumnus with the instruction that the alumnus is to pass along the employer survey to his or her employer. Self-addressed, postage-paid envelopes are enclosed in order to facilitate and encourage a response from each alumni and employer. The assessment activities of the fall semester complete the calendar year of collecting data for assessment purposes. During the follow-up spring semester, the department assessment committee begins the process of evaluating assessment data collected during the previous calendar year. The department assessment committee meets and evaluates the latest additions to the portfolios. The committee then writes the assessment progress report, which is due on March 1 of each year. In writing this report, the committee considers the scores on the ETS-MFT test, student portfolios, faculty evaluations of the students’ oral and written reports, exit surveys for majors, alumni surveys, and employer surveys. This data is evaluated with respect to the assessment criteria with the goal of measuring how well the student learning objectives have been met. All of this goes into writing the assessment progress report. A College Assessment Poster Session, where a summary of the assessment progress report is displayed on a poster, is held during March. The assessment progress reports are collected by the College Assessment Committee, consisting of one member from each department in the college. The College Assessment Committee is divided in teams of two each to evaluate the department assessment progress reports. Each team of two is selected so that at least one member served on the committee the previous year and is a continuing member; also, the team is selected so that no member of the team is from the department whose assessment progress report is being evaluated. The team evaluates the assessment progress report with a scoring rubric that is used campus-wide. The department assessment committee then considers the assessment evaluation report and all other assessment data collected during the calendar year and analyzes how well the student learning objectives are being met. At this time, the department makes data-driven decisions concerning possible changes to the mathematics curriculum. This completes the most important part of the assessment cycle, “closing the loop” by using the results of assessment to improve the degree programs. Conclusions. This case study should be considered a preliminary report. The changes to the structure of our assessment program were made during the year of our participation in SAUM. The evaluation of the newly restructured assessment cycle will not be completed until spring, 2003. A preliminary examination of our collected data has given us confidence that our assessment process has been significantly improved. For example, we have now collected faculty reviews of student portfolios. There is now an explicit link, via the inclusion of assessment criteria in the evaluation rubrics, between the data that comes from these evaluations and our learning objectives. The changes in the logical structure of our assessment process were motivated by the shortcomings that we recognized and the very good advice that we got from our colleagues and mentors in SAUM. Acknowledgements. We are both very appreciative of the MAA support that enabled us to participate in SAUM. The great value of our interactions with colleagues at other colleges and universities who face similar problems is difficult to measure. We are especially appreciative of our mentor, Bill Marion, whose suggestions have resulted in significant improvements to our assessment process. We also thank Bernie Madison, Bill Haver, and Bonnie Gold for the many informative sessions that they had with us. Finally, we thank Peter Ewell for his very instructive consultation. Exhibit A
|