 |
Additional Online Case Studies & Appendices | |
The Development,
Implementation and Revision
of a Departmental Assessment
Plan
AbstractIn the late 1990s, the Colorado School of Mines began to prepare for a visit by the Accreditation Board for Engineering and Technology. As a support department to eight accredited engineering departments, the Mathematical and Computer Sciences Department had a responsibility to assist in the accreditation process. The approaching visit motivated the creation of a departmental assessment plan. This paper traces the development and implementation of the original departmental assessment plan and the events that stimulated the revision of that plan. An emphasis will be placed throughout this paper upon what the Mathematical and Computer Sciences department has learned. I. BackgroundIn the late 1990s, the Colorado School of Mines (CSM) began to prepare for a visit by the Accreditation Board for Engineering and Technology (ABET) that was to take place in the academic year 2000-2001. Eight CSM engineering departments were to be reviewed using ABET's newly revised engineering criteria [1]. A major difference between the old and the new criteria was that the new criteria required that accredited departments directly demonstrate what students know and can do. In other words, the new criteria emphasized the direct assessment of student outcomes. All of the engineering departments and the departments that support core-engineering courses worked to develop and implement an assessment system that measured student outcomes. As a department that provides many of the core engineering courses, the Mathematical and Computer Sciences Department (MCS) began to develop an assessment plan in the fall of 1997. The first step in this process was to establish departmental goals and objectives. This was completed by the end of the 1997-98 academic year. The next two years were dedicated to developing and implementing a departmental assessment plan that would support the student attainment of the departmental goals and objectives. Much of the original assessment plan focused upon how the information that was currently being collected could be better used for assessment purposes. A number of survey instruments were also introduced with the purpose of acquiring data in a fast and efficient manner. In summary, much of the early work with respect to assessment process was focused upon ensuring a successful ABET visit. The ABET visit in 2000-2001 resulted in all eight accredited departments receiving full accreditation. The success of this visit immediately reduced the administrative pressure to produce quick assessment results. The MCS department now had the time and opportunity to review and revise the department's assessment plan. The review process began in the fall of 2001. By the end of the academic year, the department's Undergraduate Curriculum Committee had approved a revised set of goals and objectives and a revised departmental assessment plan. The MCS department is currently in the process of implementing the revised plan. This paper traces the development and implementation of the original MCS departmental assessment plan and the events that stimulated the need to revise that plan. An emphasis will be placed throughout this paper upon what the MCS department has learned. II. Conceptual FrameworkAccording to the Mathematics Association of America's (MAA) Committee on the Undergraduate Program in Mathematics (CUPM) in collaboration with the MAA's Assessment Subcommittee, assessment is a cycle that consists of the following five phases [2]: 1) articulating goals and objectives, 2) developing strategies for reaching goals and objectives, 3) selecting instruments to evaluate the attainment of goals and objectives, 4) gathering, analyzing and interpreting data to determine the extent to which goals and objectives have been reached, and 5) using the results of assessment for program improvement. When the final phase is reached, the assessment cycle begins again. This conceptualization of the assessment process is consistent with other literature on assessment [3, 4, 5]. The phases within this cycle provide a framework for developing a departmental assessment plan. Each phase can be moved through sequentially, supporting the emergence of a departmental assessment plan. Once the initial cycle has been completed, the knowledge and information that has been gained through the implementation process can be used to improve the assessment plan prior to the next cycle. III. Development and Revision of Goals and ObjectivesAs was discussed earlier, the first step in the development of a departmental assessment plan is the establishment of goals and objectives. "Goals" are broad statements of expected student outcomes and "objectives" divide a goal into circumstances that suggest whether a given goal has been reached [6]. Careful thought should be given to the University Mission Statement [2, 7] and to the requirements of any appropriate accreditation board when developing departmental goals and objectives [7]. Attention should also be given to faculty buy-in. Faculty will not work to assist students in reaching goals and objectives that they do not believe are important. Departmental goals and objectives should reflect the collective understanding of the faculty members of what students should know and be able to do, rather than the ideas of a single individual. Departments may be tempted to use the current curriculum to motivate the development of student goals and objectives. Although this method will result in the perfect alignment of the goals and objectives with the curriculum, it will also result in a missed opportunity for improving the curriculum. In other words, the process of determining what is important should not be artificially constrained by what currently exists. In order to support the development of a set of goals and objectives and to acquire faculty support for the goals and objectives, each member of the full-time MCS faculty was interviewed in the fall of 1997. They were asked: 1) What competencies do you think students should have after completing the mathematics core courses?, 2) What competencies do you think students should have after completing their major courses in mathematics?, and 3) What competencies do you think students should have after completing their major courses in computer science? The reader will notice that each of these questions refers to student competencies rather than student goals and objectives. This phrasing stimulated the faculty to identify specific knowledge and skills. A feature of the interview process was that the faculty were not directed to consider the current curriculum, but rather they were asked to indicate the competencies that students should have upon completion of their course work. Using the specific information that the faculty provided, the departmental assessment specialist created broader statements of goals that captured these competencies. Next, a departmental sub-committee was formed that consisted of the head of the department, a mathematician, a computer scientist, a mathematics education expert and an assessment specialist. Based on the faculty responses to the interview process, the requirements of ABET and the University Mission Statement, the sub-committee drafted four sets of departmental goals and objectives: 1) a general statement for all students of mathematics and computer science, 2) a statement for the core mathematics courses, 3) a statement for the major mathematics courses, and 4) a statement for the computer science major courses. The fulltime faculty approved a version of these goals and objectives later that academic year. Periodically, the established departmental goals and objectives should be reviewed to determine whether they continue to be consistent with departmental needs. In the academic year 2001-2002, the department's Undergraduate Curriculum Committee reviewed the student goals and objectives. Although the committee felt that this list continued to capture the desired student outcomes, questions were raised with respect to the phrasing of the goals and objectives. The original list consisted of short phrases that implied the desired student outcome. The revised list directly indicated what the students needed to demonstrate in order to suggest that a given goal had been reached. Both the original and the revised list of student goals and objectives are displayed in Table 1. A critical question was raised during the final review of the revised student goals and objectives, "What is the faculty's responsibility in assisting students in reaching these goals and objectives?" This question resulted in the development of the faculty goals and objectives that are shown in Table 2. The faculty goals and objectives were designed to parallel the student goals and objectives, indicating the faculty's responsibility in supporting the desired student outcomes. IV. Using Goals and Objectives to Develop, Implement and Revise an Assessment PlanOnce a program has a set of goals and objectives, the next step is the creation of a plan that will support the attainment of those goals and objectives. In order to facilitate the creation of this plan, the MCS department used the Olds and Miller Assessment Matrix [8, 9]. This matrix, which is available electronically at http://www.mines.edu/fs_home/rlmiller/matrix.htm, provides a framework for planning and recording the phases of the assessment cycle. An example of a portion of the current student assessment plan is shown in Table 3. In the academic year 2001-2002, this plan was extended to include a faculty assessment plan. Both components of the larger assessment plan, student and faculty, are available at http://www.mines.edu/Academic/assess/Plan.html. The first column, “Performance Criteria (PC)”, is a statement of an observable performance that is used to determine whether a given objective has been reached. The statements within this column describe the strategies that will be used to interpret the collected data. Stating performance criteria is a first step in the development of strategies for reaching and evaluating the attainment of goals and objectives (phase 2 of the assessment process). The reader will notice that many of the established performance criteria in the MCS assessment plan reference a specific course. For example, PC1 states, "Students in Calculus for Scientists and Engineers (CSE) I, II and III will complete common exams that assess this objective. All students will pass the calculus sequence prior to graduation." The calculus sequence consists of three courses, all of which are coordinated. Coordinated courses have multiple sections, which are taught by different instructors. A lead faculty member coordinates these sections and holds regular meetings at which instructors have the opportunity to share instructional strategies and to create common assignments and/or exams. The lead faculty member for coordinated courses also ensures that the designated program objectives are assessed through common assignments and/or exams. The MAA [2] has criticized the practice of many mathematics departments of restricting the assessment process to traditional testing techniques, which are characterized by examining individual student responses to written evaluations. Instead, the MAA has recommended that departments supplement information acquired through traditional methods with additional information that has been acquired through team assignments and presentations. One aspect of the MCS Performance Criteria is that it does not rely solely upon traditional testing methods. Several of the performance criteria within the MCS plan refer to common student assignments that require group work and/or oral presentations. For example, PC24 states, "All students are required to pass Engineering Practices Introductory Course Sequence prior to graduation. Successful completion of this course requires that students work in multidisciplinary teams for a semester on the solution of a problem that was solicited from a local business." Other papers have addressed how these team activities are evaluated [10, 11]. Performance Criteria has also been established for the evaluation of oral presentations, "PC27: Students complete team oral presentations in Field Session. All MCS majors are required to pass this course prior to graduation". Field Session is an intensive six week summer course that is completed immediately following the students' junior year. Students and their teams dedicate at least eight hours each day to solving a problem that has been solicited from a local business. At the conclusion of the course, the students are required to present their solution to the participating company and their instructors in both written and oral form. These activities are then evaluated using a common scoring rubric or scoring scheme. Although PC27 is specific to team oral presentations, other criteria has been established that addresses individual oral presentations. The second column in the assessment matrix is “Implementation Strategy". This refers to the student or faculty activities that support the attainment of given performance criteria. Stating which activities will support the attainment of the goals and objectives is the second step in developing strategies for reaching the goals and objectives (phase 2 of the assessment cycle). For the majority of the student goals and objectives in the MCS assessment plan, the implementation strategy is the students' coursework. This includes core courses (courses that are required of all CSM graduates), major courses (courses that are required of all MCS majors) and Field Session (a design course that is required of all MCS majors after their junior year). As was discussed earlier, core courses are coordinated. A lead faculty member schedules regular meetings throughout the semester among the course instructors. During these meetings, instructors have the opportunity to share the activities that they have developed to support the attainment of the appropriate goals and objectives. In the major courses, it is the responsibility of the individual faculty member to ensure that the goals and objectives are being reached within a given course. The importance of reaching the stated goals and objectives through these courses is further reinforced in the faculty assessment plan in that faculty are expected to provide students with the opportunity to attain each of the student goals and objectives. The third column in the assessment matrix, “Evaluation Methods,” specifies the measurement instrument that will be used to collect the evidence as to whether the performance criteria have been reached. This column describes phase three of the assessment cycle (i.e., selecting assessment instruments). Two types of evaluations are mentioned in the student assessment plan: 1) common assignments and/or exams in coordinated courses and 2) the student senior survey. Three methods are referenced in the faculty assessment plan: 1) student evaluations of faculty instructional efforts, 2) voluntary observations completed during classroom instruction, and 3) a review of course materials by the course coordinator. The majority of the evaluation methods that were used in the original student assessment plan were dependent upon surveying the faculty and students. Although survey instruments provide an easy manner in which to acquire information, it is not necessarily the most reliable method of data collection. When the plan was reviewed and revised in 2001-2002, the decision was made to shift the focus to measurement techniques that directly assess student performances. This resulted in the current student assessment plan, which is dependent upon student performances on assignments, exams, oral presentations and team activities that are completed in classes. Surveys have not been completely eliminated from the revised plan. Students, upon graduation, are still asked to complete a survey and this information is used to supplement the direct measurement of the student outcomes. Given that this is the first year of implementation for the revised plan, the new methodology has not yet been fully tested. "Timeline" refers to when each evaluation method will be implemented. This column contains information on when the data will be gathered, analyzed and interpreted (phase four of the assessment cycle). Examination of the presented timeline suggests that the new assessment techniques have been introduced slowly during the five years in which the plan has existed. Spacing the introduction of new techniques has allowed the department to focus upon the implementation of specific methodology within a given year or semester. Had the department introduced all of the new measurements in the first year, the department would have been overwhelmed and the assessment plan would have failed. The final column of the assessment matrix is "Feedback". This column indicates how the acquired information will be disseminated and used and is directly linked to the fifth phase of the assessment process (using results). A primary concern that has been expressed by the MAA [2] and others [1, 3, 4] with regard to assessment is ensuring that the information acquired through assessment is used for program improvement. At the end of each semester, the department's assessment specialist summarizes the results of all survey instruments and compares these outcomes to the stated performance criteria. She also meets with the coordinators of the core courses and discusses how the goals and objectives were supported and assessed. Based on this information, she then writes and submits a report to the head of the department. This report contains recommendations on how the department may improve its programs and its assessment system. To assist the MCS department in documenting how the information that is acquired through assessment is used, a feedback matrix was developed [12]. A portion of this matrix is shown in Table 4 and the complete feedback matrix can be found at: http://www.mines.edu/Academic/assess/Feedback.html. This matrix indicates in which semester the information was collected, the source of the information, the concern that the information raised, the department's response to that concern and the efforts to follow-up on whether the response was successful in addressing the concern. For example in the spring of 2001, the students in Probability and Statistics for Engineers indicated on the student evaluation that the content covered during the semester was not consistent with the statistics that they used in their engineering courses. Based on this, the MCS department asked the engineering division to review and provide feedback on the curriculum for the Probability and Statistics for Engineers course. The Engineering division made a number of suggestions, including the recommendation that error analysis become part of the curriculum. In the fall of 2001, a new textbook and lab manual was selected that was consistent with the Engineering divisions recommendations. This process is documented in the feedback matrix. V. ConclusionsA great deal of time and effort has been dedicated to the development and implementation of the MCS departmental assessment plan. Revision of this plan is ongoing. As was discussed earlier, continual improvement of an assessment plan is a natural part of the assessment process. Many of the techniques that have been used here can be easily transported to the needs of other departments and disciplines. In fact, the Olds and Miller assessment matrix is already used across the CSM campus [12]. Further information concerning the MCS department's assessment efforts can be found at the department's assessment webpage: http://www.mines.edu/Academic/assess/. Additional information concerning the broader CSM effort with regard to assessment can be found at: http://www.mines.edu/Academic/assess/Resource.htm. AcknowledgementThe plan discussed here reflects the collaborative effort of the Mathematical and Computer Sciences Department (MCS) at the Colorado School of Mines (CSM). The author gives special recognition to Dr. Graeme Fairweather, MCS Department Head, and Dr. Barbara Bath, Director of Undergraduate Studies in MCS. Additionally, special thanks is given to Dr. Barbara Olds and Dr. Ron Miller who designed the assessment matrix that was used to organize the presented assessment plan. References1. Engineering Criteria, Accreditation Board of Engineering and Technology, 6th ed., 2000, available at http://www.abet.org/accreditation/accreditation.htm. 2. Assessment of Student Learning for Improving the Undergraduate Major in Mathematics, Mathematical Association of America, Subcommittee on Assessment, Committee on Undergraduate Program Mathematics, 1995. 3. Steen, L, "Assessing Assessment,"in Gold, B., Keith, S.Z., and Marion, W., eds., Assessment Practices in Undergraduate Mathematics, 1999, pp. 1-6. 4. Assessment Standards for School Mathematics, National Council of Teachers of Mathematics (NCTM), Reston, Virginia, 1995. 5. Moskal, B. "An Assessment Model for the Mathematics Classroom," Mathematics Teaching in the Middle School, 6 (3), 2000, pp. 192-194. 6. Rogers, G. & Sando, J., Stepping Ahead: An Assessment Plan Development Guide, Rose-Hulman Institute of Technology, Terre Haute, Indiana, 1996. 7. Moskal, B. M. & Bath, B.B., "Developing a Departmental Assessment Plan: Issues and Concerns," The Department Chair: A Newsletter for Academic Administrators, 11 (1), 2000, pp.. 23-25. 8. Olds, B. M. & Miller, R., “Assessing a Course or Project,” How Do You Measure Success? (Designing effective processes for assessing engineering education), American Society for Engineering Education, 1996, pp. 135-44. . 9. Olds, B. M. & Miller, R., "An Assessment Matrix for Evaluating Engineering Programs," Journal of Engineering Education, 87 (2), 1998, pp. 173- 178. 10. Knecht, R., Moskal, B. & Pavelich, "The Design Report Rubric: Measuring and Tracking Growth Through Success," Proceedings of the annual meeting of the American Society for Engineering Education, St. Louis, Missouri, 2000. 11. Moskal, B., Knecht, R. & Pavelich, M., "The Design Report Rubric: Assessing the Impact of Program Design on the Learning Process," Journal for the Art of Teaching: Assessment of Learning, 8 (1), 2001, pp. 18-33. 12. Moskal, B., Olds, B. & Miller, R.L., "Scholarship in a University Assessment System," Academic Exchange Quarterly, 6 (1), 2002, pp.32-37. Table 1 General Statement of Student Goals and Objectives.
G: Goals, O: Objective Table 2 Faculty Goals and Objectives
G: Goals, O: Objectives Table 3 Portion of the Mathematical and Computer Sciences Departments Student Assessment Plan (Complete plan available at: http://www.mines.edu/Academic/assess/Plan.html)
Table 4 A Portion of the MCS Feedback Matrix (Complete matrix available at: http://www.mines.edu/Academic/assess/Feedback.html)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||