A Comprehensive, Proactive Assessment Program

A large technical institute using the author as assessment coordinator, creates a broad new assessment program, looking at all aspects of the department's role. Statistical studies guide improvements in curriculum, teaching and relations with the rest of the university.

Background and Purpose

Virginia Polytechnic Institute ("Virginia Tech") is a land grant, state, Type I Research University, with an overall enrollment of 26,000 students. Since 1993, the semester enrollment of the mathematics department averages 8,000 - 12,000 students. Of this group, approximately 90% of the students are registered for introductory and service courses (1000 and 2000 level). These first and second-year mathematics courses not only meet university core course requirements, but are also pre- or co-requisites for a number of engineering, science and business curricula.

A comprehensive assessment program, encompassing the activities of the mathematics department, began in the spring of 1995 at Virginia Tech. In the preceding fall of 1994, a new department head had been named. Within days of assuming the position, he received opinions, concerns, and questions from various institutional constituencies and alumni regarding the success and future direction of Calculus reform. The department head entered the position proactively: finding mechanisms which could provide faculty with information regarding student performance and learning; developing a faculty consensus regarding core course content and measurable objectives; recognizing and identifying the differences in faculty teaching styles and learning styles for students; synthesizing and using this information to help improve student learning and academic outcomes. Concurrently, the mathematics department was in the midst of university restructuring, and "selective and differential budget adjustments" were the menu of the day. From the departmental perspective, the university administration required quantitative data and an explanation of a variety of course, student, and faculty outcomes, often with time frames. The resources in Institutional Research and Assessment were shrinking, and more often data had to be collected at the departmental level. The department decided data and analyses available within the department were much preferable to obtaining data via the university administrative route. The selection of a person to analyze and interpret the data had particular implications since learning outcomes are sensitive in nature and are used within the department; another problem was that educational statistics and measurement design skills, distinctly different from mathematical expertise, were needed for the analysis and interpretation of data. Networking with the university assessment office on campus provided an acceptable option — an educational research graduate student who could work part-time with the department in the planning and implementation of the assessment program (Scruggs).

Method

Data Gathering and Organization: The departmental assessment effort roots itself in obtaining forms of data, organized by semester, with students coded for anonymity:

High school data are obtained from admissions for entering freshmen and include SAT verbal and math scores, high school GPAs, high school attended, and initial choice of major.

Background survey data from the Cooperative Institutional Research Program (CIRP) are acquired during freshmen summer orientation.

Course Data including overall course enrollment, sectional enrollment, student identification numbers by section, instructors of each section, and times and locations of courses.

Achievement Data including grades from common final for about 8 core courses, ranging from college algebra to engineering calculus and differential equations. (Initially the purpose of these examinations was to provide a mechanism to evaluate Mathematica as a component of engineering calculus.) These examinations are multiple choice and include three (3) cognitive question types: skills, concepts, and applications. These examinations help determine the extent to which students have mastered mathematical skills and concepts; and secondly, allow comparisons between and among sections, in light of instructional modalities, methods and innovations. After common exams are administered and scored, each instructor receives a printout detailing the scores for their class, as well as the mean and standard deviation for all students taking the common examination. The assessment coordinator receives all scoring and test information electronically, including the individual item responses for all students. Test reliability, validity, and item analyses are performed for each course common exam. This data is then made available to mathematics faculty to aid in the interpretation of the current test results, as well as for the construction and refinement of future test questions.

Survey Data: Virginia Tech has administered the Cooperative Institutional Research Project (CIRP) Survey every year since 1966. Student data, specific to our institution, as well as for students across the United States, is available to the mathematics department for research, questions, and analyses. The mathematics department assessment program is actively involved in the process of identifying variables from the CIRP surveys which are associated with student success. Additionally, several in-house survey instruments have been designed to augment this general data and gauge specific instructional goals and objectives. The departmental surveys use Likert rating scales to accommodate student opinion. With this procedure, affective student variables can be merged with more quantitative data.

Methodology Overview: Data is stored in electronic files, with limited access because of student privacy concerns, on the mathematics department server. SPSS-Windows is used for statistical analyses. Specific data sets can be created and merged, using variables singularly or in combination, from academic, student, grade, and survey files. More information on this process is available from the author.

Findings and Use of Findings

The department's role on campus ranges from teaching students and providing information for individual instructors to furnishing information to external constituencies including other departments, colleges, university administration and state agencies. A comprehensive program of mathematical assessment must be responsive to this diverse spectrum of purposes and groups. Departmental assessment then refers to the far-reaching accounting of the students and departmental functioning within the department and throughout the university. Assessment has different purposes for different groups, and given this range of applications, the following discussion incorporates selected examples of data analyses, outcomes, and decisions.

Academic Measurement: While tests are important, their usefulness is contingent on the quality of the instrument, course goals, and the intended purposes. In the common final examinations, close attention is given to the construction and evaluation of the tests themselves. Common finals are constructed by faculty committees using questions submitted by individual faculty members who have taught the course. The tests are then evaluated for appropriateness of content, individual question format and style, and item difficulty. Post-test analyses are performed by the assessment coordinator that include test reliability coefficients, item analyses, overall and sectional means and standard deviations, and score distributions. With each administration, faculty and student feedback regarding the finals has become increasingly positive, indicating that the tests are more representative of course content, and the questions have greater clarity. During this iterative process, faculty knowledge and involvement in assessment has grown with increasing dialogue among faculty a welcome outcome.

Departmental assessment practices have provided a mechanism for monitoring and analyzing student outcomes as innovative and different teaching methods have been introduced and technology added to existing courses, such as engineering calculus and college algebra. Did these changes have a positive effect on student learning? What effects, if any, did the changes have on long-term learning and performance in other courses? These questions were posed from inside the department and from other departments and university administration. The departmental data base allows rapid access to grade and common final data for mathematics courses, and grade outcomes for engineering and science courses.

For the freshmen students enrolled in the fall semesters of 1993 and 1994, student academic background data in conjunction with course grades were used to examine longitudinal outcomes for traditional and "Mathematica" calculus students who subsequently enrolled in advanced mathematics and engineering courses, such as differential equations, statics, and dynamics. Mean comparison studies with t-tests were performed, comparing grade outcomes of the traditional and Mathematica students. (No statistically significant differences noted except for the dynamics course [Table 1].)

A designated section of differential equations was taught in the Fall of 1996 as a response to a request from civil engineering and was nicknamed the "Green" version. Targeted toward Civil Engineers, the course utilized environmental and pollution examples to support differential equation concepts and theory. Table 2 summarizes "Green" course outcomes as compared to sections taught in a traditional format with a variety of majors in each section.

Integrating teaching methods and theory application and use as they apply to specific major areas offers intriguing opportunities. During the fall of 1997, faculty in the college algebra sequence collaboratively with other departments and individual faculty outside of the mathematics department. So designed, the sets help students recognize that mathematics is a valuable aspect of all that they do, not just a core university requirement .

Student Placement: To keep up with a changing student population and changing expectations of the university, students, parents, state legislatures, and the media (as evidenced in the charge for academic and fiscal accountability), the departmental "menu" of courses and teaching methods and student support options have been expanded, as has the requirement for assessing and justifying the changes. Appropriate placement of students under these circumstances becomes both an educational and accountability issue.

Since 1988, Mathematics Readiness Scores have been calculated for entering freshmen. Institutional Research devised the initial formula using multiple regression analysis. The formula for the calculation has gone through several iterations, with scores currently calculated from student background variables available through university admissions: high school mathematics GPA; College Board Mathematics scores; and a variable which indicates whether or not the student had taken calculus in high school. A decision score was determined above which students are placed in the engineering calculus course, and below which, in pre-calculus courses. Each semester, using the course grades, the scores are validated and the formula modified to maximize its predictive capability.

Special Calculus Sections: Even though student success in engineering calculus improved after the math readiness scores were utilized, academic achievement remained elusive for many capable students who had enrolled in the pre-calculus course. Grade averages were low in this course, and longitudinal studies indicated that many students who had earned a grade of C or above, failed to complete the second course in the engineering calculus sequence with a comparable grade. In the fall of 1996, based on a model developed by Uri Treisman at the University of California, Berkeley, a pre-Calculus alternative was piloted within the mathematics department. An augmented version of the engineering calculus sequence was begun — Emerging Scholars Program (ESP) calculus, now operating for the first and second semesters of calculus. The traditional three-hour lecture course was accompanied by two, two-hour required problem-solving sessions, supervised by faculty and using undergraduate teaching assistants as tutors. Due to the academic success of the students, as well as faculty, tutor, and student enthusiasm for the approach, 6 sections of ESP calculus were incorporated into the spring course schedule. Students enrolled in the spring ESP sections had previously been enrolled in traditional calculus or pre-calculus in the fall. Average course grades for the traditional calculus students was 0.8 (out of a possible 4.0). Comparisons of the student outcomes for the traditional and ESP versions from the spring of 1997 are shown in Table 3. The fall of 1997 has 17 sections of ESP calculus on the schedule, with a number of sections of traditional engineering calculus. Previous assessment efforts, both quantitative and qualitative, supported the departmental decision to proceed toward the ESP approach and away from pre-calculus.

Developmental Courses: The college algebra/ trigonometry course enrolls approximately 1300-1400 students each fall semester. This non-major service course, serving primarily freshmen students, requires significant departmental resources. In the fall of 1995, a computer-assisted, self-paced approach was pilot tested, involving a cohort of 75 students from the 1300 total enrollment. At the beginning of the semester, all students were given a departmental survey that is designed to ascertain student perceptions of their learning skills and styles, motivation, and mathematical ability. At the conclusion of the semester, these non-cognitive items, determined from the factor analyses of survey data, were analyzed with student grades using regression analysis. The goal was to identify predictors of success in the computer-assisted version of the course. Significantly related to success were the self-reported attributes of being good to very good in math, organized and factual in learning new material.

Two very different means of placement have been described above. One utilized cognitive achievement data, while the second made use of non-cognitive student reported information. Both approaches have provided valuable information, for student placement and for course evaluation and modification. Since the introduction of technology as the primary instructional modality in 1995, the college algebra course has undergone several iterations in response to quantitative and qualitative departmental data analyses. At the present time, this course maintains its technology-driven, self-paced instructional core. As a response to student survey responses which indicated a need for more personal and interactive experiences, a variety of instructional alternatives, such as CD lectures, have been incorporated into the course.

Technology: A variety of student outcomes and background variables were used as a means of assessing the incorporation of computer technology into the college algebra and engineering calculus courses. Assessment results and outcomes are generally positive with some concerns. One finding indicated that the use of technology allowed students to pace themselves, within a time frame beneficial to student schedules. Also the downstream results for engineering calculus indicated that students receiving technological instruction during the regular course time did as well, if not better, in the more advanced course work. Negative findings were related to computer and network functioning, certain aspects of the software, and the lack of congruity between lecture and computer assignments. Using the outcomes as a guide for modifying courses each semester, technology use within the department has increased. In fall of 1997, the mathematics department opened a Mathematics Emporium, with 200 computers and work stations, soon to be expanded to 500. Assessment has played and will continue to play a role in ideas, decisions, and educational innovation regarding technology.

Learning/Teaching: Our data base enables our department to effectively respond to issues raised from within the department and externally from other departments and university administration. For example, the common final examinations in many departmental service courses have given additional information regarding student, sectional, and course outcomes. Scores, in conjunction with course grades, have been used to examine the connection between grading practices and student learning. In the fall of 1995, there were 31 sections of engineering calculus, all relying on the same course goals and text book. After common finals were taken and grades assigned, Pearson Correlations were used to ascertain the association between sectional final scores and grades. For all sections taken together, the correlation was calculated to be a 0.45. Though statistically significant, the magnitude of the result was lower than expected, prompting further study, as sectional mean scores and grades were examined individually. The following table affords examples of the variety of sectional outcomes. Actual data is used, though the sections are identified only by number and in no particular order [Table 4].

One can note that Sections 2 and 3 are disturbing in the incongruence demonstrated between course grades and common final scores.

Making this data available anonymously to instructors offers them the opportunity to compare and analyze for themselves. The department head promoted the use of assessment data to generate an informed and potentially collaborative approach for the improvement of teaching.

Focus Groups: Seeking to evaluate the ESP calculus program, a focus group component was included to obtain students' views and feelings regarding the course format, philosophy, and expectations. Student responses were uniformly positive. About this Calculus approach, freshmen students suggested an unanticipated aspect of its value — the sense of community they experienced within the mathematics department, and by extension, the university as a whole. Student comments show that they feel the value of esprit de corps in a school that uses their input. As one student remarked, "Learning math takes time and resources. ESP is what makes Tech a good school."

Success Factors

Assessment within the mathematics department is the reflection of a variety of factors, many planned, others serendipitous. But how can success be gauged? What is the evidence of the value added to students, the department, and the institution? Who has gained? Answers to these relate to the department, the individual faculty and the students. The ongoing assessment of the department allows the department to be public, share concerns and answer questions, and allow it to better identify, compete for, and manage available resources within the department, the university, and beyond. The faculty is more able to monitor their students' outcomes, as well as that of curriculum and instructional techniques. Ensuing program planning provides faculty the opportunity for increased ownership and distinctly defined roles in instructional development. Students have certainly received the benefits of assessment by feelings of enhanced involvement and contribution to their educational process. Though probably unaware of the scope and extent of quantitative information which impacts their educational experiences, students interact with assessment and the department through opinion surveys regarding their courses and occasionally through participation in focus groups. Through the realization that their opinions matter, there is the opportunity for a strengthened sense of affiliation with mathematics, the department, and the university.

A mathematics department faculty member recently asked the question, "Whatever happened to the ivory tower?" The answer of course is that it no longer exists, or that it has been remodeled. Departments are no longer concerned primarily with their discipline. In today's educational climate, valid thoughtful information must be readily available regarding student learning and success, program development and improvement. Stewardship of faculty and financial and space resources must be demonstrated to a variety of constituents beyond the department. As a matter of performance and outcomes, everyone gains from the assessment process on the departmental level.

grades, have been used to examine the connection between grading practices and student learning. In the fall of 1995, there were 31 sections of engineering calculus, all relying on the same course goals and text book. After common finals were taken and grades assigned, Pearson Correlations were used to ascertain the association between sectional final scores and grades. For all sections taken together, the correlation was calculated to be a 0.45. Though statistically significant, the magnitude of the result was lower than expected, prompting further study, as sectional mean scores and grades were examined individually. The following table affords examples of the variety of sectional outcomes. Actual data is used, though the sections are identified only by number and in no particular order [Table 4].

One can note that Sections 2 and 3 are disturbing in the incongruence demonstrated between course grades and common final scores.

Making this data available anonymously to instructors offers them the opportunity to compare and analyze for themselves. The department head promoted the use of assessment data to generate an informed and potentially collaborative approach for the improvement of teaching.

Focus Groups: Seeking to evaluate the ESP calculus program, a focus group component was included to obtain students' views and feelings regarding the course format, philosophy, and expectations. Student responses were uniformly positive. About this Calculus approach, freshmen students suggested an unanticipated aspect of its value — the sense of community they experienced within the