Assessing Written and Oral Communication of Senior Projects

Dr. Kevin Dennis
Saint Mary’s University of Minnesota
700 Terrace Heights, 59
Winona, MN 55987

Abstract: In an evolving process to assess its majors, the mathematics department decided to use Senior Project presentations and reports as instruments to evaluate the department’s goal of developing the skill of mathematical communication.  The department developed rubrics for oral presentations and written reports that were piloted on a recent class of graduates. The findings and recommendations from these rubrics are discussed.

Introduction

Saint Mary’s University of Minnesota is a private Catholic university conducted by the De La Salle Christian Brothers with campuses throughout the world. The Winona campus offers a coeducational, residential, liberal arts program with an undergraduate student enrollment of about 1,300. It also offers graduate programs in a variety of disciplines.  The Department of Mathematics and Statistics at the Winona campus, with six full-time faculty members, offers a major and minor in mathematics and a minor in statistics.  The department graduates between four and seven mathematics majors per year.

The mathematics department has a comprehensive assessment plan for its majors.  The department uses Junior and Senior Assessment exams to assess students’ content knowledge. Senior Exit Interviews and Nature of Mathematics Survey assess students’ attitudes towards mathematics.  To assess students’ mathematical communication skills, Senior Projects and collection of homework from proof writing courses are evaluated.  In this paper, we will concentrate on the assessing of written and oral communication of the mathematics major’s Senior Projects.

All members of the Saint Mary’s Mathematics department agree that the communication of mathematics is one of the most important skills a major should gain as an undergraduate.  Because of this, the department has made the communication of mathematics a learning outcome for its majors.  To assess this learner outcome, the department has devised an assessment tool to evaluate the effectiveness of its program to this outcome.

Background

At Saint Mary’s University of Minnesota, every senior mathematics major must take a course entitled “Senior Seminar”.  This one-credit course is offered during the spring semester.  The focus of this course is an independent research project called the Senior Project, which counts for 70% of the course grade.  To help prepare students for the Senior Project, the instructor of the course selects mathematical journal articles for students to read and discuss.  These articles aid students in selecting topics and in learning about the format of a mathematical paper.  After reading several articles, students then choose a topic of interest to research.  Students typically choose topics involving mathematical modeling or mathematics education. 

After learning about their topic, students must present their project in a 25-minute talk with a 10-minute question and answer period afterwards.  All faculty and students of Saint Mary’s are invited to this talk.  Historically, the oral presentations were evaluated using questions found in Appendix A, but on a Likert scale of 1 (Low) to 7 (High).  The department felt that the Likert scale was too subjective and that a more quantifiable scale should be employed.  Hence, the scale found in Appendix A was created. 

After presenting their topic, the students must also write a 15 – 20 page technical report on their project.  Note that the term “technical report” is used instead of paper to denote the format of the paper specific to the discipline of mathematics.  For past written reports, only the instructor of the Senior Seminar course read the reports of the Senior Projects, mainly to determine the student’s course grade.  Since the department decided to assess the students’ written communication, the department created a rubric to assess the technical reports.

Method

To assess the oral presentations, any faculty member, regardless of discipline, attending the Senior Project presentations is given the form found in Appendix A.  Typically, there are five to seven faculty members attending each presentation.  After a student’s question and answer period, the faculty members are given a few moments to fill out the form while the next student prepares for his or her talk.  In addition to answering the seven questions, faculty members also write comments about the mathematics being presented, comments to the student and comments to the instructor.  After the assessment forms are collected, the students receive the averages of their scores along with any appropriate comments about their mathematics and their presentation.

To assess the written technical reports, the department decided to have three faculty members read each report.  For a particular paper, the three faculty members, with one being the instructor of the course, fill out the form found in appendix B.  The scores from these questions are tallied, usually after the semester is concluded, so the students do not receive them.  The department then meets at the end of the semester to discuss all assessment items, including the oral presentations and written reports.

Findings

The rubrics for oral presentations and written reports in the forms found in Appendix A and B were piloted during the 2002 – 2003 school year.  Due to this fact, some of the findings from the department are about the assessment forms themselves.

Student Communication Skills

With the oral presentations, the scores for the seven questions on average ranged between 5 and 6 for each student.  With these scores, the department agreed that the students for the most part were good presenters, but did not give good mathematical presentations.  The students had a tendency to give lengthy explanations for simple mathematical concepts, while quickly explaining the more difficult concepts needed for their projects.  In addition, the students used notation in their talks without explaining the meaning. The department believes that the mathematical explanations distinguished the Senior Project presentation from any other presentation that a student may have done in college.  Even so, this attitude did not carry over to the students since most students still treated the Senior Project presentation as just another presentation. 

With the written reports, the scores on average ranged between 3 and 4 for each student.  With these scores, the department felt that the students met only the minimal expectations of the project. The students did not research the history of their topic, but instead just found the minimal amount of formulas and theorems needed to solve their problem.  As with the oral presentations, the students had trouble with communicating the mathematics effectively. Many times, the students would show how to place numbers into a certain algorithm, instead of showing how the algorithm is generated.

Rubrics

The department found some problems with the rubrics themselves.  For the oral presentation rubric, the first problem was that the standard made it difficult to judge high scores.  Note that scores of 4 through 7 on the standard all dealt with just needing minor changes in the talk.  The second problem was the interpretation of the rubric standard.  Though the students’ scores averaged between 5 and 6, some students had scores ranging from 3 to 7 for the same question.  Upon discussion of this issue, some faculty members thought the rubric was used in determining the course grade and hence scored higher than the standard called for.  Another interpretation problem was that mathematically “weaker” topics were given lower scores, though the level of the topic is not incorporated into the rubric.  The department’s only trouble with the written report rubric was that it was hard to apply to mathematics education projects.

Recommendations

The department responded to the findings from the assessment by recommending changes to the format of the Senior Seminar course.  To aid in some of the communication issues of the oral presentation, the department suggested that the Senior Seminar instructor incorporate a practice session in the class.  The students would present their topic to the Senior Seminar class and be graded on that presentation, then give the talk to the Saint Mary’s community.  To aid in the minimalism that existed in the students’ reports, the department suggested that the instructor encourage “stronger” mathematical topics, with even the possibility of restricting the choice of topics.  In addition, the instructor should encourage the students to start their projects during the fall semester, instead of waiting until the spring.

One factor against the success of increasing the communication skills via changes in the Senior Seminar class is that the mathematics department encourages the rotation of its upper level courses to all department members.  With this change, some continuity in the Senior Seminar class is lost, so information gained from one year’s class may not be reflected in the next year’s Senior Seminar class.  On the other hand, an encouraging note is that the university has gone to a full year planning schedule.  In the past, the university scheduled its courses on a semester-to-semester base, which meant that the Senior Seminar instructor was not chosen until middle of the fall semester.  Now the Senior Seminar instructor is known before the school year starts, so the instructor can work with the senior mathematics majors informally in the fall, preparing them for their Senior Project.  This means that the Senior Seminar class in the spring can be used to discuss the communication of mathematics more than researching mathematics.

Besides suggesting changes in the Senior Seminar class, the department set some goals in order to elevate the students’ mathematical communication skills.  The main goal was to give students more opportunities to see examples of mathematical communication, both orally and in written form.  To see more written examples, one recommendation was to have all instructors of upper-level mathematics classes have students read journal articles and write reports over these articles. For oral communication, the department set a goal to encourage more students to go to conferences and have more speakers present at Saint Mary’s University of Minnesota.  One idea discussed was to have the top two Senior Seminar presenters be awarded free trips to the local mathematical conferences held in April.

With the information obtained through this assessment process, the mathematics department at Saint Mary’s University of Minnesota learned that while its majors present well and write well for general topics, the technical communication of mathematics is lacking.  The department has set goals to have the students see and read more examples of good mathematical communication in and outside the classroom.  Hopefully, through these examples, the students’ mathematical communication skills will increase over time.

7 – Excellent – ready for presentation at a conference
6 – Very Good – needs little changes before presenting at a conference
5 – Good – needs a few minor changes before presenting at a conference
4 – Satisfactory – needs minor changes before presenting at a conference
3 – Poor – needs few major changes before presenting at a conference
2 – Very Poor –needs major changes before presenting at a conference
1 – Unsatisfactory – presentation has little or no value

Questions:

1. How well did the student explain the purpose of the project?
2. How well did the student organize the material?
3. How neat was the student’s presentation (use of overheads, PowerPoint, etc.)?
4. How would you judge the student’s presence (voice, delivery, etc.)?
5.  How well did the student answer/respond to questions and comments?
6. How well did the student communicate his/her understanding of the mathematics?
7. Overall, how would you rate this presentation compared to other presentations you have seen (before today)?
Comments on the mathematics:
Comments to the student:
Comments to the instructor:

Appendix B. Sample Report Assessment Form

5 – Excellent – needs minor revisions in a few places (e.g., 2 or less changes)
4 – Good – needs minor changes throughout report (e.g., 1 per page)
3 – Satisfactory – needs a major change or many minor changes (e.g., over 2 per page)
2 – Poor – needs major revisions throughout report
1 – Unsatisfactory – has little to no value

Presentation of Report

1. How well written is the report (e.g., correct grammar, spelling, etc.)?
2. How well does the student present graphics/figures/equations (e.g., placement, neatness, etc.)?
Technical Report
3. How well does the student explain the purpose of the project?
4. How in depth does the student explain the history of the problem?
5. How well does the student explain the background mathematics needed to understand the problem?
6. How well does the student critique the model/study/topic?
Content
7. How in depth does the student explore the mathematics/statistics/research?
8. How accurate are the mathematical/statistical statements?
9. How well does the student justify the mathematical/statistical statements?
10. How consistent and effective is the student’s use of notation?
11. How effectively does the student use examples to clarify points made in the paper?
Overall – Report
12. How well does the student demonstrate that he/she understands the mathematics/statistics/research?
13. How well does the paper compare to other reports seen before this years?
Overall – Topic
14. The level of the topic appears obtainable with someone in a background in:
            1 – Freshman level mathematics and statistics (Calc I/Calc II/Intro to Stats)
            2 – Sophomore level mathematics and statistics (Calc III/Linear/Probability)
            3 – Junior/Senior level mathematics and statistics (First course in specialized areas)
            4 – Graduate level mathematics and statistics (Second course in specialized areas)
            NA – Educational Research Project
15. The level of originality of the topic appears to be:
            1 – Using known techniques to solve a problem found in a lower level course
            2 – Using known techniques to solve a problem covered in an upper level course
            3 – Using known techniques to solve a problem partially covered in an upper level course
            4 – Using unknown techniques to solve a problem partially covered in an upper level course
            5 – Using techniques to solve a problem unknown to the student
            6 – Using techniques to solve a problem unknown to the mathematical community or contributing new literature to the educational community