Mathematical Association of
America
Supporting Assessment in
Undergraduate Mathematics
Assessment of Quantitative
Reasoning in Applied Psychology at Portland State University
Robert R. Sinclair, Ph.D.
Dalton Miller-Jones, Ph.D.
Jennifer A. Sommers,
M.A.
Portland State University
Department of Psychology
________________________________________________________
Notes. The authors express their gratitude to our faculty colleaguesthe
Psychology faculty, graduate students,
and undergraduates who participated in this project by contributing to the
development of the assessment plan, offering feedback on the research, helping us
with data management, offering us the
opportunity to gather data in their classes, and, in the case of the
undergraduates, participating in the research.
For more information, contact the
first author at:
Portland State University
Department of Psychology
Portland, OR
97207
(503) 725-3965
sinclair@pdx.edu
Abstract
This study reports efforts by the
Department of Psychology at Portland State University to assess student
knowledge and skills in quantitative areas of Psychology. Preliminary research showed that faculty
unanimously agree that research design
(e.g., distinguishing experimental and correlational research designs), psychological
measurement (e.g., test reliability and validity), and statistics (e.g.,
calculating central tendency indices, interpreting correlation coefficients) are important
competencies for undergraduates majoring in psychology. Each of these competency areas is
either directly related to or dependent upon students’
quantitative literacy. Moreover,
at
Portland State, math faculty cover several of these concepts in two social-science
statistics courses required for psychology majors. This report
describes assessment practices and outcomes for quantitative
literacy in these areas. The
research included psychology students (majors and
non-majors) at all levels of the undergraduate program. We found relatively low levels of competence in areas
of quantitative research methodology, with psychology majors outperforming
non-majors. Senior-levels
students
who completed our intended methodology course sequence
reached acceptable mastery levels. However, there appeared to be little advantage
for students who had completed the required statistics sequence as compared
with those who had not. These
results are interpreted within a “continuous improvement” model where, based on
these data, adjustments are made to program planning and individual courses
that are intentionally designed to impact our learning goals and objectives.
Assessment of Quantitative
Reasoning in Applied Psychology at Portland State University
Background
Psychology has a long history of interest in
individual assessment acrossin a
wide variety of contexts. ConsequentlySimilarly,
psychologists have played a leading role in much of the scholarly research on
learning and behavioral change. However, only within the last decade have
psychologists begun to pay serious attention to assessment of learning in
undergraduate psychology programs.
Perhaps the most tangible demonstration of this interest is the recent
release of Undergraduate Psychology Major
Learning Goals and Outcomes: A Report, a task force study endorsed by the
American Psychological Association (Murray,
2002). The study identified learning
outcomes for ten10
educational goals in psychology [available on-line at
www.apa.org/ed/pcue/taskforcereport.pdf].
Quantitative literacy (QL) plays
a prominent role in several goals described in the report. For example, the Research Methods in Psychology goal explicitly focuses on data
analysis and interpretation. Similarly,
the Values in Psychology goal
emphasizes the utility of the scientific method and the value of using
empirical evidence to make decisions.
Many psychology programs heavily rely on mathematics
departments to provide their statistical training. This reliance most commonly occurs at the
undergraduate level, but some graduate level psychology programs
also either encourage (or require) students to
take quantitative methods courses taught by math faculty. In either case, quantitative
literacyQL is a critical component of the
psychology major, since advanced courses assume students have a grasp of basic
statistical concepts and an understanding of
how to apply those
conceptsem to psychology. Thus, mathematics
departments often play critical roles in psychology training. Consequently, strong quantitative
literacy QL assessment efforts provide both
psychology and math programs with useful data about whether their courses statistics
training accomplishes
each program’s educational objectives.
In 1998, the Portland State
University psychology program responded to a request by the Dean of the College
of Arts and Sciences that all departments identify learning goals/objectives
their majors should have achieved at graduation. One motivation for
this request was the knowledge that the next round of higher education
accreditation review, at the time 5 years in the future,
would require a focus on authentic indicators of student learning, not just the
traditional set of input data (e.g., the number and kinds of classes taught,
student enrollments). The university president also designateded assessment of student learning as one of three Portland State University Presidential
Initiatives to
emphasize the central role of assessment at the university. In
response to theise challenges, the Ppsychology Ddepartment crafted an assessment
vision involving tracking student learning from our initial introductory
courses, through our research methods and experimental psychology courses, to
our advanced seminars in industrial/organizational, applied developmental, and
applied social psychology. This design
enables us to determine
whether our programs actually affect student learning by tracking changes in
students’
performance across our curriculum, and using the empirical data generated by
our research to guide decisions about curriculum development“practice
what we preach” by basing our programmatic decisions on powerful empirical data.
Assessment in
Psychology
Our initiative began with a series of workshops
in which psychology faculty generated approximately 50 valued learning
outcomes. These outcomes were organized
into nine9 broad learning goals that closely
resembled the goals suggested by the APA task force report described
above. Faculty also indicated which
learning outcomes and goals pertained to each of their courses. We used summary ratings (by faculty) of these
outcomes and goals to establish assessment priorities. We then organized the learning goals into three four categories: Theories and Issues,
Student
Engagement, Application of
Psychology, and RPsychological
Research esearch
Methodsology and Statistics.
Consistent with our description above, the faculty ratings suggested that Research Methodsology and Statistics
was a should
be the highest priority topic because mastery of
student learning in this area is closely tied to learning about other aspects
of psychologyin the assessment initiative. . Our decision to
focus on QL issues also mirrors
one of the key ability areas identified by Portland State University’s faculty
senate for our graduating seniors:
Quantitative
Reasoning and Representation – ability to
deepen understanding of the value and need for this type of reasoning, the
ability to understand the graphical presentation of data, and to transform
information into quantitative and graphical representations.
The broad area of RAs
we conceive of it, research Mmethodology
and Statistics consists ofincludes
fourive topics: Research Design and the
Scientific Method, (e.g., use of experimental,
observational, questionnaire strategies), Scientific Method, Psychological
Measurement (e.g., rReliability
and vValidity in
psychological assessment) (in psychological
measurement), and Statistics, and Research Ethics. Quantitative literacy QLis an essential
component of knowledge
forms the foundation of several
of these topics. For example, tFor example, the
statistics area presently
fofocuses on three quantitative
literacy conceconcepts: central tendency, variation,
and association. Upon graduation, we
expect students to be able to conduct and present the findings of basic
statistical tests in each of these areas as well as to interpret and critique
presentations of these tests in published empirical literature. Similarly, we
expect students to masterobtain
basic research concepts in such
as research design and psychological measurementmetrics (reliability
and validity). AlthoughWhile research design and psychological measurementmetrics
are somewhat different than what might traditionally be regarded as
quantitative literacy, students use quantitative QL skills
as they learn about knowledge in these domains. For exampleAs one example, students
must understand the concept of correlation to be able to grasp differences between
forms of reliability and validity. Finally, we note that our decision to focus on quantitative literacy
issues also mirrors one of the key ability areas identified by Portland State
University’s faculty senate for our graduating seniors:
Quantitative
Reasoning and Representation – ability to deepen understanding of the value and
need for this type of reasoning, the ability to understand the graphical
presentation of data, and to transform information into quantitative and
graphical representations.
Our Assessment
Research
The main purpose of this case study is
to describe our preliminary research efforts to assess quantitative literacy
and related concepts. This research
provides baseline data for future assessment efforts andas
well as empirical support for changes to the curriculum. In particular, our data illustrate some of
the ways assessment data can be used to document the impact of a program and,
as well as to pinpoint areas of
particular need in curriculum development.
We view assessment as fundamentally aimed at demonstrating the effect of
the program on student behavior change, typically defined as increased student
mastery over learning goals previously identified by the department. Thus, effective programs should demonstrate
high levels of overall performance as well as desired patterns of changes in
learning over the course of the program.
Our first programmatic assessment efforts involved
the development of a 20-item multiple
choicemultiple-choice exam covering topics related
to research methodology. The test
consisted of questions on research design (e.g., distinguishing experimental
and correlational research designs), psychological measurement (reliability and
validity),
and statisticsquantitative
literacy. The statisticsquantitative
literacy section is the most directly related to quantitative literacy and
concerns the portion of the curriculum taught in the math department. The statistics questions focused on
very basic statistical concepts, such as calculating central tendency (mean,
median) and variability (range) measures and interpreting correlation
coefficients. Many other relevant
concepts were not included (e.g., hypothesis testing). We also asked four
quantitatively oriented psychology faculty
to rate the difficulty level of the questions based on the level
of challenge of the focal concept, the difficulty of the
distracters (i.e., the three incorrect response options for each question) and
the level of cognitive difficulty associated with each question. UsingBased on
these ratings, we sorted the questions into high challenge and low challenge
scales. We
also sorted the items into three substantive scales corresponding toand
formed scales for statisticsquantitative
literacy, research design, and psychological measurementreliability and
validity, as well as the total score. In each case, the scores were defined as
mean proportions of questions successfully answered.
The test was administered during two academic terms
to over 800 students taking a wide array of undergraduate psychology classes
(from freshman to senior level). We strategically
sampledThe classes were sampled
strategically so the participants would represent a broad cross section
of our students. This strategy enabled
us to capture changes in provide indications
of students’ mastery of quantitative literacy topics from the time they progress
as they entered the major (i.e., at the beginning of the courses
in our introductory sequence), toward
their advanced level
coursese middle of the major, and as they
entered senior
level advanced seminars. This strategy enabled us to compare student
performance across the curriculum and . Moreover, the
non-majors taking these courses serve as sort of a quasi control group for
examining psychology majors. That is,
we would expect to see greater change in majors, as compared with non-majors.
Analyses
and Findings
to ascertain
whether our training improved students QL knowledge
and skills. We made three
types of comparisons. First, we
compared the test scores of psychology majors (who are required to complete a
sequence of methodology and statistics courses) to non-majors. Second, we examined majors’ changes in test
performance as they progress through the curriculum. Finally, we investigated differences by class level in
quantitative literacy for all students taking psychology courses, including
majors, minors, and non-majors,,
by class level.
Findings
Table 1 (Appendix A) presents differences between
psychology majors and non-majors on each of the test scores. The total scores of 57% for majors and 46%
for non-majors represent somewhat llow levels of
mastery of quantitative literacy-related
topics. However, pPsychology
majors out-performed non-majors on all subtest scores by 9-13%, depending on the across
different subtest considereds. Interestingly, the highest scores were for
the statisticsquantitative
literacy dimension for majors. The statistics score, which
was the only subtestscore
to meet or exceed 70%, which is commonly viewed as “C” level performance in
graded classes. The pPsychology
dDepartment
requires psychology majors to complete two statistics courses, and statistical
topics are either explicitly or implicitly covered in several other courses –
perhaps to a greater extent than other research methods topics. Thus, our findings most
likely data may reflect the different levels
of focus on these topics.
The generally low levels of performance suggest
ample need for improvements in our efforts to address quantitative literacy
issues. This preliminary finding
is largely consistent with our
experience, as well as those of colleagues at other institutions. Under, with undergraduate
students’ often express a great deal of distaste for, or ambivalence toward, topics
related to research methods and statistics. F
(for example, psychological measurement and testing was
rankedted 45th out of 46 on a
recent survey of our students’ interests in topics related to psychology).
It is
important to note that students may not be to blame for these attitudes. Faculty may need to redouble their efforts
to teach these concepts in engaging ways. Finally, one piece ofOne
set of good news for psychology majors at least,
is that they showed consistently higher levels of performance than non-majors. These differences, provide some
evidence of beneficial effects of our program for our students as compared with students in other
programssuggesting
some evidence of program benefits,
as compared with other programs.
All psychology majors must complete a The
psychology major has a core set of curriculum requirements
that all majors must complete.
Many of these requirements concern research
methodology and quantitative literacy issues, including: (a)
relatively basic coverage in our two- course
required introductory
psychology sequence, (b) specific coverage of quantitative literacy issues in
statistics courses taught in the math department (but required for psychology
majors), and (c) an intense focus on research methodology issues in our upper- division
research methods course. This
curriculum is founded on the assumption that each of these courses contributes
to students’
capacity to conduct research, evaluate published studies, and interpret the
results of data analyses in applied contextsknowledge
of quantitative literacy and research
methodology, an assumption we examine in Table 2 (Appendix A). This table examines overall test performance
for psychology majors broken down by the number of these required courses they
have completed. As a whole, the majors
scored 57% on the test. These scores
were slightly higher for students who had completed the entire sequence (61%)
and lower for students who had not completed any of the sequence (52%). Interestingly, there was no difference in
overall test performance for students who had the introductory course(s) only
and those who had completed the introductory course and a statistics course
(both groups obtained an overall score of 56%). This suggests that future attention needs to be given to the
extent to which statistics research-focused courses
are having their intended effects in ourthe
curriculum.
Assessment efforts involve documenting change across
an entire educational experience.
Moreover, many of the courses that are not explicitly part of our
methodology sequence either implicitly or explicitly address methodology
issues. Therefore, a second way to
explore the effects of the program on quantitative literacy concerns showing
performance changes across class levels.
Table 3 shows the test scores both for the entire research sample and
only for psychology majors. It is important
to note that the entire sample data include the majors as well as the non-majors, so these
data underestimate the differences between non-majors and majors across program
levels.
As Table 3the (Appendix A) table shows,
students in both groups appear to make gains in
test mastery at each level of the curriculum.
The gains are somewhat slight, ranging from 5% gains from freshman to
senior level for the majors on psychological measurementreliability and
validity to 15% gains from freshman to senior level for the entire
sample on high challenge items. There
are a couple of clear trends worth noting.
First, both for majors and the entire sample, there are consistent
trends of small improvements in test scores across the curriculum. These findings suggest students are improving their
quantitative literacy skills as they progress through the psychology
curriculum. Second, psychology majors show higher performance
at each class level. Thus, psychology
majors indicating the haveir greater entering mastery of these
skills both at entry and upon completion of the program.
These findings probably reflect the fact that non-majors take psychology
courses more because of their substantive interest in
psychology and that they are less facile with and/or less interested in
methodological issues in psychologyand their
higher level when they leave. These findingsdata suggestshow
t hat students areppear
to be improving their quantitative literacy skills as
they progress through the psychology
curriculumcoursework at
Portland State University. Second, psychology
majors show higher performance at each class level indicating their greater
entering mastery of these skills and their higher level when they leave. These data can be
interpreted as showing some effects of the impact of the psychology program on
student learning. Finally,
we note that the test scores were uniformly higher for the statisticsspecific
quantitative literacy subtest than eiththe er
for rresearch design or psychological
measurement subtestsreliability and validity, particularly
for the psychology majors who, by the time they reached senior level classes, had reached a marginally a more n
acceptable mean of 76% correct on the statistics test. Although there aremay be multiple interpretations of
these data, they do appear to show that students who
have completed more of the psychology curriculum do, in fact, reach higher levels of mastery on
quantitative literacy skills.
Insights
Our preliminary analyses indicated a
couple of distinct conclusions about quantitative literacyreasoning
amon amongg
psychology majors. First, we noted
consistent patterns of improvement across levels of the major, - suggesting that our current
curriculum does appear to positively benefits
students. However, we regard the
overall levels of performance on these exams are as lower than we desire. Thus, In interpreting
these findings, it is important to note several reasons why the
test scores might be lower than might be expected
in a typical academic examination context.
First, students were not informed in advance that the tests would be
administered and were not encouraged to specifically prepare for these test
questions. Second, student test
performance was not linked to their grades in the courses in any way. Students received extra credit for
completing the tests regardless of their performance on the exam. Thus, their motivation to perform well was lowernot
as high as it would have b than een in
the typical context of testing for a grade.
This means that their scores should not be interpreted in relation to
what faculty might expect of students in a normal testing context. On the other hand, most of the questions
were of relatively low difficulty levels and did not address sophisticated
topics such as hypothesis testing or statistical significance. AlthoughTherefore, while th there are legitimater reasons to
expect students’ performance to be lower than might be expected on a graded
test, we feel see ample room for improvement in
students’ performance on future assessments.
ThereforeIn
response to this challenge, we have engaged in a series of
initiatives designed to improve our quantitative literacy training. These initiatives include:
·
Developing
introductory-level course assignments that actively engage students in
quantitative literacyreasoning
in psychology before they enter
statistics courses taught in the math department. In the past, students received relatively little instruction in
quantitative literacy in their introductory courses and were expected to learn
many of statistics topics in math-taught
statistics courses in which examples were less clearly tied to psychology. To help address this problem, we have
introduced introductory- level
assignments that systematically explore research design, psychological
measurementreliability and validity, and statisticsquantitative
literacy in hands-on student work. Examples include requiring
students to gather their own research data, conduct basic statistical analyses,
and present findings in written form.
The goal of these assignments is, in part, to help students
contextualize the knowledge they receive in their statistics courses and to
help them transfer their knowledge from the statistics courses back into the
psychology curriculum.
·
Expanding
our web-based learning resources related to quantitative literacyreasoning
topics. These efforts
include basic postings of
links to existing resources at various web sites, and
the development of an on-line lab in which introductory psychology students will
conduct and report the results of a complete research
project. The World Wide Web has many
examples of useful statistics resources, particularly for psychology courses. We are capitalizing
on those resources bBy locating, gathering, and
organizing web materialthese resources
for our students, we are simply attempting to be strategic in how
we capitalize on those resources.
·
Improving
our strategic planning with faculty who teach research methods to develop a
standard learning goals for research methods courses and other
courses focused on quantitative reasoning. This partnership involves
efforts to encourage faculty teaching methodology and statistics courses to
more actively participate in assessment research design and to draw from the
departmental assessment planning as they construct and revise their own
courses.
·
Experimenting
with performance-based grading systems in which students must demonstrate
minimum quantitative
literacy QL proficiency levels to receive a B-
grade and with “perform to mastery” systems in which students are given
multiple opportunities to demonstrate proficiency on the same set of quantitative
literacy QL-related topics.
Conclusionns
Each year, we make small but tangible improvements
to the depth and breath of our assessment initiative. Along the way, we have had many opportunities to learn from our
mistakes, and even a few opportunities to benefit from our successes. Perhaps the most important thing we have
come to appreciate is the importance of, and the challenges with, aligning
course content and course assignments with assessment goals. For example, our decision to consciously
focus on quantitative
literacy QL issues required us to add
additional course time to that topic AandND
to cut the amount of time devoted to other topics. These decisions can be complex, emotionally arousing, and even
adversarial if not handled properly.
However, all of the initiatives described above have been implemented to
some degree and in subsequent research, we hope to demonstrate improvements in our
students’ mastery of quantitative literacy.
References
Murray, B. (2002). What
psych majors need to know. Monitor on Psychology, July/August 2002.
APPENDIX A
Tables
Table 1
Psychology Mmajors’ and Nnon-majors’ Qquantitative Lliteracy Pperformance
|
|
Quantitative Literacy Subscales
Proportion of Questions Correctly Answered
|
Psychology Majors
|
Non-Majors
|
|
High Challenge*
|
50%
N=270
|
37%
N = 482
|
|
Low Challenge*
|
60%
N=270
|
50%
N = 482
|
|
StatisticsQuantitative
Literacy*
|
70%
N=270
|
57%
N = 482
|
|
Research Design*
|
57%
N=270
|
46%
N = 482
|
|
Psychological MeasurementReliability/Validity*
|
43%
N=268
|
34%
N = 480
|
|
Total*
|
57%
N |
