Introduction
The
Design of NSSE 2000
Four Sets
of Data Reviewed
The
Five NSSE Benchmarks
How the
University Fared on the Five NSSE Benchmarks
The AAUDE
Consortium Questions
Individual
Questions of Interest to the University
Summary
and Conclusion: What Can We Learn from NSSE
The University conducts surveys which
yield useful data on student satisfaction, activities, and learning
experiences, and this data has been used to improve the undergraduate
experience. But these data tell us nothing about how U.Va. compares
to other colleges and universities. The U.S. News and World Report
college ranking provides comparisons with other colleges and
universities, and U.Va. has historically performed very well, ranking
number two in 2001 among public universities and twenty-first overall.
Such rankings typically are based on student entrance exam scores,
library holdings, alumni giving, faculty-student ratios and the
like. This kind of information has more to do with the assets of
a school -- including its pool of applicants -- than how well it
is teaching its students, or what those students are learning. From
the point of view of college faculty and administrators interested
in institutional improvement, the rankings are more notable for
what they do not provide:
- information about what students
actually do in college, and what they are learning;
- information that can be used to
improve the educational experience of students;
- an answer to the fundamental question:
are our educational practices and curriculum producing desirable
learning outcomes?
The National Survey of
Student Engagement (NSSE) is designed to address these issues. One
unique aspect of NSSE is that it asks students about what they actually
do in college, unlike traditional opinion surveys. These activities
are linked to educational practicies that are known to promote student
engagement, which is defined as "the extent to which students participate
in proven educational processes that contribute to educational outcomes...
such as critical thinking, problem solving, effective communication,
and responsible citizenship," according to The NSSE 2000 Report:
National Benchmarks of Effective Educational Practice. In addition,
by providing the University with comparative data from other institutions
and establishing meaningful national benchmarks, NSSE helps the University
to identify strengths and weaknesses and to identify educational practices
which could be improved.
The University of Virginia
participated in the first NSSE, conducted in the spring of 2000,
and is participating again in spring 2002. This article reviews
the results from the first study. Results from the 2002 survey will
be released in the Spring of 2003.
NSSE was designed by
a team of national assessment experts and administered by Indiana
University's Center for Survey Research. The survey's first
national administration took place in the spring of 2000. A sample
of first-year and fourth-year students at participating colleges
and universities were sent surveys. Copies of the survey
instrument and the complete NSSE
2000 Report are available on-line at NSSE's website.
For more information about the administration of NSSE, go to Methodology.
In addition to the standard questions, NSSE allowed participating
institutions to form consortia, which were permitted to include
20 questions of their own. U.Va. participated in a consortium with
15 other universities in the Association of American Universities
Data Exchange (AAUDE). Click here for
a list of the AAUDE institutions, questions and frequencies.
We have examined four
sets of data compiled from NSSE to determine how the University
fared. The first set of data includes the University's mean scores
and frequencies on individual questions. These results can be helpful
in keeping the magnitude of comparative differences in index measures
in perspective, which are measured in percentiles. In other words,
percentiles do not reveal the absolute level measured by either
an index or an individual question --only how an institution compared
to the whole. Looking at individual question results also provides
more detail to what is shown by the summary measures and benchmarks.Click
here to see the individual question mean scores. Individual
question means were tested for statistical differences, comparing
the University to both AAUDE and the national sample. However, because
of the large number of respondents, many comparisons were statistically
significant. Therefore, the statistical differences were also measured
in terms of their substantive effect by dividing the mean difference
by the standard deviation of the mean of the comparison group, producing
an "effect size" measure between 0.2 and 0.9 (0.2 to 0.5 can be
considered a small effect; 0.5 to 0.8 moderate; above 0.8 large).
All of the University's individual question "effect size" comparisons,
both positive and negative, were between .28 and .58. This means
that the magnitude of the differences between U.Va. students and
AAUDE and students in the national sample was no greater than small
to moderate. This is an important point to keep in mind when considering
abstract index scores in percentile terms.
The second set of data
we examine is the AAUDE consortium questions. None of these questions
were used in the creation of the benchmarks, or the Institutional
Engagement Index. However, some of the AAUDE questions addressed
issues not part of the NSSE but which the University has considered
important in its assessment efforts. Most of these are satisfaction
questions, e.g., satisfaction with academic advising. Click
here to see the AAUDE consortium question frequencies.
The third
set of data is an index created by NSSE called the "Institutional
Engagement Index." This index consists of three measures: the actual
score on the five benchmark indices; a predicted score, based on
an ordinary least squares regression model using data from the Integrated
Postsecondary Education Data System (IPEDS); and a residual, the
result of subtracting the predicted score from the actual score.
The resulting
residual, either positive or negative, might be considered a measure
of "educational effectiveness." The following IPEDS variables, if
available, were used in the regression equation used to calculate
the predicted score: whether the institution was public or private,
admissions selectivity (from Barron's 1999), undergraduate
enrollment, urbanicity, percentage full-time and part-time, sex,
racial/ethnic composition, educational and general expenses per
student from 1995-96 IPEDS, endowment or assets (land, buildings,
and equipment) per student from 1995-96 IPEDS, student-reported
major field, student-reported age, percentage of students who completed
the survey via the web. Click
here to see the Institutional Engagement Index for U.Va.
Finally,
and perhaps most interestingly, NSSE established five benchmark
indices from 40 questions on the survey which measure desired educational
outcomes (critical thinking, problem solving, effective communication,
and responsible citizenship). The five indices were called: Level
of Academic Challenge, Active and Collaborative
Learning, Student Interactions with Faculty
Members, Enriching Educational Experiences,
and Supportive Campus Environment.
Each of the five benchmarks
NSSE created is composed of 6-12 questions and was tested (through
factor analysis) for reliability. More about how the indices were
created and the factor analysis statistics can be found in Appendix
B of the NSSE
2000 Report. Below we describe each of the five benchmarks, and
list the individual questions which make up the index.
Level
of Academic Challenge looks at the quantity and quality of academic
work assigned, the cognitive complexity of work, and the standards
faculty use to evaluate student work. The Academic Challenge Index
was composed of 10 items:
- How many hours per
week a student spent preparing for class (studying, reading, writing,
rehearsing, and other activities related to their academic program).
- How many assigned
textooks, books, or book-length packs of course readings a student
read.
- How many written
papers of 20 pages or more a student wrote.
- How many written
papers of less than 20 pages a student wrote.
- The extent to which
a student's coursework emphasized analyzing the basic elements
of an idea, experience, or theory.
- The extent to which
a student's coursework emphasized synthesizing and organizing
ideas, information, or experiences.
- The extent to which
a student's coursework emphasized making judgments about the value
of information, arguments, or methods.
- The extent to which
a student's coursework emphasized applying theories or concepts
to practical problems or new situations.
- Whether the campus
environment emphasized spending significant amounts of time studying
and on academic work.
- How often a student
worked harder than they thought they could to meet an instructor's
standards or expectations.
Active
and Collaborative Learning attempts to measure the extent to which
students take advantage of opportunities in and outside of the classroom
to actively participate in the learning process. Seven questions make
up this index. Each question asked how often a student had:
- Asked questions
in class or contribute to class discussions.
- Made a class presentation.
- Worked with other
students on projects during class.
- Worked with classmates
outside of class to prepare class assignments.
- Tutored or taught
other students.
- Participated in
a community-based project as part of a regular course.
- Discussed ideas
from their reading or classes with others outside of class (students,
family members, coworkers, etc.).
Student
Interactions with Faculty tries to get at how closely and how
often students interact with their professors. Six items make up the
index and asked how often a student had:
- Discussed grades
or assignments with an instructor.
- Talked about career
plans with a faculty member or advisor.
- Discussed ideas
from their reading or classes with faculty members outside of
class.
- Worked with faculty
members on activities other than coursework (committees, orientation,
student-life activities, etc.).
- Received prompt
feedback from faculty on their academic performance.
- Worked with a faculty
member on a research project.
Enriching
Educational Experiences looks at the opportunities students
have to learn in a diverse environment, both in and outside of the
classroom. Exposure to differing points of view, the use of technology
in the educational environment, and the opportunities to participate
in educational activities such as internships, volunteer work, and
study abroad are all areas which contribute an an enriching educational
experience. The index consists of 11 items:
- How many hours per
week a student participated in co-curricular activities (organizations,
publications, student government, sports, etc.).
- Whether a student
had done a practicum, internship, field experience, co-op experience,
or clinical assignment.
- Whether a student
had participated in community service or volunteer work.
- Whether a student
had foreign language coursework.
- Whether a student
had taken an independent study course or had a self-designed major.
- Whether a student
had participated in a culminating senior experience (comprehensive
exam, capstone course, thesis, project, etc.).
- Whether a student
had studied abroad.
- How often a student
had serious conversations with other students holding religious
beliefs, political opinions, or personal values very different
from their own.
- How often a student
had serious conversations with students of a different race or
ethnicity.
- How often a student
used an electronic medium (e-mail, list-serve, chat group, etc.)
to discuss or complete an assignment.
- The extent to which
a student felt the campus environment encouraged contact among
students from different economic, social, and racial or ethnic
backgrounds.
The
Supportive Campus Environment index looks at the extent to
which the school is actively committed to helping students succeed
academically and socially. This index was composed of six items:
- Whether the campus
environment emphasized providing the support students needed to
help them succeed academically.
- Whether the campus
environment emphasized helping students cope with nonacademic
responsibilities (work, family, etc.).
- Whether the campus
environment emphasized providing the support students needed to
thrive socially.
- Whether students
felt the quality of relationships with other students were friendly
and supportive and promoted a sense of belonging.
- Whether students
felt that faculty members were available, helpful and sympathetic.
- Whether students
felt the relationships with administrative personnel and offices
were helpful, considerate, available, and flexible.
On two of these indices,
Enriching Educational Experiences and Academic
Challenge, U.Va. students were near the top -- not only among
all colleges and universities but also among AAUDE peer institutions.
On two others, Student Interactions with
Faculty Members and Active and Collaborative
Learning, the University did not fare so well. The fifth benchmark,
Supportive Campus Environment, produced
mixed results. We examine each of the benchmarks and the University's
performance. For each, we have created separate tables of frequencies
for the individual response items. These charts indicate the frequencies
for U.Va. first-year and fourth-year students, as well as the national
and AAUDE frequencies for first-year and fourth-year students.
Fourth-year University
students were in the 90th percentile of the Enriching
Educational Experiences index. The participating AAUDE institutions,
in comparison, scored between the 50th and 60th percentiles on this
index. On this same measure, using the Institutional Engagement
Index, the University also fared well, moderately outperforming
what one would expect, given the University's IPEDS data. First-year
U.Va. students also scored highly on the Enriching Educational Experiences
index, between the 80th and 90th percentile, and somewhat higher
than AAUDE insitutions, which scored in the 60th-70th percentile.
However, on the Institutional Engagement Index, the score for first-year
students was a bit below what was predicted by the IPEDS data.
In examining
the University's mean scores on the individual questions which make
up the enriching experiences index, three questions stand out:
- How often a student
had used an electronic medium (e-mail, list-serve, chat group,
etc.) to discuss or complete an assignment.
- How often a student
had serious conversations with students with religious beliefs,
political opinions, or personal values very different from theirs
- How many hours per
week a student was participating in co-curricular activities (organizations,
publications, student government, sports, etc.).
On
each of these questions, for both first-year and fourth-year students,
the University outperformed the national sample and these differences
were statistically signficant, though substantively small to moderate.
For the question about having conversations with different types
of students, the University also outperformed the AAUDE group for
both first-year and fourth-year students. Sixty-five percent of
University fourth-years had serious conversations with students
whose backgrounds differed from their own, compared to 48% of AAUDE
fourth-years and 45% of all fourth-years.
For the question on
participation in co-curricular activities, the University did moderately
better than AAUDE but only for fourth-year students. Only 36% percent
of University fourth-years participated in co-curricular activities
for five or fewer hours each week; 62% of AAUDE fourth-years and
64% of national fourth-years participated for five or fewer hours
each week. Click here for a chart which
shows the eleven individual question frequencies for the Enriching
Educational Experiences index.
Academic
Challenge was another index in which U.Va. students, both first-year
and fourth-year, did well, both groups scoring in the 70th percentile
overall. Among the 46 universities participating in the 2000 NSSE
with the Carnegie Classification "Doctoral/Research Universities-Extensive,"
U.Va. was selected as one of four potential "exemplars," relative
to its peer institutions.
For this
index, six individual questions stand out, meaning that statistically
significant differences with either the AAUDE group or national
sample were identified. The six questions were:
- How much time a student
spent preparing for class.
- The extent to which
students' coursework emphasized making judgments about the value
of information, arguments, or methods
- The extent to which
students' coursework emphasized synthesizing and organizing ideas,
information, or experiences
- The extent to which
the campus environment emphasized spending significant amounts
of time studying and on academic work
- The number of written
papers or reports of fewer than 20 pages
- The number of assigned
textbooks, books, or book-length packs of course readings.
U.Va. first-years and
fourth-years both reported reading a higher number of assigned textbooks,
books, or book-length packs of course readings than either AAUDE
students or the national sample.
About two-thirds of
first-year and fourth-year U.Va. students read more than 10 textbooks,
books, or book-length packs of course readings in the past year.
For AAUDE first- and fourth-years, about half the students had read
more than 10 and for the national sample a bit less than half had
done so. Click here for a chart
which shows the 10 individual question frequencies for the Level
of Academic Challenge Index.
U.Va. did not fare well
on two of NSSE's benchmarks. Student Interactions
with Faculty Members and Active and Collaborative
Learning.
First-year U.Va. students
scored in the 10th percentile on the student-faculty index, and
fourth-year students in the 30th percentile. First-year students
from the AAUDE group scored a bit higher than U.Va. and fourth-year
students from the AAUDE group scored a bit lower than U.Va. One
in five first- and fourth-year U.Va. students had talked about career
plans with a member of the faculty "often" or "very often," compared
to about one in three students from the national sample. Click
here for a chart which shows the six individual question frequencies
for the Student Interactions with Faculty Index.
First-year U.Va. students
scored in the 20th percentile on the collaborative learning index,
and fourth-year students in the 10th percentile. Students from the
AAUDE group scored similarly or a bit lower than U.Va. When asked
how often they had made a class presentation in the past year, 24%
of U.Va. first- and fourth-years said "often" or "very often," compared
to 43% of the national sample. Click
here for a chart which shows the seven individual question frequencies
for the Active and Collaborative Learning Index.
On the fifth benchmark,
Supportive Campus Environment, U.Va.'s
results were mixed. U.Va.'s actual scores significantly exceeded
its predicted scores for both first- and fourth-year students. Click
here to see the Institutional Engagement Index results.
U.Va.'s Supportive
Campus Environment benchmark scores for both first-years and
fourth-years were higher than AAUDE institutions' scores. On one
of the questions making up the benchmark about how much support
students felt they received from the campus environment for "thriving
socially," 48% of U.Va. first- and fourth-years said the campus
environment provided the support they needed to thrive socially
"quite a bit" or "very much," while 35% of the AAUDE students and
37% of the national sample felt similarly. However, despite this
individual result and despite exceeding what was predicted on the
Institutional Engagement Index, U.Va.'s Supportive
Campus Environment was no better than average nationally for
first-year students (the 50th percentile) and below the national
average for fourth-year students (30th percentile). Click
here for a chart which shows the six individual question frequencies
for the Supportive Campus Environment Index.
Members of the AAUDE
consortium added twenty questions to the 2000 NSSE that were included
as a final supplement to the NSSE survey. A list of the questions
and table of frequencies, as well as a table of means, including
the results of statistical significance testing, are available by
clicking here. For the purposes
of this analysis, we are only reporting those questions in which
a statistically significant difference between the University and
the rest of the AAUDE group, for either first-year or fourth-year
students, was moderate or higher in its effect size. Four questions
met this criteria:
- How a student rated the academic
quality of the university in general.
- How a student rated the quality
of academic advising they received from their college or department.
- The
source, during the past year, from which the student received
most (emphasis added) of their academic advising.
- When
and if the student expected to complete their bachelor's degree.
We look at each question in turn. Both
first-year and fourth-year students rated the University's academic
quality in general very highly, moderately higher than students
from the rest of the AAUDE group. Sixty-eight percent of U.Va. first-years,
compared to 36% of AAUDE students, rated the academic quality "excellent."
The same comparison for fourth-year students is 55% of U.Va. students
and 25% of AAUDE students. U.Va.'s high rating in academic quality
did not carry over to academic advising. Only 33% of U.Va. first-year
students felt the quality of the academic advising they received
was "good" or "excellent." Fifty-eight percent of first-year AAUDE
students felt similarly. There was not much difference between U.Va.
and AAUDE fourth-year students' ratings of advising.
Another disparity between U.Va. and
AAUDE first-year students is the source from which they received
most of their academic advice. Only 19% of U.Va. first-years
claimed they received most of their advice from their advisors,
compared to 49% of AAUDE first-years. Two-thirds of U.Va. first-years
said they received most of their advice from either the
"undergraduate catalog or other publications" or "friends or family."
Only 37% of AAUDE first-years said likewise.
When and if a student
expected to receive their bachelor's degree was the question in
which the strongest difference between U.Va. and AAUDE was observed.
Ninety-three percent of U.Va. students (first- and fourth-year combined)
felt they would receive their bachelor's degree within 4 years,
compared to only 54% of AAUDE first- and fourth-years. All in all,
the AAUDE questions were good news for how the University fared
on the NSSE. The exception was academic advising; the University's
first-year advising system is not highly regarded or utilized by
its students.
Several individual questions
are of particular interest to the University. In answer to the question,
"how would you evaluate your entire educational
experience at this institution?" 52% of U.Va. first- and fourth-years
rated their experience "excellent," compared to 28% for AAUDE and
34% for the national sample.
Another
area which has received much national attention lately is how much
time students spend studying. When asked how much time they spend
preparing for class (studying, reading, writing, rehearsing, and
other activities related to your academic program), slightly more
than a third of U.Va. students claimed to be studying more than
20 hours per week. About 25% of AAUDE students and the national
sample were studying that much. Conversely, a bit less than a quarter
of U.Va. students were studying less than 11 hours per week, compared
to more than a third of AAUDE students and the national sample.
These numbers are both good news and bad news for the University.
While University students are spending more time studying and preparing
for class than AAUDE institutions and the national sample, still
nearly a quarter of first- and fourth-year students surveyed were
studying an average of only an hour and a half a day.
Another
question that might be of interest to University faculty and administrators
is how much a student's college education contributed to the student
"being honest and truthful." University first- and fourth-year students
gave more credit to the University in this regard. Nearly three-quarters
of U.Va. students said the University contributed "quite a bit"
or "very much." Fifty-four percent of AAUDE students and 61% of
the national sample said likewise.
NSSE, while only a few
years old, has been thoroughly field-tested. A pilot survey was conducted
in 1999 to test the instrument and the reliability of the five indices.
After the 2000 survey, extensive psychometric analyses were conducted
to examine the validity and reliability of the survey questions. Focus
groups were organized with respondents to determine if the questions
were being consistently interpreted by respondents and whether respondents
are interpreting questions as the designers intended. A few problems
were reported. For example, the question on "number of books read
on your own" caused some confusion among students who did not know
if this item referred to books read for pleasure or to supplement
class readings. The vast majority of questions, however, were found
to be "clearly worded and easy to complete." Conceptual Framework,
Interpreting the Meaning of Engagement Items: Results from Student
Focus Groups, NSSE College Student Report 2001. NSSE did not change
the wording of the vast majority of questions in the 2001 instrument
but rather included additional information to clarify some questions.
A few wording changes were made. NSSE staff even conducted a telephone
survey of nonrespondents to see how they may have differed from respondents.
Interestingly, nonrespondents were a bit more engaged than respondents.
But the differences were not large, and one must be cautious about
interpreting this result, since telephone interviews sometimes have
been shown to cause higher scores than paper surveys. A more thorough
examination of the nonrespondent issue can be found on NSSE's website
presenting its Nonrespondent
Analysis for the 2001 survey. In addition to the psychometric
testing, focus groups and nonrespondent analyses, NSSE staff compared
the results of the institutions that participated in both the 1999
and 2000 surveys and the 2000 and 2001 surveys. Overall, the correlations
for common questions were high, though some were higher than others.
See
"Estimates of Stability," in the "Conceptual Framework" section of
the 2001 report.
Despite the sound methodology
and substantive interest of NSSE, there are still methodological
concerns, as well as substantive problems that need to be addressed.
In the area of methodology, the primary concern seems to be around
the relatively low response rates for the surveys. Less than half
of those surveyed responded and these respondents did differ somewhat
from the populations. Women are overrepresented in the national
samples and part-time students are underrepresented. Weighting can
reduce this bias. However, if respondents differ in other ways that
cannot be accounted for in weighting, the consequences are more
serious. The nonrespondent analysis mentioned earlier indicated
that nonrespondents were slightly more engaged than respondents,
meaning that student engagement is underestimated by NSSE. This
problem is mitigated by the comparative value of the survey. If
the nonrespondent bias is distributed evenly across institutions,
the bias would cancel itself out, though the absolute level of engagement
might still be understated. Still, higher response rates both at
U.Va. and nationally would diminish the nonrespondent problem. For
the 2002 survey, the University is offering incentives to students
taking the survey, the single most effective means of increasing
response rates on surveys.
Beyond methodology,
some have criticized the value of NSSE because the more elite institutions
are less likely to participate. Colgate University has stated that
it will not participate in future surveys because its peer institutions
are not participating, according to an article in the Chronicle
of Higher Education, Nov. 17, 2000. For U.Va. this is somewhat
of a problem also. Berkeley has chosen not to participate, as well
as the University of Pennsylvania. Other top-tier state universities
have participated, most notably the University of Michigan. The
AAUDE institutions on the whole are not U.Va. peers. However, even
if the comparative value of the survey is not ideal, the University
still can use NSSE to identify its strengths and weaknesses. For
example, we have learned that University students do not have as
much contact with faculty as they do at other institutions. While
this might also be the case for peer institutions, it nevertheless
is a worthy goal for the University to improve and increase student-faculty
interaction. Another substantive criticism that has been leveled
at NSSE has more to do with how it is used. The results
for individual colleges or universities are not made public, except
by permission of the participating institution (most don't give
it). Prospective students and parents cannot use NSSE as an alternative
to the U.S. News rankings because so few schools make the
results public.
On the whole, the NSSE
results have identified several significant strengths at the University.
- The quality of the
faculty and administration has created a challenging, rich and
diverse academic environment for both its first-year and fourth-year
students.
- The
University has created an outstanding extracurricular environment
and is exceeding expectations in establishing an environment supportive
to its undergraduates outside of the classroom.
The results of NSSE
have also pointed to some areas for improvement.
- While the supportive
environment is higher than what one might expect, given the University's
size and other characteristics, this level of support is only
about the same as the national average for first-year students
and below the national average for fourth-year students.
- The
area most in need of improvement, based on its identified importance
to undergraduate student engagement, is in the area of faculty-student
interaction. Students do not have enough positive interactions
with faculty, especially first-year students whose score was near
the bottom nationwide.
- While fourth-years
fared better compared to other institutions, their score on the
student-faculty index was still lower than the national average.
Moreover, U.Va.'s scores on student-faculty interaction were below
expectations in this area for both first- and fourth-year students.
The NSSE area in U.Va.
did least well was the active and collaborative
learning index, which attempts to measure the extent to which
students take advantage of opportunities in and outside of the classroom
to actively participate in the learning process. For both first-
and fourth-year students, the University ranked significantly below
average, and less than expected. The importance of such learning
experiences in a large research university can be debated. When
looking at some of the measures making up this index--making a class
presentation, working with classmates outside of class, tutoring
other students--one can argue that such experiences are not only
less feasible in a large research university, they might also be
less appropriate in a University whose students are as highly regarded
as U.Va.'s. Nevertheless, other aspects of the index--participating
in a community-based project as part of a course and discussing
ideas from readings or class with others outside of class--would
still seem to be areas of potential weakness that the University
would not want to ignore.
The NSSE is a thoroughly-tested
and informative alternative to the U.S. News rankings from
the point of view of an institution that is looking for ways to
improve the undergraduate educational experience. To that end the
University community should take note of its strengths and weaknesses.
The good news for the University is that its strengths outnumber
its weaknesses, and the amount of change required for improvement
does not seem to be outside the realm of possibility.
For more information on this article or NSSE, contact Jonathan
Schnyer, Associate Assessment Coordinator, Office of Institutional
Assessment and Studies.
IAS
Reports: National Survey of Student Engagement |
|
|
