For work that cannot be evaluated with complete objectivity (e.g, multiple-choice exam), it is wise to develop a scoring rubric/guide that improves consistency among raters and provides useful information for those examining the results. Rubrics are used to assess complex student work, such as papers, essays, or presentations. Rubrics guide reviewers to make outcome-specific assessments and to do so using pre-determined, concrete criteria.

Please see our rubrics page for rubric examples and information on creating and using a rubric.

It is useful to create a timeline, with deadlines and meeting dates for your assessment (when will student's work be collected and analyzed?). In addition, it is important to determine in advance who will conduct the scoring and analysis (faculty or graduate students). If rubrics are used, it is recommended that at least two reviewers complete rubrics on each student's essay, report, or presentation. If the two reviewers disagree substantially, a third review may be advisable. Additional issues to consider are discussed below.

Sampling
One of the first decisions to make when gathering data is whether to assess all students or a sample of students.

If you decide to sample, there are two considerations to keep in mind. First, you must determine the appropriate sample size. The appropriate sample size will depend on the size of the population being sampled, the acceptable margin of error, and the desired level of confidence. This sample size calculator will assist you in determining the appropriate sample size. Second, you also need to consider what type of sampling procedure you will employ. WIll you sample a random number of students across all classes/sections? A set number of students from each class/section, selecting, for instance, every "nth" student on the roster? All students from one class/section? Feel free to contact IAS if you are unsure which sampling procedure to adopt.

Protecting confidentiality
Students will inevitably be worried about how assessment results will be used and how they may affect their grade; thus it is important to protect student confidentiality and communicate clearly the purpose and use of the data to them.

Qualitative Data
In cases where the use of a rubric or an objective test is not appropriate, exit interviews, focus groups, course evaluations, or survey questions may be used. Direct methods to assess student learning are preferable, but these indirect methods of assessment may still provide much needed information. It is useful to develop a protocol/prompt/question list to ensure some consistency in the questions.

Quantitative Data
There are many options available for tabulating quantitative data:

1-ITC provides support for scanning bubble-sheets.
2-Data entry in Excel.
3-Other options available at no charge from ITC are: Mini Tab, SPSS, and SAS.

Unfamiliar with quantitative data analysis? The Scholars' Lab at Alderman Library is an excellent resource, as is ITC's Research Computing Center. IAS is also available to help. Contact IAS at 924-3417 or Data Support.

This step may be a challenge for program coordinators unfamiliar with quantitative and qualitative research methods.

Qualitative Data
Generally direct measurement of students learning outcomes should be favored over indirect measures such as surveys and focus groups. However, when only qualitative data is available (text comments, interviews) qualitative methods such as quick read-throughs, tallies, thematic analysis methods can be used. Note that in many cases survey data, such as questions where students are asked to answer on a scale of possible responses (e.g. very satisfied to very dissatisfied), can often be analyzed using quantitative methods.

Quantitative Data
Analyzing quantitative data allows coordinators to determine whether students have met specific criteria set in the assessment plans (for example, whether or not 80% of students are “competent” in oral communication) or to establish baselines for future assessments.

In addition, at this step coordinators should examine the validity and reliability of the assessment tool. More specifically, did the methods used measure what you intended (validity) and are the methods likely to yield the same findings each time they are employed (reliability)? This process need not be sophisticated, including the use of advanced statistical analysis. Rather, it is important to take the time to reflect on the assessment process.

Analyzing quantitative data in MS Excel
More on validity and reliability

Reports should be kept as simple and short as possible with a focus on what will be useful and meaningful to the audience of the report, most often faculty members. A clear connection should be made between the stated outcomes, standards/criteria, results and analysis. A good example of program-level assessment reporting is the UVa Civil Engineering Program Report.

Developing an action plan/reponse to assessment results
Assessment is an iterative, ongoing process. First, once data are gathered at different points in time, it becomes possible to determine whether students' knowledge and skills have improved over time. Second, rarely does the first effort answer all your questions; rather, each iteration may raise additional questions or even concerns. Thus, your goal should not be to make major changes based on one result; rather, it is important to ask:

•Are more data needed? •Would a pilot test be useful? •Should the assessment tool be adjusted? •Should teaching practices be reviewed?

Thoughout this process, the involvement of faculty is critical (did faculty find the results informative?). Student input should be sought and considered as well. Both of these audiences are critical to solving identified problems.

Program coordinators should use the results to plan for future assessments, including determining what resources are required for improved assessment. You may decide to maintain your focus on specific outcomes and investigate your findings further, or move on to assess other outcomes.

Assess the Assessment
Finally, it is important to reflect on the process itself (how can it be modified to be more efficient, less intrusive for the faculty and students, etc).

In a new worksheet, place the name of the variables on the first row (these are likely to include a type of student identification, grader identification if there are multiple graders, a score for a particular learning outcome). Then, entries for each cases can be entered directly in the worksheet.

There is an alternative method to enter data in Excel that may be easier and decrease the likelihood of mistakes:

Alternatively, you may use the template and instuctions created by the IIAS.

(Back to Gathering Data)

One can easily produce useful descriptive statistics and charts with Excel, that is, Excel can provide mean scores, standard deviations and percentiles. Follow the steps below to obtain the necessary statistical package (Analysis ToolPack) to obtain these options:

1-In the top Menu, select Tools, then Add-ins.
		2-In the dialog box that appears (shown here), check the boxes "Analysis ToolPak," and "Analysis ToolPak VBA" 3-Click OK.

4-After these steps have been successfully completed a new option should be available in the drop down menu "Tools," named "Data Analysis."

5-The most useful functions that appear in the dialogue box will be "Descriptive Statistics" and "Rank and Percentile" (see below).

6-Once either function has been selected another dialogue box will appear asking you to select the range of data that you are interested in analyzing.

Below are the dialog box and results when "Descriptive Statistics" is the option selected.

7-Enter the set of data for analysis ("Input Range"), select the box "Summary statistics" and then click OK.

8-You will obtain the following information (numbers in the table are illustrative only):

(Back to Tabulating and Analyzing Data)

For help with data entry or analysis, please contact IAS by phone (4-3417) or email (iaas@virginia.edu).