Introduction

Universal Student Ratings of Instruction (USRI) course evaluations gather feedback from classes to help instructors, departments and faculties improve curriculum and instruction. 

This article will describe the use of USRI results Reference Data.

---

Applicability

This article was written for deans, chairs, students and instructors at the University of Alberta.

---

Details

The columns of reference data display statistics from Tukey's box­ and­ whisker plot analysis(John W. Tukey, Exploratory Data Analysis, Addison­Wesley Publishing Company, Inc. 1977).The values displayed are derived from all the classes in the indicated reference group.

These statistics are chosen to achieve two main objectives:

1. summarize skewed distributions of data, and

2. identify outliers from the general population if they exist.

The median value (middle of a ranked set of numbers) is generally preferred over the mean to identify the centre of a skewed distribution of scores. This is the value below which 50 percent of the medians from other classes lie. Please note that data for the items in the current set of mandated questions are accumulated from Academic Year 2005/06 and beyond.If an item(question) has not been used at least 15 times by the indicated reference group since then, the reference data cells will be filled with the text: "too few uses". It is theoretically possible for all median scores in a single year to be above, or below, the Reference Group median.

The 25th and 75th percentiles provide information about the spread of scores around the median. By definition, twenty­-five percent of the scores are above the 75th percentile and twenty­-five percent are below the 25th percentile. Since this occurs by definition, these valuesshould not be used to determine whether a particular score is good or bad. The lower Tukey Fence, which is the 25th percentile minus 1.5 times the distance from the 25th to the 75th percentile, defines a reasonable limit beyond which a score can be considered an outlier. Outliers are scores that appear to be outside the usual distribution of scores for the population being tabulated, i.e., for the indicated reference group. Given the nature of the USRI data, the upper Fence will usually be above 5.0 and, therefore, need not be reported.

Please note that some items can be expected to elicit higher ratings because they are closer to apple pie types of items, i.e., we would expect the item to be rated quite positively. This is illustrated by the campus-­wide results accumulated in the years 2000­2004 for the two items shown below.

Item	Tukey Fence	Reference Data
		25% 50% 75%
The instructor treated students with respect.	3.4	4.3 4.6 4.8
Overall, the quality of the course content was excellent.	2.9	3.8 4.1 4.3

 

This suggests that the median obtained for the first item in a particular class can be expected to be 0.5 of a rating above that for the second item simply because that has been found to be the case in results from thousands of classes surveyed at the University of Alberta. Note that the 25th percentile for the first item corresponds to the 75th percentile for the second item. Also, the reference group used for a particular class consists of all classes in the indicated department or faculty. One of the most consistent findings of researchers studying students' ratings of instruction is that the ratings obtained for items such as those addressing general satisfaction with a course or instructor depend on the discipline in which the course is taught.

Franklin and Theall (1995) reported that "Professors in fine arts, humanities, and health-­related professions are more highly rated than their science, engineering and math­-related colleagues." There appears to be a combination of reasons for these differences including, differences in the characteristics of the students, in the nature of the subject matter, and in the course objectives that are emphasized in different disciplines. The sizes of the differences and the conclusion that they are not necessarily related to characteristics of the instructors in the different disciplines leads to the advice that "we must continue to be very cautious about, ­­if not prohibited from­­ using, the results of student evaluations to make comparisons across disciplines" (Marincovich,1995).

For example, the item "Overall, this instructor was excellent." illustrates that results at the University of Alberta are consistent with the research studies. The reference data from some of the departments in which a large number of classes have been surveyed appear in the following table.

Department	Tukey Fence	Reference Data
		25% 50% 75%
Physics	2.4	3.7 4.1 4.5
Computing Science	2.5	3.7 4.1 4.5
Electrical & Computer Engineering	2.7	3.9 4.2 4.6
Mathematical & Statistical Sciences	2.8	3.9 4.2 4.6
Earth & Atmospheric Sciences	3.0	4.0 4.3 4.6
Biological Sciences	3.1	4.0 4.3 4.6
English	2.8	4.0 4.4 4.7
Modern Languages & Cultural Studies	2.9	4.0 4.4 4.8
History & Classics	3.4	4.2 4.5 4.7
Elementary Education	2.7	4.0 4.5 4.8
Drama	2.9	4.1 4.7 4.9

References

• Franklin, J., and Theall, M. "The Relationship of Disciplinary Differences and the Value of Class Preparation Time to Student Ratings of Teaching." in N. Hativa and M. Marincovich (eds.), Disciplinary Differences in Teaching and Learning: Implications for Practice. San Francisco: Jossey­Bass, 1995.
• Marcinovich, M. "Concluding Remarks: On the Meaning of Disciplinary Differences." in N. Hativa and M. Marincovich (eds.), Disciplinary Differences in Teaching and Learning: Implications for Practice. San Francisco: Jossey­Bass, 1995.

Keywords: tsqs, reference data, USRI reports