Stark and Freishat (2014) are pretty negative about student evaluations of teaching. To be fair the more you study evaluations the more problems you see. I agree that we have a problem with bias in student evaluations of teaching. Still many of the criticisms of evaluations, however, seem to be about the general problems of surveys. Other criticisms seem to be about poor tenure decision processes. Are student evaluations flawed? Yes, but I’m not sure their eight recommendations are a fully thought through solution. Therefore, here is me evaluating evaluations of evaluations.
Recommendations For Change
The authors have eight broad recommendations. I will give the recommendations followed by my comments. To reiterate I want to see change but I’m nervous about some of the proposed changes.
Sensible Things To Note
Recommendation 1) Drop omnibus items.
I agree that general “teaching effectiveness” measures can be misleading. They may be influenced by inappropriate factors but what measures aren’t biased in this way? The authors need more detail on why omnibus measures are uniquely bad.
Recommendation 2) Don’t average or compare averages.
Instead, report the distribution of scores, the number of respondent, and the response rate
Stark and Frieshat 2014, page 20
Average have problems. Yet averages help simplify and for this reason are standard in commercial and academic research. As academics regularly report averages in our research we can’t really complain about the use of averages in evaluating us. As to also reporting distributions I’m all for this. Let us do more of this.
Recommendation 3) Low response rates aren’t good.
This is true. Still this doesn’t necessarily mean student evaluations are bad. Low voter turnout is bad but let’s not abandon democracy. The problem here is surveys with low response rates. How can we boost response rates?
More Sensible Advice But It Is A Bit Generic
Recommendation 4) Look at student comments but understand their limitations.
Agreed, hopefully people are already doing this. I think faculty are already quite aware that student comments can vary from insightful through plain strange to unfair and offensive.
Recommendation 5) Beware comparison
Avoid comparing teaching in courses of different types, levels, sizes, functions, or disciplines.
Stark and Freishat 2014, page 20
Often professors are the only ones teaching that course. Do the authors think the scores cannot be compared to anything else? It is true that less comparable things are harder to compare. Still, I’d hate to abandon all assessment of teaching effectiveness and all courses differ in some way. To assess what works best you must compare something. You can do this while acknowledging courses are never totally comparable and the comparison is far from perfect.
Are Faculty Colleagues Better Than The Students?
Recommendation 6) Teaching Portfolios Should Be Used
Use teaching portfolios as part of the review process.
Stark and Freishat 2014, page 20
The authors worry that tenure decisions are made on just average student evaluation teaching ratings. I agree. Doing this is terrible. This is the biggest problem to my mind. Tenure is a massive decision so deserves considerable effort. Smaller decisions don’t justify as much effort. As such, is the author’s advice about all review processes? Or is it just about tenure decisions? I want to know more on the role that student evaluations should have in different decisions some of which are more consequential than others?
Recommendation7) Classroom observation is recommended.
Classroom observation seems potentially useful but far from a perfect solution. Academic observers usually have more subject knowledge than students. Still, in elective subjects especially, they often have much less knowledge than the teacher being observed. This means the flaws remain. The observers don’t really know whether what they are learning is good or not.
There aren’t as many colleagues who can observe. Observers also typically spend a lot less time in the classroom than the students. There are, therefore, fewer observations from fewer observers. On a practical matter this likely means that the idiosyncratic mistakes/bias are likely to be larger. If you have a colleague with major bias doing classroom observation of teaching effectiveness you have a big problem. The review might be a lot more biased than that of a classroom of students who have bias of varying degrees. If you have perfect colleagues that is wonderful. If you don’t classroom observation is hardly problem free.
Why Don’t We Spend Longer On Reviewing Performance Already?
Recommendation 8) The authors suggest evaluators spend more time looking at materials and observing during reviews.
Using more information will probably improve assessment but has downsides. For example, rather obviously extensive reviews take a lot of time. My guess is that the academic review process is often bad not because everyone thinks the way it is done is perfect. Instead, it is bad because no one invests the time to improve them.
If extensive reviews are not being done at the moment then why is this? Time pressure? This suggests that the challenge is not only with the students. Colleagues don’t want to spend time reviewing their colleagues because, to be honest, it doesn’t help them personally. We need to consider the whole issue of academic management. A tenure decision made solely on student evaluations is a badly made decision. We do need to find some way to motivate more effort in the process. Exhortations that professors spend time helping their colleagues are nice. Yet if professors aren’t doing that already I doubt another platitiude will change much.
Evaluating Evaluations Of Evaluations
There are very reasonable elements of the critiques of evaluations. Still, I want to know more about the changes proposed. I have two big worries whenever people talk about getting rid of student evaluations.
Firstly, I worry that these recommendations transfer power to senior faculty from students. This may be great if you are lucky in your senior faculty. Still many universities do not have perfect senior faculty. Many senior faculty have no HR training. Many would not know employment law if it bit them. Absent other changes will getting rid of student evaluations even improve the problem of bias? I’m personally not that optimistic.
Secondly, I fear that academics are too insular. Our key metrics of success are papers/citations. I.e. how much we have impressed our academic colleagues. Student evaluations are one of the few ways outsiders can penetrate the academic bubble. Any solution that involves academics being more involved in evaluating academics makes me nervous. We need outside input. I believe that students really need some way to share their ‘thoughts’. (Even if sometimes those thoughts are biased, offensive, or just plain silly).
So when we are evaluating evaluations of evaluations we should remember to compare any solution to the possible alternatives.
For more on teaching ratings see Bribery Works With Students – Marketing Thought (neilbendle.com).
Read: Philip B. Stark and Richard Frieshat (2014) An Evaluation of Course Evaluations, available here