Illusions of learning - the problem with student evaluations

"students_lecture_event" (CC BY-ND 2.0) by University of Minnesota Duluth

One of the problems with asking students to evaluate their courses and teachers is that the surveys often focus on a false customer-supplier relationship. In the commercial sphere companies want to find out if their customers are happy with what they have bought and this indicates a successful transaction. But education is far more complex and is not simply a matter of giving the customers what they want. In many cases the effects of education are delayed and cannot be assessed at the end of a course in terms of customer satisfaction. If you think back to your own education you may admit that the best teacher you had was one you hated at the time. A teacher who demanded a lot, criticised you, pushed you and challenged you. After the course you would have handed in a scathing evaluation but now, years later, you realise that you learned so much more from that experience than from the friendly teachers who made life easy and gave you a good grade.

We ask students to evaluate our pedagogy without helping them understand the process and this results in rather superficial evaluations that say very little about how much they actually learned from the course. This is the topic of a fascinating article in the Journal of Applied Research in Memory and Cognition, On Students’ (Mis)judgments of Learning and Teaching Effectiveness. Sadly this article is only available if your employer pays a subscription for access to the journal but I will try to convey the gist of it here.

The authors discuss at length what they call illusions of learning. In their studies they discovered that students tend to give positive reviews to teachers who are enthusiastic and provide easily digested material that helps them pass the tests; an understandably pragmatic approach. They base their evaluations on superficial elements such as the teachers' appearance, age, gender, accent, enthusiasm, presentation style, and use of digital tools, none of which have any real relevance for the student's learning. Furthermore, the students showed a clear preference for traditional teaching methods like lecturing and believe that the learn more from traditional teaching than from active and collaborative learning, even when test results suggest the opposite.

Students believe, for example, that they learn best from enthusiastic and engaging instructors who provide smooth and well-polished lectures that do not require active class participation. Such factors, although they readily inflate students’ judgments of their own learning, do not consistently enhance students’ actual learning. They also inflate students’ evaluations of the effectiveness of their instructors. Indeed, students’ evaluations of teaching effectiveness can be poor predictors of their actual learning in their courses, and these evaluations can be biased by external factors unrelated to student learning, such as an instructor's gender, age, attractiveness, and grading leniency.

One section of the article has the heading: Do Student Evaluations Incentivize Poor Teaching Practices? If the university gives too much weight to student evaluations then more challenging teaching methods will be discouraged in favour of strategies that enhance customer satisfaction. Teachers who push and challenge students and encourage them to investigate topics for themselves risk poor evaluations and possible reprimand. To get better evaluations, teachers will be encouraged to revert to traditional methods and we are then stuck in a vicious circle. The students will have the illusion of learning and the teachers will have the illusion of teaching. Everyone will be happy but how much have they learned?

We propose that faulty metacognition is a key contributor to the problem. Students’ misevaluations of teaching effectiveness can be driven by the same factors that underlie their misjudgments of their own learning. Although they do not enhance student learning and can even impair it, teaching approaches that minimize effort and create the appearance of a smooth, well-polished, fluent, and enthusiastic instructor readily boost students’ subjective impressions of what they have learned and their perceived effectiveness of that instructor. Because these subjective impressions are the primary basis for determining teaching effectiveness, and as such are a key metric used for decisions about hiring and promotion, instructors are currently incentivized to adopt teaching approaches that may produce illusions of learning that boost their ratings but can actually undermine students’ learning.

Sadly there is no easy solution to this problem except widening the scope of evaluations to include multiple perspectives and helping students to make more informed evaluations of their learning. Given the workload for many students this will not be easy and the authors see no clear solution at present. At least we now have evidence that the system requires a radical rethink and that awareness is an improvement.

It is hoped that future conceptualizations of teaching effectiveness include research-based evidence for improving student learning and metacognition as a strong basis in formulating measurements that accurately and reliably reflect the quality of teaching.

Reference

Carpenter, S. K., et al. On Students’ (Mis)judgments of Learning and Teaching Effectiveness. Journal of Applied Research in Memory and Cognition (2020), https://doi.org/10.1016/j.jarmac.2019.12.009

Deconferencing or unconferencing?

"Audience" (CC BY-NC-ND 2.0) by NordForsk

Educational conferences are big business and whatever field you work in you'll find a conference somewhere every week of the year. Take a look, for example, at Contact North's excellent overview of conferences in educational technology for the year ahead - a breathtaking 1700 of them! All of them involving a hefty carbon footprint in terms of air travel. As I've written before, we can't go on meeting like this.

Now there's a growing interest in deconferencing; reducing the number of conferences you attend, ultimately to zero. This is not just about reducing our climate anxiety, there is also the question of what you actually gain from attending conferences. Are the benefits to you and your organisation worth the time and effort or do you go simply because you are expected to or simply due to habit? After a while you realise that you have heard most things before and the number of eureka moments per conference is alarmingly low. Over the last year I have been trying to only attend conferences that can be reached by train. One advantage of this is that you have to think carefully before signing up for a conference. Is it really worth those lengthy train journeys and a lot of leisure time sitting on trains and waiting at stations? I'm also aware that train journeys are not always environmentally friendly, especially if the trains are hauled by diesel locomotives. But it's not just about cutting the air miles, it's about rethinking the meaning of conferences and trying to finding new arenas to meet.

I can recommend an excellent post on the topic by Alan Levine (aka CogDog), On deconferencing. He has withdrawn from the conference circuit and explains the reasons and it's not just about carbon footprints. It's also about the exclusivity of big conferences. Many of us who get paid to attend conferences tend to take it all for granted and seldom stop to think how privileged we are. We are part of an exclusive club and we tend to forget that there are millions of academics who cannot afford to attend or face visa restrictions and other barriers even if they can afford the fees. He admits that he was one of the club:

It was for work. It was just part of the job. Even later, when going independent, others still paid my fares in exchange for presenting/workshopping. I earned it, right? And it felt, yes, a bit glamorous. And I was there to to tweet out all the foibles of travel woes, missed planes, rude TSA agents, bland food.
Never thinking about how that looked to someone who did not get such opportunities.
As a road warrior, I was so… justified.

However his main reason for unconferencing is the search for deeper discussion and the limitations of the traditional conference format:

Even now, I picture these large conference halls where most folks are there tweeting slides. 

We need to rethink our conferences or better still devise new arenas for meaningful, accessible and inclusive discussion with less dependence on airlines and expensive hotels.

Conference on… but I am deconferencing. I am looking for better ways to share knowledge, ideas that can include more people and less travel, but just plain… better.

Experiments with digital conferences are promising but we shouldn't focus only on the digital arena. The concept of the unconference has been around for many years and is a physical gathering, generally small-scale, where groups get together and discuss issues of common interest without any keynotes or slideshows. These could also be arranged online. Many conferences fail to harvest the vast amount of knowledge and experience among the participants and the unconference is all about that. Maybe the future is more about more focused small-scale discussion groups, mixing synchronous and asynchronous as well as on-site and online collaboration.

I don't mean we scrap major conferences completely but we will certainly need to reduce their number and find other ways to meet. A major driver of academic conferences is their importance to researchers and demands for accepted conference papers as professional recognition. Maybe we can find new ways of recognising researchers in a more inclusive and interactive arena.

The unheard students

"studying" (CC BY 2.0) by scottfeldstein

Despite a wealth of reports and studies on the importance of lifelong learning and the need for professional development and reskilling, the focus of most higher education institutions is still firmly on traditional full-time campus programmes aimed at young students. Mature students studying part-time and mostly online are generally peripheral and invisible to the institution's core business. A new report from HEPI (Higher Education Policy Institute) in the UK, Unheard: the voices of part-time adult learners, raises this issue and gives a voice to these unheard students. There has been a dramatic reduction in the numbers of mature students studying in England over the past 5 years and even though the trend isn't completely shared by the other nations of the UK there is a need for a new approach. Higher university fees in England and Wales have certainly excluded applications from students from disadvantaged backgrounds. However it is not simply an issue of high fees. A similar trend is visible even here in Sweden where there are no fees, but a policy decision to focus on traditional campus degree programmes has led to a drop in the number of online courses available to mature students.

Of course there are open universities in many countries around the world who address this issue and offer flexible higher education to this important segment, but since demand is growing we need more institutions to widen their scope. Governments seldom make significant investments in lifelong learning and universities are still primarily research institutions with an attractive campus as their physical presence in the community. Say the word student and most people think of young people between 18 - 23 who study full-time on campus and this image is reflected in the imagery of practically every university's website. However, for most mature students full-time campus study is simply not an option. You cannot just give up your work and move your family to a campus city. The vast majority of mature students study part-time and the report outlines many of the barriers that they have to overcome. To understand this target group universities need to understand that mature part-time students have a different profile from full-time campus students:

• They do not identify themselves primarily as students. Their identities as parents or professionals are much stronger and as a result their ties to the institution where they study are much weaker than traditional students.
• They are interested in finding a course that is flexible and fits their existing lifestyle and are less interested in which institution provides it.
• They tend not to join student unions and their voice is seldom heard on faculty boards and committees.
• Because of the difficulties of managing part-time study, work and family commitments many mature students have to be very adaptive and creative to complete their studies. The existing structures do not help them.
• They are often unfamiliar with the study skills and academic terminology that are taken for granted in a campus setting and this can cause confusion as well as creating feelings of inadequacy.
• At the same time, mature students are often highly motivated and resourceful since course completion can help them into a new career or give them opportunities for advancement. They have clearer goals for their studies than many campus students.

The report raises concern that the current lack of incentives and opportunities for lifelong learning, particularly in England risks creating a serious level of inequality.

The crisis engulfing part-time adult learners in England points to an impoverished future in which higher education morphs into a purely full-time experience for 18-year olds fortunate enough to be born in the right place, attend the right school and gain the right A-Level grades. No more ‘second-chance’ transformations, no more learn-while-you-earn, no more enriching learning with contributions from adults who can bring different life experiences. Flexible opportunities for those disadvantaged individuals who cannot study full-time may all but vanish.

Changes are urgently needed and many of the report's recommendations can be applied in other countries around the world. One major factor is better recognition of prior learning, especially relevant workplace experience, and this would help reduce the study time necessary for a qualification. Another important factor is better guidance in the transition from adult education and non-formal learning into the formal university system. This is referred to as access pedagogy and can include preparatory courses and online guides focusing on study skills and academic writing and generally developing the students' confidence and resilience. Once accepted into a course the many students who are new to higher education will need initial extra support and tutoring is seen as particularly important:

... recognising some students will feel they do not belong, will feel they are transitioning ‘across separate worlds with no guidebook’, and will progress as a small and isolated cohort – so support students with tutors who will continue the higher education journey with them.

This support could come from various sources and not necessarily from university teachers. Support could come from local learning centres or from more experienced online students who could act as mentors. With such scaffolding in place there will be fewer dropouts and society will benefit from more people getting the chance to reskill and upskill. We keep hearing about the lack of qualified skilled people to fill vacancies in both the public and private sectors and it's time to do something about it.

Read more about the report in an article on FEnews, Unheard: The voices of part-time adult learners reveals the sharp decline in part-time learning.