Challenging assumptions about learning

Photo by Dom Fou on Unsplash

Do you have a clear idea of how you learn and what teaching methods help you most? I certainly don't and I'm close to retirement. However we continually ask students to evaluate their courses and draw conclusions from their feedback. Student course evaluations form the basis for future course design and administration but I wonder if we are asking the right questions. 

This is a topic discussed in a post by Zach Groshell, Do Students Have a Good Idea of What Helps Them Learn? We are asking them to evaluate a complex process that they have limited insight into and often the benefits of good teaching become clear months or even years later. He refers to a recent study, On students’ (mis)judgments of learning and teaching effectiveness. Journal of Applied Research in Memory and Cognition (Journal of Applied Research in Memory and Cognition, 9(2), 137–151), that investigates the hidden biases and subjective impressions behind student course evaluations. They found for example a mismatch between what the students thought they had learned and their test results. They preferred traditional content-rich lectures to more active sessions despite studies showing that active participation leads to deeper learning. They were also highly influenced by subjective perceptions of the teacher's status, enthusiasm, gender, ethnic background and accent (as we all are). I hasten to add that we are all subject to these biases.

As we have seen, empirical research has provided a wealth of results showing that students are poor evaluators of their own learning, and that their subjective impressions of teaching effectiveness are vulnerable to many biases that are unrelated to teaching and learning… Does this make it risky for instructors to use effective learning techniques? Particularly early in their careers and in teaching-focused positions, instructors may find themselves faced with the difficult decision of whether to incorporate teaching practices that gain them recognition as effective instructors, even if such practices do not positively impact students’ learning.

This poses a tricky question to teachers - do you teach for student satisfaction or teach for learning, even if it may lead to dissatisfaction and poor evaluations? An ambitious teacher who decides to promote active learning and student collaboration runs the risk of students misunderstanding their pedagogy and even a reprimand from the head of department when the poor evaluations emerge. Most of us can remember a teacher we hated at the time but with hindsight realise had pushed and challenged us to a higher level of understanding. What we think we want is not always what we need.

On a similar theme of assumptions and biases is an article by Perry Samson in Educause Review, Students Often Prefer In-Person Classes . . . Until They Don’t. We often assume that students prefer classroom teaching and student surveys show a clear preference for this. However, in this study the students were offered three levels of participation, the hyflex model of on-site, online or asynchronous interaction. Although they at first preferred to be in class, physical attendance declined as the course went on.

Given reasonable options, students in my class did not prefer the in-person mode of course delivery. In fact, the number of students who physically attended class dropped precipitously to an average of around 20% by mid-semester (see figure 1). At the same time, about one-third of students opted to participate synchronously during class time (see figure 2), with a growing number, reaching about 30%, participating asynchronously. The number of students who didn't participate any given day was relatively consistent throughout the semester at about 15%.

The test results showed that the on-site students performed no better than the others, in fact the first-year students who studied asynchronously (recorded lectures, forum discussion etc) had better results than those who attended classes in person. The study is of course limited and offers no exploration of the students' preferences but it does show that students appreciate the choice of participation modes more than we might assume.

My conclusion here is that we need greater dialogue with students about teaching and learning, explaining in advance why we are using a particular approach and getting them to buy into the method through together discussing rules of engagement and building a framework for feedback and reflection. Expectation management is so important and so pre-course information is so much more than just presenting a syllabus. It's setting an agenda and helping students understand how to succeed.

The problem with prizes

Photo by Joshua Golde on Unsplash

Most universities and schools have annual polls to find the best teacher of the year. This is a great honour of course and the idea is to reward good teaching and provide inspiration to other teachers. We have a great fascination for prize winners in all areas of society; from sporting awards to Oscars and Nobel prizes. Everyone loves a winner. The problem is that the focus is so often on individuals rather than the teams behind them and it is often unclear what the selection criteria are.

I started thinking about this after hearing a presentation at a Swedish online conference this week. The authors, Jeanna Wennerberg, Klara Bolander Laksov och Tore West, presented results from a study of nominations for best teacher awards at Stockholm University and examined in particular gender issues. The paper has not been published yet so I will not reveal too many details, but the study showed a clear bias in favour of male teachers (77% male 23% female). Interestingly, female students overwhelmingly nominated male teachers. The students' nominations were accompanied by criteria to justify the choice and there were clear trends there such as female teachers being seen to be better at building good relations with students and being more inclusive. However in general male teachers were seen as meeting a greater number of criteria for nomination. When we celebrate  a winner we need to consider what biases and preconceptions lie behind the nomination. What makes a great teacher?

I think most of us can admit that during our education we have had a teacher who we hated at the time. They worked us hard, challenged us, nagged and made us uncomfortable. They never gave us the right answers we needed for the exam but forced us to work things out for ourselves. But later on in life you realise that this teacher was the one who really taught you valuable lessons. They will never win any awards because their effect is only visible at a distance. How do we capture this in our best teacher awards or in course evaluation forms?

This backs up many other studies about student evaluations of teachers and raises many questions about the validity of these subjective and spontaneous assessments. Gender is one of several variables where unconscious bias and prejudices play an important role; accent/dialect, socio-economic background, nationality, ethnicity etc. How far do stereotyped ideals affect nominations and evaluations? 

Given that teaching today is becoming increasingly a team effort where several teachers design a course in close collaboration with educational technologists, librarians and media specialists, is a focus on the teacher as soloist still a valid strategy? Even if we see that teamwork is such an important factor and that the less visible members are just as important as the front figure we still revert to the urge to nominate individuals. It's similar in examination where we assess an individual's ability and seldom assess and reward a team.

I am not criticising the hard work of the teachers who do receive best teacher awards. They do a great job and deserve their award. But we do need to think a bit more deeply about how we define good teaching and how it is evaluated and judged. It's much more complex than a simple spot poll. Interestingly, the Nobel prize winners this year feature several research teams and the peace prize went to a collective, the UN World Food Programme. 

Illusions of learning - the problem with student evaluations

"students_lecture_event" (CC BY-ND 2.0) by University of Minnesota Duluth

One of the problems with asking students to evaluate their courses and teachers is that the surveys often focus on a false customer-supplier relationship. In the commercial sphere companies want to find out if their customers are happy with what they have bought and this indicates a successful transaction. But education is far more complex and is not simply a matter of giving the customers what they want. In many cases the effects of education are delayed and cannot be assessed at the end of a course in terms of customer satisfaction. If you think back to your own education you may admit that the best teacher you had was one you hated at the time. A teacher who demanded a lot, criticised you, pushed you and challenged you. After the course you would have handed in a scathing evaluation but now, years later, you realise that you learned so much more from that experience than from the friendly teachers who made life easy and gave you a good grade.

We ask students to evaluate our pedagogy without helping them understand the process and this results in rather superficial evaluations that say very little about how much they actually learned from the course. This is the topic of a fascinating article in the Journal of Applied Research in Memory and Cognition, On Students’ (Mis)judgments of Learning and Teaching Effectiveness. Sadly this article is only available if your employer pays a subscription for access to the journal but I will try to convey the gist of it here.

The authors discuss at length what they call illusions of learning. In their studies they discovered that students tend to give positive reviews to teachers who are enthusiastic and provide easily digested material that helps them pass the tests; an understandably pragmatic approach. They base their evaluations on superficial elements such as the teachers' appearance, age, gender, accent, enthusiasm, presentation style, and use of digital tools, none of which have any real relevance for the student's learning. Furthermore, the students showed a clear preference for traditional teaching methods like lecturing and believe that the learn more from traditional teaching than from active and collaborative learning, even when test results suggest the opposite.

Students believe, for example, that they learn best from enthusiastic and engaging instructors who provide smooth and well-polished lectures that do not require active class participation. Such factors, although they readily inflate students’ judgments of their own learning, do not consistently enhance students’ actual learning. They also inflate students’ evaluations of the effectiveness of their instructors. Indeed, students’ evaluations of teaching effectiveness can be poor predictors of their actual learning in their courses, and these evaluations can be biased by external factors unrelated to student learning, such as an instructor's gender, age, attractiveness, and grading leniency.

One section of the article has the heading: Do Student Evaluations Incentivize Poor Teaching Practices? If the university gives too much weight to student evaluations then more challenging teaching methods will be discouraged in favour of strategies that enhance customer satisfaction. Teachers who push and challenge students and encourage them to investigate topics for themselves risk poor evaluations and possible reprimand. To get better evaluations, teachers will be encouraged to revert to traditional methods and we are then stuck in a vicious circle. The students will have the illusion of learning and the teachers will have the illusion of teaching. Everyone will be happy but how much have they learned?

We propose that faulty metacognition is a key contributor to the problem. Students’ misevaluations of teaching effectiveness can be driven by the same factors that underlie their misjudgments of their own learning. Although they do not enhance student learning and can even impair it, teaching approaches that minimize effort and create the appearance of a smooth, well-polished, fluent, and enthusiastic instructor readily boost students’ subjective impressions of what they have learned and their perceived effectiveness of that instructor. Because these subjective impressions are the primary basis for determining teaching effectiveness, and as such are a key metric used for decisions about hiring and promotion, instructors are currently incentivized to adopt teaching approaches that may produce illusions of learning that boost their ratings but can actually undermine students’ learning.

Sadly there is no easy solution to this problem except widening the scope of evaluations to include multiple perspectives and helping students to make more informed evaluations of their learning. Given the workload for many students this will not be easy and the authors see no clear solution at present. At least we now have evidence that the system requires a radical rethink and that awareness is an improvement.

It is hoped that future conceptualizations of teaching effectiveness include research-based evidence for improving student learning and metacognition as a strong basis in formulating measurements that accurately and reliably reflect the quality of teaching.

Reference

Carpenter, S. K., et al. On Students’ (Mis)judgments of Learning and Teaching Effectiveness. Journal of Applied Research in Memory and Cognition (2020), https://doi.org/10.1016/j.jarmac.2019.12.009

Is the customer always right?

  by Marco Fieber/Ostblog.org

Education at all levels is awash with evaluation forms and needs analysis surveys. We all want feedback but are we asking the right questions? Development projects are based on needs analysis, assuming that we should only develop services that customers/staff/students really require. The problem with this approach is that you don't need what you don't know. If we only designed products and services that people needed we would never have had cars, television, computers, mobiles or the internet. No-one needed them until they had been developed. So asking for needs can help but is often not the right approach for innovation. The same applies to student satisfaction. We assume that if students are satisfied with a course then the teacher has done a good job and the students have learnt a lot. Or have they?

Education, unlike products and services, cannot be judged on simple customer satisfaction. Students are satisfied if they get a good grade but that doesn't always mean that they have learnt very much or that the course was well designed. An article on The Conversation, Students don’t know what’s best for their own learning, highlights the problems of judging education by satisfaction. Students tend to give good evaluations to teachers who don't demand too much, are entertaining and give them good grades. Those who make them work hard and don't serve all the answers on a plate are generally less popular. Teachers who try innovative new methods that force students to take charge of their learning and really grapple with working out concepts for themselves often face negative evaluations and possibly a resultant stern talk from their head of department. Many students (and teachers) mistake content for learning.

That is why many students assume that reading or highlighting passages in their text-book, or merely listening to a lecture, is enough to produce learning. They mistake the ease of the task with greater knowledge. Time-consuming and effortful tasks, like self-testing their knowledge, are consequently seen by students as less efficient for their learning, despite the fact that the more difficult tasks produce the most learning.

Student evaluations can therefore be dangerously misleading and result in "easy" courses being encouraged and more demanding courses modified to please the customers. Lectures are still popular because they provide the "answers" in one easily digestible session and many still equate lecturing with teaching. As a result students will generally give positive evaluations to teachers who give them what they expected. Seeing this as a measure of course quality can have serious consequences according to the article.

... universities that rely on student evaluations are likely to punish good teachers and encourage those who simply make it easy for students. Most universities have codes of conduct that require decisions to be made on valid evidence. Any manager discussing student evaluations when reviewing lecturers’ performance is probably breaching that part of their own job requirements. Given the evidence, student evaluations are a distraction from the responsibility to provide the best possible education for the nation.

Of course it is good to find out what students thought of a course but we need to steer away from simply measuring satisfaction and provide them with a new set of rubrics that allow them to gauge their effort, collaboration, acquisition of new skills and how they have met new challenges. This type of evaluation takes time to learn but that way we will get more reliable criteria for assessing course quality. At the end of a good course you may feel rather tired, maybe irritated and sore but you also realise that you've achieved something. We need to learn to appreciate teaching that challenges us and pushes us forward even if it hurts a bit. 

Soft skills are key to employment

  by Prashant Menon

The skills that employers value most are not the skills that tend to be examined and rewarded by universities. What employers are most interested in are soft skills such as teamwork, initiative, creativity, critical thinking and practical experience and these are all notoriously hard to quantify and examine within the traditional academic structure. This is further confirmed in an article in the Chronicle of Higher Education, College Students Think They’re Ready for the Work Force. Employers Aren’t So Sure. The article discusses the findings of a new report by the Association of American Colleges and Universities, Falling short? College learning and career success, which is a survey of 400 employers and 613 students on how well students are prepared for employment. There is a clear mismatch between the expectations of employers and graduates' self-assessment especially when it comes to so-called soft skills that are seldom visible in academic certificates.

The students indicated that they felt qualified in areas like written and oral communication, critical and analytical thinking, and applying knowledge and skills to the real world. But employers consistently rated students lower than they rated themselves. For example, while 59 percent of students said they were well prepared to analyze and solve complex problems, just 24 percent of employers said they had found that to be true of recent college graduates. (Chronicle article)

It seems that employers are less interested in grades and formal credentials and want instead to see evidence of practical work experience and proficiency in soft skills. This does not mean that formal credentials are not relevant but that they come further down the list when compared to hands-on experience. According to the report the demand is for generalists rather than specialists:

... employers say that when hiring, they place the greatest value on demonstrated proficiency in skills and knowledge that cut across all majors. The learning outcomes they rate as most important include written and oral communication skills, teamwork skills, ethical decision-making, critical thinking, and the ability to apply knowledge in real-world settings. Indeed, most employers say that these cross-cutting skills are more important to an individual’s success at their company than his or her undergraduate major. (AACU report)

There are of course many university programmes which focus on problem-based learning, work experience, real-life projects, internships and so on but studies like these also highlight the need for credentials to be more granular and include evidence of soft skills. The use of skill badges to complement traditional credentials can be one way of rewarding students especially if they can be awarded by the organisation the student has worked with during studies. The growth in interest in competency-based degrees can also be seen as a response to employers' demands. A university education can thus include a variety of certificates/badges awarded mostly by the university but also by other organisations who have benefitted by the student's work. In this way we can make academic credentials more granular and the student will have references from several sources. 

The red pen

 by cellar_door_films

Education is full of stereotyped symbols that are hard to erase. No matter how collaborative, open and innovative we make education, the standard metaphors in the media and everyday speech are still the classroom, the desk, the blackboard, chalk, homework and the exam hall. One particularly enduring symbol is the dreaded red pen. This simple writing tool has become forever type-cast as the villain, an instrument of criticism and correction and is seldom used for anything else. We can happily write in all other colours but the red pen seems forever damned.

This theme is discussed in a post (in Norwegian but try with Google translate and you'll get the gist) by a Norwegian colleague of mine, Torhild Slåtto . Comments written in red on a student's assignment (either by pen or red text in a digital document) are said to have such a demotivating effect that most teachers prefer to comment in pretty well any colour other than red. She gives an example of a student who got a high grade for an assignment but couldn't understand why since there were so many red comments on it. The comments could well have been very positive and constructive but the colour clouded that fact and the student interpreted the comments as negative. If the comments had been in blue or green the effect would've been completely different.

There are many other examples of educational stereotypes and preconceptions that can lead to misunderstanding. The traditional role of the teacher as arbitrator and corrector lies so deep that students often misinterpret efforts to encourage collaborative learning, teamwork and peer assessment as "lazy" teaching. If you expect everything to be teacher-centred then you may view a student-centred strategy as the teacher not doing her/his job. This can result in negative course evaluations, not because of bad teaching but due to very different preconceptions of the teacher's role. In such cases the student may even be disappointed that no red ink was used; "the teacher didn't even correct me."

We need to unlearn old truths before we can move on.

MOOC assortment

  by camknows

An assortment of MOOC thoughts inspired by this week's news feed.

There is some concern that many MOOCs are simply repositories of content in the form of lectures, reading and automatic tests and are not even really courses at all. This is the main point behind a post by Justin Reich, Is a MOOC a Textbook or a Course? Yes there's plenty of good material there and you can certainly learn a lot but you're largely on your own and the question is whether this can be considered a coherent course. I feel that such types of MOOC are perfectly valid as long as they don't pretend to be any more. It's vital that every MOOC describes clearly and upfront exactly what type of course it is (or not) and explains what types of study the student should expect. That can be one quality criterion; a clear statement on what's in store for the learner, the type of course, pedagogy and demands. All I've seen do this well but maybe we also need to know what the MOOC doesn't offer, just to avoid any misunderstanding.

Then there's the excellent report from the University of Edinburgh on the results of their first MOOCs under the Coursera banner. As Donald Clark reports on his blog, Report on 6 MOOCs turns up 10 surprises, students were clearly satisfied with the courses on offer with only 2% responding that the course did not meet their expectations. The report also confirms that very few MOOC students are motivated by certificates or credits and that attempts to provide a social aspect to the courses via course forums seldom work. Somehow the key element of the connectivist cMOOCs, networking and collaboration, do not translate well into the xMOOC paradigm (my wish for 2014 is that we find a new terminology for all this). Forums tend to be confusing, overcrowded and not always particularly interactive and most students seem to go it alone.

Maybe the reason for the cMOOCs succeeding in creating a dynamic learning network was that the participants were usually digitally literate academics who were already used to such an environment. The xMOOCs however attract a much wider range of participants, many of whom may not be used to online learning at all. Many sign up expecting a fairly traditional course and expecting it to be largely self-study even if the organisers might wish it to be otherwise.

So how do we get learners more involved in MOOCs and how can we foster collaboration and networking? Peer review may be the best bet and already some courses are use peer assessment as a powerful teaching and empowerment tool as well as providing meaningful feedback. Students need training to be able to offer meaningful feedback to each other and this can be built into the course at an early stage. If a set of assessment criteria for a task is given to students and they can practice on a test case first they can then assess each other. Studies indicate that peer assessment often comes very close to the grades given by experienced teachers, though the feedback may not be as insightful.

There's an article on MOOC News and Reviews,Massive MOOC Grading Problem – Stanford HCI Group Tackles Peer Assessment, that describes peer assessment in MOOCs and gives examples of how this can lift the learning experience of the MOOC to new levels, without involving faculty at all.

"I recently completed a two-week video production course on the Skillshare platform that used peer reviews. While the course was not for college credit and the only rubric used was to foster positive comments even while pointing out areas for improvement, knowing I was commenting on the work of my cohorts made me dive into the course more deeply. I felt a certain sense of fairness at play: I was going to make my project live up to the standard I set for others. In the end, I put in more hours reviewing comments and working on my projects than I spent watching the instructor’s videos and engaging in other assignments. The comments I received from my peers spurred me to rethink, revise and improve my project."

Finally this is exactly the approach being offered by NovoEd, yet another new kid on the MOOC block.
They divide students into groups of up to ten, based on location or interests and those groups then support and assess each other through the course. Creating group loyalty and a small scale study environment even in the vast sea of a MOOC could well be the road to follow.

Read more about NovoEd in an article on TechCrunch,Stanford’s NovoEd Brings Collaboration And Group Learning To MOOCs To Help Fight Attrition.

Change the focus

  by thejuniorpartner

When I was a student just about everything I wrote was for a very exclusive audience; generally one person, a teacher or examiner. No-one else would ever read it. I wrote purely to be assessed and it wasn't a very stimulating, fulfilling or realistic process. That's why today's students are writing assignments that are published publicly on the net or made available on the university's open access archive. When you know your work will be available to all you automatically raise your game, suddenly the assignment is real and not a test. Shift the focus and change the view.

A similar change of focus can inject much needed energy to the tired ritual of course evaluations. A short article by Brian Croxall, Improve your Course Evaluations by having your Class Write Letters to Future Students (Chronicle of Higher Education), describes how he choose a new angle for the end-of-course evaluation. Instead of handing out the standard evaluation form he asked the students to write a letter to next year's students giving them advice and tips about the course and the teaching. By changing the focus from writing to a faceless administrator to a group of fellow students the task suddenly became meaningful and the feedback was much more enlightening. The quantity and quality of the feedback was improved.

"... I found that the students wrote, on average, far more on these evaluations than they have on past ones that I have provided. And in writing something directed at fellow students rather than me or some faceless, unknowable bureaucracy, I’d say that the students were much more candid. This means that they are more direct in talking about my strengths and weaknesses. And while it’s nice to hear the former, it’s the latter that will actually help me do a better job the next time around."

A slight change of focus can often make a big difference.

Is it a course? Is it a textbook? No, it's a MOOC!

  by Flicktone

It's hard enough trying to evaluate regular education because learning is not easily measured; if indeed we can even hope to do so. We can of course test to see what people have remembered or whether they can explain concepts in a clear and logical manner but how do we know what they have actually learnt? So if evaluating traditional education is tough how hard is it to evaluate a MOOC? A bit like nailing jelly to the wall as a colleague once said, though I can't remember when.

MOOC providers may believe that they are offering a university course and duly organise it as such but do the participants share that belief? If you sign up for a regular university course you make a commitment, invest money and time and expect a reward for your efforts. You buy into the course concept, accept the university's terms and step into line. Everyone understands what they're involved in and accepts the rules. When you sign up for a MOOC, however, you don't buy into the concept in the same way. You join the course for your own reasons, have your own objectives and preferences and are often uninterested in the certification that may or may not be available at the end. Many have no intention of completing the "course" at all, they're simply curious to see what the fuss is all about. When there is no consensus as to what the MOOC is, an evaluation becomes extremely difficult.

Stephen Downes' post, Evaluating a MOOC, claims that we can't evaluate MOOCs with the same criteria as traditional courses since they aren't actually courses. They are many things to many people.

"I think the best way to understand success in a MOOC is by analogy with, say a book, or a game, or a trip to the city.
The person taking the MOOC is like a person reading a book, playing a game, or taking a trip to the city. It is impossible to talk about 'the objective' of such an activity - some people want to learn something (and others something else), others are doing it for leisure (and others as part of their job), others to make friends (and others to get away from their friends for a while), etc."

A successful MOOC creates enough momentum to continue after the "course" is complete. Since the motivations of each participant are so different it is more relevant to evaluate the MOOC as an entity and look at what the community has achieved and how vibrant and self-sufficient that community has become. MOOCs will develop into new forms and new terminology will emerge to differentiate between them. Some will be traditional courses with a syllabus and learning outcomes and will expect students to accept the rules of engagement whereas others will be more fluid and become learning communities with learners coming and going, some staying for months and others simply dropping in for inspiration.

"MOOC success, in other words, is not individual success. We each have our own motivations for participating in a MOOC, and our own rewards, which may be more or less satisfied. But MOOC success emerges as a consequence of individual experiences. It is not a combination or a sum of those experiences - taking a poll won't tell us about them - but rather a result of how those experiences combined or meshed together."

The important point is that evaluation is only possible when we know what it is we are evaluating and that all participants are aware of what they are engaged in. Confused? I still am but I'm trying to work it all out.

Rethinking assessment

I'm heading down to Granada in Spain next week to attend the EFQUEL Innovation Forum, which I'm also involved in organising. I'm looking forward especially to hosting a session on Thursday afternoon (6 Sept) entitled Rethinking Educational Assessment which you are welcome to join as a web conference via Adobe Connect.

The widespread use of open education today with all the flavours of MOOC as well as many informal learning arenas is changing our conceptions of education. The use of open educational resources and collaborative learning tools offer us the chance to escape from the restrictions of the traditional classroom and move the focus from content delivery to context creation. New educational models demand a rethink of how we assess learning. In a content delivery model assessment is mostly based on processing that content and knowledge acquisition can be relatively easily tested.

However employers are often more interested in soft skills than hard qualifications. The key employability factors are teamwork, initiative, problem solving and engagement and if you can show that you have these skills they will often trump qualifications (though obviously qualifications are still necessary). Is it possible to assess these soft skills and how can you do so? How can we move assessment and examination from showing what you know to how well you can do things? One recent development in this direction is the use of open badges as described here several times in the past year or so. Badges can be awarded to people who show a clear mastery of a skill in the eyes of their peers as well as teachers or mentors. Is peer review the future of assessment and if so how can we provide quality assurance?

In the session I'll be joined by two authorities in this field: Preetha Ram from OpenStudy and Anka Mulder, president of the OpenCourseWare Consortium (read more). They'll be offering their ideas on alternative assessment and have three questions for everyone to consider before and during the session:

• Traditional educational institutions are very well placed to provide campus as well as online education. At the same time they have to find ways to monetize open online education to cover their expenses. Which are the business models they can develop? 
• Students as well as employers still rely heavily on formal certificates and diplomas provided by accredited institutions. In order to become successful, recognition of informal certificates by employers is essential. What can be done to encourage this? 
• Verification of a student’s identity is essential in assessment, notably for institutions that provide accredited diplomas. How can we make online assessment more reliable? 

See the recording of the webinar.

Webinar - The missing link: connecting formal and informal learning

Next week, on Friday 16 September (09.45 - 11.15 Portugal time),  I'll be moderating a session at the EFQUEL Innovation Forum 2011 in Oeiras in Portugal (see venue pic on right).

It's part of a theme called Alternative certification and validation practices and I've entitled this session The missing link – connecting formal and informal learning. It's basically about how new forms of informal collaborative learning can somehow be assessed and gain acceptance from formal education.

I've decided to extend the reach of this session by using the e-meeting tool Adobe Connect and that means that any of you can drop in and listen to the speakers and chat with us. I'm keeping it simple in terms of technical production, using a couple of laptops, webcams and an echo-suppressing microphone so it won't be a blockbuster production.

Here's the description from the conference program:

The missing link – connecting formal and informal learning
The net offers enormous scope for informal learning and several interesting initiatives are attempting to provide frameworks for collaborative, student-driven learning, for example MOOCs, open courseware etc. In this session we will spotlight just a few of these initiatives and discuss how such informal learning can be assessed and certified and whether the formal higher education system can find ways of integrating these alternative structures. Representatives from OER University (Wayne Mackintosh), Peer 2 Peer University and Open Badges project (Erin Knight) and OpenStudy (Chris Sprague and Preetha Ram) will participate.
The session will take the form of an open webinar via Adobe Connect Pro with participants both at the conference and on the web. Speakers will have video feed but individual participants will be able to interact via a chat function. The virtual room will be opened to the public about 15 minutes before the start of the session and participants only need to click on this URL and log in with their own name.
The Adobe Connect room and technical support is provided by SUNET, the Swedish University Computer Network.

Watch the sessions

• Erin Knight - Peer 2 Peer University and Open Badges
• Preetha Ram - OpenStudy
• Wayne Mackintosh - OER University and general summary

Badges of learning

We all learn new things every day. We learn by asking, watching, imitating, practicing, testing and thinking. In fact we learn much more outside formal education than inside it but we don't get any credit for this. What counts are the degrees and certificates you get for attending classes and passing formal exams. However increasing numbers are learning on the net through informal communities, watching TED-lectures, Khan Academy or iTunes U or getting involved in collaborative work designing new open source applications. How do we recognize lifelong learning?

Peer 2 Peer University, the collaborative learning initiative that already runs a wide range of online peer-driven courses, is examining the idea of badges that can be awarded for successful completion of courses. Together with Mozilla they have started the Open Badges project to create a structure for all sorts of adult education institutions and other organisations to award digital badges that can then be shown on your website, social media sites and CV. They can be awarded for completion of courses, demonstrated skills or for significant work in online projects.

All sorts of organisations could issue badges but one issue that will undoubtedly be raised is that the actual badges will be so easy to copy. How will they ensure that you really have deserved that badge on your profile? I would imagine that each badge will link to the issuing organisation who should quickly be able to confirm that you have indeed been awarded it.

P2PU is already piloting open badges in their School of Webcraft courses. By successfully completing course assignments participants can unlock various skills badges that show what they have achieved (see more about the P2PU badge scheme). Badges can be awarded by peers also participating on the course. The framework for tyhe open badge system is outlined more fully in a draft article called An open badge system framework.

This could lead to a whole new landscape in terms of how we demonstrate our skills to future employers. The formal system is not going to be swept away by this development but the new badges offer a way forward to recognizing people's real skills and should complement existing structures. Hopefully the formal system will see this as an opportunity and not a threat. For Open Badges to gain credibility it is vital that some more forward thinking higher education institutions join up with the project.

For more thoughts on this read article by Michael Sean Gallagher, Open Badges and Acknowledging Decentralized Activity in Learning

And the winner is ...

Course evaluation is an integral part of a university's quality control process but it is also extremely difficult to get quality information from them. The best way to make sure the forms are filled in at all is to hand out papers in class and use classroom time to fill them in. On-line evaluation forms are notoriously difficult and returns are often pitifully low unless incentives are offered (eg fill in the evaluation and you enter a prize draw).

However millions of students are happily contributing to a completely voluntary evaluation tool called RateMyProfessors. This site claims to cover over 6,000 universities and colleges in the US, Canada and the UK and asks students to rate their teachers according to a variety of criteria such as easiness, helpfulness, clarity, textbook use and interest level. Lists are then compiled presenting, for example, a top ten "hottest" teachers, whatever that may mean.

This service is clearly controversial and is outside university jurisdiction (a possible reason for students' interest). Teachers may well feel unfairly treated and this is clear in the discussion forum on the site. There is an interesting feature called Professors strike back where teachers state their case on video.

Is this a case of positive consumer action or a place to get your own back on a teacher who gave you bad grades? Whatever it is it is definitely a factor to be reckoned with and it will be interesting if the phenomenon spreads to the rest of Europe.