Skip to main content

testing

Desirable difficulty for effective learning

When we are presented with new information, we try and connect it to information we already hold. This is automatic. Sometimes the information fits in easily; other times the fit is more difficult — perhaps because some of our old information is wrong, or perhaps because we lack some of the knowledge we need to fit them together.

When we're confronted by contradictory information, our first reaction is usually surprise. But if the surprise continues, with the contradictions perhaps increasing, or at any rate becoming no closer to being resolved, then our emotional reaction turns to confusion.

Confusion is very common in the learning process, despite most educators thinking that effective teaching is all about minimizing, if not eliminating, confusion.

But recent research has suggested that confusion is not necessarily a bad thing. Indeed, in some circumstances, it may be desirable.

I see this as an example of the broader notion of ‘desirable difficulty’, which is the subject of my current post. But let’s look first at this recent study on confusion for learning.

In the study, students engaged in ‘trialogues’ involving themselves and two animated agents. The trialogues discussed possible flaws in a scientific study, and the animated agents took the roles of a tutor and a student peer. To get the student thinking about what makes a good scientific study, the agents disagreed with each other on certain points, and the student had to decide who was right. On some occasions, the agents made incorrect or contradictory statements about the study.

In the first experiment, involving 64 students, there were four opportunities for contradictions during the discussion of each research study. Because the overall levels of student confusion were quite low, a second experiment, involving 76 students, used a delayed manipulation, where the animated agents initially agreed with each other but eventually started to express divergent views. In this condition, students were sometimes then given a text to read to help them resolve their confusion. It was thought that, given their confusion, students would read the text with particular attention, and so improve their learning.

In both experiments, on those trials which genuinely confused the students, those students who were initially confused by the contradiction between the two agents did significantly better on the test at the end.

A side-note: self-reports of confusion were not very sensitive, and students’ responses to forced-choice questions following the contradictions were more sensitive at inferring confusion. This is a reminder that students are not necessarily good judges of their own confusion!

The idea behind all this is that, when there’s a mismatch between new information and prior knowledge, we have to explore the contradictions more deeply — make an effort to explain the contradictions. Such deeper processing should result in more durable and accessible memory codes.

Such a mismatch can occur in many, quite diverse contexts — not simply in the study situation. For example, unexpected feedback, anomalous events, obstacles to goals, or interruptions of familiar action sequences, all create some sort of mismatch between incoming information and prior knowledge.

However, all instances of confusion aren’t necessarily useful for learning and memory. They need to be relevant to the activity, and of course the individual needs to have the means to resolve the confusion.

As I said, I see a relationship between this idea of the right level and type of confusion enhancing learning, and the idea of desirable difficulty. I’ve talked before about the ‘desirable difficulty’ effect (see, for example, Using 'hard to read' fonts may help you remember more). Both of these ideas, of course, connect to a much older and more fundamental idea: that of levels of processing. The idea that we can process information at varying levels, and that deeper levels of processing improve memory and learning, dates back to a paper written in 1972 by Craik and Lockhart (although it has been developed and modified over the years), and underpins (usually implicitly) much educational thinking.

But it’s not so much this fundamental notion that deeper processing helps memory and learning, and certain desirable difficulties encourage deeper processing, that interests me as much as idea of getting the level right.

Too much confusion is usually counter-productive; too much difficulty the same.

Getting the difficulty level right is something I have talked about in connection with flow. On the face of it, confusion would seem to be counterproductive for achieving flow, and yet ... it rather depends on the level of confusion, don't you think? If the student has clear paths to follow to resolve the confusion, the information flow doesn't need to stop.

This idea also, perhaps, has connections to effective practice principles — specifically, what I call the ‘Just-in-time rule’. This is the principle that the optimal spacing for your retrieval practice depends on you retrieving the information just before you would have forgotten it. (That’s not as occult as it sounds! But I’m not here to discuss that today.)

It seems to me that another way of thinking about this is that you want to find that moment when retrieval of that information is at the ‘right’ level of difficulty — neither too easy, nor too hard.

Successful teaching is about shaping the information flow so that the student experiences it — moment by moment — at the right level of difficulty. This is, of course, impossible in a factory-model classroom, but the mechanics of tailoring the information flow to the individual are now made possible by technology.

But technology isn't the answer on its own. To achieve optimal results, it helps if the individual student is aware that the success of their learning depends on (or will at least be more effective — for some will be successful regardless of the inadequacy of the instruction) managing the information flow. Which means they need to provide honest feedback, they need to be able to monitor their learning and recognize when they have ‘got’ something and when they haven’t, and they need to understand that if one approach to a subject isn’t working for them, then they need to try a different one.

Perhaps this provides a different perspective for some of you. I'd love to hear of any thoughts or experiences teachers and students have had that bear on these issues.

References

D’Mello, S., Lehman B., Pekrun R., & Graesser A. (Submitted). Confusion can be beneficial for learning. Learning and Instruction.

The changing nature of literacy. Part 4: Models & Literacies

This post is the fourth and last part in a four-part series on how education delivery is changing, and the set of literacies required in today’s world. Part 1 looked at textbooks; Part 2 at direct instruction/lecturing; Part 3 at computer learning.. This post looks at learning models and types of literacy.

 

Literacy. What does it mean?

Literacy is about being able to access information locked up in a code; it's also about being able to use that code. To be literate is to be able to read and write.

There's also another aspect of literacy that goes beyond mere decoding. This is about reading with understanding, with critical awareness.

Argument around the dangers of modern technology tends, in the way of arguments, to simplistically characterize the players: Internet = short, shallow; Social media = frivolous, distracting; Games = frivolous; Textbooks, Lectures = serious, deep, instructive.

But of course this is ridiculous even if we restrict ourselves to the learning context. Even social media have their uses. Even games can teach. And even textbooks and lectures can be shallow, or uninstructive, or inaccurate. (Indeed, way back in my first year of university I experienced a calculus lecturer who, I believe, reduced my understanding of calculus!)

The internet is, as we all recognize, a two-sided tool (but every tool is). Many people worry about the misinformation, the shallowness of much of the information, the superficiality of surfing, the way people might get stuck in a little corner that reinforces their vulnerabilities or prejudices, and so on.

We can say the same about infographics (data visualization, visual communication, call it what you will). It’s fostered as a way of helping us deal with the complexity and quantity of information (and I’m a big fan of it), but some people have criticized it for its potential for misinformation. Of course, text (wherever found) is far from pure in this respect!

But we don’t deal with misinformation by banning it (well, some of us don’t); we deal with it by providing the tools and the education so that people can recognize when something is being dangerously misleading or just plain wrong.

So, one of the important aspects of literacy (once you get beyond the decoding level) is being able to evaluate the information.

Why do we talk about digital literacy? Do we really need a new term (or terms)?

It comes down to skills. Because that is what literacy is: it's a skill (with all that that implies). And the new literacies do, undeniably, require new skills.

As far as the decoding aspect is concerned, well, text is still text. And textbooks have always included illustrations, so you could say that that is not new either. But that would be a mistake. The problem with visualizations is that it is not obvious that there's a skill to reading them — they're not as transparent as most believe (hence the misinformation claim). Humans have always used pictures to communicate; it is only recently that these have become sufficiently sophisticated to warrant the term 'language'.

So one of the modern literacies must be visual language, which like verbal language (and math and music), comes in different flavors. We wouldn’t use the same strategies to interpret and analyze a novel as we would a chemistry text, or a poem. We need to develop the same understanding of the taxonomy of visual language.

So I think we should include visual literacy in our new literacy set.

But of course, the new information delivery systems have requirements that go beyond content. Being able to use the code goes beyond reading text and pictures. It involves being able to navigate the delivery system. With a book, you just have to turn the pages. But with hyperspace, learning spaces, video-books, and so on, 'reading' is more complicated.

This is the important thing, the qualitative shift: the shift from linearity. Having a space, be it the whole of the internet or a confined learning space, in which you can go in many directions, in which there is no one path, may be empowering and richly layered, but it is not a place you can throw anyone into without training. Not if they are going to truly benefit from it. Like the need for visual literacy, this is another under-recognized need.

The complexity of these spaces and their navigation has, however, led to a number of useful distinctions being made — between digital literacy and computer literacy, information literacy, and media literacy (among others). Basically, these point to the need to distinguish between an ability to use technology (know the language of software — What’s a window? What’s the difference between a browser and a search engine? Do you hashtag your tweets? Do you use folders?) from the ability to find, filter, and evaluate information, and from the ability to actively participate in the information flow across media (Do you change your verbal style appropriately when you move from a tweet to a YouTube script to a written report to a comment on someone’s blog? Do you use different modes of analysis and evaluation when viewing different media?)

Given that we want students to become adept at all of these, how should we teach them?

In an interview, Will Richardson, a teacher whose experiences with interactive Web tools in the classroom led him to write Blogs, Wikis, Podcasts, and Other Powerful Web Tools for Classrooms talks about the need for teachers to have a visible presence on the Web, to participate in learning networks, and about how this openness is a huge culture shift for the closed shop of teachers. About network literacy as a key skill: “students should be able to create, navigate, and grow their own personal learning networks in safe, effective, and ethical ways. It’s really about the ability to engage with people around the world in these online networks, to take advantage of learning opportunities that are not restricted to a particular place and time, and to be conversant with the techniques and methodologies involved in doing this.” About how kids may be more technologically savvy, but they need help sorting out which information, and which people, to trust.

My favorite bit: he talks about Rethinking Education in the Age of Technology: The Digital Revolution and Schooling in America (Amazon affiliate link), which apparently discusses how historically we used to have an apprenticeship model of education, which moved to the factory model, where it’s all about training everyone the same way, and now we’re moving back to a more individualized, self-directed and flexible lifelong-learning model. Put in those terms, it seems clear that we can’t just keep tweaking; the changes are more fundamental than that.

He also asks why no one is consciously teaching kids how to read and write in linked environments — which relates back to my point about learning to traverse non-linear spaces.

But the onus shouldn't (and can't) be all on the teacher. They need a structure that supports them.

But as with learning networks and digital tools (Facebook, Twitter, blogs, RSS, Scribd, Flickr, TumblrMashable, ...), the structures too keep changing under their feet. Blackboard, Moodle, Udemy, Instructure (to pick out some old and some new).

It's perhaps easier when the structure is purely online. (In the U.S., the Keeping Pace with K-12 online learning 2010 report tells us that state virtual schools or state-led online learning initiatives now exist in 39 states, and 27 states plus Washington DC have at least one full-time online school operating statewide.) But mostly online learning occurs side-by-side with face-to-face learning. (The report estimates that about 50% of all districts are operating or planning online and blended learning programs.)

A report profiling 40 schools that have blended-learning programs has found six basic models of blending learning:

  • Face-to-face-driver: face-to-face teachers deliver most of the curricula. The physical teacher deploys online learning on a case-by-case basis to supplement or remediate, often in the back of the classroom or in a technology lab.
  • Rotation: within a given course, students rotate on a fixed schedule between learning online in a one-to-one, self-paced environment and sitting in a classroom with a traditional face-to-face teacher. The face-to-face teacher usually oversees the online work
  • Flex: uses an online platform to deliver most of the curricula. Teachers provide on-site support on a flexible and adaptive as-needed basis through in-person tutoring sessions and small group sessions.
  • Online-lab: relies on an online platform to deliver the entire course but in a lab environment. Usually these programs provide online teachers. Paraprofessionals supervise, but offer little content expertise.
  • Self-blend: encompassing any time students choose to take one or more courses online to supplement their traditional school’s catalog. The online learning is always remote.

As we can see (and as was also discussed in the Keeping Pace report), online learning is not about making the teacher redundant! No surprise when you consider that a major aspect of online learning (and its attraction for many students) is that it personalizes learning.

This is also echoed at university level. A spokesman for the Pearson Foundation, discussing a survey of over 1,200 college students, said: "There seems to be this belief among students that tablets are going to fundamentally change the way they learn and the way they access what they are learning. Students see these devices as a way to personalize learning." Students don't see tablets as means of accessing digital textbooks as much as a means to access e-mail, manage assignments and schedules, and read non-textbook materials such as study aids, reports, and articles. (You might also like to read about one university's experience introducing iPad's into the classroom)).

In the same way, a study involving students in China and Hong Kong found that Facebook was being used to let them connect with faculty and other students, provide comments to peers/share knowledge, share feelings with peers, join Groups established for subjects, share course schedules and project management calendars, and (via educational applications) organize learning activities.

So one aspect of online learning is management and collaboration.

Of course online learning is not only about personalizing learning. It's also about broadening access to quality educational resources. The open course movement is perhaps more advanced at university level (exemplified by MIT's OpenCourseWare (updated: now Open Education Global), Yale's Open Courses, the Open University's Learning Space), but in Iowa, schools will soon have access to wide variety of digital materials from a central repository using Pearson Education's Equella. Certainly the internet is rife with educational materials aimed at K-12, but there are great benefits from the more formal structure of such a repository.

But this wonderful cornucopia is also the biggest problem. So many resources. And so many structures, programs, digital tools. It takes a lot of time and effort to master each one, and who wants to put in that effort unless they’re sure it’s really important and going to last?

There's no good answer to that, I'm afraid. We are living in a time of transition, and this is the price of that.

But we can try and develop our own 'rules of engagement'. Something to filter out the deluge of new tools and new systems and new resources.

When doing so, we need to consider the two principal, and different, issues involved in this revolution in information delivery systems, which should be kept quite distinct when thinking about them (however muddled together they will be in application). One concerns their use in learning — do textbooks need 'bells and whistles' to be more effective means of learning? what is the best way to frame this information so that students (at their grade level) can understand and remember it? This is the how question. For this we need to work out the different strategies that each delivery system needs to be an effective learning tool, and the different contexts in which each one is effective.

The other issue concerns the world for which the education system is supposedly training students. How is information delivered today? How do people work with information? This is the what question; the issue of content — though not in the 'core knowledge' sense.

Although, part of the issue does concern this question of core content. Because the fact is, however we may pine for the days when we all knew the same things, read the same books, could recite the same poems (no, we never really had those days; we just had smaller groups), there is too much information in the world for that to be possible. And society needs the diversity of many people knowing different things, because there's too much for us all to know the same thing. So what we need from our education system — and I know it's a truism but there you go, doesn't make it less true — is for our students to learn how to learn. Which means they need to know the best strategies for learning from the various information delivery systems they're going to be trying to learn from.

And there's something else, that stems from this point that there's too much for us all to know the same thing. We have this emphasis on doing well as an individual — individuals graduate, become famous, get Nobel Prizes, get remembered in the history books. But science and scholarship, and politics and community development, have always benefited from the stimulation of different minds. The complexity of the world today means that we need that more than ever. The complexity of science today means that most discoveries are the results of a team rather than a single person. Even in mathematics, the archetypal home of the solitary genius.

For example, the Polymath Project began with one mathematical genius who decided to take one of the complex mathematical problems he had struggled to solve to his blog. He threw it out there. And readers threw ideas back. Since then, several papers have been published in journals under the collective name DHJ Polymath.

An example of the open science movement (see the Open Knowledge Foundation and the Open Science Summit), raising the question — is the ‘traditional’ way of doing science really the best way? Let’s bear in mind that the ‘traditional’ way is not in fact all that traditional. It’s a product of its times (and rather recent times at that). We shouldn’t confuse the process of scientific thinking with the institutionalization of science. Proponents of Open Science argue that the advent of the internet can break right through the inertia of the institutions, can allow collaboration and the processing of huge data-sets in ways that are far quicker and more efficient.

This is the world we need to educate for. Educate ourselves and our children. And the heart of it is collaboration. Which is one of the reasons we shouldn't be keeping social media out of the classroom. We just have to use it in the right way.

I began this series with Denmark allowing internet access during exams. So let's finish by returning to this issue.

As with the wider question of education, we need to ask ourselves what testing is for. First of all there's the point that, like note-taking, testing has an obvious purpose and a less obvious one. The obvious one is that it provides a measurement of how well a student knows something (we’ll get to the squirrelly ‘knows’ in a minute); the less obvious is that testing helps students learn. (For note-taking, the obvious purpose is that it provides a record; the less obvious is the same as for testing: it helps you learn.) Many tests may be (or perhaps should be) primarily for learning.

Final exams, on the other hand, are usually solely about assessment. But then we must ask, assessment of what? What do we mean by 'know'? There are topics within subjects which are 'core' — crucial details and understandings without which the subject cannot be understood — cell division in biology; atomic structure in chemistry. But there are many other details that you don't need to have in your head — but you do need to 'know' them enough so that you can find them readily, and fit them into their place readily.

Anyone who can write well and develop an argument in depth on a specialist topic in a three-hour exam period from the internet deserves to pass (I'm assuming, of course, that there are adequate guards against plagiarism!). As with course-work, access to the internet simply raises the standard.

 

These posts have all been rather a grab-bag. This is such a wide topic, with so many issues and everything is such a state of flux. To write coherently on this would require a book. Here I have simply tried to raise some issues, and point to a random diversity of articles and tools that might be of interest. Do add any others (issues, articles, tools) in the comments.

Successful remembering requires effective self-monitoring

We forget someone’s name, and our response might be: “Oh I’ve always been terrible at remembering names!” Or: “I’m getting old; I really can’t remember things anymore.” Or: nothing — we shrug it off without thought. What our response might be depends on our age and our personality, but that response has nothing to do with the reason we forgot.

We forget things for a number of short-term reasons: we’re tired; we’re distracted by other thoughts; we’re feeling emotional. But underneath all that, at all ages and in all situations, there is one fundamental reason why we fail to remember something: we didn’t encode it well enough at the time we learned/experienced it. And, yes, that is a strategy failure, and possibly also a reflection of those same factors (tired, distracted, emotional), but again, at bottom there is one fundamental reason: we didn’t realize what we needed to do to ensure we would remember it. This is a failure of self-monitoring, and self-monitoring is a crucial, and under-appreciated, strategy.

I’ve written about self-monitoring as a study skill, but self-monitoring is a far broader strategy than that. It applies to children and to seniors; it applies to remembering names and intentions and facts and experiences and skills. And it has a lot to do with cognitive fluency.

Cognitive fluency is as simple a concept as it sounds: it’s about how easy it is to think about something. We use this ease as a measure of familiarity — if it’s easy, we assume we’ve met it before. The easier it is, the more familiar we assume it is. Things that are familiar are (rule of thumb) assumed to be safe, seen as more attractive, make us feel more confident.

And are assumed to be known — that is, we don’t need to put any effort into encoding this information, because clearly we already know it.

Familiarity is a heuristic (rule of thumb) for several attributes. Fluency is a heuristic for familiarity.

Heuristics are vital — without these, we literally couldn’t function. The world is far too complex a place for us to deal with it without a whole heap of these rules of thumb. But the problem with them is that they are not rules, they are rules of thumb — guidelines, indicators. Meaning that a lot of the time, they’re wrong.

That’s why it’s not enough to unthinkingly rely on fluency as a guide to whether or not you need to make a deliberate effort to encode/learn something.

The secret to getting around the weaknesses of fluency is effective testing.

Notice I said effective.

If you intend to buy some bread on the way home from work, does the fact that you reminded yourself when you got to work constitute an effective test? Not in itself. If you are introduced to someone and you remember their name long enough to use it when you say goodbye, does this constitute an effective test? Again, not in itself. If you’re learning the periodic table and at the end of your study session are able to reel off all the elements in the right order, can you say you have learned this, and move on to something else? Not yet.

Effective testing has three elements: time, context, and feedback.

The feedback component should be self-evident, but apparently is not. It’s no good being tested or testing yourself, if your answer is wrong and you don’t know it! Of course, it’s not always possible to get feedback — and we don’t need feedback if we really are right. But how do we know if we’re right? Again, we use fluency to tell us. If the answer comes easily, we assume it’s correct. Most of the time it will be — but not always. So if you do have some means of checking your answer, you should take it.

[A brief aside to teachers and parents of school-aged students: Here in New Zealand we have a national qualifying exam (actually a series of exams) for our older secondary school students. The NCEA is quite innovative in many ways (you can read about it here if you’re curious), and since its introduction a few years ago there has been a great deal of controversy about it. As a parent of students who have gone through and are going through this process, I have had many criticisms about it myself. However, there are a number of good things about it, and one of these (which has nothing to do with the nature of the exams) is a process which I believe is extremely rare in the world (for a national exam): every exam paper is returned to the student. This is quite a logistical nightmare of course, when you consider each subject has several different papers (as an example, my younger son, sitting Level 2 this year, did 18 papers) and every paper has a different marker. But I believe the feedback really is worth it. Every test, whatever its ostensible purpose, should also be a learning experience. And to be a good learning experience, the student needs feedback.]

But time and context are the important, and under-appreciated, elements. A major reason why people fail to realize they haven’t properly encoded/learned something, is that they retrieve it easily soon after encoding, as in my examples above. But at this point, the information is still floating around in an accessible state. It hasn’t been consolidated; it hasn’t been properly filed in long-term memory. Retrieval this soon after encoding tells you (almost) nothing (obviously, if you did fail to retrieve it at this point, that would tell you something!).

So effective testing requires a certain amount of time to pass. And as I discussed when I talked about retrieval practice, it really requires quite a lot of time to pass before you can draw a line under it and say, ok, this is now done.

The third element is the least obvious. Context.

Why do we recognize the librarian when we see her at the library, but don’t recognize her at the supermarket? She’s out of context. Why does remembering we need to buy bread on the way home no good if we remember it when we arrive at work? Because successful intention remembering is all about remembering at the right time and in the right place.

Effective encoding means that we will be able to remember when we need the information. In some cases (like intention memory), that means tying the information to a particular context — so effective testing involves trying to retrieve the information in response to the right contextual cue.

In most cases, it means testing across a variety of contexts, to ensure you have multiple access points to the information.

Successful remembering requires effective monitoring at the time of encoding (when you encounter the information). Effective monitoring requires you not to be fooled by easy fluency, but to test yourself effectively, across time and context. These principles apply to all memory situations and across all ages.

 

Additional resources:

If you want to know more about cognitive fluency and its effect on the mind (rather than memory specifically), there's nice article in the Boston Globe. As an addendum (I'd read the more general and in-depth article in the Globe first), Miller-McCune have a brief article on one particular aspect of cognitive fluency -- the effect of names.

Miller-McCune have have a good article on the value of testing and the motivating benefits of failure.

Flashcards

Flashcards are cards with a word (or phrase) on one side and its translation on the other. You can buy ready-made flashcards, and these can certainly be helpful, particularly if you're inexperienced at learning another language. However, it is more effective if you make them yourself. Not only will the cards be customized to your own use, but the activity of selecting words and writing them down help you learn them.

A standard way of using flashcards is simply to go through a set number each day, separating out those you have trouble with, so you can review them more often. Keep these ones handy so that you can go through them at odd moments during the day when you're waiting for something.

Use the flashcards as a handy way to group words in different ways. Deal out the cards and move them around, looking for connections.

If you have word-family flashcards (recommended) - e.g., cards with various related forms of a word - you can make different sentences with your cards. You could also play cards with them, if you have others to play with. You could play a version of rummy, for example, where the sets are infinitive, present tense, future tense, past perfect. Use your imagination!

A bingo game with flashcards is another fun way to practice. Construct bingo cards (large cards divided into a certain number of spaces the same size as your flashcards) with the native language words on it. While this is better played with others, you can at a pinch play with yourself, simply picking out a flashcard from the pile and seeing how quickly you can match it with its counterpart.

Learning words in isolation will not help you much in dealing with words in context. You do need to practice reading/writing/speaking/listening sentences. But flashcards are a useful means of memorizing vocabulary.

Flashcard software

VTrain (Vocabulary Trainer): is flashcard software apparently used in the language labs of 40 Universities and hundreds of high schools; it's free for educational establishments. It's shareware.