There are so many different coursebooks, worksheets and other resources available for teaching English nowadays; how can teachers or directors of study choose which ones to use?
Additionally, as other posts in EFL magazine have discussed, how can we use them most effectively? (See, for example, “Making the most out of your coursebooks” by Charlie Ellis.)
The best way to choose a set of ELT materials for use in your classes is to analyse and evaluate key features of the material to make an informed decision. In this article, I want to share some advice on how to do this in a systematic way.
“Many evaluations are impressionistic, or are at best aided by an ad hoc and very subjective list of criteria.” (Tomlinson, 2013: 24).
Of course, any evaluation is a judgement – in this case about a material’s suitability – and never entirely objective. To avoid basing judgement on a flick through a book and our gut feeling, it is helpful to come up with a system for analysis. In this way, we can make our evaluation more rigorous and reliable. This usually requires a set of criteria that we can use to make evaluative judgments based on our principles of what makes materials ‘good.’ These will include knowledge of the target audience, general points, more detailed elements, and thinking beyond the specific materials at hand.
What are your principles?
As a first step, teachers or materials evaluators need to be aware of their own beliefs about what makes for good teaching, what leads to learning, and therefore, what makes materials effective in general. These principles might be related to learning and SLA theories, and/or be based on what we have studied, read, or researched, and on our own experiences as teachers and learners. We need to aim to make our beliefs transparent to others, so they can understand how we are approaching the evaluation. Then, we can draw up focused criteria, to create, for example, a form that can be used to analyse points in a thorough and systematic manner and to serve as the basis for evaluation. And, because evaluating with this list of criteria shows our beliefs explicitly, it is also good for professional development because it means reflecting on our own principles and where they come from.
Kath Bilsborough has written several useful articles on discovering your own principles as a teacher and/or materials writer, for example this one for IATEFL’s Materials Writing Special Interest Group entitled “More on emerging principles”.
Who is the material for?
Before we move on to creating our criteria checklist to analyse a material, we need to be aware of the fact that we are usually not evaluating material in isolation, or in a vacuum. Instead, we are evaluating the effect it is likely to have on the learners and teachers who use it. Therefore, we need to know about those people and the kinds of contact they will have with the materials.
To be as specific as possible, our evaluations need to assess the suitability and effectiveness for certain learners in a certain context.
Regarding the learners, we need to know a whole range of things that will impact upon our evaluation of the material: How old are they? What level are they working at? For what goals or purposes do they need English? What are they interested in? What are their expectations and preferences for learning? What other languages do they speak that might influence their English? And so on.
We also need to know about the teacher(s) and their preferences, about class format, and about what equipment and infrastructure is available to them. These factors form something of a reference point which takes account of the people and context, and they need to be kept in mind when we do our evaluative analysis.
What do we look at when evaluating materials?
In getting to the heart of the evaluative analysis the key question is: What is it that we look at when we evaluate a material?
For my trainee teachers, I group features to analyse and evaluate into three categories.
1) General points to evaluate
Firstly, a group of features that are quite general: we don’t need to look too closely to judge these points. We can start with the overall design: Is it easy on the eye and will it appeal to the learners? For example, analytical points here might look at the amount of text, or the images: the number or type (for example – photos/cartoons), the representation of different social groups (professions, races, disabilities), the images’ accessibility (colour schemes), and so on. You can then evaluate how suitable this makes the material, based on what we know about the specific learners and teachers.
We also need to judge whether the design and layout are logical to follow and not confusing. This is linked to overall clarity: Is everything clear – the order, the instructions, the language input? Note that ‘clear’ is not a good criterion for analysis. Instead, we should aim to first define measurable criteria on aspects that contribute to clarity. For example, are symbols or numbering systems used, and are they consistent? When it comes to evaluating clarity, we also need to consider whether the input and the explanations are comprehensible for the level, and so need to analyse specific markers of this.
Naturally, we will also want to look at what type of content and input there is. For example, what Englishes are used/presented? What topics or input sources are included? How are they presented and approached? Is it mainly text based or does the material work with other media, such as videos or audios? If so, where are these found – CDs, online? Our tool will need to include concrete measurable items to analyse these aspects.
The last point in this first category is what kinds of activities there are; we can analyse the constellations of working (individually, pairs, groups), or numbers and sequencing of closed or open-ended exercises. This can help us to evaluate whether the content and activities are repetitive or varied, whether they target different thinking skills, and in general, whether they fulfil our criteria for effective materials.
In sum, in this category, you are evaluating overall whether the material looks and seems like something that you deem “good/effective” in general, based on your principles of what makes for good teaching, and what leads effectively to learning.
2) Digging deeper into materials evaluation
Secondly, we have a group of aspects to evaluate that require us to look at the material more closely and to inspect things in more detail. These aspects are a bit more specific, and the evaluations will depend more on the particular context that you are thinking of using the materials in.
For me, the most important facet here is the material’s relevance to the learners and the course context. This means considering the points from above on the learners’ needs, goals, interests, ages, etc. For an evaluative analysis, it helps to think about how you can match features of the material to characteristics of the learner group and context. For example, if it’s a conversation class for learners wishing to holiday in Australia, how relevant is a material on writing business emails?
In this category, we should also assess the validity of the language points, skills, and topics covered. Does everything logically relate to what the material and course are trying to achieve? And link this to the viability of the material and activities, and whether it seems likely that these are going to successfully generate the learning we’re aiming for: are the learning objectives achievable with the input and activities in the material?
2a) Practical considerations for ELT materials
As a sub-category here, there are some concrete considerations regarding actually employing these materials in lessons. Thinking about practical usability in the context will mean considering whether the activities are intended for individual lessons or include group work, for example, or whether you will be able to manage to cover the material(s) in the time you have. Analytical questions here will include how many units or sections there are or how long the activities will take, which will then enable you to gauge whether this fits with the lesson format.
Within this list of practical considerations, we should also think about teachability and flexibility. Can the relevant teachers teach with the material, or does it not fit with their teaching approach, preferences, or classroom personality? A helpful point to analyse here is the guidance given for teachers; is there input on preparing lessons, running activities, and assessing learning using these materials and activities? And further, can the material be adapted for different groups of learners, local contexts, set-ups, learning preferences, and so on, if necessary? Is there guidance for teachers on how to do this?
As you can see, there are quite a few in-depth points to consider in this category. Still, many can be covered by yes/no or Likert-Scale points in our analytical tools. Moreover, these things are arguably more important than, for example, an appealing design. This is why I feel these points really form the heart of any evaluation.
3) Evaluating aspects beyond the materials
Finally, there are some aspects or considerations that move us further “off the page” so to speak: They require us to make calculations and judgments that go beyond the immediate use of the materials. We are still thinking about a particular context/group of learners, and attempting to assess how well accepted the materials will be, how learners will work and learn with them, and the less concrete outcomes of using these materials with these learners/in this context.
One facet here is motivation: will the materials motivate the learners to engage both cognitively and affectively, and thus be drawn into learning through active interaction with the materials? This is a hard aspect to break down into measurable criteria – but thinking back to the characteristics of the learners can help: If they’re interested in different cultures and traditions, for example, then we could count how many are presented in the material.
Another factor is the level of difficulty and support. Ideally we want learners to be working in their zone of proximal development. This ‘zone’ is just a little bit above the level they are currently using English at, and so it needs materials that build up to a manageable challenge, supporting the learners in moving to the next step in their development – both linguistically and cognitively. This idea is from Vygotsky’s constructivist theory, and you can read about how it can apply to ELT materials in Gargi Sarkar’s post “Constructivist theory, connection and rubrics”.
The aspects of motivation and difficulty are perhaps easier to define, and in my experience, likely to be linked to personalisation. So, an analytical point could ask whether there are opportunities for individual practice which allow learners to make connections to their own lives. Does the material enable users to personalise their own learning, the way they learn and activities they enjoy, according to their preferences?
The final consideration in this category takes us further “off the page” and looks beyond the course at hand: to the long-term effects of the material. This is hard to predict, but nonetheless worth considering: do the materials do what we need in the long term? For example, will working with these materials help learners to continue to learn after the lesson/course? Will they take something away that they can use in future?
Evaluative analysis: worth it in the end
Overall, what I aim to show my trainees and the readers of this article, is that a good materials evaluation is not something that can be done in a rush. There are quite a number of points that all deserve attention in any analytical tool or form we draw up as the basis for our evaluation. But I believe the time and thought that goes into looking at these aspects and applying our beliefs to them to formulate criteria with which we can set up a system to analyse materials evaluatively, is helpful for teachers’ and directors of studies’ professional awareness and development – as well as giving us a tool we can use over and over to evaluate teaching materials in a systematic, principled, thorough and reliable manner.
Over to you
What experience do you have with drawing up criteria for evaluating ELT materials? Is there anything you’d like to add to this list of points to consider?
Reference
Tomlinson, B. 2013, ‘Materials Evaluation’, in B. Tomlinson (ed), Developing Materials for Language Teaching. London: Bloomsbury
