Do we even know what we assess when we assess learning?

‘It took me quite some time to get the little girl to let me know what was bothering her,’ said Prof. A. K. Sharma, the former Director of NCERT. The year was 2000 and he was telling me about an incident from a class 2 maths period in the model school in the NCERT campus. The teacher had just completed teaching children subtraction of two-digit numbers with ‘borrowing’, and he had found two children hesitating over the problems they had been given to solve.

The first, a girl, had made a ‘mistake’ as she had failed to borrow from the tens side. Being a grandfatherly and kindly figure, he was able to cajole the girl to speak up. Very softly, looking down and away from him all the while, she said, ‘We learnt in the moral science class that borrowing is bad.’

Reeling from this, he approached the other child, a boy, and discussed why he had not completed his work on the problem. After much exchange, the boy said, ‘But why should I borrow 1? I want to borrow 2.’

Taking part in a recent session on ‘error analysis’, I was reminded of Prof. Sharma’s advice to engage with children to understand their ‘errors’ rather than rely on their work on paper. In numerous assessment experiences since, I’ve seen children who are otherwise very competent falter because of an issue at home or a fight with a friend or because they are being bullied. In open-ended questions in language, teachers are hard put to identify if there really is an ‘error’ or if the child’s view is a valid, logical interpretation. (And asking only close-ended questions is hardly sufficient to understand children’s abilities.) It becomes even more difficult when it comes to children from marginalized backgrounds – as they encounter discrimination and even denigration (of their background, language or culture), they often resist by ‘not-learning’ or do not answer out of fear of being ‘disciplined’.

As the evaluation industry expands in the Indian context with more and more professionals taking in rigorous analysis of children’s responses and analyses of their ‘errors’, the tendency is to interpret these within the framework of the subject for which the test was conducted. But do we know what we really assess when we look closely at children’s responses? What if it’s not a maths or language issue but something else altogether?

Detention For Adults?

To all those who are convinced that the non-detention policy is harming education…

Children’s apparent lack of learning becomes an issue mainly because it is easy to see that they have missed out on something. The fact that at a younger age learning is very fast and that clear milestones are available helps us perceive this – and therefore apply all kinds of expectations, tactics, at times even coercion to ‘ensure’ learning – one such being the detention system which, many believe, is needed in order to maintain ‘quality’. By making children lose a year because we couldn’t ensure their learning (and blaming them for it), we feel we can generate the fear required to make them ‘serious’ and learn.

If we are convinced about this, why should it apply only to school education? What if we could lay out clear benchmarks for adults to learn and grow – in general as well as in the work they do. Certainly it is possible to have a life-long ‘curriculum’ with two-year benchmarks (over their entire careers, and even post retirement) for educationists and curriculum developers, teachers, HMs, government officials, managers, businessmen, fathers and mothers (and grandparents), journalists, artists, municipal staff, auditors, accountants, administrators, intelligence agents and politicians. What if there was a ‘detention system’ (in terms of not being allowed to be promoted or get a pay increase or being sent back to some lower ‘grade’)? Yes, in some government jobs there is an ‘efficiency bar’ and the supposed HR policies and internal competition are expected to sort this out. But do they?

Can we as a nation claim that we have, every year, demonstrated the improvement required to declare ourselves ‘promoted’ to the next level (whatever that is)?

And what happens when police are unable to reduce crimes, leaders are unable to ensure the welfare of the poor, systems are unable to deliver basics such as electricity / water / education / health, or societies are unable to get men to have basic respect for women?

Who should be ‘detained’?

Why Measuring Learning Outcomes Does Not Improve Accountability in Education – Or Outcomes

In the last few years, the clamour for measuring learning outcomes and using that as a means to ensure accountability has grown louder. In fact the current Five Year Plan insists that learning outcomes be measurable and be measured. Corporate houses funding various foundations and NGOs are big on learning assessment and look to it as a means of bringing about improvement. Many sensible people are voicing views to the effect that if a teacher is unable to generate learning outcomes, he should be shoved aside and replaced by someone better. And, of course, the feeling persists that we are not measuring the quality of learning enough.

This is unfortunate. Not because measuring outcomes is not important or somehow wrong but because the present formulations of the issue are simplistic to the extent that they prevent underlying issues to be addressed. Here is how.

First, it is not as if the quality of learning is not being measured, or has not been measured in the last 20 years. The first all-India survey of learning levels was conducted by the NCERT in 1995, and there have been many since. Several large-scale independent studies of students’ learning levels have been run, including ASER and surveys of Education Initiatives. Small-scale learning assessments have been conducted for innumerable research studies (e.g. of 1 lakh children in Tamil Nadu to assess the state’s Activity Based Learning Programme) or pilot projects (for instance, several states have piloted their textbooks and used learning achievement as a benchmark). And of course at least hundreds (if not thousands) of NGOs/NGO-run programmes (often in government schools) have incorporated assessment as an effectiveness measure.

There are thus any number of assessments available – and they've been telling us for the last twenty years that our children are not learning. Only, this doesn’t seem to have resulted in improved learning, thus questioning the assumption behind the clamour for measurement.

This is a little like weighing a child to assess the level of nutrition – unfortunately, merely weighing the child will not lead to better nutrition... Something else is clearly required, and that doesn’t seem to be happening.

Second, insisting on having 'measurable' outcomes is hugely misleading – just because you can measure something doesn't make it more worthwhile (e.g. we do want students to be creative or considerate or civic though there are no easy measures for these). Several of the assessments mentioned suffer from this. Thus an Adivasi child who displays great resourcefulness, knowledge of the environment and concern for others would be called poorly educated since the ‘tests’ measure only basic literacy and numeracy.

Measuring outcomes would be useful only when we measure what matters most to us. Not whether a child can read something aloud but whether he can form an opinion on it and give the reasons behind them. Not whether a child can do calculations but whether she can apply it in real world contexts to solve problems or take a decision. Some of these may be hard to measure, but it would be useful to remember that it is not the purpose of education to be assessable, but the purpose of assessment to measure what is considered most worth learning.

Third, measuring outcomes does not account for contexts and tends to disadvantage (and label) those facing adverse conditions. Which then makes it even more difficult for them to improve. There are many teachers who work very hard in difficult conditions – but don't attain the kind of outcomes expected because the curriculum assumes children will be able to attend daily or speak the school language at home (and several other such notions), which don't apply to the children they work with (some 60-70% in India). We'll end up shoving these teachers out if we take the advice to replace them – instead of overhauling the system which has designed itself in such a way that marginalized children WILL fail.

Fourth, there is a danger that the present focus on outcomes is actually obfuscating – instead of increasing – accountability. India's challenges now arise from its success in rapidly expanding the school system to bring in so many children. The consequence is that we now have students (at all levels) who traditionally never attended schools - working children, migrant groups, girls from various communities, children with disabilities, socially excluded communities…. the list is endless. What this means is that while the nature of our students has changed, the curriculum, pedagogy and assessment remain as they used to be and so, the DESIGN ITSELF leaves these learners out.

At a second level, when it comes to implementation, there is a tendency in those responsible to ignore laxity on the assumed ground that it is only happening to those who do not matter. (Just as it is easier to ask a poor person to push a stalled car rather than a well-dressed one, similar prejudices operation in all facets of our society, including government officials.) Even now, therefore, it is mainly those from better-resourced families who continue to succeed, and we continue to have poor education for the poor. So the accountability really needs to be demanded at the level of the system (NCERT, MHRD, Departments of Education) and state / district / block officials.

As long as people keep pointing fingers at teachers as the main villains, the really responsible will continue to escape accountability. For instance, when the NCERT's own national survey shows low levels of learning, why does nothing happen to anyone at any level, including the NCERT itself (whose curriculum has been taken by many states now performing poorly)? How come officials at various levels continue exactly as they have been for decades with impunity when every measure  brings out dismal levels of learning in their watch? Recently, when our group, IgnusERG assessed class 9 students in a district we found 68% of them to be at class 4-6 levels, 7% below class 3 level, and only 4% at the class 9 level where they were expected to be. When this finding is shared, everyone finds a way to blame some one else!

Finally, let me leave you with this – in the current form, knowledge of outcomes attained does not help bring about improvement. Most states will be implementing SLAS (State Learning Assessment Survey) in the coming months. But once a state finds out it is performing poorly, say, in mathematics, that will not inform it of the reasons why this is so. It could be the poor curriculum (e.g. overambitious expectations) or weak syllabus (less time allocated than required), or inappropriate pedagogy (no use of concrete materials at an early age) or bad textbooks (poorly sequenced or giving discrete rather than contextual examples) or demotivated teachers or insufficient teaching time (because the state continues using teachers for non-teaching tasks even after RTE and court orders to this effect) or home vs school language issues or at least 10 other problems that can be named, each of which can seriously lead to poor outcomes. So where will the improvement begin?

The point, as mentioned earlier, is: do ask for outcomes, but don't keep it simplistic, or we'll continue to get the poor outcomes we've been documenting over the last 20 years.

So exactly what IS CCE, and what it is NOT!

Take a look at the video here. Do let me have your comments.

The Myths of Tests - Or Going Around In a Circle

Sometimes, it's plain amazing to see the kind of debates taking place around the world. Here's a piece dealing with something we've often ranted against, the 'standardized tests'. Some of the phrases and terminology will seem remarkably familiar to readers here.

http://www.drdouggreen.com/wp-content/Myths-of-Tests.pdf

A great deal of this discussion took place in India when the MLLs were formulated, with behaviorism coming under attack. The new National Curriculum Framework 2005 is as 'anti-behavioral' as it can be. And the Right to Education Act mandates Continuous Comprehensive Evaluation rather than standardized testing. Board exams too have been pushed beyond Grade 10. Nevertheless, the pressure to use learning achievement data as a proxy for assessing the quality of the education system remains high (this is what Pratham's ASER implements every year). This summary, by practitioners as well as academics / researchers, captures some of the myths associated with testing.

PS: I'm NOT trying to promote sales of this book!

The Dangers of Relying on Testing as a Means of Improving Quality of Education

Over the last two decades, if there's one thing we've learnt it is that testable results alone do not account for educational quality or the quality of educational experience. The processes and relationships are as important (if not more) than what gets tested. Using test results as a proxy for quality, and hoping that an emphasis on testing will lead to improvement, is a managerial rather than educational approach (this sounds a little screechy, I know, but I feel screechy looking at the kind of damage this naive assumption is doing all around).

Take a look (link below) at emerging fallouts of one such large-scale testing based improvement programme, enacted through the No Child Left Behind law in the US.

   82 percent of US schools may be labeled 'failing'

Admittedly, this is only a news report, and there are also defenders, but the difficulties being faced by schools are real. There is also no evidence that whatever improvements that might have been brought about are long lasting, or that the nature of classroom processes and relationships has improved, or that children and teachers both find the school more worth coming to.

If you're an administrator or accountability activist or from an organisation that funds educational efforts - do consider that you might be doing yourself a disservice by insisting on testing as the means of assessing impact or of ensuring improvement. There's a lot more to education in the 21st century, and it does not lend itself to such simplistic frames!