Why Measuring Learning Outcomes Does Not Improve Accountability in Education – Or Outcomes

In the last few years, the clamour for measuring learning outcomes and using that as a means to ensure accountability has grown louder. In fact the current Five Year Plan insists that learning outcomes be measurable and be measured. Corporate houses funding various foundations and NGOs are big on learning assessment and look to it as a means of bringing about improvement. Many sensible people are voicing views to the effect that if a teacher is unable to generate learning outcomes, he should be shoved aside and replaced by someone better. And, of course, the feeling persists that we are not measuring the quality of learning enough.

This is unfortunate. Not because measuring outcomes is not important or somehow wrong but because the present formulations of the issue are simplistic to the extent that they prevent underlying issues to be addressed. Here is how.

First, it is not as if the quality of learning is not being measured, or has not been measured in the last 20 years. The first all-India survey of learning levels was conducted by the NCERT in 1995, and there have been many since. Several large-scale independent studies of students’ learning levels have been run, including ASER and surveys of Education Initiatives. Small-scale learning assessments have been conducted for innumerable research studies (e.g. of 1 lakh children in Tamil Nadu to assess the state’s Activity Based Learning Programme) or pilot projects (for instance, several states have piloted their textbooks and used learning achievement as a benchmark). And of course at least hundreds (if not thousands) of NGOs/NGO-run programmes (often in government schools) have incorporated assessment as an effectiveness measure.

There are thus any number of assessments available – and they've been telling us for the last twenty years that our children are not learning. Only, this doesn’t seem to have resulted in improved learning, thus questioning the assumption behind the clamour for measurement.

This is a little like weighing a child to assess the level of nutrition – unfortunately, merely weighing the child will not lead to better nutrition... Something else is clearly required, and that doesn’t seem to be happening.

Second, insisting on having 'measurable' outcomes is hugely misleading – just because you can measure something doesn't make it more worthwhile (e.g. we do want students to be creative or considerate or civic though there are no easy measures for these). Several of the assessments mentioned suffer from this. Thus an Adivasi child who displays great resourcefulness, knowledge of the environment and concern for others would be called poorly educated since the ‘tests’ measure only basic literacy and numeracy.

Measuring outcomes would be useful only when we measure what matters most to us. Not whether a child can read something aloud but whether he can form an opinion on it and give the reasons behind them. Not whether a child can do calculations but whether she can apply it in real world contexts to solve problems or take a decision. Some of these may be hard to measure, but it would be useful to remember that it is not the purpose of education to be assessable, but the purpose of assessment to measure what is considered most worth learning.

Third, measuring outcomes does not account for contexts and tends to disadvantage (and label) those facing adverse conditions. Which then makes it even more difficult for them to improve. There are many teachers who work very hard in difficult conditions – but don't attain the kind of outcomes expected because the curriculum assumes children will be able to attend daily or speak the school language at home (and several other such notions), which don't apply to the children they work with (some 60-70% in India). We'll end up shoving these teachers out if we take the advice to replace them – instead of overhauling the system which has designed itself in such a way that marginalized children WILL fail.

Fourth, there is a danger that the present focus on outcomes is actually obfuscating – instead of increasing – accountability. India's challenges now arise from its success in rapidly expanding the school system to bring in so many children. The consequence is that we now have students (at all levels) who traditionally never attended schools - working children, migrant groups, girls from various communities, children with disabilities, socially excluded communities…. the list is endless. What this means is that while the nature of our students has changed, the curriculum, pedagogy and assessment remain as they used to be and so, the DESIGN ITSELF leaves these learners out.

At a second level, when it comes to implementation, there is a tendency in those responsible to ignore laxity on the assumed ground that it is only happening to those who do not matter. (Just as it is easier to ask a poor person to push a stalled car rather than a well-dressed one, similar prejudices operation in all facets of our society, including government officials.) Even now, therefore, it is mainly those from better-resourced families who continue to succeed, and we continue to have poor education for the poor. So the accountability really needs to be demanded at the level of the system (NCERT, MHRD, Departments of Education) and state / district / block officials.

As long as people keep pointing fingers at teachers as the main villains, the really responsible will continue to escape accountability. For instance, when the NCERT's own national survey shows low levels of learning, why does nothing happen to anyone at any level, including the NCERT itself (whose curriculum has been taken by many states now performing poorly)? How come officials at various levels continue exactly as they have been for decades with impunity when every measure  brings out dismal levels of learning in their watch? Recently, when our group, IgnusERG assessed class 9 students in a district we found 68% of them to be at class 4-6 levels, 7% below class 3 level, and only 4% at the class 9 level where they were expected to be. When this finding is shared, everyone finds a way to blame some one else!

Finally, let me leave you with this – in the current form, knowledge of outcomes attained does not help bring about improvement. Most states will be implementing SLAS (State Learning Assessment Survey) in the coming months. But once a state finds out it is performing poorly, say, in mathematics, that will not inform it of the reasons why this is so. It could be the poor curriculum (e.g. overambitious expectations) or weak syllabus (less time allocated than required), or inappropriate pedagogy (no use of concrete materials at an early age) or bad textbooks (poorly sequenced or giving discrete rather than contextual examples) or demotivated teachers or insufficient teaching time (because the state continues using teachers for non-teaching tasks even after RTE and court orders to this effect) or home vs school language issues or at least 10 other problems that can be named, each of which can seriously lead to poor outcomes. So where will the improvement begin?

The point, as mentioned earlier, is: do ask for outcomes, but don't keep it simplistic, or we'll continue to get the poor outcomes we've been documenting over the last 20 years.

If there were just three things you could change in elementary education, what would they be?

We're now at a point where we probably have far too many ideas of what needs to be changed in order to bring about real and lasting improvement in elementary education in India. It's a confusing, mammoth list of to-do items! We need to prioritize, both on basis of what is important and what is easy to start with. Hence this request - if there were just three things you could change, what would they be?

Pseudo Solutions for Real Educational Problems

Ask an intelligent question and get a ________ reply!

Here's an experiment. It seems to work well with functionaries from educational systems in India, Bangladesh and several other countries in South Asia and beyond.

Bring together a group of educational personnel such as academic supervisors, district and state / provincial educational officials. Pose a critical educational problem before them. Of the kind that they probably deal with on a daily basis, such as:

• How to improve learning among children? Or
• What action to take so that classroom processes become more interactive than they are at present? Or
• How to enable children to enjoy learning mathematics (rather than being afraid of it)? Or
• How to ensure and increase teacher attendance?

 Now, monitor the responses you get. They will usually include answers such as:

Teachers must be dedicated / devoted to the profession.

• We must ensure that the system functions well.
• We must increase monitoring and do it properly.
• Teachers must be made aware of their responsibilities.

That didn't surprise you, did it? These are the typical answers one hears (you can probably increase the list greatly). But why should these answers worry us?

Because these answers are positively dangerous!

Either they reveal that our education system is in the hands of people who don't know what to do. Or, what is worse, it is in the hands of those who know what to do but are trying to hide behind these kinds of answers.

You can decide for yourself – after taking a look at the explanation below.

THE RICH VARIETY OF PSEUDO SOLUTIONS

The key issue is that instead of actions and concrete steps, those responsible come up with other things instead. From an educational planning point of view, a step or an action is something that you have to do, that you can set in a clear time-frame, that can be budgeted, broken down into clear parts to be implemented. It is not a vague statement of good intent.

And by coming up with statements that are not actions or steps, those making these statements are actually preventing solutions from really coming about. Here's how.

Give a quality instead of an action

A commonly offered 'solution' – 'teacher must be dedicated to his profession' – is not an action but a quality, the outcome of many other steps that we would have to take. Since those talking are often even responsible for recruiting teachers (and they did not take into account whether the potential teacher had a sense of 'dedication' or not), they need to discuss exactly how this dedication will now be ensured. E.g. by conveying to teacher that they matter, are valued, by visiting them, developing and disseminating performance standards (and using them to identify good performance, in an objective manner), or by setting role models in the form of the seniors themselves following a code of conduct, or a thousand other activities…. But instead of concrete action, we are presented sermons. Basically, offering a quality instead of a step merely looks like a ploy to avoid the necessary!

Defer the solution through 'action' that keeps on requiring further action

Do you remember those little 'Russian' dolls we used to get long ago – you lifted one and found another doll inside it, and another one inside, and so on. This variety of 'pseudo-solution' is just like those dolls. Here, the proposed solution is nothing but a guise to postpone committing oneself to actual action. For instance, you commonly hear suggestions such as 'Teachers must be trained properly', which begs the question: 'What should we do to ensure that teachers are trained properly?' Answer: 'We must have proper trainers.' But: 'How will we get proper trainers?' Answer: 'By recruiting them properly.' And so on. The solution is never really in your grasp; it keeps on evading you because it contains in itself yet another question, the answer to which contains another one…. Even Socrates with his Socratic Method would have had a tough time pinning down the actual action required. Lesser mortals like us just go mad and give up!

Show resolve, not necessarily solve!

Here, the answer to the critical problem is in the form of some very resolute-sounding statement. It gives the feeling that people are 'very serious' about doing something (never mind if scratching the surface shows that it can't really be converted into action). Pseudo-solutions of this category sound like this: 'We must ensure discipline.' Or 'We have to cover every single school.' Or 'The inputs must be made regularly.'

Nothing wrong with these statements, except that they are only resolutions and not clear steps or concrete action. They don't take into account that the present action, which is so strongly being proposed to be improved, may itself not be the right action to start with. Or may not even have anything wrong to begin with. For instance, before concluding that inputs must be made regularly, we need to take into account that perhaps the inputs may be inappropriate, and making them regular will not help. Also, the feeling is that having said that they will be regular, what are the steps to make them regular? (E.g. use of scheduling software and training everyone it its use, or interactions to discuss the needs of the different components of our programme in terms of regularity as well as the nature of inputs needed, and exploring whether more than just regularity it is how well they are implemented that needs to be improved…)

Once again, the feeling is that having declared something solemnly, it will now happen. Unfortunately, it doesn't.

Everyone except us!

This is encapsulated in statements that exhort everyone to pull up their socks (or equivalent), except the people making these statements. Hence in a discussion on the kind of improvements required to increase the effectiveness of an educational system or a programme, it will be said that teachers must be devoted / dedicated, that supervisory staff must be capable, that managers must be professional and administrators sensitive and flexible apart from being committed. Such statements will be made about categories other than that of the solution givers, of course! And of course it is still not clear as to how the suggested change will be brought about.

The monitoring myth

For some reason as yet not very clear, a lot of proposed solutions have to do with monitoring – it is pointed out that monitoring is very poor, ineffective, irregular, and several other words that I'm sure you can reel off. Well, excuse me, but monitoring is extremely limited as a solution. A commonly used example: I'm monitoring the weight of a child regularly and it keeps on decreasing – all this regular monitoring does not help me if I don’t know what to do – the kind of nutrition to ensure, how to obtain / procure and prepare the required nutrients and enable the child to ingest these in an appropriate manner over the required period… I can keep monitoring without necessarily bringing about any improvement.

The dangerous part is the feeling that programmes and systems work only if they are monitored. Not necessarily – in order to work well, those involved need to feel that they are doing something worthwhile, that someone cares that they are there, that the task is challenging yet doable and enjoyable, that they are equipped to do it, enjoy doing it and are supported in their actions. Under these circumstances monitoring can indeed play a role to enhance effectiveness, but it is no substitute for the basics that need to be in place. It's a little bit like a car that has a very good speedometer and odometer (monitoring devices), but no engine (implementation requirement)! Good monitoring is not necessarily equal to good implementation.

An accompanying myth is that better planning is the solution. In fact, if you look at the kind of technical professionals brought in by donor agencies, multilaterals, development partners and even governments, there seems to a far greater concentration on the planning and the monitoring/evaluation parts, but very little on the stuff that comes in between the two – i.e., implementation! And that is why, when educational functionaries are asked to come up with solutions or steps that will lead to specific outcomes, they tend to suggest action related to better planning and monitoring, rather than improved implementation.

The thesis, and a question

So that's the thesis – that when asked to identify actions / steps / solutions to address critical educational issues, those responsible come up with things that might look like them but are not the real thing. And it is this that has kept us back, preventing the huge amounts of money and effort being invested from translating into reality.

But if this is actually the case, is it due to sheer incompetence, or is it a deliberate ploy to ensure that real change does not happen (because behind it all, people are very uncomfortable with an education system that actually works). If you're a conspiracy theorist too, let me know!

Listening Workshops - Or the Simplest Step to Educational Reform

Is 'bottom up' change really possible?
If you are an educational functionary, by now you must be  fed up of hearing how planning and change have to be 'bottom up'. By which is usually meant that those who are 'under' you must somehow begin to contribute, own and implement a range of actions. And you inwardly wonder if this is ever going to happen!

It was during a discussion on precisely such views that the idea of a listening workshop emerged. Colleagues in the Institute of Educational Development (IED) in BRAC University, Bangladesh felt that a 'listening workshop' might help them understand teachers and grassroots functionaries better.

Listening workshop - a straightforward structure
It was agreed that before forming any views, it is critical to simply listen to teachers and head teachers. Hence a straightforward meeting / interaction / workshop was designed around the following three questions that would be asked of teachers and head teachers:

• What do you really do? Exactly what does your work involve?
• What do you like doing?
• What do you find difficult or dislike doing?

It was also agreed that IED colleagues initiating the discussion would only listen, and not prompt or provide leading questions or offer any comment from their side. In other words, they really had to listen rather than talk!

So why is all this worth writing about? Because around ten such listen workshops were actually conducted, and most turned out to have  a very interesting pattern, followed by an unexpected twist.

What teachers felt
The listening workshops, it transpired, tended to proceed in the following stages.

• Teachers found it really difficult to believe that anyone could come down from the capital only to listen to them! There had to be a 'hidden conspiracy' or an 'agenda' they were not aware of... It would take anywhere from 40-60 minutes to convince the participants that the intention really was to listen to them. (What do you think this tells us about the functionaries that teachers usually deal with?)
• Once teachers believed the above, their initial reaction was that of giving vent to all their frustration and anger at 'you people who sit up there and form all kinds of views about us without ever visiting the field and observing the realities for yourself.'
• Finally, teachers would pour their hearts out on the three questions given above.

The teachers' replies have of course begun to inform the work of the institute in many ways. However, it was the completely unanticipated outcome below that left everyone (cautiously) elated.

The unexpected 'reform'
In the case of a large number of teachers who participated, a few days after the listening workshop it was found that they were implementing many new pedagogical actions in their classrooms! In the entire discussion, at no point had they been asked to make any improvement in their classrooms. So it was not as if teachers did not know improved methods - a large number of in-service interactions had ensured that they had had exposure. It's just that they were not using them. But for some reason the listening workshops triggered a change process in the classrooms!

What do you think this tells us about teachers, about their motivations, and about the kind of relationships they experience? If you can bear the initial first hour, isn't holding a listening workshop the simplest way to initiate educational reform at the local level?

The Poor Make an Educational Choice

Though it had been around for a long time, in 2003-04, a disturbing trend began to be dramatically visible in the government school system: a large number of districts began to report a decrease in the number of children enrolled. However, this decrease was not due to any slowing down in the growth rate of child population. Nor was it because accurate data was now available in place of the earlier inflated numbers. And since the number of children reported to be out of school was not increasing either, what accounted for the children missing from government schools? Yes, you guessed it – they were shifting to the ever-spreading network of the low-fee private schools.

The number of districts reporting such decreased enrolment stood at 180 or nearly one-third the number of districts in the country. Nor was this confined to the so-called ‘backward’ states – for Andhra Pradesh, Delhi, Gujarat, Himachal Pradesh, Karnataka, Maharashtra and Tamil Nadu also reported the phenomenon. In the year 2005-06, six new states reported districts with decreasing enrolment in government schools. And the situation hasn't really improved since.

The private schools that children migrate to come under the ‘unrecognized’ category, hence few government records are available on their numbers or growth. However, it is apparent that the increase in their numbers is astonishing. A World Bank study estimated that 28% of the rural population in the area studied had access to private schools in their own villages, and nearly half the private schools were established after 2000. Studies in Punjab showed that around 27% children studied in such schools and a similar picture obtained in Andhra Pradesh and Orissa. 

This large-scale exodus has been occurring at a time when the government is spending an unprecedented amount of money and effort on education. Since 2000, tens of thousands of new schools have been opened in underserved areas and the infrastructure of existing schools boosted. Around 8.5 lakh teachers have been appointed and around 87% teachers in place provided 20 days of in-service training every year over the last few years.

Despite such efforts, anybody with any means whatever is choosing to walk across to a (usually) nearby school and pay for what they consider good education. This is in a context where education is available free in government schools, along with other incentives such as free textbooks and mid-day meals.

Like mobile phones, private education is no more the preserve of the elite. Surveys have found that 20% students in such schools are first-generation school-goers, with another 14% having parents with four (or less) years of education. Visits to such schools in the poorer regions of a state like UP put all doubts to rest. Without fail, it is the poor who are sending their children to schools that charge fees in the range of Rs. 30-100 a month. Schools manage this by paying teachers Rs. 1000-1200 per month – well below the minimum wage for unskilled labour. It is usually the educated unemployed who take this up as a means to gain experience while being on the lookout for other jobs. Therefore, teacher turnover is high, but there is a continuous stream of cheap labour available. The result is a commercially viable venture that provides subsistence level education.

In the meantime, who remains in the government system? For those hovering around the poverty line or below, there is no other recourse. Over 80% of SC and ST children in school are in government schools, which also have a higher proportion of girls and children with disabilities. In a telling comment, it is common for families with meagre resources to educate their sons in private schools and daughters in government schools. Indeed children are often enrolled in the government schools (for entitlements such as mid-day meals or uniforms) but actually attend the nearby private schools (for education)!

Unfortunately, the exodus of the more powerful and influential families has led to a greatly reduced sense of accountability in government schools. Those who are ‘left behind’ are usually the more disadvantaged groups, already disempowered due to economic and social reasons. Teachers, school heads and education officials tend to feel that it is almost ‘pointless’ to serve ‘these people’. In fact, a common refrain across the country is to complain of the ‘poor stuff we get to teach’ (and by 'poor stuff' they mean children!). There is an increasing tendency to blame the poor for not being able to support their wards at home or provide educational resource and the like. What is forgotten in all this is that education is not a favour being done to the poor – it is their right!

This is perhaps one of the reasons why the dramatic increase in inputs into the education system has not led to outcomes in terms of children’s learning levels, which continue to remain abysmal. Surveys by the NCERT and the NGO sector have repeatedly brought out how only half the children seem to learn half of what they should! During field visits to government schools, it is very common to come across children sitting unattended in class, with the teacher either absent or simply not teaching. Often, of course, the teacher has more than one class to handle and is therefore unable to teach. However, it is the sheer lack of concern for children that strikes any observer the most.

Many take the view that the expanding number of private schools is contributing to universalisation of elementary education in the country. While that is certainly true to an extent, a greater impact seems to be that in leading to reduced accountability, private schools are also contributing to a reduction in the government’s ability to universalise education in its own schools.