Do we even know what we assess when we assess learning?

‘It took me quite some time to get the little girl to let me know what was bothering her,’ said Prof. A. K. Sharma, the former Director of NCERT. The year was 2000 and he was telling me about an incident from a class 2 maths period in the model school in the NCERT campus. The teacher had just completed teaching children subtraction of two-digit numbers with ‘borrowing’, and he had found two children hesitating over the problems they had been given to solve.

The first, a girl, had made a ‘mistake’ as she had failed to borrow from the tens side. Being a grandfatherly and kindly figure, he was able to cajole the girl to speak up. Very softly, looking down and away from him all the while, she said, ‘We learnt in the moral science class that borrowing is bad.’

Reeling from this, he approached the other child, a boy, and discussed why he had not completed his work on the problem. After much exchange, the boy said, ‘But why should I borrow 1? I want to borrow 2.’

Taking part in a recent session on ‘error analysis’, I was reminded of Prof. Sharma’s advice to engage with children to understand their ‘errors’ rather than rely on their work on paper. In numerous assessment experiences since, I’ve seen children who are otherwise very competent falter because of an issue at home or a fight with a friend or because they are being bullied. In open-ended questions in language, teachers are hard put to identify if there really is an ‘error’ or if the child’s view is a valid, logical interpretation. (And asking only close-ended questions is hardly sufficient to understand children’s abilities.) It becomes even more difficult when it comes to children from marginalized backgrounds – as they encounter discrimination and even denigration (of their background, language or culture), they often resist by ‘not-learning’ or do not answer out of fear of being ‘disciplined’.

As the evaluation industry expands in the Indian context with more and more professionals taking in rigorous analysis of children’s responses and analyses of their ‘errors’, the tendency is to interpret these within the framework of the subject for which the test was conducted. But do we know what we really assess when we look closely at children’s responses? What if it’s not a maths or language issue but something else altogether?

What happens when you seriously try to empower children, teachers and community through large scale education initiatives?

The pervasive notion that 'nothing has been done in education in India' could not be further from the truth. In fact not only has a great deal been done, but its consequences have been faced over decades. In particular, what follows applies to introducing educational designs based on local context, using the experiences and strengths of the stakeholders, creating a situation where they play an active role in determining and implementing processes.

Though obviously much must have been done over the decades till the 80s, my experience ranges from mid-80s, when I was part of a team working on such classroom practices, textbooks and educational designs from 1986 onwards. Implementation of the programme called Prashika (Prathamik Shiksha Karyakram) focused on marginalised groups, with the team living in a tribal area as well as in a rural, deprived pocket and introducing the innovation in government primary schools. The work in Prashika was pathbreaking in many, many ways (integration of 5 subjects at the primary level, incorporation of multiple local languages, a hugely localised textbook/workbook that could only be completed with each child contributing, called Khushi-Khushi - still not matched anywhere, I believe). It provided hope that much was possible despite the difficulties faced and informed many of the later efforts that followed, both in the government and the NGO sector.

Later in DPEP - particularly Kerala, Assam, Karnataka, Haryana, UP, Bihar, TN, Nagaland and later with SSA Gujarat further work was done. Localised training, contexualisable textbooks (some really brilliant stuff still not matched anywhere - and that's a professional opinion), teacher determined assessment system, involvement of community knowledge, children constructing local histories / local environment books, peer learning and assessments, textbooks that would be 'complete' only along with a set of 50 district-specific books kept in the school library.... many, many innovative and large scale measures were conceived and actually implemented using a strategically developed implementation plan. 

In each first five states we were able to see 2-3 years of implementation, development of hundreds / thousands of teachers who implemented contextualised learning, a high degree of in-class practice backed by supportive, localisable material. These states changed their position in the national achievement surveys too, with Kerala rising to the top (it had been fairly close to the bottom before this, below Bihar in the first national survey). In the case of Gujarat, field testing was done in 630 schools, researched by MSU Baroda with very encouraging findings. 

However, as long as we were not visibly successful there were no problems. When change began to be visible on some scale and a palpable sense of energy was witnessed among teachers and communities, alarm bells began to ring. in each of these states, the powers that be - especially at state level, state institutions, administrations, political parties - found that this went against the command-and-control structures conducive to them being able to assert their authority. Schools didn't want to be told what to teach when and how - they had their own plans. Empowered teachers / school heads / even some VECs refused to kowtow to mediocre ideas or corruption oriented bosses - leading to huge conflicts all over the place. Unfortunately these never got reported, recorded or researched. The results were mass scale transfers, cases against state project directors who encouraged this (Kerala SPD was charge sheeted, Karnataka SPD given punishment posting in North Karnataka, Assam SPD sent to conflict zone during worst riots, Bihar SPD transferred to PHED and later kept without posting), the re-casting of State Resource Groups from those selected for tested capabilities to those stocked with ex-officio positions, the emasculation of the BRC-CRC structures from genuine teacher support institutions into data collection centres (believe it or not, we did have functional BRCs CRCs at one time!), the centralisation of powers away from the VECs and re-casting into SMCs with a different function, and major shift in recruitments away from districts to states (in one state the Education Minister held a Recruitment Mela in a stadium to personally appoint 3000 para-teachers). 

Interestingly, Prashika in MP faced a similar adminstrative backlash and was closed down.

Yes, like it or not, this is what ideas of empowerment through education come up against - and they fall short not because of lack of any purity in the idea itself or absence of rigour, but because after a point when it goes into implementation an idea is something else, and not its original pure self. You might look at the actual work and find it is not 'up to the standard' - yet when trying to create it for those who need education the most, other aspects need to be taken into account. Basically, empowering the weak is clearly seen by the strong as disempowering them - and the empire strikes back! One of the outcomes is that a few years later, it appears as if nothing has been done, and people gear themselves up to again come up with 'innovative' ideas, often weaker than might already have been tried, uninformed by the past.

Why Measuring Learning Outcomes Does Not Improve Accountability in Education – Or Outcomes

In the last few years, the clamour for measuring learning outcomes and using that as a means to ensure accountability has grown louder. In fact the current Five Year Plan insists that learning outcomes be measurable and be measured. Corporate houses funding various foundations and NGOs are big on learning assessment and look to it as a means of bringing about improvement. Many sensible people are voicing views to the effect that if a teacher is unable to generate learning outcomes, he should be shoved aside and replaced by someone better. And, of course, the feeling persists that we are not measuring the quality of learning enough.

This is unfortunate. Not because measuring outcomes is not important or somehow wrong but because the present formulations of the issue are simplistic to the extent that they prevent underlying issues to be addressed. Here is how.

First, it is not as if the quality of learning is not being measured, or has not been measured in the last 20 years. The first all-India survey of learning levels was conducted by the NCERT in 1995, and there have been many since. Several large-scale independent studies of students’ learning levels have been run, including ASER and surveys of Education Initiatives. Small-scale learning assessments have been conducted for innumerable research studies (e.g. of 1 lakh children in Tamil Nadu to assess the state’s Activity Based Learning Programme) or pilot projects (for instance, several states have piloted their textbooks and used learning achievement as a benchmark). And of course at least hundreds (if not thousands) of NGOs/NGO-run programmes (often in government schools) have incorporated assessment as an effectiveness measure.

There are thus any number of assessments available – and they've been telling us for the last twenty years that our children are not learning. Only, this doesn’t seem to have resulted in improved learning, thus questioning the assumption behind the clamour for measurement.

This is a little like weighing a child to assess the level of nutrition – unfortunately, merely weighing the child will not lead to better nutrition... Something else is clearly required, and that doesn’t seem to be happening.

Second, insisting on having 'measurable' outcomes is hugely misleading – just because you can measure something doesn't make it more worthwhile (e.g. we do want students to be creative or considerate or civic though there are no easy measures for these). Several of the assessments mentioned suffer from this. Thus an Adivasi child who displays great resourcefulness, knowledge of the environment and concern for others would be called poorly educated since the ‘tests’ measure only basic literacy and numeracy.

Measuring outcomes would be useful only when we measure what matters most to us. Not whether a child can read something aloud but whether he can form an opinion on it and give the reasons behind them. Not whether a child can do calculations but whether she can apply it in real world contexts to solve problems or take a decision. Some of these may be hard to measure, but it would be useful to remember that it is not the purpose of education to be assessable, but the purpose of assessment to measure what is considered most worth learning.

Third, measuring outcomes does not account for contexts and tends to disadvantage (and label) those facing adverse conditions. Which then makes it even more difficult for them to improve. There are many teachers who work very hard in difficult conditions – but don't attain the kind of outcomes expected because the curriculum assumes children will be able to attend daily or speak the school language at home (and several other such notions), which don't apply to the children they work with (some 60-70% in India). We'll end up shoving these teachers out if we take the advice to replace them – instead of overhauling the system which has designed itself in such a way that marginalized children WILL fail.

Fourth, there is a danger that the present focus on outcomes is actually obfuscating – instead of increasing – accountability. India's challenges now arise from its success in rapidly expanding the school system to bring in so many children. The consequence is that we now have students (at all levels) who traditionally never attended schools - working children, migrant groups, girls from various communities, children with disabilities, socially excluded communities…. the list is endless. What this means is that while the nature of our students has changed, the curriculum, pedagogy and assessment remain as they used to be and so, the DESIGN ITSELF leaves these learners out.

At a second level, when it comes to implementation, there is a tendency in those responsible to ignore laxity on the assumed ground that it is only happening to those who do not matter. (Just as it is easier to ask a poor person to push a stalled car rather than a well-dressed one, similar prejudices operation in all facets of our society, including government officials.) Even now, therefore, it is mainly those from better-resourced families who continue to succeed, and we continue to have poor education for the poor. So the accountability really needs to be demanded at the level of the system (NCERT, MHRD, Departments of Education) and state / district / block officials.

As long as people keep pointing fingers at teachers as the main villains, the really responsible will continue to escape accountability. For instance, when the NCERT's own national survey shows low levels of learning, why does nothing happen to anyone at any level, including the NCERT itself (whose curriculum has been taken by many states now performing poorly)? How come officials at various levels continue exactly as they have been for decades with impunity when every measure  brings out dismal levels of learning in their watch? Recently, when our group, IgnusERG assessed class 9 students in a district we found 68% of them to be at class 4-6 levels, 7% below class 3 level, and only 4% at the class 9 level where they were expected to be. When this finding is shared, everyone finds a way to blame some one else!

Finally, let me leave you with this – in the current form, knowledge of outcomes attained does not help bring about improvement. Most states will be implementing SLAS (State Learning Assessment Survey) in the coming months. But once a state finds out it is performing poorly, say, in mathematics, that will not inform it of the reasons why this is so. It could be the poor curriculum (e.g. overambitious expectations) or weak syllabus (less time allocated than required), or inappropriate pedagogy (no use of concrete materials at an early age) or bad textbooks (poorly sequenced or giving discrete rather than contextual examples) or demotivated teachers or insufficient teaching time (because the state continues using teachers for non-teaching tasks even after RTE and court orders to this effect) or home vs school language issues or at least 10 other problems that can be named, each of which can seriously lead to poor outcomes. So where will the improvement begin?

The point, as mentioned earlier, is: do ask for outcomes, but don't keep it simplistic, or we'll continue to get the poor outcomes we've been documenting over the last 20 years.

What Do We Actually 'Celebrate' In Our Schools?

'So, you're 'celebrating' again.'

'Yes, it's Independence Day tomorrow.'

'Oh, so another round of ritualistic speeches?'

'What do you mean, ritualistic speeches?'

'You know what I mean. The same old 'important' people will be called. They will be welcomed, garlanded and they will walk around, feeling even more self-important.'

'You're being very unkind, you know.'

'But close to the truth, isn't it?'

'I'm not sure if this is really ritualistic…'

'No, it is what follows afterwards…. The same formal atmosphere will be created. Children will sit in neat rows and told not to talk too much. The LIP (or your Local Important Person) will be praised, invited to give us the benefit of his wisdom, children will be asked to shush, and then the LIP will give the same speech as every year – you are the future of the country… freedom is very important.. our great leaders were so very great… you must work hard… you must try to like the great people of the past… And all this while instead of experiencing freedom on Independence Day of India, children will be sitting bored, stiff, not allowed tomove around or talk or express themselves….'

'You're being really harsh!''

'OK, tell me, didn't you hear the same speeches when you were a child?'

'Ye-es.'

'Did you really enjoy those celebrations? Were they a celebration for you?'

'Actually, to be honest, no, not really.'

'Aren't you surprised that the same speeches are being made even now?'

'Yeah, now that you mention it…'

'And shouldn't children be more like the leaders of tomorrow rather than the leaders of long past. After all, every kid is not going to experience walking 17 kilometres to school!'

'Hmm… something to think about, there. And come to think of it, why was every great man's school 17 kilometres away?'

'See, it's getting you too!  And when it comes to – no, no, better not to say that.'

'Well you can tell me... I'm not going to shout at you!'

'I know you won't. But I don't want you to feel depressed either.'

'Come on, I can handle it. Tell me what you were going to say.'

'Well, if you insist. The thing is, children attend all these functions year after year, experience the same thing over and over again. And what do they learn? They learn that they don't matter. Their job is to listen. Their role is to be passive, not think for themselves. And look at you – you were a child who once found these functions boring but you are organizing exactly the same kind of function again! Independence Day isn't quite an experience in Independence, isn't it? My thesis is that these National Day type of 'celebrations' only teach us to be slaves, to accept that we have no freedom to be different or better, to allow ourselves to be defined by the limited vision of those limited adults who were similarly made limited by the experience they went through as children themselves…. Hey, you're suddenly very silent now. This is not look good… come on, say something.'

'What do I say? I'm feeling so…'

'So… what?'

'So depressed!'

-------------------------------------------------

What do you think about the other 'celebrations' we have in our schools? Is the birthday of a child celebrated 'more' or differently or better than that of other children whose families are considered to be less important or not influential?

What are the festivals celebrated in schools? Whose festivals are left out? Many communities / religious groups never see their festivals even discussed in school? What do they feel about it? And what do they 'learn' from this?

On Sports Day, do most children have the scope to participate and gain something? Or is only the 'victory' of a  few celebrated again and again? And what do the rest learn from this?

And on Results Day, whose achievements stand out? And what does everyone learn from this? (Maybe CCE will make a difference here?)

Perhaps all these celebrations, in the end, make us realise that their isn't much about us that is worth celebrating. When I was younger I would have said that this happens even though the intention is quite different. But now, a little more battered and older, I think the intention was always this – to make you realise that only a few can be 'important' persons worth celebrating, not you.

-------------------------------------------------

So what should we celebrate in our schools?

For starters, children and learning. Simply the presence of every child is worth celebrating (rather than 'Oh God, another one!'). And how to celebrate? By smiling, by welcoming, by genuinely talking with the child, giving space to her questions, by looking for ways to ensure she is comfortable, involved and engaged in an actual learning.

Children will ask unexpected questions, offer different points of view, find innovative ways of doing things, or help each other… celebrate this. Point out what they have done which is so good, and why it is so.

There will be times when those who usually 'fall behind' will make an effort, come up with something of their own (of course, only if you ensure they have the opportunity to do so). Celebrate their efforts, point out their good parts, and indicate what else they can do that will earn them similar 'celebration'.

If you find a fellow teacher, a staff member, a parent, an SMC member who is doing something successfully and contributing to children and the school, that's worth celebrating.

And on Independence Day? Start a few days before. Discuss with children what Independence Day means to us. Ask them how they think it should be celebrated. Come up with ideas that puts the children in the front, not adults or LIPs. Maybe they make drawings and posters related to freedom. Maybe they hold a debate on what freedom means and whether we really are a free people. Maybe they decide not to do a 'function' in the  school at all and instead spend time with children who are unable to be in school because they are not really free…And maybe they will learn something very different from such 'celebrations' than we did.

Why Corruption Is An Easy Issue To Raise

It's interesting to observe why the issue of 'corruption' seems to attract attention. Right now, across the country (and the world), a huge majority of people are oppressed by the accepted notion that it is OK for some to be considered 'above' others. That is why it is OK for some of 'us' 

• to go to high fee private schools (we have 'earned' it), 
• to sit in AC coaches in the train (we paid for it after all, never mind that the others' capacity to pay for the same is hampered by systemic and systematic obstacles), 
• to feel that we belong to 'big' or 'important' families... 

Such societal hierarchies have a far greater impact and preserve disadvantage.

Isn't it corruption too to believe that one belongs to a 'better' or 'purer' religion / caste / class / background / family than others. Yet Anna and co don't raise issues of social fracture (conveniently forgetting that Gandhi spent far more of his life on these issues, and regarded true independence as one from social oppression too). It's worth thinking on why the issue of corruption really suits the middle class - it's so neutral and harmless, and avoids the really frightening ones. It's also something where you can blame 'others' without feeling that you are part of the problem...

As an educator, therefore, if I had to teach children any value, it would not be an ordinary thing like 'do not be corrupt' but the more difficult concept of 'though you are unique and deserve the best, do not think you are more important than others or have a birthright to more than they do'.

'What We Learn Cannot Be Burnt - 'An Afghan Neo-Literate Woman

As we work in education, it often tends to get too 'sanitized' - as if it is not about real people in real situations, where education has a meaning that's almost impossible to comprehend. Here's a story from Afghanistan, from a programme called Learning for Life that sought to provide initial literacy and health awareness to enable women to become CHWs (community health workers, sorely needed in the country). This story was documented in June 2005, by Judie Schiffbauer, and shared by Katy Anis.

Each morning, six days a week, 40 year old Zeba Gul wraps a light gray shawl around her head and shoulders and leaves her family’s mud-walled compound in the Afghan village of BegToot.  She follows a path that winds through dusty alleyways and then along green fields to arrive at a two-story building constructed of unbaked brick made from mud and straw.  Inside, a set of narrow stairs leads to the Learning for Life classroom, where other women are already gathered.  Removing her shoes at the doorway, she enters and lowers herself to the mat-covered floor, tucking her long legs beneath her.  

In December 2004, when the LfL health-based literacy program began in BegToot, wind whipped snow against the classroom windows, but on this fine summer day, the windows are open to admit a pleasant breeze.  The room looks out over groves of mulberry trees, for which the village is named.  Tall, creviced mountains rise high in the distance, still bearing traces of winter snow.

But the 26 women in the class are not admiring the view.  Instead, each attends to Qotsia, their 21-year-old teacher, who stands beside a small blackboard at the front of the room.  One of millions of Afghans who fled the war-torn country, Qotsia grew up as a refugee in Iran, where she received 12 years of formal education.  Now she has returned to BegToot, and the women are grateful.

 Dressed and coifed in black, Qotsia begins to write with a piece of chalk.  Carefully demonstrating each stroke, she writes a word in Dari composed of several letters from the alphabet displayed on a poster on the wall.  The dark black letters on the poster are easy to see, but six months ago, no woman in the class could have named or written a single one.  Today, hands shoot up when Qotsea asks someone to spell out and then read what she has written.  One woman rises and comes forward:  k a r u m (worm).  “Very good, Pashtoon Jan!” says Qotsia.  Pashtoon Jan smiles as her fellow learners sound out the word, repeat it in unison, and write it in their notebooks:   k a r u m.  Worms are the topic of today’s lesson.  

To the left of the blackboard, a series of drawings depicts women busy with women’s chores:  one is cleaning vegetables; one is boiling water to be stored in an earthenware jar; another is feeding a sick baby; and one is washing a child’s dirty hands.  Now the women in this Level One literacy class are going to learn how worms and a child’s dirty hands are related. 

As one of six Community Health Workers enrolled in the class, Zeba Gul already knows a lot about worms.  Their life cycle and method of transmission were explained to her when BRAC, a REACH NGO-grantee, trained her as a CHW.  But until now, Zeba Gul has never known how to spell, read or write the names of the parasites-- roundworm, tapeworm, and pinworm—that sicken so many children and adults in the village. 

As Qotsia begins the lesson, Zeba Gul leans forward and points to a young woman sitting nearby: “That’s my daughter,” she whispers. “Because of this class, she is learning to read and write before her hair turns gray.”

Later, the class at an end and women lingering to talk, Zeba Gul told her story.  She was born in Paghman, but she has not always lived there.  When she was sixteen, she married and moved to Kabul with her husband to live with his family. Her daughter and several other children were born in the city.

“It was good,” says Zeba Gul.  “My husband had a small shop.  He worked hard.  In the morning, he opened the shop.  In the afternoon, he had a second job in a government building.”

Even during the dark days of war, the family chose not to leave Afghanistan for sanctuary in Pakistan or Iran.  “We stayed,” she says, remembering their struggles with a hint of pride in her voice.  “We were hard workers, and we stayed.” 

For a time after the Russians left, Zeba Gul thought the worst was behind them.  But peace did not last long.  “After that,” she said, “I wasn’t sure what the fighting was about; I know only that it did not stop.  So much fighting.”

When Zeba Gul explains that both the family’s shop and home were near Damazang in Karte Seh, the room grows very quiet.  Everyone knows that Karte Seh was virtually destroyed during the civil war.  “Ay, Khoda!” the women whisper, as Zeba Gul continues her story:

“One night, our shop was ablaze. How it burned!  And our house burned too.  Everything we had was swallowed in fire.  Oh, God.  What could we do?  We had nothing left!  So we returned to Paghman.  It was more than ten years ago.  Here, my husband is a farmer.  Thanks to Allah, he is alive.” 

Many of her listeners have been less fortunate, and the widows nod in agreement as Zeba Gul utters her prayer of gratitude.  The women in the room have known great sorrows, but it is resilience that binds them. 

“Now,” continues Zeba Gul, “I am a CHW.  And I am learning to read and write in this class.  See there: my daughter is also here! Faz l’Khoda--Give thanks to God.  What we learn cannot be burned.”  

Over-aged -- and loving it! -- Part 1

It's difficult, but imagine for a moment that you're an 11-year-old who wasn't able to attend school. When you were very young, you can remember, your parents moved from place to place, working on construction sites. A few years ago, they got work back in your village as a canal made agriculture more possible. And you yourself started off being an assistant cattle-herder. Now, though, you've graduated to full cattle-herder, with knowledge of all the grazing areas, the watering places, the dangers to look out for (that unexpected ditch into which all the young cattle are always falling) and the idiosyncrasies of owners who don't always pay on time. As you saw children going to the nearby school carrying weird little bags or screaming insults at you, you wondered what they did holed up the whole day in that building. Even the cattle seemed to be more free than they.

Then one day, the newly appointed teacher organised a meeting with all community members and explained to them something called 'Right to Education'. Basically, this meant that your parents decided you should go to school. No one asked you. Your father only said, 'Now work is more regular here, we can manage.' So off to school you were dispatched. Being alone with a hundred cattle in the nearby jungle (with the possibility of that nasty jackal) seemed so much less fearful than entering that stark building, all yellow and white with blue things written on it here and there.

What are the children in there going to say? Your mother made you have your bath and put on the other pair of clothes, so no one would say you smell -- but the beloved odour of cows isn't going away from you and your clothes anytime soon. There are some green-painted metal play things on small play ground. The smell of food being cooked mingles with the smell of something else (it's paper and chalk and sweat, though you don't know it yet). Your heart is in your mouth as you step onto the ramp climbing up to the school. The teacher comes out and is looking at you -- and you're doing your best not to run away. Away, back to the beloved forest, with the hundred cattle who know you so well.

Is Education for Girls Different from Education for Boys?

If we were to educate only girls, would we develop an education different from the one that prevails now? And would it be different from an education created only for boys?

Before you lynch me for raising blasphemous ideas and restricting girls to things such as reproductive health and sewing/knitting, let me explain. If we were to look at education only from the boys' point of view, we would find that everything we wanted is probably already there. But that is not the case when it comes to girls. Surely, education for both boys and girls would be much better off if the girls' perspective, experience and world view were, in fact, included.

Think, for a moment, of recipes, and how they would be wonderful material for learning mathematics (interesting, isn't it?). Or the kind of abilities girls have with fine motor skills and patterns. Or multi-tasking. Or giving value to emotions and relationships. Or being able to share rather than dominate. Or how to make use of meager resources. Or a range of other things which I'm sure you can list (endlessly). Are we not depriving ourselves in not exploring this? There is no doubt that, on the whole, education is much the poorer from having been defined by the male perspective. Which then applies to the world itself as well.