With a per capita income of less than USD 1,500 and a population of 1.25 billion, India runs arguably the largest citizen-led assessment in the world. This year, the Annual Status of Education Report (ASER) celebrates its 10-year anniversary. ASER’s remarkable track record offers valuable lessons for developing countries trying to build demand for quality education.
A common problem in many developing countries is that citizens do not know the quality of their public schools. According to the 2014 World Gallup Poll, 78% of Indonesians, 62% of Colombians, and 46% of Brazilians said that they were satisfied with the quality of their school system. However, in the 2012 Program for International Student Assessment (PISA), a global test of math, reading, and science for 15-year-olds, 76% of Indonesians, 74% of Colombians, and 67% of Brazilians could not even answer the easiest questions on the math exam.
The gap between expectations and reality is not limited to the poor. In the 2014-2015 Global Competitiveness Report, which asked business executives around the world to rate the quality of education in their countries, the mismatch between perception and reality in some developing countries was stark. For example, Qataris ranked their education system third in the world, a position above Singapore’s. However, Qatar ranked third from the bottom in the 2012 PISA math assessment, while Singapore was the best-performing country.
In fact, many of us suffer from what psychologists call “comparative optimism”. Even when we acknowledge our school system is under-performing, we believe that our local school is outperforming the rest. Statistically, we cannot all be right. This phenomenon is not limited to developing countries. It was first highlighted by the annual poll conducted by Education Next and the Program for Education Policy and Governance in the United States in 2008.
Many find the low demand for education quality troubling. The World Bank’s landmark 2004 World Development Report (WDR) argued that improving education requires teachers and government officials to do things that are costly for them: teachers need to show up for work, arrive on time, and teach for the entire day; government officials need to overcome political opposition from vested interests to adopt reforms that improve the efficiency of schools. Without pressure from the public, the report contended, neither is likely to happen.
…
Developing countries have tried to “build demand” for quality education in one of two ways. The traditional approach has been for governments to design and implement student assessments, and for civil society organizations to distill the results for non-technical audiences. Mexico’s Mejora tu escuela was one of the first web-based platforms that allowed citizens to compare the student achievement of their local school against that of other schools, districts, and states.
A far less common approach has been for civil society organizations to take matters into their own hands and develop student assessments that regular citizens can administer and understand. This approach was pioneered by Pratham, the largest education non-profit in India, when it began administering the Annual State of Education Report (ASER) in 2005.
ASER has multiple characteristics that set it apart from other assessments. First, it is household-based. Most learning assessments are school-based because they have been created in developed countries, where nearly all children are in school. Yet, in developing countries, many children are either not in the formal education system or not in school. Including these children is crucial to providing a complete picture of the state of learning in these countries.
Second, it is simple to administer. Most learning assessments are pen-and-paper exams. Yet, many children in developing countries lack even the most basic reading and arithmetic skills; they simply cannot answer such tests. The ASER tool is remarkably simple. In reading, the administrator asks students to identify letters and words, read a paragraph (at a grade 1 level) and a story (at a grade 2 level). In math, the administrator asks students to recognize one- or two-digit numbers, solve two-digit subtractions, and divide three-digit numbers. These outcomes are not just easy to evaluate; they are also easy to express in a language everyone can understand. It is now common in India to hear that “X% of students in grade 5 cannot read a grade 2 text.”
Third, its administration is participatory. The ASER Center (which became its own non-profit in 2008) does not administer the survey by itself. Instead, it trains and relies on universities, non-governmental organizations, government departments, and private companies across India. To date, more than 2,000 partners have administered ASER. (For a description of ASER’s partners, go here). The benefits of this strategy are multiple: it makes administration more manageable, it offers an opportunity for partner organizations to learn how to administer student assessments, and it increases the buy-in and ownership of civil society over the results of the survey.
Fourth, it has an impressive coverage. To provide a representative snapshot of student achievement, since 2006, ASER randomly samples 30 villages per rural district, and 20 households per village (i.e., 600 households per district). To track achievement over time, since 2007, ASER resamples 10 villages from two years before, 10 villages from one year before, and selects 10 new villages in each district. ASER has achieved an impressive coverage. In 2014, it covered 577 districts, 16,497 villages, 341,070 households, and 642,911 3-to-16-year-olds. (As a comparison, PISA 2012, the largest global assessment, covered 510,000 15-year-olds in 65 school systems).
Finally, it is remarkably flexible. To stay relevant, ASER has undergone important changes almost every year since its inception. In 2005, it began administering a school observation sheet in addition to its household survey and student test. In 2006, it started evaluating higher-order tasks in reading and math (e.g., reading comprehension). In 2007, it began assessing students’ English proficiency. In 2008, it started collecting data on villages. (This brief document summarizes the most important changes since ASER began).
…
After a decade, ASER’s main achievements in India are clear. As the only annual source of information on children’s learning outcomes, it has shed light on the abysmal academic achievement of children and young people in the country, and it has positioned learning as a crucial metric by which many Indians judge the quality of their school systems.
ASER’s success in India raises the question of whether non-governmental organizations in other developing countries should follow its steps. Eight have already started: Pakistan, Uganda, Kenya, and Tanzania conducted their first nationally-representative citizen-led student assessments in 2010, and Senegal in 2013. Mali, Nigeria, and Mexico have conducted smaller pilots with the intention of conducting national assessments soon. These countries have come together under the People’s Action for Learning (PAL) network.
The countries in the PAL network have three characteristics in common that offer some guidance on where and when citizen-led assessments can be most helpful. First, nearly all of them lack a regular, robust, and nationally-representative student assessment whose results are publicly disseminated. Some, like Mali or Nigeria, have only recently started participating in regional assessments, such as the Program of Analysis of School Systems of the Confemen (PASEC). Therefore, in these countries, citizen-led assessments are not duplicating other efforts; they are providing a service that would not be available otherwise.
Second, the average student in most of these countries has not yet developed basic competencies. Uganda consistently ranks in the bottom half of the Southern and Eastern Africa Consortium for Monitoring Education Quality (SACMEQ) test. Tanzania and Kenya are usually near the top of the rankings, but over half of Tanzanians and more than a third of Kenyans performed in the lowest levels of the latest sixth grade math test. The basic literacy and numeracy skills that student in these countries need to develop can be adequately captured by citizen-led assessments.
Third, in most PAL countries, capacity for administering student assessments is very low. Therefore, citizen-led assessments offer an opportunity to train a cadre of experts on how to design, administer, and analyze assessments. This typically entails observing a citizen-led assessment in action, participating in trainings held by ASER in India, and conducting pilots.
…
Any country considering conducting citizen-led assessments, however, ought to be clear about what it hopes to achieve. ASER’s experience in India suggests that these assessments can catalyze policy discussions on student learning. Yet, the evidence on the capacity of these assessments to mobilize communities to improve schools is less encouraging. A study in Uttar Pradesh, India found that training communities to administer and disseminate the ASER survey did not increase the involvement of parents or teachers in schools.
These limitations are not specific to citizen-led assessments. Impact evaluations of initiatives to involve parents in school management in Niger, Gambia, Mexico, and Senegal have consistently found that these programs are less successful when parents lack the capacity and/or authority to act. A study in Indonesia suggests that disadvantaged communities need considerable scaffolding to choose their representatives and make demands on government officials.
In short, citizen-led assessments can be a useful tool to address common obstacles to low demand for quality education in developing countries. Yet, the frontier challenge remains ensuring that the information they produce leads to greater grassroots mobilization. This has recently led ASER to launch its Lakhon mein Ek (One in a million) campaign, an alliance of institutions, non-profits, businesses, foundations, and individuals that aims to reach 100,000 villages and their communities in India by January 2016 to involve citizens in assessing and improving the quality of their local school systems. Ten years after its inception, ASER should be commended for remaining committed to Pratham’s motto: “Every child in school… and learning well.”
– Guest Blogger Alejandro J. Ganimian
Alejandro J. Ganimian is an education post-doctoral fellow in the Abdul Latif Jameel Poverty Action Lab (J-PAL) South Asia.
