Human Rights and Statistics – Some Reflections on the No-Man's-Land Between Concept and Indicator

By Nancy Thede
2000

1. Introduction

Indicators for human rights respect are viewed by most practitioners as extremely desirable, as a means of tracking progress or a lack thereof, as a means of holding governments accountable, and as a means of gauging the impact of their own action. But then the debate begins : what do we mean by indicators? What exactly are we trying to measure? How to ensure that the indicators we identify tell us what we need to know? Should indicators be used to establish a comparative index? In short : under what conditions can indicators be considered a valid and useful tool?

This conference occurs in a context of increasing efforts on the part of international institutions to develop quantitative indicators for human rights. At the same time, very little serious dialogue has taken place between human rights workers and statisticians to determine the needs of the former and how they might be addressed by the tools of the latter.

To date, statistical information in the field of human rights is relatively underdeveloped. The notable exceptions regard enquiries into massive violations of human rights, especially the right to life and to physical integrity. Increasingly also, we have seen efforts to quantify economic and social rights (e.g.: CEDAL, 1999). It is therefore difficult to generalise to the entire field of human rights. It can be argued though that each right has both an objective and a subjective aspect. Certain rights – and especially the objective aspects of certain rights - lend themselves more easily than others to quantitative measurement.

My concern in this paper is with political rights and, in particular, participation rights. These comprise, for instance, highly subjective rights such as freedom of expression. Such a right is not nearly as clear-cut as the right to freedom from arbitrary execution or freedom from torture. Each political right involves numerous facets and a complex interrelation amongst those various facets. This is particularly so with the right to participate, or the right to choose one's political representatives. This area has seen several attempts at quantification (e.g.: Arat 1991, Freedom House, Gastil 1991). Despite the fact that they have been severely criticised on the basis of serious methodological and conceptual flaws, they continue to be widely used and quoted. A similar phenomenon has occurred with respect to the UNDP's Human Development Index (HDI). Considered by its creators to be the weakest part of the annual Human Development Reports, the HDI is nonetheless the most often quoted component of the Reports, and has acquired a status of autonomy with respect to its context (i.e., the Reports) 1. The problem, therefore, is that once a quantitative measure is created, it will be used independently from the analysis that originally generated it. The underlying question we must face is ultimately: are some statistics (even faulty ones) better than no statistics at all?

This question must be addressed in the context of the several distinct tracks in the discussion of the application of statistical methods to human rights issues. One track concerns the effort to generate information on the national level that can orient policy development and programmes to address human rights abuse. A second track is related to the international attempt to ensure accountability for massive violations of human rights and crimes against humanity. The third track – and the one that most informs the analysis in this article – is the renewed debate surrounding the development of a human rights index, or even a democracy index.

This article addresses in particular the development and implementation of indicators for policy orientation in the field of democratic development. It begins with an overview of the experience of my institution, Rights & Democracy, in developing and implementing a series of indicators for democratic development based on human rights criteria. This experience has allowed us to identify a number of issue which caution against quick translation of political rights phenomena into quantitative measures. It will then examine the broader debate on human rights indicators, looking at the principal trends and attempting to identify the issues that need to be addressed in the construction of indicators. A third section will underline the expectations of practitioners with respect to human rights and democracy indicators. Finally, some conclusions and recommendations will be formulated on the basis of the preceding considerations.

2. The Rights & Democracy « Democratic Development Framework » : some lessons learned

The International Centre for Human Rights and Democratic Development – now known as “Rights & Democracy” – is an institution created by the Parliament of Canada in 1988 with the mandate of promoting international human rights and the development of democracy around the world. Through this work Rights & Democracy has come to premise its approach on the notion that not only are human rights and democracy linked and interdependent, but they are mutually constitutive. In other words, human rights are the product principally of social movements for the recognition of new rights and new social actors. The social and legal recognition of such rights constitutes both democratic processes and democratic institutions. Democratic institutions and democratic societies are constituted and evolve rooted in this process of social struggle. Human rights are thus not simply a characteristic or an outcome of democratic societies, they are at its very core. Citizens' movements and organisations for rights have historically been the motor of development of democratic institutions and processes. As such, they are intricately interrelated in ways that are unique to each society, and which change over time.

As early as 1992, Rights & Democracy began to develop an analytical grid for democracy based on a series of human rights criteria (see Gillies 1993). It is composed of a series of six interdependent categories of rights, representing the entire range of human rights. The categories are: participation, security, well-being, non-discrimination, collectivities and democratic institutions. These categories are drawn from the entire family of human rights, and thus underscore the indivisibility and interdependence of all rights. Rather, however, than attempting to examine each of the sixty or more specific rights that can be derived from the major international instruments 2 , we have chosen a small number of rights in each category as proxies. The reasoning on which the selection of proxies is based is extremely crucial for the validity of the observations with respect to the overall assessment: the case for each of the proxies will be set out below for each of the categories.

Finally, it is important to mention that, after lengthy analysis of the debate on the issue (see Arat 1991; Carter Center 1992; Humana 1986; Gastil 1991; UNDP 1991, 1992), Rights & Democracy explicitly opted not to establish a ranking system nor a cumulative scorecard for purposes of comparing different countries. The grid therefore establishes a qualitative, not a quantitative measure. It is an underlying premise of this analysis that each of the categories under consideration is necessary for the existence of a democratic society. Taken together, the six categories and their component criteria are representative of those rights which make a life of human dignity and meaning possible. By the same token, the absence or abuse of any one component diminishes the prospects for the dignity of the human person or group and thus, the overall quality of democracy in that country. Hence, "political participation can be meaningless without adequate security of the person; it can be very restricted in its scope if discrimination is rife in society; it can appear close to meaningless where basic welfare needs are not met" (Taylor & Muntarbhorn, 1994).

Following is a brief overview of the criteria composing each category of the grid, and the reasons for choosing them. It should be remembered that each right is examined both from the point of view of institutional guarantees established at the level of the State, and actual practice, in recognition of the fact that formal guarantees are frequently not respected.

Participation

The proxies we use in this category are the right to vote, the right to take part in government, freedom of association and assembly, freedom of opinion and expression.

• The effectivity of the right to vote is considered to be the fundamental indicator of freeness and fairness of elections, which in turn is the classic measure of the existence or not of a democratic system. It also is in most cases an indicator of the broader state of health of the political institutions in general.

• The application of the right to take part in government provides an indication of the quality of political participation that goes beyond elections and the formal electoral process to allow us to assess the strength of civil society and its capacity to actively involve itself in significantly influencing the public policy agenda.

• The freedoms of association and assembly are essential for democratic participation. They indicate the quality of and even the potential for political expression of the unorganised as well as the organised sectors of society.

• The freedoms of opinion and expression, finally, are essential to the existence and development of a vigorous public sphere where all citizens, without distinction, can participate in debating society's political and social priorities.

It is interesting to note here the view expressed by Guillermo O'Donnell (1999), to the effect that the core characteristics of a democratic regime are, on the one hand, an institutionalised system of fair elections and, on the other, a series of basic freedoms which stand in a causal relationship to democracy and which, moreover, are the condition for the generation of other rights. These rights he identifies as being the freedoms of expression, information, assembly and association.

Security

A climate of personal security, in addition to the moral imperative of respect for life and human dignity, is essential for the development of an active civil society that exercises its rights to democratic participation. The proxies we use here are again four:

• The first two - extra-judicial killings and torture - represent the most extreme forms of insecurity. When the right to life and freedom from torture are systematically violated, all other individual and group rights are at risk.

• Arbitrary arrest and detention allows us to assess whether or not a climate of repression exists, and offers an indicator of the direct involvement of the state and its agents in maintaining such a climate.

• The occurrence of violence against women – whether in the public or the private sphere 3 - constitutes an indication of the nature and extent of barriers to participation by women in the public sphere.

Well-being

This category purports to assess the material aspects of existence and touches principally on economic, social and environmental rights. In addition to the fact that these rights are important in terms of the quality of life of members of a given society, they also have an impact on the quality of democracy in that society. Clearly, extreme inequality of access to economic and social rights affects the quality of democracy by inhibiting the real exercise of democratic rights by the poorest segments of society. Furthermore, some empirical research suggests that extreme material inequality or the absence of public policies to redress such skewed distribution contribute to democratic decay (Arat 1991). In addition, the neo-liberal vilification of the role of the State in the economy and its consequent withdrawal from furnishing public goods undermines not only economic and social rights, but also threatens the entire family of rights (Broadbent 1996).

The four proxies used in this category - education, health care, labour rights, environment - allow us to assess the broader category of fulfilment of basic human needs (which would include food, shelter, employment, etc.) from a strategic perspective. The justification for each is the following:

• Education provides an indication in most contexts of social mobility, but also of democratic participation and the quality of the public sphere more broadly. The access of women to education always indicates a greater degree of emancipation from domestic oppression for them and a greater capacity for democratic participation.

• Health care is a proxy for social services more generally. For women in particular, public health care services can be a means of emancipation from exclusively domestic responsibilities.

• Labour rights are an important indicator of economic rights, given the "demonstration effect" they have on salaries and on working conditions, given the historical role of trade unions in defending the right to organise. Moreover, labour rights straddle both international covenants and are central to many of the ILO conventions states may have ratified. Moreover, labour rights in general give an accurate insight into the level of respect of the democratic rights of association and assembly.

• Environmental rights allow us to assess, here again, the economic well-being of significant sectors of the less powerful members of a society. They also are a proxy for cultural rights in the broadest sense. On the first count, the degree of environmental protection (from industrial pollution, for example) is an indicator of the priority accorded by the state to small producers (in general, the majority of the population) versus powerful economic sectors. Secondly, it is most often the case that mega-projects with major environmental consequences (dam construction or petroleum development, for example) affect communities whose entire way of life can be drastically altered as a consequence of the project: this thus touches upon the respect for cultural rights and minority rights. Such communities are in general a tiny minority of the national population and cannot impose their point of view through the formal democratic process. The extent to which the State takes their position and situation into account will indicate its commitment to cultural rights and minority rights.

Non-discrimination

In this category we check for discrimination on the basis of gender, social status, religious beliefs and ethnic origin in all other categories of rights. The principle of non-discrimination is central in all human rights instruments as well as to the modern concept of democracy. An assessment of trends in discrimination in a given country can serve as an early warning system for democratic deterioration.

Collectivities

Within this category we examine the rights of indigenous and tribal peoples, and the rights of minority groups.

• The rights of indigenous and tribal peoples, taken from the point of view of the recognition of their collective self-determination, is indicative of respect for cultural rights and the treatment of diversity, a key issue for an effective democracy.

• The examination of the rights of minority groups provides insight into the degree if inclusiveness and tolerance of the democratic system. How minorities are defined and in what terms they are recognised is a measure of democratic participation as well as of formal inclusiveness.

Democratic Institutions

This category is designed to give a clearer assessment of the role of the state in promoting and protecting human rights and democracy. The two sub-categories examined here are the political institutions of the state and the rule of law.

Political institutions must themselves be structured so as to ensure the permanence of democratic processes. The proxies we have chosen here are the following:

• free and fair elections (ensuring genuine expression of the popular will in the choice of political representatives)

• a competitive party system (ensuring the existence of a choice of political options)

• separation of powers (ensuring accountability of all branches of government)

• decentralisation of government (allowing for direct access to participation in the political process)

• civilian control of the military (ensuring subordination of the security forces to the democratic process).

Rule of law, understood as an independent and predictable legal framework and process, is essential to a society in which human rights violations are not tolerated and where democratic participation is enabled. The proxies we have selected here are:

• access to justice (as a measure of the availability and credibility of the court system for the entire population);

• fair trial (as a measure of the independence of the judiciary);

• a democratic constitution (as a fundamental guarantee of the institutionalisation of human rights, including the right to democracy).

Results

We have applied this analytical grid in seven countries to date (Kenya, Tanzania, El Salvador, Guatemala, Thailand, Peru and Pakistan). Even though the goal was not, from the outset, to establish a comparative index, our experiences of application have confirmed that view. We have seen that, despite the fact that we use a common methodology and series of categories and criteria, the results of each study vary vastly from one country situation to another. In large part, this appears to be due to the fact that the concrete situation in each country has a determining effect on the interpretation of the data.

A second major lesson of our experience with this set of indicators is that their interpretation is often highly contested. Despite the fact that a great deal of effort is put into identifying areas of common ground and consensual strategies amongst the various actors involved, especially between state and civil society, it is ultimately rare that consensus can be attained around the major conclusions of the studies. Human rights criteria may be characterised as universal, but they are not neutral in concrete country situations. On the contrary they are highly politically charged, their definition is the subject of acrimonious dispute, and their application affects the material and political interests of many (if not all) social actors. Human rights indicators cannot therefore be established as if they did not refer to subjects of conflicting interests. It is for that reason as well that we consider that the idea of value-free measurable indicators is highly misleading, and all the more so if they purport to establish an "index". Human rights are values - they cannot be approached as value-free.

3. The Debate on Indicators: Issues and Context

There is a strange conundrum in the discussion surrounding indicators: despite the fact that a large number of analysts identify major conceptual and methodological problems in the definition and construction of indicators for human rights (some as early as McCamant in 1981), very few of them wholly reject the use of indicators even though it is unclear whether the problems they identify will ever really be resolved. It is therefore important to examine some of the major trends and issues in the debate on indicators. It is also important to recognise that, given the variety and seriousness of the problems raised with creating indicators, in the words of the UNDP itself, none of the various proposed systems for classification and measurement of human rights “has so far gained universal, or even common, acceptance” (1991:19).

The numerous problems raised with respect to the construction and use of indicators, even by the proponents of their use, revolve principally around the subjectivity of human rights per se, and the paucity and unreliability of data for various reasons. These problems include the following:

• developing adequate definitions of concepts in the human rights area (McCamant 1981; Goldstein 1992:37; Barsh 1993: 92; Arat 1999:16)

• problems of data collection - and data collectors - which undermine the reliability and comprehensiveness of the data (Goldstein 1992; Bollen 1992; Lopez & Stohl 1992; Claude & Jabine 1992; UNDP 1991: technical note 6; UNDP 1992: 30)

Above and beyond these immediate problems, however, we can identify a few overriding concerns which merit further attention. It appears to me that there is a tendency on the part of the proponents of statistical indicators and indices to brush off what they may perceive as “soft” issues, those of a conceptual nature. In my view, we ignore them at our peril. By refusing to deal with them, our efforts to construct indicators will be at best irrelevant, at worst misleading or even harmful. I will briefly explore several of these concerns now.

Conceptualisation: turning rights into indicators

Statistics can be useful as an analytical tool, but – obviously – only if they are well constructed and appropriate to the problem to be addressed. The question we must ask is therefore under what conditions can statistical indicators for human rights be valid? Five major issues must be addressed in this discussion. They are: 1) how to conceptualise indicators for human rights purposes; 2) the complex nature of certain rights; 3) the need for interpretation and contextualisation; 4) the problem of mystification of statistics by users; and 5) the need for theory in order to link the concept to the indicator.

There appears to be no general agreement even on what an indicator is. The UN “Common Country Assessment” framework defines an indicator as “a variable or measurement, conveying information which may be qualitative or quantitative, but consistently measurable” (OHCHR 1999: 1). In practice however, in the vast majority of proposals for “indicators” or “checklists”, including those of the CCA in many instances and of the UNDP's proposed indices, and that of my own organisation, we find not measurable indicators at all but rather factors or phenomena that require rigorous unpacking in order to arrive at anything that could conceivably be consistently measurable. And even then, as Naila Kabeer rightly signals: “Indicators can be seen as highly compressed summaries of information, meanings and values. They combine explicit empirical information with implicit assumptions about the meaning of that information… indicators also embody certain values about the kinds of information that ‘count' in capturing the phenomenon being measured” (1999:2).

Indicators thus necessarily narrow the complexity of meaning(s) and facets of a right by selecting only a certain type and number of its characteristics for measurement. Barsh (1999) maintains that this in itself makes the development of indicators untenable, since by so doing we distort the very meaning and intent of the right in question. This position merits serious consideration. Are we, under the pretext of promoting respect for human rights, creating indicators that will impoverish the very concepts human rights defenders have striven so long and hard to enrich? In addition, Caceres (1999: 82) makes the point that, on a more global level, the increasing primacy of a quantitative, productivist approach to development ultimately undermines human rights.

At the very least, due consideration must be given to the manner in which a right is represented by one or more indicators. Lopez and Stohl (1992: 224) make the disquieting affirmation that existing efforts at measurement of human rights have entirely bypassed the crucial phase of specifying and disaggregating the relevant dimensions of each human rights concept. Instead, they have leapt directly from the specific right to the operationalisation of an indicator for measurement. This observation is unfortunate still true, eight years after it was first made. It boils down, in essence, to accepting as valid indicators whatever readily measurable form of data exists. This is extremely problematic, since we need to be certain that the information we are collecting tells us not just “something about that right”, but “something significant about that right”. Each right needs to be problematised, to be understood in the fullness of its potential for human dignity, and on that basis its significant aspects should be identified and translated, if possible, into indicators. This is not being done.

Complexity of Human Rights Concepts

Many authors underline the fact that human rights, as set out in the international instruments, are inherently subjective, value-based and highly politicised concepts. Barsh (1999: 7) argues, for example, that Article 25 4 of the International Bill of Human Rights has no fixed meaning, but is in a process of continual redefinition in the “ongoing and unsolvable dialectic between the procedural ideal promoted by the United States … and the recognition that democracy involves ‘complex and multifaceted concepts that defy precise and universally accepted definition'”. Similar arguments could undoubtedly be made with respect to the majority of the political freedoms 5, at the very least.

Others, however, attempt to account for that subjectivity explicitly in their analysis. Klug (1993 : 18), for example, argues that rights are not concrete phenomena, and it is therefore difficult to measure them directly. She maintains, however, that “international rights norms are effectively human rights indicators. However, without further clarification, they are too broad to provide a reliable gauge.” This then leads us back to the issue of how those norms themselves can be disaggregated into measurables. Her insight nevertheless does sketch out a potentially useful intermediate step between the concept and the indicator.

Kabeer (1999: 30-31) draws attention to the need to establish a structural means of verifying that the indicator means what we presume that it does. She argues that there is a “critical need to triangulate or cross-check the evidence provided by an indicator in order to establish that it means what it is believed to mean. Indicators compress not only a great deal of information into a single statistic, but also assumptions about what this information means.” We need to render our assumptions explicit and also cross-check for interference from unidentified assumptions. This is particularly important when attempting to apply the same indicator in widely varying social, cultural and political contexts.

The Need for Interpretation and Contextualisation

Virtually all authors agree that indicators cannot, in and of themselves, provide an assessment. They must be interpreted in the light of knowledge and analysis of the context to which they are applied. This appears to me to be the single most forceful argument against the establishment of a rating system and a comparative index. The OECD (2000) states that indicators give a signal, they do not provide an analysis. Claude and Jabine (1992) see statistics as a complement to analysis, and not the contrary. Goldstein discusses at length the overriding importance of putting statistics in context: “Even if human rights terms can be adequately defined and reliable quantitative information can be obtained, making intelligent assessments of such data will often be extraordinarily difficult, especially if the data are interpreted out of the context of other, non-quantitative sources, such as interviews, on-the-spot observation, and background reading” (1992: 49). He clearly subordinates statistics to context: “Statistics can unquestionably be helpful when used in an intelligent way and by a user who can put them in context” (p. 55). He even quotes a former president of the American Statistical Association “who in 1972 told his organisation that statistics are ‘a crutch, indispensable, but still a crutch' which ‘cannot walk by itself,' and ‘if not proportioned to the needs of the user… can hinder as well as help'” (p. 56).

Lopez and Stohl (1992: 217-218) assert that measurement can only be valid if interpreted intelligently on the basis of solid political judgement. They emphasise the “centrality of conceptualisation and judgement in any measure of human rights”. They maintain that “informed judgements by experts with a detailed knowledge of a particular situation over time are necessary to ensure the validity of any particular measure”. They see this as the only possible means of accounting for the diverse political contexts that human rights assessments must confront.

Most social scientists would agree with Barsh's statement that “reported data does not establish anything by itself… It must be interpreted” in historical and political context, but also in a theoretical context. And such a theoretical context is normative. Be it with respect to human rights or with respect to democracy, we ascribe to a particular theory, even if it is implicit, as to what constitutes human rights or democracy. Analyses that purport to be “ideologically neutral” or “theory free” are based on implicit theories which are more usefully rendered explicit where their value can be examined and debated. This is particularly the case when dealing with democratic rights and political participation.

The Mystification of Statistics

Goldstein points out insightfully that once a statistic has been created, even if it is faulty, it will be used. Social scientists are undoubtedly more prone to close their eyes to the methodological foibles of statistics than are professional statisticians. There is an ever-increasing pressure to quantify, and it manifests itself on researchers through funding priorities, for example. And so it comes to pass that shoddy data are often accepted by social scientists “as though they are methodologically sound quantitative data” (1992 : 48). Once created, the process of fabrication of the statistic becomes invisible and the statistic therefore becomes ‘autonomous' from the method that generated it. Clearly inappropriate statistics are thus often repeatedly and unquestioningly used.

Let me cite a personal example. In a paper on civil security in transitional democracies, a researcher cited a statistic showing that the homicide rate in sub-Saharan Africa was twice as high as that in Latin America. Familiar with both regions, the figure seemed very out of line to me and I questioned it. The researcher contacted the authors of the study, who had in turn themselves quoted the statistic, and obtained the response that they too found the figure hard to believe but that it was produced by a reliable source. When we checked the methodology that the source in question had used, we judged it to be totally inappropriate and based on invalid assumptions. In fact, the methodology was found to be questionable on several counts. For example, in the case of developing countries where data was not readily available, the study used the Lorenz model of calculating probabilities. It is assumed that the mortality rate - including the rate of deadly violence – is necessarily higher in remote rural areas than in urban centres. This method may be better adapted to the calculation of the general death rate than the homicide rate, since other studies indicate that violence tends to be higher in urban centres. The model also employs a standard differential calculated on the basis of all the other causes of mortality examined. The rate of mortality for Latin America and sub-Saharan Africa attributed to other causes thus directly influences the calculation of the homicide rate. In addition, the source of data used as a basis for the estimates differs: whereas the figures for Latin America were provided by the respective governments (for the year 1990), those for sub-Saharan Africa come principally from studies by the World Bank for 1993. Finally, the method of data collection is not the same, since the homicide rates for countries of the northern part of sub-Saharan Africa were estimated on the basis of data collected for countries of the southern part of the continent 6.

I have a hard time trying to answer the question: why would anyone invent such an outrageously misleading methodology simply to obtain a statistic? The issue bears some serious consideration. Of even greater concern is the fact that other researchers, clearly unconvinced of the validity of the statistic, would repeat it in their publications without going to the trouble of double-checking the methodology used. We are all somewhat under the spell of the “statistical mystique”: statistics create an “illusion of precision” (Barsh 1999) that policy-makers at any level find appealing. But particular care must be given to ensuring transparency and critical review in the fabrication of quantitative measures in an area as complex and value-laden as human rights and democracy.

The Need for Theory to Link Concept to Indicator

If we are to arrive at a manageable set of human rights indicators, two things are necessary: to have a relatively small series of indicators (tens, rather than hundreds) and that those indicators be highly significant in terms of the right in question.

Many existing frameworks for assessment of human rights treat each right or freedom as if it were an indicator. That is clearly an untenable approach, particularly if we are attempting to quantify. It is possible to disaggregate any single article of one of the International Covenants into (at least) several potential indicators. At a working group on indicators for participation and democracy last year 7, a preliminary discussion by participants came up with over twenty possible indicators for Article 19 (freedom of expression) alone. Clearly, a framework with a comparable number of indicators for each right would be extremely unwieldy. But if we are seeking to establish a small number of indicators for each right, it is necessary first to establish what are the component aspects of each right, what the relationship of each component is to the overall respect of that right, and finally what are the indicators that can generate meaningful information about one or the other of the components of that right. A theory of the relationship between the overall object of the right, its component parts and the relevance of indicators needs to be developed in each case. In general, this has not been done.

The issue of theory has to do also with normative standpoint and how we account for it in the process of analysis. Any analyst has a standpoint: we are all situated individuals imbued with interests (personal and institutional) and with values. Presumably we all adhere to values of human rights and democracy. But even there, we may each subscribe to substantially different visions of what these mean. After all, almost every expert on democracy proposes his or her own definition of its basic characteristics. The very phenomena we are attempting to measure are also value-laden and, moreover, the specific values and the relative importance accorded to each of them may vary from one context to another. The very point about democracy is its quality: how can that be captured by quantitative measures? To attempt to ensure that our own values and standpoint do not interfere with the analysis, we cannot simply eradicate them – we can only make them as explicit and open to critical debate as possible.

Since standpoint is unavoidably present in social analysis, although often dissimulated by objectivist discourse, the best way of controlling for it is by theoretically grounding the standpoint adopted. In the words of Kabeer (1999: 42), “the problem that this raises is not one of a normative standpoint per se … but in determining the extent to which this normative standpoint expresses values that are relevant to the reality it seeks to evaluate.”

4. What Practitioners want from Indicators

I cannot purport to speak from the point of view of practitioners in general, for they are as varied and their approaches as diverse as those of the producers of indicators. My institution is at one and the same time a modest producer and a practitioner, and we produce our analyses hand-in-hand with civil society practitioners in the countries where we work.

Based on those relationships, I think that it would be fair to say that, in general, such practitioners (principally human rights NGOs, democracy movements and research centres) consider that indicators can be useful tools for their work. The nature of the tool and the use to which it might be put vary from one context to another : some want to use indicators as a political tool, to bring national and international attention or pressure to bear on a government that is demonstrated to be violating rights and abusing democratic process. Others use indicators as a policy tool, to develop and advocate proposals to their own governments or to international institutions in order to address problems identified through the use of indicators. Still others see indicators principally as a tool for their own strategic planning and programming, as individual organisations or in coalition. In such cases, indicators are used to identify problem areas and to monitor the impact of activities designed to address them through the interventions of the NGOs themselves.

However, the use of indicators by practitioners is quite incipient, despite a few notable exceptions (e.g.: CEDAL 1999), and there is not yet what one might call a « culture of indicators » in the NGO community. By this, I mean not only that indicators are little used and little developed by NGOs, but also that there is no real habit of handling them, no real understanding of how they are constructed, what the variety of options are, no consciousness of their faults and limitations. In short, indicators are somewhat mystified by NGOs. There exists a certain naiveté regarding what they can and cannot accomplish. Many NGOs tend to think that indicators will give them knowledge they don't already have – whereas, in most cases, indicators will confirm knowledge that NGOs already possess about the environment they are working in. Indicators are a way of ordering, structuring, systematising knowledge in a more rigorous manner, but they do not often teach us something new about a reality with which we are already familiar. So, NGOs that don't work with indicators often tend to have an exaggerated idea of what indicators will do for them.

Concomitantly, there is very little training available for NGOs that want to establish indicators for their own work. Here, perhaps, we begin to tread the thin line between what NGOs would like to monitor for their own purposes and the growing insistence by funding institutions that grantees develop « performance indicators ». Those of us who are funders need to thoroughly and critically examine this bureaucratic trend which in many cases imposes superfluous reporting duties on already over-extended NGOs. But this has little to do with the issue of indicators on substantive rights and democracy issues.

Some NGOs have examined the question of indicators and have taken a clear decision not to use them. This is the case of Amnesty International and of Human Rights Watch. Certainly the quality of their work and their credibility is not affected by the fact that they do not use indicators. Many women's rights organisations, on the other hand, are calling for more and better statistics on violence against women. Aware that, for many reasons linked to lack of confidence on the part of women, fear of reprisals, lack of diligence on the part of public institutions, existing sources of statistical information (such as hospitals and police stations) are unreliable, women's organisations are developing their own research (see Swiss, Jennings et al., 1998) and at the same time pressing governments to commit to more systematic reporting processes. These organisations, although fully cognisant of the need for qualitative information, push for better statistical indicators as well because they find they have more impact when dealing with governments and the men who run them.

In short, there is no general consensus amongst NGO practitioners as to whether statistical indicators should be developed, nor how that might be done. What is clear, however, is that NGOs expect that indicators will be useful for their work, be that on the political, the policy or the programming level. It is also clear that, for those few who do use indicators and statistics now, that they rely heavily on contextual analysis as a complement to statistics per se. Those NGOs that do possess expertise in the field of statistics and indicators are much more interested in developing and applying them in a flexible and participatory manner, always in close relation with the visions, priorities and needs of the communities with whom they work. They propose innovations such as the idea that indicators be applied at the sub-national (rather than national) level and that both quantitative and qualitative indicators be relativised (Joseph 1999: 24); that specific combinations of indicators be used according to the specific problem to be addressed (Barrios 1999: 100). Across the board, they emphasise a participatory approach not only to the application but also to the design of indicators, an approach that includes first and foremost the local community. They also underline the importance of the contextualisation of indicators, in application as well as in interpretation (see Barrios 1999: 95; Caceres 1999: 82; Joseph 1999: 22; Swiss, Jennings et al. 1998). Undoubtedly, any attempt to develop national-level indicators would also need to respond to these concerns on the part of NGO practitioners.

5. Conclusions

No one can be against indicators if those indicators are sound and reveal meaningful information about the right being examined. Some contend that that is simply not possible, given the numerous conceptual problems and distortions in translating rights into measurable data. These authors are against the creation of indicators, period. Others, on the contrary, argue that any statistical data, even if incomplete, is a step in the right direction. Still others – the majority, it would appear to me – see the usefulness of a combination of analysis and quantified data, the latter being a useful complement to the former but never under any circumstances being able to stand on its own.

Therein lies the major problem. We know by now that if a statistic is produced, it will be used, in many cases without contextual analysis and without any awareness of the methodological constraints under which it was generated. This tendency to « autonomisation of statistics » is enhanced by the numerous proposals for rating systems and comparative indices based on calculations that are riddled with unadmitted subjective judgements and uncontrolled variables. Those of us who are minimally aware of the conceptual and methodological issues attendant upon the development of quantitative indicators for human rights and democracy must take it upon ourselves to broaden and deepen the debate as to what constitutes a well-founded indicator. The problems are not necessarily insurmountable, but they must be seriously addressed. To paraphrase a famous democrat: “Statistics are too important to be left to the statisticians!”

Some Recommendations

The underlying objective informing any work in the human rights field is ultimately to improve the respect and promotion of human rights in all countries. That being the case, two conclusions follow. First, most would agree that a common grid for assessing respect for human rights in all countries would be a useful tool. Second, however, it is equally clear that a comparative index of rights across countries is a conceptually risky operation and often a misleading one: what purpose therefore would it really serve? Our overriding concern must be how to ensure that such a set of common indicators effectively bind together both quantitative data and its qualitative interpretation. How to ensure, in other words, that the analysis is not “shaved off” and we are left with the bare statistics?

The Human Development Report 2000 argues that statistics can help create a culture of accountability for realising human rights. If that is true, we must also ensure that there is a culture of accountability in the very process of creation of those statistics. To that end, two related initiatives are necessary in order to improve efforts to construct a common grid.

One is fostering a culture of statistics amongst international human rights and democracy organisations and national partners in the field. The debates generated by the 1991 and 1992 UNDP proposals for a comparative index, the workshop organised by the Office of the High Commissioner for Human Rights in September 1999 on indicators, and hopefully the HDR 2000, play an important role in stimulating interest in the issue, in giving it some profile amongst NGOs. We now need to go further in developing knowledge and capacity amongst NGOs in reading quantitative data and in generating their own should they consider it useful. The prevailing notion that « any statistic is a good statistic » must be reversed. A capacity must be developed from the field to determine what quantitative measures are necessary, under what circumstances they are valid, and how they should be interpreted.

The second initiative is promoting transparency and dialogue in building an international indicators framework. This will take time. A major step was made by the OHCHR last year, at its workshop which brought together representatives of UN bodies, human rights experts and a small number of specialised NGOs and research centres. Experiences of broadening efforts have recently taken place in other fields. A good example is perhaps that which has occurred in the wake of the 1995 Beijing Conference, where « the general approach in the development of gender statistics has involved efforts to promote dialogue and understanding between statisticians and the various users of statistics – policy makers, representatives of non-governmental organisations, activists and researchers… Responding to the Beijing Platform, centres for women's studies and research organisations, both at the national and international levels, are becoming more involved with statistical producers in developing and testing appropriate methodologies to strengthen gender analysis, as well as in monitoring and evaluating the implementation of the goals of the Beijing Conference » (UN 2000 : xix). A concrete example is the WHO Violence Against Women Database, which relies for its research « on an extensive global network of non-governmental and governmental organizations, as well as individuals and institutions » (UN 2000 : 157).

Such initiatives can provide a useful inspiration for the type of process that should be engaged in the attempt to identify indicators for human rights and democracy. A transparent, on-going dialogue involving national and international NGOs, government policy-makers, academics, and international bodies to determine objectives and construct and test appropriate concepts and methodologies is without doubt a complex challenge, but the only way to arrive at an outcome that will effectively be used by those who have the most to gain from it.

REFERENCES

Arat, Z. (1999). The Level of Respect for International Human Rights: Recognition, Realization, and Protection. Paper presented at the conference “Towards an Indicators System in Human Rights”. 16-17 September. International Institute for the Sociology of Law. Oñati, Spain.

Arat, Z. (1991). Democracy and Human Rights in Developing Countries. Lynne Rienner Publisher. Boulder and London.

Barrios, A. (199). Indicadores de Medición: Aspectos metodológicos y Estratégicos de Exigibilidad, pp. 94-102 in CEDAL (1999). Indicadores para la Vigilancia Social de los DESC. Lima, Perú.

Barsh, R. L. (1993). Measuring Human Rights: Problems of Methodology and Purpose. Human Rights Quarterly 15, 87-121.

Barsh, R.L. (1999). The Fortunate Unmeasurability of Human Rights. Paper presented at the conference “Towards an Indicators System in Human Rights”. 16-17 September. International Institute for the Sociology of Law. Oñati, Spain.

Bollen, K.A. (1992) Political Rights and Political Liberties in Nations : An Evaluation of Human Rights Measures, 1950 to 1984. In Human Rights and Statistics : Getting the Record Straight (eds. T.B. Jabine and R.P. Claude), 188- 215. University of Pennsylvania. Philadelphia.

Broadbent, E. (1996). Things fall apart; the centre cannot hold; Mere anarchy is loosed upon the world...' or Post-Referendum Canada. Speech delivered to the Joint Meeting of the Canadian Political Science Association and la Société québécoise de science politique, Ottawa-Hull, January 27.

Caceres, E. (1999). La Necesidad de Instrumentos Cualitativos, pp. 82-84 in CEDAL, Indicadores para la Vigilancia Social de los DESC. Lima, Perú.

Carter Center, (1992). Quality of Democracy Indicators. Atlanta.

CEDAL (1999). Indicadores para la Vigilancia Social de los DESC. Lima, Perú.

Claude, R.P. and Jabine, T.B. (1992). Exploring Human Rights Issues with Statistics. In Human Rights and Statistics : Getting the Record Straight (eds. T.B. Jabine and R.P. Claude), 5 - 34. University of Pennsylvania. Philadelphia.

Gastil, R. D. (1991). Freedom in the World: Political Rights and Civil Liberties. Greenwood Press. Westport, Connecticut.

Gillies, D. (1993). Human Rights and Democratic Governance: A Framework for Analysis and Donor Action. International Centre for Human Rights and Democratic Development. Montreal.

Goldstein, R.J. (1992). The Limitations of Using Quantitative Data in Studying Human Rights Abuses. In Human Rights and Statistics : Getting the Record Straight (eds. T.B. Jabine and R.P. Claude), 35-61. University of Pennsylvania. Philadelphia.

Gupta, D., A. Jongman and A. Schmid, 1994. "Creating a Composite Index for Assessing Country Performance in the Field of Human Rights: Proposal for a New Methodology", Human Rights Quarterly 16, 1, 131-162.

Humana, C. (1986). World Human Rights Guide. Facts on File Publications. New York and Oxford.

Joseph, J. (1999). Métodos alternativos en la Construcción de Indicadores, pp. 22-24 in CEDAL, Indicadores para la Vigilancia Social de los DESC. Lima, Perú.

Kabeer, N. (1999). The Conditions and Consequences of Choice : Reflections on the Measurement of Women's Empowerment. (Discussion Paper No. 108). United Nations Research Institute for Social Development. Geneva.

Kapoor, I. (1996). Indicators for Programming in Human Rights and Democratic Development : A Preliminary Study. Canadian International Development Agency. Ottawa, Canada.

Klug, F. (1993) Human Rights as Indicators of Democracy. Human Rights Centre, University of Essex, Great Britain.

Lopez, G.A. and Stohl, M. (1992). Problems of Concept and Measurement in the Study of Human Rights. In Human Rights and Statistics : Getting the Record Straight (eds. T.B. Jabine and R.P. Claude), 216-234. University of Pennsylvania. Philadelphia.

O'Donnell, G. (1999). Democratic Theory and Comparative Politics. Paper presented at the annual meeting of the American Political Science Association, August 26-29. Atlanta, United States.

Organisation for Economic Co-operation and Development, OECD (2000). Indicators for the International Development Goals : A Suggestion for Some Qualitative Indicators. Paris.

Pinheiro, P.S. (1999). Human Rights Indicators in New Democracies. Paper presented at the Workshop on Civil and Political Rights Indicators, Geneva. Center for the Study of Violence. University of Sao Paulo. Brazil.

Radstaake, M. (1999). Human Rights Assessments and Indicators Equally Needed to Enhance Human Rights Policies. Paper presented at the Workshop on Civil and Political Rights Indicators, Geneva. Human Committee on Human Rights. Netherlands.

Samuelson, D.A. and Spirer, H.F. (1992). Use of Incomplete and Distorted Data in Inference About Human Rights Violations. In Human Rights and Statistics : Getting the Record Straight (eds. T.B. Jabine and R.P. Claude), 62 - 77. University of Pennsylvania. Philadelphia.

Sen, A. (2000). "A Decade of Human Development". Journal of Human Development, 1, 1, 17-24.

Swiss, S., Jennings, P. et al. (1998). “Violence Against Women During the Liberian Civil Conflict”. Journal of the American Medical Association, 279, 8, 625-629.

Taylor, C. and Muntarbhorn, V. (1994). Roads to Democracy: Human Rights and Democratic Development in Thailand. International Centre for Human Rights and Democratic Development. Montreal.

United Nations (2000). The World's Women 2000 : Trends and Statistics. (Social Statistics and Indicators Series K, No. 16). New York.

United Nations Development Program, UNDP (2000). Human development Report 2000. New York.

United Nations Development Program, UNDP (1992). Human development Report 1992. New York.

United Nations Development Program, UNDP (1991). Human development Report 1991. New York.

Endnotes

1.   The principal author of the HDI, Amartya Sen, writes that « it would be a great mistake to concentrate too much on the Human development Index, or any other such aggregative index… These are useful indicators in rough and ready work, but the real merit of the human development approach lies in the plural attention it brings to bear on developmental evaluation, not in the aggregative measures it presents as an aid to digestion of diverse statistics » (2000 : 23).  Return

2.  The instruments from which we have drawn our criteria are the following: the International Bill of Human Rights (comprising the Universal Declaration and the Covenants on Civil and Political Rights and on Economic, Social and Cultural Rights), which is the standard reference for all human rights assessments; the Convention on the Elimination of All Forms of Discrimination Against Women (CEDAW), the Declaration on the Right to Development, ILO Convention 169 Concerning Indigenous and Tribal Peoples in Independent Countries, which constitutes the only internationally-recognised instrument in this field. Although it does not touch on the issue of self-determination, it does specify the obligations of states regarding a broad spectrum of civil, political, economic, social and cultural rights of indigenous peoples.  Return

3.   We define violence against women in the same manner as the United Nations' Declaration on the Elimination of Violence Against Women (A/48/629), as follows: "any act of gender-based violence that results in, or is likely to result in, physical, sexual or psychological harm or suffering to women, including threats of such acts, coercion or arbitrary deprivation of liberty, whether occurring in public or in private life." In the terms of the Declaration also "women's opportunities to achieve legal, social, political and economic equality in society are limited... by and endemic violence."  Return

4.  Article 25 deals with the rights of citizens of a given state to participate in the conduct of its political affairs.  Return

5.   This statement is inspired by O'Donnell's affirmation that the political freedoms (assembly, association, information, expression) are « subjective » rather than « individual » rights (1999 : 25).  Return

6.   I am indebted to Geneviève Lessard for her research into the methodology of the study.  Return

7.   Workshop on Civil and Political Rights Indicators, 27-30 September 1999, organised by the Office of the High Commissioner for Human Rights, Geneva.  Return