Progress measuring progress
Forums are quite commonplace these days, but really useful ones are rare. We tend to think of Davos or Porto Alegre, or perhaps even the OECD’s own annual public forum. We imagine headlinemaking topics, on climate change, security, society, telecommunications or business innovation. But a recent OECD world forum held in Palermo, Italy, was different than most, for it had as its billing the rather unspectacular subject of statistics.
Curiously, the Forum was a hit. For nearly four days in November 2004, despite sometimes complicated technical presentations, the great amphitheatre holding the first OECD World Forum on Statistics, Knowledge and Policy was packed. Like in all good forums, business people and civil society activists, government officials, policymakers and experts converged on the wonderful Teatro Massimo to interact, network and busily exchange arguments and ideas. The media watched closely and, according to our data (yes, more counting), thousands of people from all over the world followed the debates live from their desktops.
Heartening stuff for us statisticians, but why such broad interest? Quite simply, this forum was not about the environment, or social exclusion, or new technology. Rather, it was about all of these things, about the data that underlies our knowledge and hence our action. The point is that today the role of statistics goes well beyond the production of figures, and touches upon people’s everyday lives.
When governments set a new budget, when households decide which school their children should attend, when businesses invest, when NGOs launch campaigns: all these decisions are often based on some sort of statistical information, which we turn into knowledge and use to inform our choices.
We live in a data-rich world in which ordinary people have become familiar with notions like GDP, or inflation rates. There seems to be no shortage of surveys, on the cost of living, on corruption, economic freedom, or on the state of our hospitals. There are surveys on the environment, or even on how depressed or happy we feel. But people quickly see the limits of such surveys. Some of them are serious, others are there just to entertain us, but which ones? And they paint so many different pictures about us that we end up being confused.
On the other hand, people want to see statistics that capture the quality of their lives, and not just any one aspect. They want to measure progress based on a fuller picture of where they stand.
People inevitably turn to statisticians to sort out what is measurable and what is not. But given the stakes, everyone wants to be involved. Hence the interest in Palermo. And the overarching question people from all backgrounds were asking was: how can we measure our lives better?
True, our life expectancy is higher, we seem healthier and smarter, and many countries are richer. But are we really making progress? Even in developed countries, we sense that the progress we have made is uneven, and perhaps not sustainable or secure. How can we be sure? Without sound data, advice sounds rhetorical, and policy prescriptions ideological.
There are several ways to measure the overall situation of a country, but three attract particular attention. The first extends traditional economic accounts based on GDP, for instance, to capture the environment or social concerns. A second way is to develop composite indicators of wellbeing that combine detailed information into a single measure. And the third seeks to identify a certain number of key indicators covering economic, social and environmental domains, without deriving any particular single measure. Each of these approaches has strengths and weaknesses.
The first approach is quite an investment in terms of the amount of data to be collected and the resources needed to make them coherent. It encounters theoretical and practical difficulties in expressing some aggregates in monetary terms, such as environmental resources. And while it may be very powerful analytically, for instance, for simulating the economic, social and environmental effects of different policy measures, it is hard to see how such a complicated approach could easily win over governments, investors or the public.
Composite indicators seem more attractive from this perspective, as they can combine a wide range of indicators – even “apples and oranges” – into a single measure. Human development indices and quality of life indicators are produced this way. They lend themselves easily to rankings among countries, and are loved by journalists and readers alike. On the other hand, they can be very misleading, depending on the selection of indicators and weights used to aggregate the results. Their authors exercise their own tastes and judgements, yet do not always warn readers how individual components might distort the final results. So, if air quality gets a high weight, this may benefit the position of countries with low urban density.
An intermediate approach is the choice of sets of key indicators, based on a more holistic view of what matters most for development and progress in each country. It has the advantage of covering a wide range of topics – say, 20-30 or so – without the need of estimating individual weights. For instance, countries might focus on a list that covers economic growth, healthcare, air pollution, youth employment and new investment. The responsibility of making a comprehensive performance assessment lies with the users, according to their vision, expectations and culture. Such key indicators may not provide a single measure of progress and do not have the methodological soundness of some accounting frameworks. But they are gaining popularity because they provide users – the stakeholders – with a manageable amount of information whose selection reflects their choices, while providing a yardstick against which performance can be openly scrutinised. As David Walker, the US comptroller general, would put it, such indicators help people keep score.
The OECD is working, in one way or another, on all these approaches. We produce statistical measures of economic, social and environmental conditions in both countries and sub-regions. We undertake methodological research on accounting frameworks and composite indicators, and we hold international workshops, including on key indicators, with the Palermo forum being a prime example. Naturally, some critics wonder whether all of this work is nothing more than statistical overindulgence, with number crunchers just looking for yet new ways to skin the same cat. There is certainly a danger of statistical overload, and of according too much importance to certain figures. As some speakers at the forum showed, governments and other decision-makers can over-react to the power of numbers, too easily allowing rankings and targets to set policy priorities.
But what counts most is to know how to treat numbers, and to develop the knowledge we need to act on them. As European Central Bank chief, Jean-Claude Trichet, points out, good decisions also depend on good judgement. At the same time, there is so much we do not know, and such are the uncertainties of life, from climate change to market behaviour, that no one can afford to be complacent.
The forum in Palermo showed that statistics represent a fundamental tool in developing knowledge, so making better evidencebased decisions possible. Statisticians do not take the decisions, but they do an important job. As Thomas Jefferson said, “I know of no safe depository of the ultimate powers of the society but the people themselves; and if we think them not enlightened enough to exercise their control with wholesome discretion, the remedy is not to take it from them, but to inform their discretion by education".
©OECD Observer, No 246-247, December 2004
Where are we in the current economic crisis?