Chasing reality: Rule of Law measurement is lagging years behind current developments

This Comment piece has been cross-posted  with the kind permission of Verfassungsblog, where it originally appeared.

This year, like every year, saw the usual spate of data and publications aimed at tracking and analysing changes in the Rule of Law. This year, unlike every other year, has seen a global pandemic of hitherto unknown proportions. We have seen extreme changes to institutional powers, the balance between institutions, and new innovations in digital courts and parliaments. These changes render much of the painstakingly collected and analysed data on the Rule of Law out of date.

But despite the shockwaves that have rippled through many countries, this year is not the only year to have seen changes in the relationship between the executive, judiciary and legislature, and changes in adherence to human rights and the Rule of Law. It is therefore not unusual to see quantitative measures of the Rule of Law age quickly. Moments that shake the independence of the judiciary, or see regimes violate fundamental rights often happen with little forewarning. Whether change arrives in form of an army coup forcing the evacuation of senior members of the judiciary like in Uganda in 2006 , or through clashes between Constitutional Courts and the Government as in Poland since 2015 , change happens rapidly; even if the undercurrents pulling towards change have been bubbling away for years.

The present turbulence therefore lays bare the shortcomings of methods of Rule of Law measurement that rely on data that can rapidly become obsolete. It is a vivid example of a general problem - that we struggle to capture change as it happens. Despite the extensive writing on Rule of Law crises, and its importance for EU politics (see the recent debates over Rule of Law budget conditionality ) obtaining up to date information on Rule of Law crises is virtually impossible.

The Current State of Rule of Law Measurement

At present there are nearly 10 data sources on the Rule of Law. Foremost among these are the World Justice Project's Indicators , and the Worldwide Governance Indicators  (which contain data on six dimensions of governance, including the Rule of Law). These are published annually, but contain data from an up-to three year period, covering six dimensions of governance. The data published this year therefore includes data collected in 2017, 2018, and 2019.

Looking back at these indicators feels like peering into another world. These surveys captured views of the state of the Rule of Law in what now seems like the long-distant past: the days before COVID-19, States of Emergency, lockdowns and social distancing. A world before Trump's dismissal of five inspectors general  and Barr's single handed erosion of trust in the justice department's independence . It took place before Hungary's use of the pandemic to consolidate of its move towards authoritarianism  through rule by decree, and further uses of autocratic legalism .

The indicators, when published, are based on opinions collected too long ago to capture Poland's blatant violation of constitutional norms regarding presidential elections  and the postponement of Benjamin Netanyahu's corruption trial on the basis of Covid-19 . While invariably crises will always lead to the rapid obsolescence of data, and those collecting the data could not have anticipated the change wrought by the pandemic, the other examples demonstrate that this problem is endemic to measurement approaches that rely on annual surveys.

A rapid shift in context, such as a terrorist attack, declaration of war, revolution or the establishment of a state of emergency can precipitate radical changes in the rule of law, in a particular country. Yet none of this information will be picked up by surveys, where the process of establishing, conducting, disseminating and evaluating the responses can take a couple of years. For example, the questionnaires put out to gather data by the World Justice Project  (WJP) need to be translated into a wide range of languages to ensure data can be collected on the indexes' 9 factors and 47 sub-factors. The extensive demands of data collection means that the general population polls (GPP) that supplement the Qualified Respondents' Questionnaires (QRQs) which comprise the data for the index are only conducted every other year. Functionally this means for many countries we will not see data on how the rule of law has been affected by Covid-19 until 2023. The WGIs collect data from a wide range of sources  including surveys of households and firms, commercial business information providers, non-governmental organizations, and public sector organizations. The cadence of release of these documents varies, but similar to the WJP, comprehensive data on the changes over the course of 2020 is unlikely to appear until 2022.

While Rule of Law indices provide invaluable, longitudinal sources of information on the evolution of the Rule of Law situation they can hardly be called timely. Functionally this creates huge problems for the users of governance and Rule of Law data who require up to date information. These issues are amplified by the limitations of the data sources helpfully set out by Tom Ginsburg  and Stefan Voight  including inter alia the difficulties of capturing the difference between de jure and de facto judicial independence, and the issues associated with aggregating the subcomponents of the Rule of Law.

Gathering accurate, reliable and timely data on the Rule of Law is crucial. Rule of Law data enables comparisons between countries and over time. It enables countries to make strategic reforms, citizens to demand change, policymakers to monitor risk areas and changes over time and journalists to provide statistics that supplements their narratives.

The Oxford Coronavirus Governance Tracker , and various projects to monitor the states of emergencies (see Verfassungblog , COVID-DEM ) are interesting in this regard. They provide fast updates on a rapidly evolving situation. The Oxford Tracker even converts qualitative assessments into a quantitative tool, by measuring policies  on a simple scale of severity / intensity while allowing for supplementary information in text to capture additional, relevant, information. We know that once states of emergency are declared they "generally lead to more government repression " and we need to find ways to easily illuminate this phenomena. Quantification is a crucial tool for shining a light on change. Numbers help to make sense of change by reducing the complexity inherent in countries' legal and institutional reality to a common form; thus making commensurable vastly different systems and enabling data users to compare and contrast between countries, and over time. Much of the enthusiasm for measurement comes from the significant practical and intellectual gains that accompany tracking and measuring changes.

New options for gathering data on the Rule of Law

The importance of collecting data, and limitations of the present approach mean that we should therefore be thinking creatively about what to do next. There are many different ways to extract the qualitative knowledge created by the mentioned projects and blogs, alongside media and civil society reports and turn it into quantitative, up-to-date, data. Those producing Rule of Law indices must therefore think hard about critically retooling measurement apparatus, mechanisms and approaches in response to the changing technological abilities, political and legal environment.

Options include drawing from the work of Environment, Social and Governance ("ESG") analytics companies - such as Refinitiv - that harness the power of Big Data and Machine Learning, such as using natural language Programming ("NLP")  to draw insights about corporate governance, and climate change by scraping relevant terms (for example "governance" from media and social media to identify controversies and calculate risk. Another approach could see new partnerships using AI to extract data from new sources, such as Project AIMS . Project AIMS builds upon the Business and Human Rights Resource Centre's ("BHRRC") Modern Slavery Registry with the intention of combating modern slavery by building a machine learning tool to automatize the analysis of statements produced by businesses under the UK and Australian Modern Slavery Acts. We could try to do the same with the analysis produced by international organisations on the Rule of Law situation.

In addition to thinking through new methods of data collection allowing for real time information updating are new types of insight on Rule of Law risk. For example, it would be useful to collect data that builds on Stefan Voight's conceptualisation of emergency measures  that track changes in approach to pandemics, specifically or changes in emergency legislation, generally.

We can also use technology to increase the frequency and reach of surveys, for example by embedding questionnaires into establishing constitutional and public law fora, or by sending short-form, online, surveys to existing recipients. This is not a recommendation to get technology to substitute for expert opinion about the Rule of Law, invariably an algorithm cannot substitute for an expert on the ground, but options that promote more frequent responses to questions make it simpler to reach that expertise.

Ultimately in a world where threats to the Rule of Law abound better information gathering is crucial, this moment shows the importance of action. Yet, this is not just about capturing the changes to the Rule of Law that have taken place during the pandemic, this is about using a sea-change moment to make better measures that are more responsive to changing situations. A multiple year lag is never very helpful, pandemic or no pandemic.

Nyasha Weinberg is a Research Fellow in Rule of Law Measurement at the Bingham Centre for the Rule of Law.