Skip to main content

The Problems with Evidence-based Policy

In contested times, it isn't unusual to witness a race to shape public opinion not just by visibly subverting the media and suppressing free speech outright but also by more subtly shaping speech to mould the contours of public discourse. This may be achieved by various means including by issuing outrageous statements and then either confirming or retracting them (possibly oneself or by having a colleague tone down one's words) after the statement’s acceptability can be gauged (based on public response), by developing narratives which comprise little but fact-independent fake news and sycophantic statements by those who kowtow to power, or by privileging one form of discourse over the other (such as by idealising ostensible civility to the exclusion of all else).

We've seen that rhetoric can lead to significant change even if it is not always based on anything which actually exists in reality. And, knowing that passionate rhetoric has the ability to manipulate us by playing on our fears and hopes, it is entirely reasonable to ask for evidence supporting its propositions, and to develop the desire to rely on what we think is hard evidence.

It is, however, all too easy to fall into the trap of advocating nothing but the dispassionate and suggesting that policy be based entirely on what, at first glance, appears to be calmly-considered, cogent evidence. Unfortunately, this approach is doomed to fail since evidence is not a panacea. Evidence can only provide a façade of neutrality when it comes to determining policy.

While it is impossible to legitimately make the argument that policy should be framed independent of reliable evidence, there remains the argument to be made that we should be wary of evidence. There are questions about how evidence can be manipulated and about what constitutes reliable evidence in the first place.

We are often encouraged to believe that evidence comes in the form of data collated from a large sample and that other forms of evidence are inferior. This understanding doesn't always take into account that data can be flawed, or that what we consider reliable evidence often privileges the supposedly-credible voice of that fraction of the ostensible collective which has a voice to start off with over the marginalised, often individual, voice that may not even be heard (or be the subject of any collation of data).

Take the case of Section 498A of the Indian Penal Code which is intended to address cruelty to wives, for example. It is claimed that it is often misused: there is ‘data’ in the form of NCRB statistics which are not helpfully compiled for the purpose of assessing the extent of misuse and which are often misinterpreted (not least by assuming every acquittal is indicative of a false case), and there are many people (who usually seem to be men) who provide their own experience in support of the claim that 498A is rampantly misused.

It is also claimed, far less audibly, by abused wives and their allies (who, in the latter case, it must be said, are invariably few and far between) that women often find it difficult to invoke IPC Section 498A and that, far from invoking it at the drop of a hat as many men seem to believe they do, approaching the police is often the last resort of abused women. Their claim is, however, invariably drowned out in the clamour raised by men relating to supposed rampant misuse of the law ostensibly necessitating its repeal. That men's voices would drown out those of abused women in a patriarchy, which India is, isn't surprising.

The case of IPC Section 498A highlights many of the problems presented by ardently relying on evidence: firstly, data is always susceptible to misinterpretation; it can be cherry-picked or misinterpreted accidentally,intentionally or perhaps even subconsciously on account of one's own worldview. Additionally, data itself be flawed and inherently problematic due to issues with collection, collation, and reportage. (In the case of NCRB reports, mistakes of law & those of fact tend to be clubbed together, for example, as are discharges & acquittals, making it impossible to reach sensible conclusions about the misuse of a law.)  

Secondly, testimonies may conflict and tell distinct stories which are not easy to harmonise. According them credence requires the making of a value judgments about who is credible (or not). Testimonies may not all converge to tell a tale both unidimensional and identical, and the value placed on them may vary depending  on where in the socio-economic hierarchy its originator stands.

The answer to the question of who has more credibility in a patriarchy, of course, is men even if their claims are supported by the flawed analysis of imperfect data and statements which would fall flat if it were not for the authority accorded to them by patriarchy on account of their gender.  

Making it even more difficult for abused women to be heard and believed is that those who oppose them often sound calm, civil, reasoned and reasonable, sometimes in complete contrast to abused women whose speech can, quite understandably, be visceral and vociferous. Considering that civility is prized even though it tends to come most easily to the already-privileged who have no need to shout to make themselves heard, the expectation of civility can put abused women at a significant disadvantage.

These are the dynamics which tend to be at play regardless of what the issue is. Neither data nor testimony, which together tend to comprise evidence, exist in vacuums unaffected by the biases and complexities of the real world. Ultimately, even if one relies on evidence, one has to make a series of fraught value judgments to determine which evidence is reliable and which should be dismissed out of hand.

Having reliable evidence though is not enough. It cannot alone will not provide a solution to a non-technical problem. It can, assuming it is reasonably accurate itself, explain what the effect of a specific policy either is or would be but it cannot comment on the rectitude of specific policy choices.

This is simply because in complex, plural societies, there are always competing interests and there are no perfect answers that benefit everyone and accommodate all of their interests equally particularly since some interests (such as those relating to matters of faith) deal in intangibles in no small measure. What we choose to do with evidence available to us is a choice which is always overlain by our politics and our beliefs about entitlements, rights, and justice. There is never a single ‘right’ solution.

Coming back to the example of IPC Section 498A: with the evidence at hand, does one choose to strength perjury laws to address misuse or does one choose to strike down the provision itself in order to counter misuse? Whatever choice one makes, claiming that one's decision is ‘evidence-based’ alone would be disingenuous especially since the latter option would make it more difficult for abused wives to seek the redress of criminal law. It is certainly not a ‘neutral’ choice.

Evidence can, at best, explain the world as it is. It cannot independently and exclusively provide guidance about what should be done. It is only in conjunction tion with one's value system and one's politics that evidence can help determine what is to be done. And, so, even though the world may be overrun by those seeking power by shaping discourse largely without reference to facts, evidence and evidence-based policy will not save anyone.