Advocate, Causes

Four Critiques Of RCTs That Aren’t Really About RCTs

As most people who follow international development know, the use of randomized controlled trials (RCTs) has sparked impassioned debate between so-called “randomistas” and the researchers, practitioners, and evaluators who oppose RCTs. Academics have been arguing about the merits and ethics of RCTs since they first came into use, and bloggers of all stripes have  contributed to the debate over the last few years. (A compilation of posts can be found here.) I hear these same conversations take place among my graduate school classmates and professors, whose opinions on RCTs are also divided.

In each of these arenas,  strong criticisms of RCTs have been made, some of which I don’t have good answers to. But many of the common critiques I hear really apply to research, aid work, and evaluation in general, and I fail to understand why these criticisms are levelled at RCTs specifically. I find it particularly troubling when researchers, aid workers, or evaluators make these critiques of RCTs without applying them to their own work.

Below are some examples of common critiques that I don’t believe are really aimed at RCTs.

“Studying People”

Many of my classmates and professors dislike RCTs because they are uncomfortable with the idea of “studying people,” particularly if nothing is provided to those people (in the case of a control group). In particular, for many RCTs, teams of surveyors go to people’s homes, schools, or workplaces to observe them or ask them questions, often of a personal nature. Opponents of RCTs argue this is voyeuristic and intrusive and that it interrupts people’s daily lives – and all for the unsavoury purpose of “studying people.”

Yet, as Kirsty Newman has explained, these complaints apply to much social science research, which is similarly based on going to other places, talking to people there, and gathering stories or data about them. Research is, in its essence, the study of people, done with the aim of extracting knowledge about them and their community, lifestyle, or past experiences and without providing benefits to them. I don’t think it makes sense to oppose RCTs on the grounds that “studying people” is bad, without also opposing the many forms of research that affect people in the same ways.

“Withholding Treatment”

I also frequently hear opponents of RCTs argue it is ethically wrong to “withhold treatment” from some people (again, in the case of a control group). RCTs rely on the ability to compare those who received an intervention with those who did not. Opponents believe that “withholding treatment” from anyone is unethical and that organizations should offer their interventions to everybody. They argue it is unfair for some people to receive a service while others don’t.

However, a similar critique could be made of aid programs in general, which Kirsty also noted. Like all providers of aid, those conducting RCTs have the funding and capacity to provide programming to a certain number of people. All aid projects provide services to some people and not to others, and no organization can include everyone in its interventions. In their inability to provide assistance to everyone, and the feelings of unfairness that result, organizations implementing RCTs are largely representative of NGOs in general.

External Validity

A more technical critique of RCTs is the argument that they lack external validity – a program that an RCT finds to be effective will not necessarily work in other settings. Because their results cannot be generalized, the argument goes, RCTs are useless.

But alternative types of evaluations (which are often based on before-after differences or comparisons of non-random treatment and control groups) face exactly this same problem, as Timothy Ogden explains. Organizations often use non-randomized evaluations to determine whether a program should be scaled up or expanded to additional areas, based on results from its initial implementation. However, there is no reason to think these evaluations would have higher external validity than RCTs.

Little Use

A final critique I want to address is the belief that the knowledge generated through RCTs rarely gets used to create improvements in aid. The results of RCTs are generally published in journals and primarily read by academics, not by people designing projects or making policy. For this reason, critics argue, RCTs do not actually lead to improvements, because they are inaccessible to the people in a position to them.

This is another way in which RCTs are no different from other types of evaluation, a point Kelly Steffen makes here. It’s no secret that organizations are typically resistant to admit failure or make changes, and a common complaint among evaluators is that their reports often go unread and their recommendations are frequently ignored. Though I think people in both areas are working to overcome this problem, neither RCTs nor other types of evaluations get used as much as they should.

These four arguments aren’t necessarily incorrect, but they’re not really criticisms of RCTs. Unless we consider them reasons to stop doing research, aid work, or evaluation in general, these critiques do not offer compelling reasons to stop doing RCTs. Instead, we should think more about how to best counter these shortcomings in all facets of development and social science work.