World Development symposium on RCTs

World Development has a great collection of short pieces on RCTs.

Here is Martin Ravallion’s submission: 

….practitioners should be aware of the limitations of prioritizing unbiasedness, with RCTs as the a priori tool-of-choice. This is not to question the contributions of the Nobel prize winners. Rather it is a plea for assuring that the “tool-of-choice” should always be the best method for addressing our most pressing knowledge gaps in fighting poverty.

… RCTs are often easier to do with a non-governmental organization (NGO). Academic “randomistas,” looking for local partners, appreciate the attractions of working with a compliant NGO rather than a politically sensitive and demanding government. Thus, the RCT is confined to what NGO’s can do, which is only a subset of what matters to development. Also, the desire to randomize may only allow an unbiased impact estimate for a non-randomly-selected sub-population—the catchment area of the NGO. And the selection process for that sub-sample may be far from clear. Often we do not even know what “universe” is represented by the RCT sample. Again, with heterogeneous impacts, the biased non-RCT may be closer to the truth for the whole population than the RCT, which is (at best) only unbiased for the NGO’s catchment area.

And here is David Mckenzie’s take: 

A key critique of the use of randomized experiments in development economics is that they largely have been used for micro-level interventions that have far less impact on poverty than sustained growth and structural transformation. I make a distinction between two types of policy interventions and the most appropriate research strategy for each. The first are transformative policies like stabilizing monetary policy or moving people from poor to rich countries, which are difficult to do, but where the gains are massive. Here case studies, theoretical introspection, and before-after comparisons will yield “good enough” results. In contrast, there are many policy issues where the choice is far from obvious, and where, even after having experienced the policy, countries or individuals may not know if it has worked. I argue that this second type of policy decision is abundant, and randomized experiments help us to learn from large samples what cannot be simply learnt by doing.

Reasonable people would agree that the question should drive the choice of method, subject to the constraint that we should all strive to stay committed to the important lessons of the credibility revolution.

Beyond the questions about inference, we should also endeavor to address the power imbalances that are part of how we conduct research in low-income states. We want to always increase the likelihood that we will be asking the most important questions in the contexts where we work; and that our findings will be legible to policymakers. Investing in knowing our contexts and the societies we study (and taking people in those societies seriously) is a crucial part of reducing the probability that our research comes off as well-identified instances of navel-gazing.

Finally, what is good for reviewers is seldom useful for policymakers. We could all benefit from a bit more honesty about this fact. Incentives matter.

Read all the excellent submissions to the symposium here.

How to increase mass employment in Nigeria (and other developing countries)

David Mckenzie of the Bank writes:

The modal firm size in most developing countries is one worker, consisting of only the owner of the firm. Amongst the firms that do hire additional workers, most hire fewer than ten. In Nigeria survey data indicate that 99.6 percent of firms have fewer than 10 workers. This is in sharp contrast to the United States, where the modal manufacturing firm has 45 workers. Are there constrained entrepreneurs in developing countries with the ability to grow a firm beyond this ten worker threshold? If so, this raises the questions of whether such individuals can be identified in advance, and of whether public policy can help them overcome these constraints to firm growth.

In an attempt to figure out if policy can help grow firms in developing countries, Mackenzie evaluated a program in Nigeria that awarded 1,200 winners about $50,000 each (out of an initial application pool of 24,000; the top 6000 applicants were in the study). See a summary here. And the paper is available here.

………. winning this competition has large positive impacts on both applicants looking to start new firms as well as those aiming to expand existing firms. Three years after applying, new firm applicant winners were 37 percentage points more likely than the control group to be operating a business and 23 percentage points more likely to have a firm with 10 or more workers, while existing firm winners were 20 percentage points more likely to have survived, and 21 percentage points more likely to have a firm with 10 or more workers. Together the 1,200 winners are estimated to have generated 7,000 more jobs than the control group, are innovating more, and are earning higher sales and profits.

Two quick thoughts. First, this is a really cool finding that should get African central bankers excited about how the financial sector can be put to use in boosting mass employment. Second, it is a caution against the odd idea prevalent in development programs of trying to turn poor people into entrepreneurs (see below). The best solution to poverty is jobs. Entrepreneurship is a risk that shouldn’t be imposed on people with already super slim margins of error in terms of income security. As Mckenzie rightly observes:

The results of this evaluation show that a business plan competition can be successful in identifying entrepreneurs with the potential to use the large amounts of capital offered as prizes, and that these individuals appear to be otherwise constrained from realizing this potential. The prize money generates employment and firm growth that would not have otherwise happened. However, the results also highlight the difficulties of picking winners. Conditional on reaching the semi-finalist stage, neither the scores for the business plans, nor individual and business characteristics have much predictive power for predicting which firms will grow faster or benefit most from the program. This remains an area for active research, but also highlights the inherent riskiness of entrepreneurial activity.