More on the apparently *transient* effects of unconditional cash transfers

Berk Ozler over at Development Impact has a follow up post on GiveDirectly’s three-year impacts. The post looks at multiple papers analyzing results from the same cash transfer RCT in southwestern Kenya:

First, on the initial studies:

On October, 31, 2015, after the release of the HS (16) working paper in 2013, but before the eventual journal publication of HS (16), Haushofer, Reisinger, and Shapiro released a working paper titled “Your Gain is My Pain.”  In it, they find large negative spillovers on life satisfaction (a component of the psychological wellbeing index reported in HS 16) and smaller, but statistically significant negative spillovers on assets and consumption. The negative spillover effects on life satisfaction, at -0.33 SD and larger than the average benefit on beneficiaries, imply a net decrease in life satisfaction in treated villages. Furthermore, the treatment (ITT) effects are consistent with HS (16), but the spillover effects are not. For example, the spillover effect on the psychological wellbeing index in Table III of HS (16) is approximately +0.1, while Table 1 in HRS (15) implies an average spillover effect of about -0.175 (my calculations: -0.05 * (354/100)). There appear to be similar discrepancies on the spillovers implied for assets and consumption in the HRS (15) paper and HS (16). I am not sure what to make of this, as HRS (15) is an unpublished paper – there must [be] a good explanation that I am missing. Regardless, however, these findings of negative spillovers foreshadow the three-year findings in HS (18), which I discuss next.

Then on the three-year findings:

As I discussed earlier this week, HS (18) find that if they define ITT=T-S, virtually all the effects they found at the 9-month follow-up are still there. However, if ITT is defined in the more standard manner of being across villages, i.e. ITT=T-C, then, there is only an effect on assets and nothing else.

… As you can see, things have now changed: there are spillover effects, so the condition for ITT=T-S being unbiased no longer holds. This is not a condition that you establish once in an earlier follow-up and stick with: it has to hold at every follow-up. Otherwise, you need to use the unbiased estimator defined across villages, ITT=T-C.

To nitpick with the authors here, I don’t buy that [….] lower power is responsible for the finding of no significant treatment effects across villages. Sure, as in HS (16), the standard errors are somewhat larger for across-village estimates than the same within-village estimates. But, the big difference between the short- and the longer-term impacts is the gap between the respective point estimates in HS (18), while they were very stable (due to no/small spillovers) in HS (16). Compare Table 5 in HS (18) with Appendix Table 38 and you will see. The treatment effects disappeared, mainly because the differences between T and C are much smaller now, and even negative, than they were at the nine-month follow-up.

And then this:

If we’re trying to say something about treatment effects, which is what the GiveDirectly blog seems to be trying to do, we already have the estimates we want – unbiased and with decent power: ITT=T-C. HS (18) already established a proper counterfactual in C, so just use that. Doesn’t matter if there are spillovers or not: there are no treatment effects to see here, other than the sole one on assets. Spillover estimation is just playing defense here – a smoke screen for the reader who doesn’t have the time to assess the veracity of the claims about sustained effects.

Chris has a twitter thread on the same questions.

Bottom line: we need more research on UCTs, which GiveDirectly is already doing with a (hopefully) better-implemented really long-term study.



Effects of Conditional Vs. Unconditional Cash Transfer

Baird, McIntosh and Ozler have an upcoming paper in the QJE investigating the differential impacts of conditional and unconditional cash transfer in Malawi:

Starting with schooling outcomes, we find that although dropout rates declined in both treatment arms, the effect in the UCT arm is 43% as large as that in the CCT arm. Evidence from school ledgers for students enrolled in school also suggests that the fraction of days attended in the CCT arm is higher than the UCT arm. Using independently administered tests of cognitive ability, mathematics, and English reading comprehension, we find that although achievement is significantly improved in all three tests in the CCT arm compared with the control group, no such gains are detectable in the UCT arm. The difference in program impacts between the two treatment arms is significant at the 90% confidence level for English reading comprehension. In summary, the CCT arm had a significant edge in terms of schooling outcomes over the UCT arm: a large gain in enrollment and a modest yet significant advantage in learning.

The paper then gets nuanced:

When we turn to examine marriage and pregnancy rates, however, unconditional transfers dominate. The likelihood of be- ing ever pregnant and ever married were 27% and 44% lower in the UCT arm than in the control group at the end of the 2-year intervention, respectively, whereas program impacts on these two outcomes were small and statistically insignificant in the CCT arm.

……….. Our findings show that UCTs can improve im- portant outcomes among such households even though they might be much less effective than CCTs in achieving the desired behav- ior change.

Check it out here.