Fact checking comparison between trachoma surgeries and guide dogs

Cross-posted in effective-altruism.com. Interesting discussion in comments there, Peter Singer himself commented.

In a 2013 TED talk Peter Singer claims

“It costs about 40,000 dollars to train a guide dog and train the recipient so that the guide dog can be an effective help to a blind person. It costs somewhere between 20 and 50 dollars to cure a blind person in a developing country if they have trachoma.”

Unfortunately, this claim is not accurate. To begin with, blindness from trachoma is irreversible so it’s only possible to prevent blindness from trachoma, not to cure it. According to a GiveWell blog post, it does cost ~$20-60 to perform one trachoma surgery but “there can be a small improvement in vision following surgery”. According to their back-of-envelope calculation with some assumptions, 1 case of full-blown blindness is averted for every 6-20 successful surgeries. However, ~$20-60 seems to be the cost for surgery of one eye. I’m not sure GiveWell took that into account and I don’t know how to do it correctly. In any case, my point is that people who use this example to advertise GiveWell don’t read what GiveWell has to say about it.

——-

EDIT (2017-05-16): Even though GiveWell haven’t made such claim and may have a different opinion, one doctor (who has a much deeper understanding of these issues than me) commented that she “would be comfortable with saying that for about $100 we can prevent trachoma-induced blindness” and that Singer’s claim was not as nearly as inaccurate as I made it seem.

——-

As of 2017-05-10, Giving What We Can also gives a similar example:

“In the developing world there are more than a million people suffering from trachoma-induced blindness and poor vision which could be helped by a safe eye operation, costing only about $100 and preventing 1-30 years of blindness and another 1-30 years of low vision, according to GiveWell.org

They also do something EAs (including me) don’t do often enough — provide a source. The source is a GiveWell page which is published in 2009 and has a disclaimer

“The content on this page has not been recently updated. This content is likely to be no longer fully accurate, both with respect to the research it presents and with respect to what it implies about our views and positions.”

The page has the following text:

“We have not done thorough cost-effectiveness analysis of this program. Because such analysis is highly time-consuming – and because the results can vary significantly depending on details of the context – we generally do not provide cost-effectiveness analysis for an intervention unless we find what we consider to be a strong associated giving opportunity.

We provide some preliminary figures based on the Disease Control Priorities in Developing Countries report, which we previously used for cost-effectiveness estimates until we vetted its work in 2011, finding major errors that raised general concerns.

We have relatively little information about the likely impact of this program, so it’s difficult to estimate the cost-effectiveness.”

[…]

“Using a simple conversion calculation, we estimate that $100 prevents 1-30 years of blindness and an additional 1-30 years of low vision when spent on surgeries (though insignificant benefits, in these terms, when spent on antibiotics). The source of the Disease Control Priorities in Developing Countries report’s estimate is unclear and these figures should be taken with extreme caution.

It seems unfair to just provide the numbers and skip all these disclaimers. Despite knowing about this uncertainty, sometimes I feel temptation to also omit disclaimers and just present the numbers to be more convincing. After all, the goal is very admirable – to help more people living in extreme poverty. But I believe that in the long run EA will achieve more if we are being totally honest and upfront about uncertainties and never take any shortcuts. Otherwise we might not be trusted the next time we have something to say. Furthermore, to influence the world we need our community to have a correct model of the world.

On the other hand, trachoma is a horrible disease. Just watch this excerpttl;dw: eyelids turn inwards and eyelashes scrape the eyeball, causing intense pain on every blink. That scraping eventually causes blindness. People treat themselves by pulling out their eyelashes with tweezers. One woman said she does it every 2 weeks. Horrible.

If you worry about being convincing, you can talk about that and then honestly talk about uncertainty regarding numbers. Most people are scope insensitive anyway. Or you can talk about cataract surgery instead of trachoma because disclaimers in this page seem slightly less severe. Or just talk about your favorite charity and then add “imagine that suffering could be prevented so cheaply in our country, action would be taken urgently”. But the main points of this post are

  • many of us were overstating the point that money goes further in poor countries
  • many of us don’t do enough fact checking, especially before making public claims
  • many of us should communicate uncertainty better

——-

EDIT (2017-05-15):

Many people in the comments gave other reasons not to use the comparison but if you decide to use it anyway and want to quote GiveWell, you could also use figures from this Peter Singer’s comment. Alternatively, you can use one of the other comparisons proposed by Ben Todd.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s