‘More Than Good Intentions’ and ‘Three Cups of Tea’
Posted on April 25, 2011 at 2:29 pm
It seems fitting that the Three Cups of Tea scandal coincides with the release of the book More Than Good Intentions: How a New Economics is Helping to Solve Global Poverty. And no, although our titles are similar, I’m not the book’s author. It was written by Dean Karlan and Jacob Appel.
One of the issues around the Central Asia Institute (CAI), the nonprofit started by the author of Three Cups of Tea, is the apparent lack of any evaluation of the schools it constructed. In response to the question posed by 60 Minutes “Has CAI ever commissioned an independent assessment of the effectiveness of its schools in Afghanistan and Pakistan?” the governing board’s answer was:
“No. CAI is unaware of any organization qualified to undertake such a study. However, it is clear that the effectiveness of its schools and its programs have been independently assessed by citizens of Pakistan and Afghanistan, in that there are hundreds of requests for new schools in these countries as well as the programs CAI promotes. In addition, the Government of Pakistan has awarded Greg Mortenson the Star of Pakistan related to the promotion of education in that country. Review of test scores of the children at schools built or sponsored by CAI will reveal that the children at these CAI related schools score higher on average than students at other schools. Over the years, many independent observers have visited the schools.”
CAI’s answer is immediately suspect as it claims increased test scores without any data to back it up. In addition, there are many people qualified to undertake an evaluation of their schools. Related studies are discussed in More Than Good Intentions. In Chapter 9 – “To Learn” – the authors discuss research on a variety of factors leading to student and teacher success. These include:
- purchasing school uniforms
- paying students to attend school
- providing deworming pills for students
- Providing bonuses to teachers that showed up to teach
- Using cameras to record teacher attendance
- Breaking students into classes based on their skill level
- Providing small group tutoring for struggling students
These studies were all Randomized Control Trials (RCT’s), which is the type of research used by pharmaceutical companies to determine the effectiveness of medication. More Than Good Intentions does a great job of explaining how Randomized Control Trials (RCT’s) work and what organizations can learn from them.
The book explains in detail how each of these different educational programs were tested to determine if they had their intended impact. The findings were rather interesting. Armed with this knowledge, nonprofits and governments can more confidently implement programs.
These and similar studies would have helped CAI determine which type of assistance had the greatest impact on the educational success of students. But instead of using research findings and evaluations, CAI seems to have spent millions of dollars based on the simple assumption that school buildings have the most impact on student learning. I’d be curious to find out if there is any study out there that proves that the lack of a building is the biggest impediment to education.
The Three Cups of Tea scandal should be a wake up call to all nonprofit governing boards. It takes far more than feel-good stories to prove actual impact. CAI is not alone in being unable to prove the impact of their work. I’m certain that thousands of other nonprofits would find themselves in the same position if their organization received the same amount of scrutiny.
I’d recommend More Than Good Intentions to any nonprofit board member interested in learning more about RCTs and how they might be used to determine how their nonprofit can have the greatest impact. Other people that would benefit from reading this book are students studying either international development or nonprofit management.
While not every program can be evaluated with an RCT and there are limitations, the book does a great job of laying out how they work and why evaluating the impact of a program is important. More Than Good Intentions is written in a very approachable fashion so that even people with little experience in international or community development can still understand the issues and main points.
Perhaps the best thing that could come out of this scandal is an increased focus on evaluating the impact of all nonprofit projects.
—–
Related Posts:
The Three Cups of Tea scandal and my greatest concern
Collecting Three Cups of Tea posts
Would you hire a doctor or contractor whose work was never evaluated?
It’s time to stop telling pretty stories and start evaluating the impact of aid
« The ‘Three Cups of Tea’ Scandal and My Greatest Concern • The shocking speed of social media »
While you may be most concerned about impact, it’s not illegal to fail at impact. So, agreed that it’s a moral issue, but it’s not the major lesson that will be learned. When the IRS goes after GM and the CAI board, everyone will be hyper aware of the illegality of mixing personal and NGO finances.
I really do hope this is a wake up call for not only nonprofits but for everyone, that we all nee d to be more careful with what we invest our money in and how much faith we put in these people and organizations. And yes it may not be illegal to fail at having an impact but when an organization fails and is not truthful about why it failed or, the amount of effort even put forth to have an impact then there is something wrong. Being in a fraternity which is a non profit, i know that we are very strictly managed and must do alot in regards to money and what we do. Before an organization is allowed to function, there should be some type of monitoring and evaluation system set up before it can continue.
From the NGO side, all of this call for evaluation sounds as effective as calling for more testing of kids doing poorly in school. Most foundation funding is for projects, not general operating, and there are lots of reasons why evaluation is rarely included in project budgets (it is considered overhead, grants are usually capped at $20K+/-). Local NGOs have no capacity for evaluation– they both don’t know how to do it and have little time to figure it out. I hope that we don’t pile on more Western-developed evaluations to try and avoid a 3 Cups problem. I hope we do look at new ways to evaluate with small NGOs at the table for the conversation.
I agree that we need to be flexible in how we evaluate, but we do need to evaluate. I’ve pushed for it long before this particular controversy. We also need to start educating donors about the importance of evaluations and the fallacy of the admin ratio – and this has to be more than just aid bloggers doing this. Nonprofits themselves have to stop selling the fantasy and start being honest about their work. That’s another point I’ve made for a long time as well, but is also a take away from this whole scandal.
I think it is important for NGO’s like CAI to be held accountable. People need to see that the money they are giving to these institutes is being used in the way it is suppose to. But I think it is equally as important for donors to have realistic expectations of organizations and what can be accomplished in the severely impoverished areas. CAI did some good things but honestly just reporting the number of schools being built would not be enough to get the financial support they needed. Maybe the focus should switch from look at what we did to look at how bad these areas are and we are going to do our best to make it even just a little better.
Saundra I’ve just recently found and thoroughly enjoy your website. I particularly appreciated the exam for prospective donors (I got a 100%). I think there’s a tough line to balance here. I don’t think there’s much point for the sector in piling on the “gotcha Greg Mortenson” 60 Minutes/Krakauer perspective. Calling for better evaluation is fine, with the important caveats that the administrator above mentioned. I also think we need to do a better job of communicating the real needs and limits of the sector – we must educate as part of programming, often – and we can’t work miracles with no money, most of the time (sadly). I’ve just posted on some of what I’m hoping the Mortenson scandal teaches us:
What Three Cups of Tea Teaches Us #1 – http://t.co/17TYuUP – Nonprofits need education and outreach. No book tour, no movement, no schools.
I think it’s sad how Mortensen and the CAI had lied about the work they have done, because obviously they have been doing good and there is really no need to inflate the numbers. I also think it’s ridiculous how much money is being spent on promoting his book (via the CAI) when there is no follow-through with a lot of these schools. I don’t mind that Mortensen is making a profit off the book, it is his work and he doesn’t allege to be giving the proceeds to the CAI. However, the fact that the CAI is spending more on promoting the book than building schools and continuing to help them out with funding and supplies is definitely ridiculous. Although Mortensen isn’t all bad, as he is obviously helping, I’m disappointed about the exaggerations and lack of follow through.
[...] Cups of Tea Scandal has highlighted many issues in the nonprofit world. I’ve written about two of them already and over 160 posts and articles have been written about the various aspects of this [...]
[...] Good Intentions Are Not Enough: “’More Than Good Intentions’ and ‘Three Cups of Tea’” [...]