My response to Philanthropedia’s post

Posted on January 4, 2010 at 4:32 pm

My response to Philanthropedia’s posts which responded to GiveWell’s post Philanthropedia’s report on Microfinance

I was originally directed to Philanthropedia by two people with whom I had discussed the possibility of crowd sourcing charity evaluations. They both wanted my opinion of your system, but after I reviewed your site I came away with many of the same questions that GiveWell outlined in their post:

• Who are the experts?
• How are they chosen?
• What criteria is used to judge the charities?
• Is the criteria meaningful or is it simply easy to evaluate.

Without answers to these questions it was impossible to evaluate your rating system. Thus, I was very interested to read your response to GiveWell’s post. Unfortunately, instead of helping me understand your processes better, it left me with even more questions than before.

What criteria does Philanthropedia use to choose experts?

I am still interested in more information on how experts are chosen than you’ve provided. You state:

“When we decide to expand into a new social cause, our first step is to prepare by researching and mapping out the space. In doing so, we identify – to the best of our abilities – all relevant experts in a given field. We list all foundations, nonprofits, and academic institutions and then find the experts’ contact information in these organizations. In addition, in some social causes we include journalists, policy makers, researchers, and other types of experts. The end result is a list of hundreds of experts that we then invite to participate in our surveys.”

What specific criteria do you use in choosing these experts? Are these just US based people or do you seek out experts from other developed, developing, and under developed countries? Is there a certain number of years of experience they must have, and at what level? Do they have to have worked for or with that particular organization or done research on it, or can they evaluate it just having heard about it through the grapevine? How large do foundations have to be, can they qualify if they are a small family foundation? Do you extend the invitation to everyone from an organization, if not how do you pick which people to invite? What percentage of experts accept your request and which percentage decline to participate?

How does Philanthropedia weigh and evaluate expert opinions?

There is bound to be disagreement, as there are in so many aspects of aid, how do you handle conflicting opinions or information?

“What is more, not all experts are “equal” in terms of their expertise or experience. And some – most notably foundation professionals and academics – bring disproportionate value to our research. However, we strongly believe in our current inclusive model, which also invites nonprofit executives and senior staff to participate for several reasons.”

Do some expert opinions carry  “disproportionate value”, and if so how is that reflected in the final result?

What are your “monitoring and curating processes.”

“Of course, not every invited expert participates today. And of those who participate, we don’t necessarily include all responses from all experts because we do additional sorting of experts during the data collection process.”

What additional sorting do you do? What would cause you to disregard the opinion of an expert you invited to participate?  What are your “internal checks that makes manipulating the results difficult.” How do you “aggressively punish any offenders”?

What criteria are used to evaluate the charities?

Although you refer to changing your survey and methodology, you give no indication of what you currently use. When I looked for the answer to this on your website this is what I found.

“We believe that there is no one metric or measure of nonprofit effectiveness. Therefore, we take a holistic approach to answer this question. Rather than tell the experts which criteria they should be considering when thinking about strong nonprofits, we ask our experts to identify what three criteria they use to judge nonprofit effectiveness. By gathering this information, we are able to capture the views of a variety of professionals and provide unique insight into the best way to evaluate nonprofits in a particular social sector.”

Do I understand correctly that you have no set criteria? How many and which nonprofits have you evaluated and decided against funding, and why? Do you inform the nonprofits when they have been evaluated but not chosen, and is there an appeals process or the ability to provide proof to rebut negative opinions?

While looking for the expert evaluation on Alliance for Climate Protection I found that this organization was promoted as having similar strengths as Kiva.

Strengths

Outstanding public visibility.

“They have great media/PR that can help to change hearts and minds on the issue. They also have the cache of Al Gore, and a new investment in a field component.”
“They have good visibility because of linkage to Al Gore.”
“They have a strong television/media presence.”
“They have an excellent website and e-mail alerts on climate protection issues. They have engaging and clever internet “ads” and videos.”
“The ACP produces remarkably compelling videos. I don’t watch TV, but I know that they’re catching the attention on non-activists based on what I hear anecdotally.”
“They are high-profile.”

Just as Holden found Kiva’s marketing ability to be a questionable strength, I too question whether the ability “to capture the attention and the imagination of the individual” is any indication of the quality of the nonprofit. McDonalds owns more playgrounds in the US than any other private or public organization. While this is a great marketing tool, it doesn’t mean that their burgers are better than Wendy’s or that burgers and fries are the healthiest food to have for dinner. Is good PR one of the criteria commonly used to evaluate nonprofits?

Transparency

I am a strong advocate for transparency, both with charities and with charity rating systems. I am glad to hear that Philanthropedia is working towards more transparency. I urge you to be clear and honest with donors about Philanthropedia’s strengths, weaknesses, and areas for improvement. They have the right, and the need, to evaluate the information they are presented with before donating.

Set an example for the agencies you are evaluating. We cannot ask them to be more transparent if we ourselves are not.

Good luck with your work improving Philanthropedia’s systems. I look forward to seeing your progress and to having a clearer understanding of your evaluation system.

—–

*** Philanthropedia’s response to this post can be found on their own blog and has been crossposted on this blog ***

« New Research on Orphanages   •   Would you hire a doctor or a contractor whose work was never evaluated? »