What is the most effective way to measure the impact of ICT for development (ICT4D) initiatives?

Dear Shehzaad,

Good to see all these contributions and I just like to add couple of points from our working experience in e-Agriculture.

I think there is no short cut way to measure impact o ICT4D initiatives. It is always complex to measure the impact, specially the direct impact (in terms of income or productivity) of ICT usage in development.

There are couple of critical factors in this process, one is considering the social-cultural factors including behavioral pattern of the target groups while setting indicators for measurement of impact, and second is consideration of changes in the eco-system of development dynamics i.e. integration of ICT in the value chain.

What do others think?

Thanks,

Shahid 

I see two different categories to consider when trying to measure "impact" of ICT4D initiatives.

I. Impact of Project

The first deals with the initiatives impact on the community, nation, market, etc. that it's trying to serve. In this way, I think you can begin by measuring ICT4D initiatives in the same way that you measure any development project. The same debates, considerations, and challenges apply. 

Within this impact analysis category, ICT4D initiatives often have an unique advantage. If smartly built, a ICT system will routinely capture a lot of important data that will make an impact analysis faster and cheaper.

II. Impact of technology

Secondly, it's important to consider the role that technology is playing in an initiative. ICT is a means, not an end, so it is crucial we ask ourselves if ICT is the correct tool for the job.  Could the same information, service, resource, etc. be delivered in a less ICTish way? Does the information, service, resource, etc. already exist without ICT?

A big part of this is the cost. Does introducing ICT lower the cost? But that isn't the only consideration. Perhaps ICTs speed up the process or make M&E easier or increase transparency. All of these benefits are tangible and should be measured/estimated before a new tool (in this case ICTs) are fully adopted.

Answering to Michael's query, M4P is an acronym for Making markets work for the poor. To elaborate it is an approach whereby market growth or development is facilitated but which leads to pro-poor growth.

Dear Shehzad and Dear All,

Interesting to follow the forum. I see the reasoning behind defining some common indicators that can be aggregated, compared, benchmarked across different projects working in ICT4D initiatives; however don’t you think doing so might limit projects to only measure against a predetermined ‘wishlist’ of indicators? I think it is more important for ICT4D initiatives to first map out the logic of their work and then determine what indicators can be used to measure those different levels of changes. Indeed at a higher level, common indicators for market development project often include scale reached, additional jobs and additional income created but I think it’s very risky and pre-mature to have a standard set of indicators that fits all projects. Market development is also about thinking of innovative solutions to engage the private sector, so just as solutions are different, the indicators related to interventions can be different.

Just as an example, I recently did some work for a programme doing some ICT work in Cambodia. One of the projects is about setting a system to pay bills, transfer money using mobile phones. Now one of the many impacts of the project is that traders are using mobile phones to make instant payments to farmers, which saves them the hassle of often travelling long distances to get money. This time can be spent in doing alternate activities. However it would be very difficult to attribute indicators such as increased productivity, increased income as a result of this.

Would be interested in hearing others thoughts on this matter.

Nabanita.
 

Dear Shehzad & Dear All

Well, when I said a universal set of indicator I meant something like Poverty index or consumer price index or human index. CAPRI (Common Agricultural Policy Regionalised Impact) can be a good example.

Common set of indicators does not mean that measuring the impact should be confined only by those. There are many different types of ICT4D project and other projects using ICT as a tool for access to information, access to market and for similar facilitation of opportunities for the poor. Of course the scope, timeframe and many other factors differ among those projects. But there should be some common set of indicator irrespective of other factors to measure the impacts. Apart from those factors there should also be other indicator to measure the specific intended or unintended impacts of the project based on its objectives. The idea is to create a standard measure against which some of the impacts of all ICT/ICT4D projects can be measured. But then again the merit of the project should not be considered only by those indicators. In this regard I completely agree with Nabanita that there might be a risk of limiting the projects by the common set of indicators. 

And finally indicators can be qualitative also, whenever the word indicator is mentioned generally a set of numbers are considered by most of us. But qualitative indicators are also very important to see the change in the whole system. To tell the complete story the use of the qualitative indicators are no less important than the quantitative ones.

Thank you everyone, have a good day

Asad
 

Isn't it worth discussing about these indicators and trying to list out both quantitative and qualitative ones to be able to start thinking in developing an ICT index for agriculture. This will build the base the future databasis and comparative studies across states or nations. At this forum we can think and try to do this.

Surabhi 

The comment till now seem to underestimate end user comments and evaluation of impact./

In this context, I would like to draw your attention to a recent publication by UNCTAD on measuring the impact of ICT. You can find it at http://www.unctad.org/en/docs/dtlstict2011d1_en.pdf.

Kind regards, Torbjörn

".....efaminu wrote: � You raised deep and probing questions. There is need to track down how ICT is affecting production and productivity, in relation to farmers' behaviour in terms of adoption rates, and what factors facilitate adoption of a particular technology - cost, availability, accessibility etc. coupled with farmers' access to other information sources. For emerging economies these are very important statistics we need especially as penetration of ICT is low.

My comment:

The free, public domain e--book "ICT Adoption in Agriculture" provides a wealth of information - specifically on ICT Adoption  affecting production and productivity. The link to the book is http://departments.agri.huji.ac.il/economics/gelb-main.html
The authors of each of the chapters review their  long and comprehensive experience in ICT Adoption in their fields of competence with a focus on decision makers faced with ICT Adoption and development. Last year the book was accessed several hundred thousand times. 
Comments will be most welcome.
Ehud Gelb [email protected]

Probably a bit late in to the discussion but I was wondering if someone can share their experience (if any) with application of Randomize Control Trial (RCT) in evaluating ICT projects. Basically I was hoping to get answers to the following questions:

1. The time and cost involvement from inception to results.

2.  The sample size that was required (control +treatment group) and the unit of sample (individual, class, household etc)

3. The size of the project (in terms of their geographic scope and number of interventions, other than the one tested, that they were working on)

4. The challenges faced.

The reason for asking this is that from what I have heard and read, it seems RCT has been applied quite successfully in projects which have few interventions, with very precise crisp measurable goals (literacy, health indicators, reduction in absenteeism etc) and significant access to resource (time and money). I am bit apprehensive when it comes to applying such methodologies in  projects with 20-30+ interventions, working on light touch approach, private sector driven and focusing on more complex indicators like competitiveness, sustainability, systemic change etc.

Hello everybody.

I would like to add a different angle of defining impact of ICT4D initiatives.

Successful ICT initiatives in agriculture often requires the collaboration and participation of private sector companies. When it comes to the measurement of impact, the private sector entities are mostly focused on issues such as financial feasibility, market scenario, penetration statistics, etc. These information, although required, do not serve the prime purpose of development organizations. They require information more emphasizing on the social and pro-poor impact of the services offered. Thus we can see a gap between the information need of the private sector and the development agencies.

Let me refer back to the rural ICT initiative that my colleagues from Katalyst have already shared. One of our learning was that most of the Community Information Centres (CICs) – those we facilitated to be developed – were more or less commercially viable. Our private partner recognized the business case, and is continuing and expanding. However, in some cases we found CICs flourishing through revenues from mobile phone top-ups, photocopying, internet browsing, etc. (definitely positive impact on pro-poor social development). For these few CICs, provision of agro information remains a sporadic and somewhat philanthropic endeavour.

My query to all participants: how can a project specifically geared towards agricultural development tackle this issue?

Ehud Gelb
9 Hagalil St.
Rehovot, 76601, Israel
Tel: 972 8 9411677;
Mobile: 972 52 4459721
Fax: 972 8 9417882
skype: ehud.gelb