Forum: "Challenges and Opportunities for Capturing Impact in ICT initiatives in Agriculture" September, 2011
Introduction and Question 1
25/09/2011
Pressure is growing on all development agencies to report on their achievements in a credible way - yet progress on the ground is actually slow.
Communicating credible results in ICT4D projects is as important as achieving them!
One part of the solution, suggests Jim Tanburn, is that implementers can be much more clear about the logic of their work. In other words, if they could clarify exactly what they expect that the sequence of events will be (A leads to B, which then hopefully leads to C, and so to D) - then it would be much more feasible to check whether events are indeed unfolding in the expected way.
Logframes are fine as a summary of this logic, but strip out much of the detail that a programme manager needs; it is difficult to show the sequencing of outcomes, for example, or which things are likely to happen in parallel, in the logframe format.
Everyone has an idea in their head about the logic of their programme, but getting it down onto paper in sufficient detail is a particular skill that many of us need to develop. The Donor Committee for Enterprise Development (DCED) is working with many programmes that are developing that skill. You will find some case studies from DCED in the Resource section of this forum.
Once the logic or results chain is sufficiently clear, a monitoring system can be used to check whether it is valid, and also to estimate the results being achieved. The alternative, according to Jim, is that someone will come in from outside, and measure things that might not be relevant; even a clever evaluator or researcher needs to understand well the logic of a programme - yet it is often not made explicit.
In ICT4D there are additional challenges: its diffused nature; involvement of partners; attribution; scalability. This all make the field more interesting but at the same more demanding in how we go about measuring and capturing impact.
Let’s explore this line of thinking together now.
Dear participants,
With great pleasure and excitement, we can announce that our online forum on 'Challenges and Opportunities for Capturing Impact in ICT initiatives in Agriculture' is now open. I will be facilitating our conversation with the Subject Matter Experts and the other forum participants from all around the world.
So without further ado, I would like to ask you - In your experience,
What is the most effective way to measure the impact of ICT for development (ICT4D) initiatives?
Do you explicitly construct logical and linear paths for your programmes?
Do you carry out regular monitoring during projects, or do you prefer ex post facto studies?
Share your thoughts with us and discuss these critical issues with the Subject Matter Experts this week. Thank you for agreeing to take part and looking forward to a fruitful conversation.
Hello everyone! Just thought of sharing our/Katalyst experience of dealing with impact measurement for our ICT initiatives.
As discussed by Jim Tanburn, We are also using results chain for monitoring and capturing the impact for our Tele Center and Mobile Telephony based initiatives. And these kind of results chain also enable us to keep atrack of the out come of our activities on a continuous basis. We are not just doing a post assessment after the implementation work is done. Rather it is utilized as "True" monitoring tool for assessing the progress and impact of our activities. It also allow us to bring in changes at activity level based on the feedback we receive at different levels of the market system and end beneficiery.
In addition to that, I must say using impact chain for monitoring ICT4D initiatives is even more convenient and usefull becasue of the different "system generated" data for the usage of different ICT based solutions or services.
For an example: Our Agri help line project (Jigyasha 7676) with one of the largest Telecom operator in Bangladesh is generating good amount of usage data from the CRM that is being used in the call centre. It does not require us any extra cost for monitoring the impact at usage level. It also enabled us to design effective promotional campaigns, addition of new content and etc just by using these kind of system generated data.
Now, I would also like to know how other projects are dealing with this issue of impact measurement. Please share your experiences and thoughts!
Cheers!
Hello Arafat,
Welcome to the forum and thanks for sharing your experience. If I could ask you a few questions so that our participants could get some more insights on you have just shared -
1. When you say 'Rather it is utilized as "True" monitoring tool for assessing the progress and impact of our activities', how exactly do you perceive this as 'true' and what could be its relative strengths?
2. Do using 'impact chains' and 'system generated' data allow you to change any of your plans during the course of the interventions, say at regular intervals or you prefer not to be so flexible all throughout and prefer to wait unless the available data is sufficient to make an assessment?
Thanks for your time and hope to know more from you!
Among the several things that I have learned while implemeting interventions with ICT in agriculture, is that result measurment is something that starts off at the very get-go. Some have tendency to start thinking of capturing results only after interventions are proving to be successful. This outlook I believe make result measurement a very reactive tool. On the other hand when you have the clarity about results measurement from the very begining, in terms of what key factors are we trying to change, how would it change, how would we capture it, etc it will not only help you design better intevenions but also allow the result measurement processing to be a tool for proving and improving your work; hence a pro-active tool
Also when it comes to ICT initiatives, having such clarities beforehand, helps you, to an extent find the right areas of partnership and allows you to draw a much clearer picture of the overlapping areas of interest in capturing results between your organization and your partner organizations, if any.
Hi Shehzaad,
Thanks for further stimulating this discussion. Please find my response as below:
From the remarks of different forum participants of this on line conference, it is apparent that most of the development projects are following an impact measurement methodology that enables them to report the impact of a program or its activities designed beforehand. And following that kind of methodology does not allow the program team to reevaluate and bring in changes in program activities. In contrary to that, our program uses impact chain for each of the activities/intervention following logical consequences of the activities. Thus that enables us to tract whether the planned activities are resulting in intended outcome. If the planned activities are giving the program with an anticipated results well and good. But if the planned activities are not yielding anticipated results, then we redesign the planned activities based on the feedback from the intended beneficiary and/or market system. Thus we thrive for more result oriented activities for our intended beneficiaries following a non-linear process.
I believe flexibility is a key feature of our methodology and we also believe there is no alternate to this. We use impact chains for keeping a track on the changes facilitated in market system/dynamics and if things are not happening as expected, we take the liberty to bring in changes in the activity plan. And usually these kind of decisions are made based on system generated data. Where we primarily look at the impact scenario at the beneficiary level from the system generated data. Here, it is necessary to mention that for ICT interventions system generated data is quite readily available. For an example, our Agri help line project with Banglalink, we get monthly reports of service usage/service usage growth/service user retention and etc. And all these reports are generated from the CRM that is being used in the call center for the agri help line.
However, we are keen to improve our MRM system and looking forward learn about other good practices in monitoring and evaluation of ICT projects.
Cheers!
Dear All,
I would like to see contributions from the private sector on these issues which may give us a different dimension of this dicussion.
In general, when we (as a private initaitive) frame and develop the business plan as a commercial venture, it is must to see the RoI (Return on Investment) is most impotant factor. And this return basically measured from financial perspective where consideration goes to Opportunity cost, Net present value or time value of money. Here indicators are mostly in terms of revenue, time line and customer satisfaction.
All entities carry out regular monitoring regardless private or development organization. It has been observed that post facto studies are more often used in development initiatives.
Thanks,
Shahid
Agricultural information access is a right to farmers and ICT is a swift way to do that. But the ICT should use in correct way. The quality of information and quality of media should be well organised. Here the media should be closed with research.
Daer Bulbul,
Very interesting when you say that 'agricultural information access is a right to farmers' - Certianly worth thinking.
However I was wondering if you could elaborate a bit more when you say 'media should be closed with research'? Thanks and will wait to hear from you.
Dear Shehzaad,
Thanks for your asking. I get much pleasure having your quest. However Here I cited media word representing those organizations who are dealing with information dispersion. The time is changing. Attitude towards agricultural information, excepting ability of farmers, thinking powers all are changing as well as climatic environment. The theme that was highly accepted before, may be next not. The innovations are timely updating by research. But I think their is a unknown gap between the research organizations and extension. I am not claiming anybody or any system.
ok, then thanking you again and wishing next discussion.
cheers to all
Bulbul
Dear Emefe,
I think what you mentioned in a simple manner is worth a lot of value - that 'access to information should be simplified'. I am not a subject matter expert in this case, but from my experience of dealing with software development and digital projects, I can positively confirm that 'user experience' and 'user journeys' are at the heart of modern day software development so that these can be programmed keeping the end user in mind and not the programmers preferences or the company's objectives.
I wonder to what extent impact assessment frameworks in use are being prepared keeping donor requirements and implementing agencies' objectives in mind - whereas assessing impacts from say a farmer's point of view may be very simple and away from a lot of jargon and anomalies.
It would be interesting to see what our Subject Matter Experts might have to say regarding your comment. Thanks.
Hi,
We need to be clear about the objective or purpose of delivering agriculture/ extension information to the farmers. The programmes or models through which this information is delivered, it is important for them to run their system efficiently and thus through a regular internal process they keep a close watch on if the information is being delived and how to improve the delivery. On the other hand it is also important to see if the information that farmers receive has any impact on their socio-economic conditions. Anectodal evidences from various earlier studies in India and South Asia (including mine ICRIER WP 246) has proven this fact. But the big question remains if actually a critical mass of farming community expecially the small farmers surviving on subsistence agriculture - mainly cereal based economy able to benefit from this information? How have they used this information? What mechanisms exist to see or to evaluste a continuous and rigorously acceptable (statistically) impact analaysis is something which I am also trying to find answer to. As an economist I have the options to do experimental trials or do an ae-ante and ex-post analysis. But then we work with a small sample of population. Is is possible to link the service provders with the researchers and through good data mangement infact using ICT itself, build up data to abe able to see the real impact in terms of income change, reduction in cost or yield improvements, reduced use of harmful chemicals, or even social changes. Important is also to observe and track the behaviour changes?
Cheers
Surabhi Mittal
Hi Surabhi
The challenges that you mention are very similar to what we face in our implementation. We are working in partnership with a mobile phone operator and helped established 500 telecenters across the country. These centers are self sustaining and provide a host of services include agriculture information. We hve done baseline and post intervention impact assessment and have found that among others better crop disease management (leading to "loss saved/not incurred') is the primary means of how this ICT initiatives impacts on the poor. However the challenge that present itself, is farrmers tend to share information with another and also that there is multiple information sources. Hence how does one attribute the extent of impact to an interventions. For us, we are trying different approaches, among others; first being to acknowledge all significant and relevant information source, second probing beyond our ICT sources to see whether other factors where also responsble and thirdly doing control and treatment groups. (although considering our scale, the third option sometimes is quite difficult to undertake)
Dear Surabhi,
You advised to observe and track behaviour changes in relation to tracking impact of ICT in agriculture. Curious to ask you if there exists any set group of performance indicators which are 'must have', should be tracked as a priority, hard indicators such as income etc.? Also may be another group of 'soft indicators' such as 'behaviour', 'lifestyle' etc.? Do you follow any such classification or prioritisation in your ICT impact assessment?
Thanks!
Shehzaad,
The indicators according to me as neccasary to track the change is
1. What % of messages do farmers act on?
2. With the new information received do qunatity/ cost of inputs used change?
3. What si the observed change in yields, income?
4. Do farmers change their behaviour over time by adopting to new technology or even trying to use new technology (seed technology, non- seed technology etc.)
I am trying to develop methodology on these lines and run some experimental trials, hopefully will be able to start some of these soon for India.
Surabhi
I am posting this on behalf of the undersigned who is not able to post his message here and has thus requested me to do so.
The recent 8th EFITA Conference in Prague http://www.efita2011.cz/index.php?idScript=29
dedicated a Keynote to the Impact of Technological Innovations. In several sessions various
aspects of information-providing mobile systems were presented and discussed. An aspect
of particular interest included the impact Social Network information-exchanging (Facebook?)
have on Agricultural Productivity. This facet specifically under the title:
Collaborative platforms and Social Media for Agriculture.
The format was a query:
Are Social Networks A Technological Upgrade or a Paradigm Shift?
This questions and findings in this Indian study add universal clarity to the issues as
discussed in Prague - issues such as:
- What kind of information do farmers value the most to improve agricultural productivity?
- Do mobile phones and mobile-enabled agricultural services have an impact on agriculture?
- What are the factors that impede realization of the full productivity enhancing potential of mobile phones?.
- What are the interrelations between mobile phones (social networks), agricultural productivity and social factors?
best
Ehud Gelb,
Center for Agricultural Economic Research, Israel
Dear Ehud,
We have just replied to your email. Please try to log in and post now.
We also noticed that you are an e-Agriculture member since 2007! that's great! It is very encouraging to see that members who joined our Community back in 2007 are still active in e-Agriculture forums and activities!
We look forward to hear more from you and future discussions,
Kind regards,
Lisa
e-Agriculture Team
You raised deep and probing questions. There is need to track down how ICT is affecting production and productivity, in relation to farmers' behaviour in terms of adoption rates, and what factors facilitate adoption of a particular technology - cost, availability, accessibility etc. coupled with farmers' access to other information sources. For emerging economies these are very important statistics we need especially as penetration of ICT is low.
Dear All,
It seems that the forum is warming up very well with great contributions from Surabhi, Shahroz, Arafat, Bulbul and John so far. Now building on Jim's suggestion as above, let me pose the following questions to the forum for further thoughts and opinions -
What is the most effective way to measure the impact of ICT for development (ICT4D) initiatives?
Please let us know what you think is the most effective way and why you think so.
Using logframes is one model.But i think relying on a logframe to develop an evaluation system is dangerous as in most cases logframes are designed at an early stage of planning of the project and also the logframes are rarely updated during course of project implementation and so it may not necessarily reflect real situations at the time of evaluation.
Developing a good Monitoring and evaluation systems to track progress, performance and to generate good outcomes following and impact assessment is good way to go.Also as part of Monitoring and evaluation, selecting indicators to measure progress and determine outcomes must be critically done.I work with an ICT based Organisation in Uganda called Women of Uganda Network (WOUGNET) and we implement ICT4Dev initiatives and would really be interested in understudying what OUR external evaluators will use to really help us comprehensively measure the impact of our ICT4Dev initiative in the near future !
Moses
I agree with Jim that logframes can be a very useful tool for evaluating the impact of ICT for agriculture programs. While ICT is innovative, and offers new opportunities for outreach and engagement with program participants, at its heart, it is like a lot of other programs.
But there are some ICT-specific challenges that are necessary to consider while thinking through the program. These might seem evident to all of you -- if so, please forgive me for repeating them -- but these are lessons we learned with our IMAC program (a mobile phone-based price information program) in Niger:
1. Sometimes the ICT for ag projects involve developing new partners with the private sector. How can we bring these partners into our planning process and support?
2. Does the program manager have the necessary technical support within the organization (or outside of it) to ensure that the ICT interventions run smoothly?
3. If we observe that farmers are "doing better" after the ICT for agriculture program, does this mean that we should replace the "traditional" way of doing things with ICT? Or are they complements? In other words, what would have happened in the absence of the program? (This is the attribution issue).
We faced many of these challenges in our IMAC program in Niger. We learned that figuring out these partnerships and ensuring that the necessary technical support is available is something that must be decided early on, during the project planning stage. More is available here: http://sites.tufts.edu/projectabc
Hi Jenny,
I agree to all the three points that you have highlighted. To answer to the third point- ICT (mobile / internet based indformation delivery systems) are complementary to the all other existing extension networks. Traditional extension chanels help in strengthening the existence of modern ICT. We have observed this during our study as well as in present day field works in various low income states of India.
Cheers
Surabhi
Dear Jenny,
Drawing your attention to Moses' opinion that
'Using logframes is one model.But i think relying on a logframe to develop an evaluation system is dangerous as in most cases logframes are designed at an early stage of planning of the project and also the logframes are rarely updated during course of project implementation and so it may not necessarily reflect real situations at the time of evaluation'
What are your views on this?
Dear colleagues,
The discussions are quite interesting indeed. The issue is about how best we can provide evidence of impact of ICT4D to those who are sponsoring us. We mention the dreaded 'log frame' because that has been the traditionally accepted tool for planning linear interventions, people have been trained to use it and draw conclusions from it. However, it assumes that there is a logic flow or some sort of linearity in the activities carried out and the outcomes, with some degree of flexibility to allow for unexpected outcomes of the activities, which you could always explain with the use of the assumptions made and the risks mentioned upfront.
The ways in which we have adapted to the log frame approach has been to try to straighten what in general is a complex, interrelated set of factors that lead to a change in attitude, behaviour which may lead to socio-economic impact. For example, we have tried to target specific communities for which we have baseline data and then compared them during and after to see what changes have happened. By narrowing ourselves to a segment of the complex, we have been able to present a 'linear' and logical approach to our interventions and are able to satisfy the measurement criteria, to a certain extent. Unfortunately, we may also be missing the spill-over effects of our intervention by just looking through the log-frame lens.
So are we stuck with the log frame - or rather, are the sponsors and funding agencies stuck with the use of this tool to justify their investments? - the aim of evaluation from a 'sponsor' is that they are provided with evidence that their investment has done some 'good'. If it were a person you were explaining it to, perhaps you could address their feelings and make them 'feel good' that they have invested in a project. Unfortunately, we are also restrained by having to report in writing and it tends to be difficult to create the sensation of 'feeling good' through text, unless there are figures that express a sense of magnitude, which then is able to make us 'feel good'. So, part of the answer to me would be to explore new ways of reporting that can be acceptable to the 'sponsors' that represent the impact of a project. Thus, we find nowadays that there is a lot of focus on getting 'stories' from the target communities, short videos, or even field visits and 'immersions' in the target community etc. that can be used both to report on progress and at the same time 'touch' the feelings of the sponsor.
So, as innovators in this area of work, what are the means of reporting that we can include to complement what the 'log frame' indicators cannot capture? Is there also some advocacy to be made at the level of the 'sponsors' to enable them to quantify the 'feel good factor' in monetary terms?
Krishan Bheenick
Food and Agricultural Research Council, Mauritius
Daer Krishan,
I really liked your views on the need to capture issues that logframes can't capture. I wonder if you have seen views from Jim Tanburn in this relation?
Would love to hear your thoughts.
Shehzaad,
I could not agree more with Jim's views on the differences that exist between conceptualisation and implementation of the log frame. Definitely it seems that some donors are stuck by regulation to have a logframe in project proposals. How many of us have been through this cycle where a project proposal is being prepared: a consultant (No1) is recruited to develop a proposal as per the formats and requirements of the donor/sponsor. A lot of documentation on what should be done as part of the project is provided to the consultant by the stakeholders, the final output is a project proposal that summarises these proposed activities into clusters that are now linked to objectives and result areas. The ideas are present in the text in less details and the actual action (minus the spirit in which the action should be made) is represented in the logframe. If the stakeholders or beneficiaries are not well organised in formulating their views, they are generally happy that a document has been prepared by an expert which increases their chances of being funded.
Some time later (in some cases a year or more later) the project is launched. Consultant No2 is recruited to implement the project. Consultant No 1 is not able to be involved in implementation, so his/her views may not be considered. The new people involved in implementing may simply just start off with the project proposal and elaborate an activity plan based on that - unless, again the stakeholders are clear about their original proposals of what should be done as part of the project and bring these to the attention of the consultant. Consultant No2 may also adopt the attitude that they are here to implement the project so as to satisfy the indicators (again quantified, but with no indication of the spirit in which these have to be implemented), no more no less. This scenario is even more important when we are dealing with information and knowledge management because there are a lot of soft skills involved - hence my stress on the 'spirit of implementation' of an ICT4D project being as important as the number of publications printed and distributed for example.
I also support the idea of unpacking the process into steps (results chain exercise) so as to document each step of the process and how well it has worked. I will come back to this under the discussion on 'Do you construct logical and linear paths
Krishan