Week 2 - Day 1, question 1 - 24 November 2008

There is an old adage which says seeing is believing.The impact of ICT can be felt by the user and here by small and marginal farmers.It can be demonstrated by capturing the preintervention scenario and comparing that with the post intervention scenario.There are so many factors which needs to be taken in to account for the success of the project.If it is not a lopsided project then the impact on the livelihood can be demonstrated to others.For example in eKutir project all the risks that a farmer faces have been factored so that a farmer uses this facility by paying a service charge.It has not only enhanced the productivity of the crops but also price realisation of his product.

Hi everybody there, We can take up some pilot interevention in certain agriculture clusters and outcome of the intervention can be compared with that of control cluster with similar characteristics with that of pilot testing area. The outcome in terms of economic benefits and other social impact can be publicised. Prior to the intervention priliminary survey has to be done in the pilot testing area to come up with a status in terms of economic and social issues and select few parameters like attitude, practice and knowledge etc. Impact of the intervention then be easily monitored during or at completion of the intervention. We are looking for taking up such activities at our end in Northeast India. I believe it may interest the forum. Dr. Monjul Islam Secretary Genreral FARMER

An important consideration in evaluating interventions is to consider the way that data is collected an interpreted. We are not just asking 'are mobiles beneficial?' - or at least I don't think that should be our question.Each project has a different model of what the mobiles are doing, who is operating the mobile, how people & organisations are working together, who is providing information and responses, how services are paid for, etc. At one level, we need independent evaluations of individual projects to inform funding decisions so that the "best" projects can be supported, extended & developed. But that work needs to be conducted by researchers who are able to understand the subtle technical, social & organisational differences between projects and between contexts. One thing we may want to do is to exert influence on network service providers to provide preferential rates (cf. the GrameenPhone model). At another level, there is the broader policy argument about getting funding for this kind of work. In this case, we may be competing with other types of agricultural intervention (e.g. price support, government agri extension programmes, input subsidies), or interventions in other areas (e.g. building schools or training health workers). I really don't know what we can do in that wider competition.

Hi from Hastings UK. Having a quantitative evidence base for monitoring and evaluating the impact of investments in mobiles in rural areas is important, but until activities are beyond pilot scale this is kind of data is probably going to be inconclusive. At this early stage to get enthusiastic support from leaders in all the roles you mention I am inclined to think that anecdotes are the best bet. This brings to mind a quote attributed to Tom Davenport by Patti Anklam in her book "Net Work". Talking about the measurement that kept early communities of practice sustained in the eyes of corporate bosses (before they could prove their return on investment quantitatively) Tom recommends the approach of "serious annecdote management". By serious I take this to mean rigourous, planned and tactical collection of stories of changes in real people's livelihoods. If these stories are woven in to rich and compelling narratives that will influence leaders to make the investments (social, financial, political) to get beyond pilots and begin activities at scale, then is the time to start collecting the baselines, setting indicators etc (ideally using a blended qualitative / quantitative approach like Outcome Mapping or Most Significant Change becuase of the complexity of livelihoods outcomes that you'll be seeking to understand).

I agree with Carl regarding the difficulties of doing impact evaluations for small/pilot projects. But unfortunately I am afraid anecdotes are not enough. Donors or Governments want data and prove that investments are somehow effective and productive. But then question is: what are good indicators for the evaluation of the impact of ICTs in rural areas? Then, which impact - economical, social - we want to consider?

I just posted under resources a [draft] Inventory of Innovative Farmer Advisory Services in Africa: http://www.fara-africa.org/media/uploads/File/Announcements/Innovative%… What I did not mention in the report is the number of innovative and pilot projects which do not know one another (or would even ignore one another!). But still, some initiatives on rural telephony in the agricultural sector in Africa are a real eye opener. A number of coordinators of services mentioned in the inventory reacted with enthusiasm about projects they had never heard about before!

Hi all, from my perspective the impact assessment at local level should always start from the livelihood perspective (i.e. considering the social, economical and ecological dimension). Some weeks ago I read a paper which could interest you all, entitled "Can the ubiquitous power of mobile phones be used to improve health outcomes in developing countries? - There is mixed evidence to support the idea that mobile phones are an effective healthcare intervention in developing countries", which was written by W. Kaplan and published by "Globalization and Health" in 2006. Here you can find the link to the paper: [url]http://www.globalizationandhealth.com/content/2/1/9[/url]. Cheers, Simone

First of all I wonder whether it is all useful to focus on the mobile phone itself to measure impact? Isn't the mobile phone just an mean for another end? Wasn't that the mistake done by older ICT4D projects to expect technology will solve itself something? But let me add some other perspective. How about mobile phones themselves can make a difference to measure impact. One example is a cooperation between Voices of Africa, Mobile Reporters (http://voicesofafrica.africanews.com/site/page/mobile_reporters) and Akvo.org, an open source initiative for water and sanitation. On Akvo all kind of projects are presented and mobile reporters now report from the field about the implementation and the impact. Mobile phones and collective action or crowdsourcing have a great potential for transparency. There is a project called Ushahidi (http://legacy.ushahidi.com/), which made it possible by all challenges, that people in Kenya during the post-election conflict faced, to report through their mobile phone about the critical situation. This way they collected information from all over Kenya and documented incidents such as riots, deaths, property loss, looting, rape etc. This degree of transparency was hardly achieved by the media and certainly not intended to be publicised by the government. Now imagine the potential to measure development projects from a grassroot level. Or to collect information about how many governmental services have arrived in villages. This could be possible by harnessing the wisdom of crowd. * Using mobile phones to collect information. * Present all information on a website with maps and databases. * Use the website to connect the people who send information and aim to get more accurate information. Beneficiaries of projects could collect information in teams, send feedbacks to the platform and create their own map of development projects or their timeline with accurate information on how government services are fulfilling their duties. This kind of transparency should be an all-win-situation.

It obviously is subjective , but what we are experimenting is pricing these services. Even if the farmer is taking first time, more due to curiosity , next time he will not subscribe to it, if the service offered is not valuable. this is a simple measure tape that we are using in our pilot work in India.

Michael Riggs' question: [color]How can we monitor the impact [of rural mobile telephony in the agricultural sector] [/color]requires a better understanding of the farmers' context for the adoption and adaptation of an innovative information tool. There are many initiatives on ICTs and small-scale farmers in Africa. However, these tend to be un-coordinated, and information on the different initiatives is not easily accessible, let alone information on their impacts (see FARA inventory). The nature or mobile technology development for farmers itself is highly contentious and requires careful research and development to make it "right' especially when it comes to livelihoods improvement and poverty reduction in Sub-Sahara Africa. Although it is recognized that the uptake of promising information technologies like rural telephony can be influenced greatly by the availability and/or functioning of input supply, credit systems, land-tenure arrangements, organization of marketing, distribution of benefits, etc., such social-organizational phenomena have mostly be considered as conditions that hamper or enhance adaptation of rural telephony. When trying to measure the impact of rural telephony the question is thus not just to seek to develop an appropriate information dissemination technology but also to alter the boundaries and conditions that affect the space for change. Resource poor farmers in high risk and diverse, rain-fed environments face very small windows of opportunity for innovation. The mobile phone projects as listed in the FARA inventory often create special conditions to enable and stimulate farmers to utilize the recommended technologies (f.i. Questions and Answer Services - QAS based on text messages). Such special conditions might include access to subsidized inputs, guaranteed marketing of the surplus generated, the creation of special credit schemes, the availability of highly qualified staff or in the case of QAS accurate and timely information. But, equally invariably, such projects turn out to leave few traces after the special conditions have been withdrawn. Replicability of the development gains is a key issue. The best guarantee for such replicability is to ensure that new communication technologies work within the prevailing physical, socio-economic, cultural and institutional conditions and, if necessary, to stretch those conditions. This approach requires special procedures to adopt a new communication tool on the basis of decision making that is informed by an understanding of the farmers' context. [img]http://www.mobileactive08.org/files/images/STA40008_0.full%20view.JPG [/img]A good example is the adoption of mobile phone conferencing. In a video interview I took with Mary Nyakira of BROSDI/CELAC Uganda during the MobileActive 2008 conference, she explains how the mobile phone conferencing works and how farmers are enjoying it. It not only contributes to a particular form of democracy and transparency but farmers like having group discussions around a mobile phone with the loudspeaker facility on. The extension worker is "beeped" when the group of farmers is ready to start the training session. The discussions are a follow up on a previous field visit. But this time the extension worker gives advice out of his/her office. Taking into account the considerable distances and the fact that extension workers can not afford visiting on a weekly basis a particular group of farmers, mobile phone conferencing is having a tremendous impact. [url]http://farastaff.blogspot.com/2008/11/mobile-phone-conferencing-among-f…] Rural telephony technologies can only work if they fit within the small windows of opportunities that African small-scale family farmers face. As pointed rightly by Ravi Shankar - even if the farmer is taking first time a specific service, more due to curiosity, next time he will not subscribe to it, if the service offered is not valuable. Contrary to political activism in Christian Kreutz' example, the majority of farmers is not that well organized, they have no political clout and cannot exert effective demand on agricultural information services. Despite the number of Market Information Prices Services using mobile phone for price information dissemination the market prices information remain often not freely available so that prices are set locally and rather arbitrarily given the actual relative scarcity. More often than not, market prices are determined by the vagaries of weather, transport, monopolistic traders, and so forth. [color]Conclusion.[/color] In order to answer the question: How can we monitor the impact? We need to look into the broader innovation opportunities of farmers. To monitor the impact of the tool we need to look into: the most effective ways of reaching farmers with timely agricultural information and knowledge (indigenous and external); mechanisms for harnessing the potential of FM radio stations and digital telephony as technologies for communicating agricultural information; options for repackaging agricultural information and knowledge for small scale farmers; the potential role of and an e-repository (of local agricultural content) in Africa for purposes of disseminating local agricultural content. [img][/img]

Carl, I also agree with your position. It may seem very strange or fantastic or vague, but in my experience there is nothing like the power of GOOD EXAMPLES. :razz: I want to say that things can be said or written or emailed or shouted in all the possible ways and still not produce any result. Vice versa, showing a good example to someone is, most of the time attractive, persuasive and self-explaining. It really helps in getting in people's mind and move them to change consolidated habits to start adopting new ones. So I'm convinced that a good collection of serious, tangible, quantitative cases can be an effective tool to convince leaders to adopt new innovative solutions.

I really like the user-evaluator model you describe Christian. I think the kind of content you describe delivered through maps etc would present some pretty powerfull narratives of change. Subsidising the cost to the user of their mobiles to capture the stories could be build into the M&E budget (e.g. airtime credits for posts)

I believe the impact of mobile telephones on economic development can to some extent be assesed. We have conducted a study on the use of mobile telephones by micro and small enterprises in less urban and rural areas of Ghana. From focus group discussions, and questionnaire administration the operators were able to state what they perceived as impact on their businesses. We recognised the dangers about the credibility of the financial data the respondents gave us but some insights which are interesting were gained.

Hi, Christian's suggestion of using mobile phones for evaluation is such a great one! I think tools like [url=http://www.frontlinesms.com/]FrontlineSMS[/url] or eventually [url=http://www.openrosa.org/index.php/javarosa]JavaROSA[/url] will make this fairly easy for projects of all size to engage in. I am wondering whether any of you have turned to organizations specializing in doing assessment such as [url=http://poverty-action.org/]IPA[/url]? Thanks, Héléne Martin [url=www.applab.org]www.applab.org[/url]

Dear Michael, A few observations regarding assessing impact on rural livelihoods as a result of mobile technology. 1. Pretty much like Maslow's hierarchy of needs, it is important to identify the need value chain with regards to mobile phones. For example, as the very term ICT indicates, Communication precedes the need for Information, reliable, credible, relevant communication needs to take place first between more than two or more parties so that information can be exchanged. The challenge lies how you translate a communication service through mobile phones into an impact on a rural farmer's or SME's business health. As not all communication is business, livelihood related and there is no way to track each and everyone's communication behaviour or purpose. 2. Mobile phones come with inherent features of character limits, smaller screens, graphics and interface issues, which implies that NOT all type of information can be delivered through mobile phones. Information that is more 'volatile' in nature i.e. market prices of agri-produce, weather information etc. perhaps can be transferred through SMS, WAP etc. However, for other types of information, internet (a PC screen with a wider screen) still remains a better medium to disseminate need based information for farmers and SMEs. As a result, I believe it is challenging and too early at this stage to be completely able to identify the impact on rural livelihoods as a result of mobile phones only. However perhaps a combination of related technologies i.e. mobile phones and PC based technologies still could be a better approach to assess initially the impact. 3. The mobile companies need to get on board to assess this impact. Although its clear that they might be more interested to keep track of data usage of their technologies and the associated revenue it might generate, it will not be entirely possible for outsiders i.e. NGOs to assess the impact unilaterally. 4. Finally, when we talk about impact, we can certainly categorize it into social, personal, commercial, community-wise etc. and also the impact could be gauged not only at the beneficiary level but also at the service provider level i.e. to assess if the mobile operator's rural penetration has increased or not and other key indicators.

Héléne, You are right, using mobile phones for evaluation is promising. An example:  [u]Mobiles max Mexican coffee industry:[/u] Digital ICS mobile phone software for collecting survey data for agricultural collectives is helping coffee growers in Mexico. Yael Schwartzman tells about the package and process of collecting and analysing data from coffee farmers to improve their produce. Posted on YouTube [url]http://www.youtube.com/watch?v=vKB8HQB0w7s[/url]

Hi, When you combine the easy usability of the mobile phone with the resourceful creativity of rural people, you often come up with great ideas you never thought of. I think the best example of this is the way people use airtime as a virtual currency, mentioned earlier. Another example is "beeping" (or "flashing"), where you dial a number, let it ring, then hang up; this communicates to the other person that you want them to call you back. My point is that if you want to look at the impact of a pilot project, for example, you may have to look well beyond the project itself to see all of the impacts. I'm sure others of you have got some good stories. I think this is where some of the qualitative / quantitative methods that Carl talks about can be useful. Nigel Gamos

[quote=Christian Kreutz]First of all I wonder whether it is all useful to focus on the mobile phone itself to measure impact? Isn't the mobile phone just an mean for another end? Wasn't that the mistake done by older ICT4D projects to expect technology will solve itself something? [/quote] I think Christian is getting at an important point here. I am not convinced the 'mobile phone' is the best unit of analysis for measuring the impact of interventions. The projects represented in this discussion are doing very different things with their mobile phones. The projects are also intervening in different types of local organisations and social structures. A better analysis might be conducted by looking at the impact of different 'services', or 'service interventions'. Evaluation needs to be asking questions like: What is the value of investing in mobile phone based advice services to individual farmers? What is the value of investing in mobile-based advice services within farmers co-ops? What is the value of investing in mobile-based news networking between NGOs / CBOs? What is the value of investing in mobile-based weather information services? ... Of course, there may be some macroeconomic evaluations that show how government investments in infrastructure are related to development outcomes, but these will just be broad correlations.

Shehzaad, Picking up your comment: "Perhaps a combination of related technologies i.e. mobile phones and PC based technologies still could be a better approach to assess initially the impact" is another promising link between RADIO and MOBILE. Mobile's answer to radio is f.i. Freedomfone. Freedomfone (still in Beta version) gives users access to dial-up information and services over their mobile. Dubbed 'dial-up radio', the service will be invaluable in societies where many people own cellphones but draconian governments have restricted access to newspapers and the airwaves. The project is run by Kubatana out of Zimbabwe with funding from US based Knight Foundation. See interview with Brenda Burrell at the MobileActive 2008 conference: posted on YouTube: [url]http://www.youtube.com/watch?v=vKB8HQB0w7s[/url] Unfortunately, when this project Dial-up Radio: Agricultural Information on Demand was presented at the September 24-26 World Bank Development Marketplace it did not get selected! See: [url]http://farastaff.blogspot.com/2008/09/sustainable-agriculture-for-devel…]

For macro impact check out this publication by the Worldbank from June 2008: THE ROLE OF MOBILE PHONES IN SUSTAINABLE RURAL POVERTY REDUCTION on page 9 http://siteresources.worldbank.org/EXTINFORMATIONANDCOMMUNICATIONANDTEC… [quote]Impact of Mobile Telephony Before we can make policy recommendations concerning the role of mobile phones in sustainable rural poverty reduction, let us consider some of the benefits (or impact) that results from the provision of affordable access to mobile telephony. Although, as we have demonstrated, mobile telecommunications is a substantial driver of economic growth, there are very few indepth studies which have been carried out to document the impact of the mobile phones on economic development and on sustainable poverty reduction. This section makes an attempt to bring together a few of the existing impact evaluation studies and draw lessons from these for sustained development.[/quote]