Since its first appearance online in 1996, the FAO Web site has grown exponentially to more than half a million pages. The need to obtain data to measure the effectiveness of FAO’s Web publishing activities, into which considerable resources are invested, was the impetus to start user research. While some quantitative data on the numbers of visitors and pages viewed and visited is available, they raise as many questions as they answer. There is a need for qualitative analysis to examine how well FAO information is being disseminated and supporting sustainable agriculture. Some basic questions are leading the user research endeavour:
Is FAO reaching its target audiences?
Where and how do target users look for information?
What do they use the information for?
Is relevant information available to the appropriate people?
The main purpose of user research at FAO is to then find ways to make information more accessible to FAO’s target audiences on the basis of their information needs, online behaviour and goals, and the possibilities that disseminating information in a Web environment offers.
A pilot Web site user study that comprised three rounds of surveys and interviews with delegates at major meetings at FAO Headquarters was initiated in 2003. This study had the following purposes:
to open dialogue with users about the FAO Web site.
to obtain indicators about the types of information most important to users, how users look for information, and to identify some immediate solutions to improve the FAO site.
to develop a model and methodology for future user surveys
to develop a mechanism to evaluate FAO’s Web publishing activities from the point of view of Web site users
During this pilot study 85 questionnaires were received and 21 delegates were interviewed about their FAO Web site use, information seeking preferences, and priorities for the future development of the FAO Web site. Because direct dialogue with participants allows for more complex issues to be addressed, interviews yielded complementary results and discussion not possible in questionnaires. They provided good insights into how this target audience uses or would like to use the FAO Web site, and the information obtained from the interviews were consistent with the questionnaire results. Delegates who gather at FAO’s major meetings include Permanent Representatives to FAO, officers at National Ministries of Agriculture, and policy advisors and are a specialized audience of the FAO Web site. They were found to be a group of users with similar information needs and usage of the Internet and the FAO Web site.
At the end of this pilot study, four basic components of a user research strategy to pursue with a broader audience were laid out:
Develop profiles of information needs and preferences of major user groups
Conduct qualitative research (observability tests, focus groups) on major Web sites (both at early design stages and as evaluation methods).
Continue with surveys with identified groups of users on major FAO Web sites
Improve analysis of Web statistics (How can meaningful data be collected? How can log analysis provide insight into how users behave on the Web site?)
This strategy has been pursued through two major recent activities: an online survey of a wide range of members of FAO’s target audience and usability testing in a sample developing country.
Building on the survey and lessons learned from the pilot survey at Headquarters, a survey aimed at a wider audience was developed. A number of divisions within FAO with an interest in disseminating information participated in its design, and technical officers and divisions were consulted to obtain relevant e-mail lists to which an invitation to fill out the survey was sent. To date, all survey respondents have received the invitation by e-mail to fill the survey out online, or in a MS word document that can be returned by e-mail. While an invitation to fill out the survey will be posted on the FAO Web site at a later date, the approach of targeting members of FAO’s audience through various FAO staff members or FAO networks ensures relevancy in the response that a general posting on a Web site would not.
To date, the survey has received a very good response, with more than 2700 responses. Some of the preliminary findings include: the high use of statistical data (70% of respondents access FAO statistics), high use of browsing the site (75% of people access information through a thematic site or browse through the site as opposed to 66% using the search box on the home page), the high use of local libraries and colleagues to supplement FAO online information (more than 35% of respondents use local libraries and colleagues to obtain FAO information), and that more than 80% of people either agree or strongly agree with the statement “It is easy to locate information on the FAO Web site”. There is also a high demand for both e-mail updates on all new information as well as subject specific information (each option was chosen by more than 60% of respondents). A third of the participants would also like to receive information by CD-ROM and/or printed materials shipped to their offices (36% and 32%, respectively).
The user research strategy has also recently been pursued through usability testing in Ghana. Ghana is the location of the FAO Regional Office for Africa and was therefore selected as an appropriate sample country to begin conducting usability tests in developing countries. These tests provided the opportunity to get in-depth perspectives of real users in a sample developing country, and to understand how location of information on the FAO Web site takes place. They provided qualitative information on:
the user experience with the systems;
the level of expertise with computers, the Internet and database searching;
the effectiveness of thematic aggregation of FAO online information;
reasons for accessing and using FAO online information.
The tests were undertaken with 22 people from FAO’s target audience who use the FAO Web site: policy makers, statisticians, and extension workers at the Ministry of Food and Agriculture, programme coordinators at NGOs, staff at FAO’s regional office for Africa, information management professionals, and lecturers, researchers and students at universities and research centres. They consisted of asking the participants to demonstrate how they would locate an item they have previously looked for on the FAO Web site, as well as to locate specific items through major FAO search tools, the WAICENT portal and the FAO Country Profiles. Participants were requested to think aloud while performing the tasks to allow the FAO evaluators to understand what were their expectations and goals in searching for the information, as well as their experience with the interface and the content organization.
This approach yields findings not possible with surveys or log analysis. For example, the majority of the participants were very positive about the FAO site in the questionnaire they were asked to fill out, and usually said it was easy to locate information and that they saw no problems with the site. Yet, in demonstrating how they look for information, they stumbled on numerous difficulties that are apparent to people familiar with the systems and who know when something that is available is not being found. This approach is also an excellent public relations tool, as many participants became aware of FAO systems they said they would use again and gained a familiarity with the extensive amount of information available to them.
A few of the overall findings from the mission to Ghana were:
Based on the positive feedback on the information people often chanced upon in doing the tasks, as well as on learning about the major FAO information systems, it appears the FAO site is underutilised in comparison to the variety and quantity of information it offers.
There was a wide range of Internet and database searching skills. In many cases, the need for developing skills in searching the Internet and databases was either explicitly mentioned or observed. This is likely related to the length of experience with the technology and the medium
While reliable Internet access is not currently accessible to many of FAO’s target audience in Ghana (several tests had to be abandoned due to virus outbreaks or slow connections and several more had to be relocated), clearly the medium holds great promise for the future. Use of the Web and email is widespread, as witnessed by the booming Internet café business. Interviews with university professors and students also confirm a growing use of online resources and use of Internet cafés. Internet costs are still prohibitive to many people, but visits to Internet cafés for work and study reasons are far from unusual and pricing schemes are comparable to leisure expenditure. For example, in Accra, usage of a fast Internet connection in the city’s most popular and well-known Internet café is roughly the price of a beer.
Overall, observation-based usability testing revealed contextual factors that influence ease of access: experience with the medium, quality of Internet connection, and user understanding of the design and results of information systems. It complemented the quantitative data from log analysis and surveys by connecting data with the experience of the people the systems were designed for.
Both surveys and usability testing have contributed to increasing understanding FAO’s Web site users information preferences and needs. The survey in progress is contributing to the development of profiles of information needs and preferences of major user groups as well as measuring the satisfaction with FAO search tools. A methodology for conducting further qualitative research through observability tests has also been developed. Future work will include continuing with this approach as well as developing a more sophisticated log analysis.
