Guideline

The evaluation of the use of a Web site is an important and often overlooked step. It helps to identify areas of the Web site to adapt to the intended users.

A number of methods of evaluation can be used - they can highlight different aspects of the user experience. Deciding which to use may be determined by the resources available for both evaluation and the subsequent re-design.

Log analysis

Server and application log analysis can be used to track numbers of views and visits. The number of visits from referring Web sites can also give an indication of how visitors are finding your Web site, as well as their area of interest.

Useful for:

• demonstrating the growth or the increasing reach of a Web site;
• basic knowledge of visitors (geography, browsers used);
• identifying major referrers to your site; and
• indicating the parts of your site that are the most popular.

Considerations:

• growth in Internet use in general should be considered when drawing conclusions about the growth of the Web site;
• provides no indication of the usability of a site, nor directions for improvement;
• Some data can be unreliable, notably geographical distribution of users due to IP identification, caching and registration issues; and
• search crawlers inflate numbers and should be separated from human activity; internal FAO users (requests from PC's with FAO IP addresses) should also be excluded.

ClickStream analysis

ClickStream analysis involves analysing visits more thoroughly to identify common paths through a Web site.

Useful for:

• identifying the labels users click on the most;
• identifying how users arrive at pages within a Web site; and
• identifying links users are not using.

Considerations

• ClickStream analysis software has been tested at FAO but was not adopted as it was unable to handle the size of FAO log files. It may be useful for individual sites; and
• additional logging may have an impact on site performance.

Search log analysis

Analysis of words used by users in the search box of a site can provide a more in depth traffic analysis.

Useful for:

• identifying the main areas of interest of users of the site;
• identifying areas to provide optimized searches for popular items (i.e. [eu OR "european union"] AND [trade OR trading] AND [policy OR policies]); and
• analysing exclusively human activity (crawlers do not type).

Considerations:

Grouping search terms requires a manual/human review, otherwise sophisticated stemming algorithms may be required to account for:

• plurals or other grammatical structures;
• mistyped search terms; and
• different languages used.

To draw useful conclusions there needs to be a relatively large number of search expressions gathered over months rather than weeks.

Surveys

Surveys can complement log analysis in obtaining a user perspective of a Web site. They obtain the most responses when they are short. The most useful information is collected when questions are clear and specific.

Useful for:

• asking about interest in services and preferences (i.e. e-mail updates);
• rating the satisfaction and priorities of users; and
• identifying themes for improvement (need for updates, thematic access, etc).

Considerations:

• effort needs to be made to contact relevant users and have them fill out the survey. Care needs to be taken over representativeness - ad hoc survey response rates will differ from those where a group of users is actively sought, or where there is some reward offered for participation. In addition, online surveys tend to be completed by those people who have time to fill in online surveys (who may not be the target group);
• when surveys have been conducted during conferences and meetings, it has been found that print versions are popular (people at meetings may not have easy access to the Internet);
• feedback and results can be ambiguous and hence difficult to interpret into the best way to improve the Web site (this can be addressed by improved survey design); and
• users are often unaware of many aspects of the site and are not able to comment succinctly on many potentially important issues. Usability testing or interviews are a better way to address functionality and complex issues.

Software that has been used at FAO to conduct and analyse online surveys includes:

• SurveyMonkey: it is inexpensive and easy to use. It cannot handle Chinese and Arabic. The URL for the survey is at http://www.surveymonkey.com/. Interpreting data in English, French and Spanish can be time-consuming as it is managed separately;
• SurveyWorld.Net: this userfriendly online tool is inexpensive. Its main advantages are that it can host multilingual surveys (in three of FAO's languages) and that users do not have to complete the survey in one session.

Usability testing

Usability testing is a method by which representative users of a site are asked to perform certain tasks in an effort to measure the site's ease-of-use, task time and the user's perception of the experience. Usability testing can be done formally, in a usability laboratory with video cameras, or informally, with paper prototypes of an application or Web site.

After the test, the success rates of the tasks are measured based on the data collected. Changes are made to the application or Web site based on the success rates of the usability tests. Whether the test is formal or informal, usability test participants are encouraged to think aloud and voice their opinions. The overall idea of usability testing is to replicate the real world situation where a potential user would use the Web site.

Testing with a small group of users (even as few as five) can be enough to obtain useful results and concrete actions needed for improvement. Obviously, the more representative the users are of the people a Web site is intended for, the more valid are the results.

Useful for:

• seeing what people actually do (as opposed to what they report);
• seeing problems users would not think to report;
• identifying specific problems in usability and functionality;
• deciding on usability and labelling issues where the Web site staff differ in opinion; and
• gaining more in depth knowledge of Web site use (i.e. can ask participants what the information is used for, how it compares with other similar sites, etc.).

Considerations:

• often difficult to recruit typical users from FAO headquarters; and
• can be difficult to conduct tests in an environment that resembles real life, while allowing participants to think out loud.

Link popularity

Google and other tools on the Web can give an indication of how many other sites are linking to a Web site.

Useful for:

• identifying need for promotion; and
• benchmarking with other sites.

Considerations:

Data is not always reliable (it does not count every link).