PC
87/5  |
| PC
87/5 |
Eighty-seventh Session |
Rome, 6 - 10 May 2002 |
Modernization of FAOSTAT - An Update |
1. At its 86th session, 17-21 September 2001, the Programme Committee discussed the proposal to proceed with a full requirement analysis for the modernisation of the FAOSTAT working system. It was noted that the FAOSTAT working system supports FAO's substantive work of compiling the statistical data disseminated through WAICENT on food and agriculture, including fisheries and forestry. Given the importance of this work, it was agreed that the modernization of the FAOSTAT working system be accorded high priority.
2. The Committee gave its endorsement to proceed with the modernization of the FAOSTAT working system and to seek alternatives for its funding. The Committee emphasized that the conceptual design for the new system should take into account the need to improve data quality, both at the country level as well as within the Organization. It agreed to recommend to the Conference that funding for the FAOSTAT working system be included in the list of proposals for the use of arrears. To this end the 31st Session of the Conference approved the use of arrears for IT infrastructure support and related technical work of the Organization.
3. The Committee requested an update on progress made in identifying the scope, approach, and cost of a new FAOSTAT system. This information document presents the analysis undertaken to define the scope of new system, its objectives and how these would be met, and the implementation approach.
4. The definition of requirements for a new FAOSTAT system commenced in mid-September 2001. Interviews were conducted with representatives from all FAO organizational units that are currently providers of data to FAOSTAT as well as those who compile data that potentially could be included in a new FAOSTAT system. Principal internal users of FAOSTAT data were also contacted for their views on needed improvements to the system. A small representative sample of external users of the system was also consulted. The output of this exercise was a draft project proposal for the modernisation of FAOSTAT.
5. An Inter-Departmental User Group, under the sponsorship of the Assistant Director-General, Economic and Social Department, was assembled to review the draft project proposal and to discuss and resolve any outstanding issues arising from the requirement analysis. A presentation of the draft project proposal was given to the User Group in mid-December 2001. Two subsequent meetings of the User Group were held in January and February 2002 to reach consensus on the scope of the project and to finalize the project proposal. It was agreed that the new system should be referred to as FAOSTAT2.
6. The anticipated end result of the FAOSTAT2 initiative is a new system that meets FAO Strategic Framework Objective E1 of "an integrated information resource base with current, relevant and reliable statistics, information and knowledge made accessible to all FAO clients." To achieve that goal, FAOSTAT2 must specifically address the following objectives:
7. The foundation of the FAOSTAT2 working system would be a new technical architecture based on state-of-the-art industry standards. Major components of the system would be Internet-based, thereby eliminating the users' reliance on their local computers and greatly expanding system accessibility.
8. The proposed project approach recommends the reuse of components that have been created for similar projects already underway within FAO as a "quick start" method for building critical system functionality. Initial project outputs would address the more pressing technical problems and major user requirements. Major improvements in system stability and reliability should be realized early in the project.
9. Several mechanisms for ensuring data quality (in terms of accuracy and completeness) are envisioned for the FAOSTAT2 system. All data entered into the system would be submitted to a series of rigorous editing and consistency checks. System editing would be performed in real-time to identify data errors early in the process and increase data entry efficiency.
10. Where a country has not provided complete times series data, the system would provide appropriate tools for estimating missing values. These tools would include interpolation and extrapolation algorithms as well as statistical models developed using analytical software packages.
11. The development and publication of an XML-based standard for the exchange of food and agricultural statistics would also facilitate improvements in the quality of FAOSTAT2 data. This standard would be distributed to member countries and other external partners who would be requested to provide their data in files formatted according to the published standard. Data provided in this fashion would be loaded, edited and validated, thus saving FAO staff time and improving data quality.
12. FAOSTAT data quality ultimately depends on the quality of the data reported by member countries. This could be facilitated if countries entered and processed data online, which is currently possible, to a limited extent, through the Web-based Virtual Questionnaire. While this method would continue to be an option under the new FAOSTAT2 system, another option would also be made available: CountryStat, a scaled-down version of the FAOSTAT2 application, which would provide countries with functionality to compile, validate, analyze and disseminate their national data. Outputs from CountryStat could then be easily loaded into FAOSTAT2 for further dissemination through FAO's web site, publications and CD Roms.
13. While countries would be encouraged to implement CountryStat, it is understood that all existing methods for data capture and transmittal - paper questionnaires, Virtual Questionnaire, electronic files - would continue to exist under the new FAOSTAT2 system.
14. A number of new user requirements would be incorporated into the new FAOSTAT2. These include improved or new system functionality, as well as expanded data requirements.
15. Improved or New System Functionality: New functionality would focus primarily on tools to facilitate data capture and improve data quality: data exchange standards; flexible, yet comprehensive, data editing routines; data estimation support and tools to be used at country-level. Software would be developed to enable users to easily extract and analyze FAOSTAT2 data.
16. Expanded Data Requirements: the requirement analysis revealed a need for FAOSTAT2 to accommodate diverse types of data, which go beyond the standard cross section and annual time series currently maintained within the system. Many data sets currently maintained outside of the FAOSTAT system could be incorporated into an extended FAOSTAT data model. To accommodate these requirements, the FAOSTAT2 system should be capable of supporting the following:
17. The original proposal for the modernisation of FAOSTAT focused solely on the working system, i.e., the underlying system used to compile, validate and analyze statistical data. The working system clearly has the most pressing problems and is in need of replacement. However, the analysis of system requirements revealed the need to also address the redevelopment of the dissemination system.
18. The dissemination system is the visible portion of the FAOSTAT, the component that most external users are familiar with. It is through the dissemination system that data can be accessed for further use. Enhancing the functionality of the dissemination system is seen as a means for strengthening the capacity of internal and external users of statistical data to perform more substantive analytical work.
19. Several opportunities exist to enhance and extend the functionality of the dissemination system. An improved interface, with the capability to visualize FAOSTAT2 data through graphs and maps, would enable users to highlight trends, anomalies and areas of concern. Key country indicators would also provide a useful vehicle to assess a situation at a glance.
20. Data dissemination mechanisms would be enhanced to support the automatic generation of high quality system outputs. New alternatives for publishing FAOSTAT2 data, such as dynamic online reporting, would be introduced. Dissemination monitoring tools would be expanded to enable FAO to analyze patterns of system usage and to better tailor its services to meet the needs of its users.
21. As the scope of the FAOSTAT system expands to include data from more diverse sources and to address the needs of more organizational units within FAO, the need to expand and reconfirm common data concepts, definitions, codes and methodologies becomes increasingly apparent. The FAOSTAT2 project offers an excellent opportunity to revisit existing methodologies and to identify areas for improvement, simplification and standardization. The requirement analysis recognized the need to address data management and statistical methodology issues. This includes identifying the appropriate organizational mechanisms to support ongoing data management and statistical methodology review. A working group has been established to define the terms of reference for this work, which should be undertaken prior to the commencement of the FAOSTAT2 project.
22. FAOSTAT2 would be implemented using a phased approach. The project would be divided into three sub-projects (working system, dissemination system and integrated corporate database). Work on these sub-projects would proceed relatively independently of each other.
23. Initial outputs of the project would focus on improvements to the current system and resolution of the most pressing problems. New system functionality would be capable of running on the current FAOSTAT system, thus enabling users to gain early experience with individual system components, before the complete, integrated system is delivered.
24. The FAOSTAT2 project would attempt to incorporate or reuse software components that have already been developed within the Organization. Rapid application development and prototyping would be employed, so that users can react to the proposed system and quickly provide feedback to developers.
25. The FAOSTAT2 development effort would be user-driven, involving key FAO staff at every level of the project. Data producers and data users would participate in the project from its inception, working as part of the project team to design and test the new system.
26. This approach to the development of FAOSTAT2 should provide numerous benefits:
27. It is too early to provide reliable estimates of cost, which will depend upon several decisions concerning the scope of the project and the approach to be taken. However, it is noted that the order of magnitude is likely to be such that the current provision of resources under the Resolution for the Use of Arrears (6/2001) for IT infrastructure to support technical and other work of the Organization would enable the project to commence and for substantial progress to be made. In the event that the arrears are not paid during the current biennium, either extra-budgetary contributions would need to be sought, or a reallocation of resources made within the existing approved budgets. In the latter case, it is expected that there would be considerable difficulty in finding the full amount and hence full implementation would probably be delayed.
28. This document is provided to the Committee for information as requested. The status of the proposal outlined above is that it will shortly be submitted to the Information Management and Technology Committee which will recommend a course of action to the Director-General. The Committee will be kept informed of progress in this regard.