Accurate data is key to achieving FAO’s mandate and driving progress toward the SDGs. Yet, despite its critical importance, official data is not always readily available and may be difficult to obtain due to limited statistical capacities, as well as insufficient funding and dissemination processes.

This, unfortunately, results in significant data gaps, especially in emergency or crisis situations, which is when access to timely information is most crucial.

At the end of 2019, as part of its efforts to improve data collection, organization and comprehensibility, FAO created the Data Lab for Statistical Innovation to contribute to addressing these gaps. By employing data science and Artificial Intelligence (AI) techniques, the lab can efficiently analyse enormous amounts of data, documents, reports and websites—often too complex to process with traditional tools.

This innovative capability allows the lab to obtain valuable insights and structure them in ways that are both accessible and actionable for further analysis. 

Reflecting on the COVID-19 pandemic, which began just a few weeks after the lab’s creation, Carola Fabi (FAO Statistics Division), explained, "The Data Lab’s innovative methods in text and data analysis, especially using 'natural language processing', were very valuable during that time. They extracted information and insights from unstructured sources like social networks, news and other online resources. For example, the Data Lab developed procedures to scrape daily food price information for key items, providing crucial insights into the effects of value chain disruptions caused by the pandemic."

Pulling data from unconventional sources through web scraping, text analytics, data validation and statistical modelling (learn more about each of these here) plays a crucial role in modernising how we collect, analyse and use agricultural and food security data globally.

The Data Lab develops innovative methods and tools to deepen our understanding of global agrifood systems. In this, traditional agricultural statistics crossover with modern data science, and, more broadly, AI. Moreover, it builds and maintains different databases to produce real-time information, facilitate analysis and support evidence-based policymaking in a timely manner.

For example, the Food Loss and Waste Database is the most important resource for food loss and waste data in the world and has been created and maintained by the lab by gathering food loss and waste data points from scientific articles, technical reports, and other types of research documents not typically found in regular publications.

In tracking food security in remote regions, traditional surveys are often expensive, time-consuming, and limited in scope—especially in low-resource or crisis settings whereas the Data Lab’s methods deliver faster, cost-effective insights, especially when data is scarce. It also creates vulnerability maps that identify socio-economically vulnerable populations at subnational levels in countries where this data is unavailable or outdated.

"By developing tools to generate indices and natural indicators from text, the Data Lab helps highlight trends, patterns, and relationships that would otherwise remain hidden.” Carola Fabi continued, “These insights are intended to complement traditional data sources, providing additional perspectives to support well-informed decision-making processes.”

By structuring this available information and validating it with official sources, the Data Lab ensures the reliability and relevance of its outputs, especially in data-limited environments.

It also serves as a knowledge hub, fostering innovations that are actively shared within FAO to strengthen internal expertise and collaboration. These advancements can support organizations such as National Statistical Offices worldwide.

More recently, the Data Lab has developed enhanced tools to extract information, generate indices based on text, analyse the mood or tone of news articles on topics related to FAO’s mandate, and identify trends. The use of large language models and generative AI has only enhanced the lab’s abilities. It has streamlined the process of converting “messy” or disorganised data, into useful insights.

In the context of global challenges like climate change and food security, this is even more critical. The Data Lab’s innovations help track changes in agricultural patterns, monitor the impact of climate events on food production, and provide early warnings for potential food crises.

The Data Lab has also just started collaborating with Kenya and Viet Nam – and country collaboration is very new for the lab, as its collaborations have usually been with various FAO divisions. Currently in very early stages, the project connects satellite data with socio-economic variables and use of AI methods. This is set to continue through the new year.

In general, the struggle with limited resources for agricultural data collection prevents a truly global look at agrifood systems.

Christian Mongeau (FAO Statistics Division), explained further, “Looking ahead, the Data Lab aims to continue exploring innovative applications of AI and text analysis to support FAO’s work. This includes further improving its ability to process and analyse unstructured data, expanding its range of data sources, and developing more efficient tools for decision-making.”

One of the lab’s latest projects is the development of an autonomous AI research agent, which can take a large set of documents and generate a research report or article, accurately citing its sources. It can optionally browse the internet to obtain information. Christian continued, “the output it generates then serves as a first draft for analysts, significantly streamlining the research process and making it more efficient.”

National and international actors, especially FAO and other UN entities, need to engage with new sources of data, cutting-edge methods, and databases, and find innovative solutions to generate information that is relevant for food security, nutrition, and agrifood systems transformation.

Into the future, the Data Lab is expected to remain a practical resource for generating insights and addressing challenges in food security and related areas, adopting more new and innovative methods and expanding its reach throughout the world.