Rapid demographic ageing substantially affects socioeconomic development^1,2,3,4 and presents considerable challenges for food security and agricultural sustainability^5,6,7,8, which have so far not been well understood. Here, by using data from more than 15,000 rural households with crops but no livestock across China, we show that rural population ageing reduced farm size by 4% through transferring cropland ownership and land abandonment (approximately 4 million hectares) in 2019, taking the population age structure in 1990 as a benchmark. These changes led to a reduction of agricultural inputs, including chemical fertilizers, manure and machinery, which decreased agricultural output and labour productivity by 5% and 4%, respectively, further lowering farmers’ income by 15%. Meanwhile, fertilizer loss increased by 3%, resulting in higher pollutant emissions to the environment. In new farming models, such as cooperative farming, farms tend to be larger and operated by younger farmers, who have a higher average education level, hence improving agricultural management. By encouraging the transition to new farming models, the negative consequences of ageing can be reversed. Agricultural input, farm size and farmer’s income would grow by approximately 14%, 20% and 26%, respectively, and fertilizer loss would reduce by 4% in 2100 compared with that in 2020. This suggests that management of rural ageing will contribute to a comprehensive transformation of smallholder farming to sustainable agriculture in China.