Low inherent fertility of tropical soils and degradation, nutrient deficiency and water stress are the key factors that hamper rainfed agriculture in semi-arid West Africa. Conservation Agriculture (CA) is currently promoted in the region as a technology to reduce soil degradation, mitigate the effect of droughts and increase crop productivity while reducing production costs. The most prominent aspect of CA for degraded lands in the semi-arid tropics would be the organic soil cover that impacts on the soil water balance, biological activity, soil organic matter build-up and fertility replenishment. Yet, the organic resources are the most limiting factor in Sahelian agroecosystems due to low biomass productivity and the multiple uses of crop residues, chiefly to feed the livestock. Hence, CA as such may hardly succeed in the current Sahelian context unless alternative sources of biomass are identified. Alternatively, we propose: (1) to gradually rehabilitate the biomass production function of the soil through increased nutrient input and traditional water harvesting measures that have been promoted as “soil and water conservation” technologies in the Sahel in order to restore soil hydrological properties as prerequisite to boosting biomass production; (2) to encourage during this restorative phase the regeneration of native evergreen multipurpose woody shrubs traditionally and deliberately associated to crops and managed the year around and; (3) to shift to classical, less labour intensive CA practices once appropriate levels of soil fertility and water capture are enough to allow increased agroecosystem primary productivity. The CA systems we propose for the Sahelian context are based on intercropping cereal crops and native shrubs building on traditional technologies practiced by local farmers. Traditionally, native shrubs are allowed to grow in croplands during the dry season; they reduce wind erosion, trap organic residues and capture the Harmattan dust, influence the soil hydraulics and favour soil biological activity under their canopies. This paper offers a state of the art around native shrubs and their integration in relay intercropping CA systems, discusses challenges and the main research needs to address them.