This paper builds on lessons learned from case studies of organization-building and collective action as a means of eradicating poverty in small-scale fisheries. The Voluntary Guidelines for Securing Sustainable Small-Scale Fisheries in the Context of Food Security and Poverty Eradication, endorsed by FAO Member States in 2014, recognize that addressing poverty depends in large measure upon the collective agency of small-scale fishers and fish workers themselves. We first discuss the nature of poverty in small-scale fisheries and argue that lack of rights and debilitating power relations are among the factors contributing to poverty. Secondly, the paper explores the possibilities of collective action and suggests that the support—but not the domination—of government and civil society is crucial. Finally, we look into the characteristics of fisher and fish worker organizations and emphasize the importance of autonomous decision making, and the need to address internal obstacles and opportunities, including those related to gender.