In 2011, as part of a broader effort to understand organisational effectiveness, a team within Oxfam GB began implementing impact evaluations on a small sample of projects largely focused on women's empowerment. The resulting ‘Women's Empowerment Effectiveness Reviews’ employ a quasi-experimental evaluation design, and have been undertaken in more than a dozen sites over three years. This article briefly presents the Women's Empowerment Effectiveness Reviews and their short history, then focuses on five key areas where the approach has presented limitations. Acknowledging that there remains room for improvement, the project has also been a major step forward for Oxfam's monitoring, evaluation, and learning practice. Through this critical reflection, we aim to share some learning to date.