DFID Madhya Pradesh Impact Evaluation of the Safe Cities Initiative
SDDirect led the three-year DFID-funded Impact evaluation of the Madhya Pradesh Safe Cities Initiative (2012-2016) together with Columbia University and local research partner New Concept Information Systems. The Safe Cities Initiative aimed to tackle violence against women and girls (VAWG) in urban slums in four cities of Madhya Pradesh (Bhopal, Indore, Jabalpur, and Gwalior).
Unlike most other VAWG initiatives, the programme aimed to tackle both intimate partner violence as well as violence experienced in public spaces through targeted interventions with women’s self-help groups (SHGs) and work with men and boys. This was a complex evaluation into highly sensitive issues and as such, it needed a highly specialist team of VAWG and M&E experts.
Our team developed an evaluation design with two integrated components:
- A randomised control trial (RCT) in 250 slums
- A broader analysis looking at the programme’s relevance, cost-effectiveness and sustainability and effects at the city and state level.
The data collection strategy was based on a mixed-method approach. This included a quantitative survey with over 7,000 direct and indirect beneficiaries across treatment and control slums, direct observation techniques, participatory focus group discussions, key informant interviews, and analysis of programme monitoring data. As part of the evaluation, the SDDirect team developed and implemented robust ethical procedures to ensure careful and appropriate data collection and handling on this highly sensitive topic.
In August 2016 the team finalised the endline report. The quantitative component of this evaluation found little evidence the interventions or combinations of them had positive impacts on the hypothesised outcomes, including reductions in intimate partner violence, violence and harassment in public spaces and associated social norms and attitudes. The qualitative analysis highlights some important perceived benefits of self-help group membership including improved social networks and confidence. The Life Skills module may also have been effective in encouraging men and boys to challenge unequal gender roles in the household. However, the qualitative data also highlights persistent harmful norms and attitudes, which continue to drive and sustain VAWG in urban slums in Madhya Pradesh. The report offers important recommendations and practical insights for future VAWG programmes and evaluations, including:
- Robustness of VAWG measures - as part of this evaluation, we assessed the reliability of commonly used VAWG prevalence measures by both comparing consistency over time and by comparing direct measures with indirect measures. Both analyses highlighted weaknesses in the standard sensitive measures commonly used in evaluations of VAWG programmes.
- Achieving change in VAWG - programmes with long-term investment, sustained resources and realistic timeframes are likely to be more effective and to have both greater and more sustained impact. Although there is no well-established minimum dosage for VAWG prevention interventions, our findings support emerging learning from another of SDDirect’s projects, the DFID What Works to Prevent VAWG Programme, that intensity of delivery is key to achieving results. In particular, this involves ensuring sufficient programme duration (typically three+ years for social norms programmes), enough regular hours of delivery, effective mobilisation and maintaining high regular attendance.
- Strengthening local VAWG response - prevention programmes should engage in strengthening VAWG response as a key strategy to ensure sustained social change and adhere to no harm principles. It is important to ensure that those who are mediating are provided with appropriate training and support to ensure adequate response to reports by mediators and referrals to appropriate bodies.
- Importance of monitoring data to measure implementation fidelity (the extent to which the intervention was delivered as intended) - evaluations can benefit from more rigorous quantitative monitoring data to be included in quantitative analyses.
Photo credit: Marji Lang