At the cutting edge>>Indicators of change: Measuring impact in voice and accountability


First published: 2009-10-09

Support is growing for initiatives that raise the voice of people in developing countries so that they are able to talk with governments and hold officials to account. Jeremy Holland and Allyson Thirkell report on a paper they wrote in response to a demand from a range of advisers for tools to effectively measure the results of the UK Department for International Development's work in voice and accountability.

Governments need to listen to the communities that they serve in order to understand issues, develop appropriate policies and deliver effective services. Knowing whether these initiatives are working is problematic as 'voice and accountability' work (V&A) is complex, the effects are often intangible and any change depends very much on where you are in the world. Whilst measuring change may be complicated, donors are determined to demonstrate the impact that they are having in all areas of their work. This renewed rigour in measuring change and demonstrating results has prompted this piece of work which was commissioned by DFID's Evaluation Department, in response to a demand from a range of advisers for tools to measure more effectively the results of DFID's work in V&A. The paper examines existing ways of measuring V&A, and proposes accessible and usable indicators to evaluate V&A. It will be published as a DFID Evaluation Working Paper soon.

"The work of governments alone will never be enough. For lasting change, states must interact with voluntary groups, charities, faith and diaspora groups, trade unions, cooperatives and others. These organisations can and do often deliver basic services where states cannot or will not ... and they can help citizens hold their states to account."
DFID White Paper, 2009

Based on DFID's Capability, Accountability and Responsiveness (CAR) framework, the report presents a practical approach to mapping V&A programmes. It separates out horizontal and vertical accountability and responsiveness components. Vertical accountability is the direct engagement that individuals and groups have with governments and other duty bearers, while the extent to which governments and duty bearers respond to their voices marks their degree of responsiveness. Horizontal accountability involves state institutions engaging in mutual scrutiny to be held accountable, for example, a public audit function which reviews probity in public spending. Equally the changes they adopt as a result of these measures signify their responsiveness to such accountability mechanisms.

The work presents a new tool through which it is possible to map out initiatives along these lines, identifying any gaps and designing programmes accordingly. This tool provides a good opportunity for donors to do joint planning around V&A work, overlaying their individual programmes, understanding their approaches jointly and the potential synergies between them. It also offers opportunities for multi-donor evaluation across a range of V&A programmes.

The report sets out a list of indicators that can be used to measure change in voice and accountability work. Evaluating V&A initiatives is notoriously difficult as the range of approaches, targets and methods is highly diverse and there is often a lack of understanding as to what is being achieved across the board. Reports often highlight individual successes in specific policy or geographical areas or even by single agencies. Using a set of commonly understood indicators across a range of activities and actors means that consistent data across programmes can be collected and the overall impact of such work can be measured and understood. This report also elaborates on accompanying data collection instruments and provides some guidance on methodology for collecting evidence of change.

Read full report: Measuring change and results in voice and accountability work