Sampling strategies for gendered impact evaluations

Alexia Pretari Gender, Real Geek

How can evaluators ensure that gendered power dimensions are accounted for in impact evaluations? Alexia Pretari reflects on the relative merits of two approaches.

Oxfam’s approach to building resilience involves bringing about changes ‘in the very structures that cause and maintain poverty and injustice’ (transformative capacity), and building active citizenship involves addressing power imbalances, including building power within. While working on impact evaluations of programs about both these topics, I asked myself, how can I ensure that our impact evaluations are not blind to the different power dimensions, which underlie social inequalities?

Gender is one dimension of power which is relevant and common to these evaluations, and is at play at different ‘scales’ or levels, including within the household. Exploring the gendered impacts of Oxfam projects means going beyond looking to see if ‘female headed households’ and ‘male headed households’ have equal impact gains from our intervention. It means designing evaluations that enable us to consider intra-household dynamics in particular, and power dimensions and their intersections more broadly.

Developing an appropriate sampling strategy, is key to enable analysis on outcomes at household and individual levels, and for different social groups in different positions of power
This is easier said than done, and something as geeky as developing an appropriate sampling strategy, is key to enable analysis on outcomes at household and individual levels, and for different social groups in different positions of power. Others in the sector have developed approaches (see links below for more info) which we have built on in our recent evaluations.

Here is a summary of my experience of designing impact evaluations that attempt to systematically take gender into account in our sampling strategy. Two approaches were trialled: the first approach aimed at creating a balanced sample of women and men respondents across households, and the second one aimed at surveying both women and men within the household.

Randomly picking the gender of the respondent

In our impact evaluations in rural Burkina Faso (a resilience building project focused on food security and nutrition), and rural Tanzania (building active citizenship and improving leaders’ accountability), as well as assessing the overall project’s impact we wanted to test if individual level information differed for men and women, and whether the programme had differential impacts on men and women. To do that, in addition to collecting household level data, we gathered data at an individual level. This individual data included participation in community groups, access to information, opinions, perspectives and experiences on a specific topic.

Our team regularly works in contexts where a comprehensive roster of individual household members is not available, nor is it feasible to conduct the full listing prior to data collection. Given these constraints, the ideal situation from a statistical point of view would have been to randomly pick the gender of the respondent from potential respondents on arrival (e.g., any adult household member, or any household decision-makers), which digital technology makes easily possible. If chance is the only factor determining who is surveyed within the household this would ensure men and women respondents’ samples are representative and comparable.

However, this sampling approach would make it challenging from a logistical point of view to match the gender of the interviewer and interviewee. Something we do because we recognise the role of gender power dynamics in the interviewee – interviewer relationship, particularly when respondents may be prompted to share personal or emotionally hard stories.

To overcome this problem, we randomly allocated the gender of the enumerator in charge of each household, which then determined the gender of the respondent. In order to do this, we shaped the survey protocol and logistics (e.g., number of enumerators who identified as men and women in each team; flexibility to redeploy enumerators during data collection if the household was composed of no adult of the randomly assigned gender).

However, there are two limitations to highlight with this approach. Firstly, if there are systematic biases in the quality of the household level information depending on the respondent’s characteristics (e.g., role in the household and/or gender in our setting), due to different access to information or other biases, this approach will affect the household level analysis. For example, if men tend to underestimate household food expenditures, because they are traditionally not in charge of food purchase, or because of other biases, compared to women respondents, this would affect the validity of the comparison between households. Secondly, if there are systematic differences in the quality of the surveys depending on the gender of enumerators, this might affect the quality of the analysis done when comparing men and women’s information.

Surveying two respondents per household

In two effectiveness reviews (of resilience projects in rural India – disaster risk reduction in flood prone river basin – and rural Ghana – building climate-resilient agricultural and food systems), we wanted to change our measurement framework in order to understand our three resilience capacities at both household and individual levels. To do this we needed to gather information on intra-household decision making processes, access to information and control over resources, in addition to household level ones. We recorded the opinions and livelihood conditions of men and women separately, as main partner decision-makers (spouses in most cases), as we were particularly interested in the dynamics between them, which the sampling protocol focused on. This also enabled us to systematically look at the differential impacts on these women and men.

Inspired by the Women’s Empowerment in Agriculture Index’s approach, we surveyed the main partner decision-makers separately to generate individual level information, and then surveyed them together, as much as possible, for the household survey. Here, survey protocol and logistics were arranged to enable each household to be visited simultaneously by a pair of enumerators, a man and a woman.

Three main limitations or challenges need to be highlighted here. First, having two enumerators conducting multiple interviews as opposed to one enumerator conducting a single interview, increases the time and costs of the process.

Second, from a monitoring perspective, good tracking is needed within the team of enumerators, to avoid mistakes in the identifiers, and ultimately to ensure adequate matching of the individual and household surveys.

Third, surveying two respondents per household is no easy task since all respondents are rarely present at home at the time of the enumerators’ visits, due for example to the agricultural season or short-term migration, which are not gender-neutral. Additional monitoring and coordination is needed as the survey protocol requires greater flexibility compared with survey approaches with a unique respondent.

What next?  

These evaluations focused on exploring dynamics between partners mainly, and differential impacts on these men and women. e should remember that gender dynamics will certainly vary depending on the position within the household, among other dimensions: a daughter-in-law and her mother-in-law living in the same household may experience a different situation. Following similar sampling strategies, different criteria for identifying the respondent(s) would have been needed to explore this difference. In fact, a similar approach has been used in the Household Care Survey (HCS), and the 2017 HCS also included a module administered to young household members.

Sampling strategies in large scale surveys are key to enable representation of different voices and living conditions, shaped by different dimensions of power and their intersection.

If you made it to here, you probably are thinking, ‘This is a lot of data’. The data will be made public ultimately, but yes, there is a lot to work through for a rather small team! Impact analyses are on their way and a report will be published soon.

Have you got any experience of working with enumerator and gender biases? Are you a fellow feminist social scientist interested to know more about our work and this data? Please get in touch:

Read more


Katie Whitehouse