by Mahran Alhayek, Melissa Harris and Sarah Nijholt
Credits: Abbie Chilton
As the World Health Organisation declared COVID as a pandemic on March 11, 2020, a group of staff from Oxfam, its partner organization and consultants were preparing to conduct interviews with women affected by ongoing conflicts and the presence of ISIS in Kirkuk Governorate, in Iraq, as part of Oxfam’s Effectiveness Review series.
We put the work on hold and reconsidered the whole approach, asking questions such as: How can we conduct these interviews in a safe manner? Can technology enhance safety and privacy? If so, How? What are the social realities in which the interviews will take place, that the technology is blind to? What are the risks associated to these technological blind spots and how can these be mitigated?
Research, monitoring and evaluation activities are often the work of many people. Today we hear from Mahran Alhayek, Sarah Nijholt and Melissa Harris, as they reflect on this experience. Sarah and Mahran led the data gathering and translation processes for this impact evaluation, Melissa set up the Computer Assisted Telephone Interview (CATI) platform that was used for data collection. The interviews were conducted by Farah Abdulrazzaq Salih and Kayghan Muhamed Saeed Taher.
First, how did you find yourselves involved in this evaluation and why was it decided to conduct phone interviews?
Sarah: I saw the terms of reference and my interest spiked when I realized this was an evaluation of a livelihood project working only with women. In Iraq, social norms can prevent women, especially in positions of vulnerability, from engaging in economic activity. I wanted to apply for this evaluation in order to further my knowledge on the barriers that women in Kirkuk face when engaged in livelihood activities.
Mahran: Initially, the evaluation was designed to conduct interviews in-person, which is usually how we collect data. However, when the COVID-19 pandemic hit Iraq and public health measures prevented us from collecting data in-person for a number of months. During these months, we focused on alternative ways to collect data: phone and online means. While there are drawbacks to these alternative ways, in the context it was not possible to safely collect the data in-person.
Doing phone interviews is not new, although it was new to Oxfam impact evaluations on that scale. Which technology did we use and why?
Melissa: This is correct! Phone interviews are not new for Oxfam and have frequently been used in contexts where face to face data collection is challenging, perhaps because of access or security issues. However, we’ve never fully digitalised the process and have instead relied on combination of manual and digital processes to collect the data.
We were however fortunate that around the time of the COVID-19 outbreak, SurveyCTO launched their Computer-assisted telephone interview (CATI) Starter Pack. SurveyCTO is a mobile data collection platform and is one of Oxfam’s ‘approved’ digital tools, meaning they meet a range of criteria, particularly around data protection and privacy. This functionality essentially centralises and digitalises the entire process of a phone interview.
Across the Oxfam confederation, SurveyCTO is currently used in around 25 country programmes for a range of data collection so becoming familiar with this gave us a really exciting opportunity to improve our phone interview practices at scale, using a platform that our teams were already familiar with.
What are the features that this technology allows when it comes to data privacy?
Sarah: The integrated case management system allows for secure and restricted access to personal information of interviewees. It also made it automatic to reschedule calls, which makes it easier for researchers to adapt to interviewees’ availability. It also provides real time information on the calls being scheduled, made or rescheduled. For Mahran and me, this was very helpful…
Also, the quality control features within SurveyCTO are quite advanced. For example, you can measure the time the researcher took to answer one or a set of questions. This, opposed to checking the total time of an interview, can help to determine which parts of a questionnaire are not working well or not well understood.
Data privacy is key and switching to phone interviews also raised questions related to the act of carrying out an interview. Particularly in a context where women are to be interviewed and their access to a phone may be restricted. What are the main differences between phone and face-to-face interviews when it comes to privacy and associated risks?
Mahran: The main difference is that we have much less control over the setting in which respondents find themselves. When doing in-person interviews, we can ensure the interview takes place at a location that is quiet and where no other people are listening or interfering with the interview. While we can take certain measures to ensure that the respondent is in a comfortable place and alone when conducting the interview by phone, we can never be 100% certain this is indeed the case. Also, even when a respondent is alone at the start of the interview, there is a chance that someone could walk in during the interview. This could place the respondent in a difficult situation in which they no longer can speak freely.
Sarah: I fully agree with Mahran. For this study in particular, there were a number of sensitive questions which revolved around the position of women within their household and community. For women to honestly answer such questions, it is critical that they are alone and know that their answers will be treated anonymously. More importantly, if others hear the answers of the respondents, it may place them in a risky situation, which needs to be prevented at all cost.
How did you try and mitigate these risks?
Mahran: We developed a protocol to ensure that the researchers would be able to speak with the respondents during a time that was comfortable for them and when they were alone and able to speak freely. The first time that the researchers would reach out to the respondents, they would not conduct the interview. Instead, they would ask which day and time would be most suitable and clarify that the respondent should be alone and able to speak freely during the interview. On the planned date for the interview, the researchers would call again 15 minutes before the planned interview to check with the respondent if the planned time for the interview was still suitable or whether they would like to reschedule the interview. When the interview started, the researchers asked the respondents once more whether they were alone and gave the respondents a ‘safe word’ to interrupt the interview in case someone walked in and they were no longer able to answer the interview questions.
Sarah: During the training of the researchers, we covered the different aspects of privacy, protection and safeguarding important to this project. As part of the training, researchers were also given a few scenarios in order to spark discussion. Overall, we tried to create an atmosphere in which the researchers knew that protection concerns were to be treated very seriously and that whenever they were confronted with potential protection risks, they would need to report those immediately.
What went well? What did not go well?
Mahran: We had one instance in which the husband controlled the phone of the interview respondent and answered questions on her behalf. The researcher found herself in a tricky situation as she knew that we would want her to discontinue the interview and schedule it for another date. However, the husband was very insistent that he would be present during the interview and this made the researcher believe that it would be better to continue and not give the impression to the husband that something secretive was being discussed.
Sarah: The situation Mahran described confirmed that situations are very specific. In this case, we felt that the researcher was indeed correct to continue the interview in order to comfort the husband and prevent tensions between husband and wife. However, the researcher did the full interview (excluding the sensitive questions) and we determined that it would have been better if she kept the interview as short as possible by skipping a large number of questions. This would not have compromised the data since we were not going to use this interview in the analysis due to the fact that the husband answered the questions instead of his wife. To ensure both researchers would feel comfortable in dealing with such situations in the future, we conducted an additional training before continuing with interviews.
What would you do differently?
Melissa: I’m not sure there was much I would do differently – my role in the project was to ensure the technology side operated smoothy and to document learnings for future data collection. Whilst there were certainly challenges using the tech for the first time, for a pilot these are really useful to learn from!
Sarah: There were a few times when we faced issues around the use of the case management system. After those issues were resolved, it really showed that SurveyCTO’s system is very efficient. However, I would have made changes in the training materials knowing what I know now by setting aside more time to go over the integrated case management system so we would not face issues at the implementation stage.
Mahran: This was a great learning experience for us and I think that, indeed, some changes to the training materials we had developed would benefit the researchers and the data collection process. For a pilot, however, I think the process went quite smooth.
Any pending features that could potentially enhance privacy that we did not use so far – which ones and what are the trade-offs?
Melissa: There are a couple of additional features that I’m really excited about trying in the future! Firstly, SurveyCTO has a feature through which consent can be pre-obtained through SMS; the interviewee receives a text in advance of any phone calls, asking if they would like to participate in the interview. Whilst this wouldn’t be appropriate for every context, for example in this pilot it would have perhaps been a risk to have text interviewees in advance, in some data collection exercizes this could be used as another opportunity to improve the process of asking for informed consent (acknowledging the power dynamics at play which is a whole other topic we don’t have time to get into here!).
Secondly, SurveyCTO has the capacity to include a ‘call masking’ feature. If activated, this would mean that even interviewers don’t need to see that phone numbers of the people they are calling. This again would be an extra step in terms of data privacy and protection by further limiting access to personal information
Sarah: Although SurveyCTO’s CATI Starter Kit does offer an audio recording functionality (known as audio audit), we were unable to use the feature during the pilot. This had to do with the type of phones we used for this exercise. Improving the compatibility of this feature with more devices would be a great addition.
Resources: The sixth opus of the Going Digital series share the learning from using CATI in this pilot and is accompanied by a resource pack with guidelines and templates to use CATI and set-up phone interview protocols
About Authors:
Mahran Alhayek: Mahran is Optimum Analysis’ Co-Founder and Fieldwork Manager. In his role, he trains and supervises researchers during data collection for all projects implemented by Optimum Analysis. He is dedicated to ensure that Optimum Analysis always utilizes local knowledge when collecting data, thereby ensuring that the local context is taken into account.
Melissa Harris: Melissa is one of Oxfam GB’s Digital in Programme Advisors, joining the team just over a year ago. Her background is in monitoring, evaluation, accountability and learning (MEAL) and she is passionate about how digital tools and practices can make for safer and more efficient data collection.
Sarah Nijholt: Sarah is Optimum Analysis’ Founder and Managing Director. She has over eight years of experience in research, with a focus on monitoring and evaluation in conflict and post-conflict settings. She is committed to support and aid humanitarian organizations to strengthen their monitoring and evaluation practices, thereby improving the effectiveness and impact of humanitarian interventions.