Simone Lombardini compares duration, estimates and enumerator’s bias from two different time-use survey modules from the same impact evaluation survey in Indonesia.
Unpaid care work and ‘ Time Poverty‘ are increasingly recognised as relevant to development efforts, and interest in measuring time-use data is growing. However, gathering information on time use is not easy; time-use modules are known for being long and tedious, and time-use is rarely the only information we’re interested in. The challenge is to strike the balance between accuracy and duration required for collecting the data. As a result we’re looking at how to design a time use module which elicits accurate information but is not too challenging or time-consuming.During a recently conducted impact evaluation in Indonesia, we explored this question by randomising two different questionnaire modules. Below, I compare the time taken to complete them, the estimates they produce, and how these estimates were subject to enumerators’ bias.
Questionnaire designs and data
The first approach (Option A) reviews the previous 24 hours, asking respondents for information on the primary and secondary activity they performed hour by hour. Option B on the other hand, presents a list of activities and asks respondents to estimate how many hours they spent on each activity yesterday. It includes 11 activities and allows for multiple activities to be conducted at any one time.
We conducted 806 interviews from a random sample of women living in project intervention and comparison areas. 402 were interviewed using Option A, and 404 were interviewed using Option B.
Duration
Unsurprisingly, the median time for Option A was almost 7 minutes, compared to under 2 minutes for Option B.
Estimates
Except for one activity Options A and B provide estimates which are statistically different. There are no clear patterns to explain these differences; however, a couple of considerations can be made.
Firstly, responsibility for the care of children, elderly or other household members is a slippery concept. While Option B directly asks the respondent to provide an estimate on the number of hours spent on this activity, estimates for Option A are obtained by combining primary and secondary activities where the respondent reported ‘Caring for children’, ‘Teaching/tutoring/training children’, and ‘Caring for disabled ill, elderly’. The fact that estimates in Option B are twice as big as estimates in Option A could suggest that respondents tend not to consider being responsible for the care of children or elderly as an activity when considering their day on an ‘hour by hour’ basis.
| Table 1: Comparing estimates for Option A and Option B (average number of hours spend by activity) | |||
| Option B mean | Option A mean | Difference | |
| Responsible for the care of children, elderly or other household members | 3.007 | 1.328 | 1.679*** |
| Fetching water and firewood | 1.544 | 1.393 | 0.151 |
| Cooking | 1.311 | 2.896 | -1.585*** |
| Cleaning the house | 0.925 | 1.811 | -0.886*** |
| Washing clothes | 1.673 | 0.565 | 1.109*** |
| Cultivating land / tending farm animals | 2.345 | 1.818 | 0.527*** |
| Formal/Informal labour | 1.396 | 0.468 | 0.928*** |
| Leisure time | 0.974 | 2.246 | -1.272*** |
| Sleeping at night | 8.022 | 8.495 | -0.473*** |
| Personal care | 1.148 | 2.129 | -0.981*** |
* p<0.1, ** p<0.05, *** p<0.01
Secondly, when comparing time devoted to ‘Leisure activities‘, Option A provides significantly bigger estimates than Option B. Meanwhile Option A provides significantly smaller time estimates for ‘Cultivating land and tending farm animals‘ and ‘Formal and informal labour‘. One possible explanation could be that short but frequent activities, which are conducted in units smaller than the hour, might be misrepresented when using Option A, potentially leading to overestimating and underestimating the time spent on these activities.
Finally, it is also possible that simultaneous activities might be underreported. The classic example is time dedicated to cooking which lends itself to multitasking. The questionnaire design attempted to reduce this issue by asking the respondent for information on primary and secondary activities (Option A), or by allowing the sum of the hours spent on activities to add up to more than 24 hours in a 24 hour period (Option B). From this data, however, it is not clear which option gives us a more reliable estimate.
Enumerators’ bias
How questions are asked has a bearing on data quality and so enumerator bias is also an important issue for us to explore when considering which time-use module should be used. For both options, the ability and willingness of enumerators to conduct the interview as designed is questionable. Option A may be considered long and monotonous, while Option B could be perceived as more difficult for respondents to estimate time accurately. Anecdotally, enumerators involved in the data collection in Indonesia reported finding it easier to perform Option A despite the longer duration of the questionnaire.
| Table 2: Comparing R-squared values regressing estimate’s enumerator fixed effects for Option A and Option B | ||
| R-squared Option B | R-squared
Option A |
|
| Responsible for the care of children, elderly or other household members | .33457536 | . 11367234 |
| Fetching water and firework | .1790417 | .1679908 |
| Cooking | .32493287 | .2558101 |
| Cleaning the house | .34904729 | .25122279 |
| Washing clothes | .25898003 | .12998045 |
| Cultivating land / tending farm animals | .51325526 | .16461814 |
| Formal/Informal labour | .32738732 | .24394941 |
| Leisure time | .27386424 | .20048464 |
| Sleeping at night | .26396948 | .29905843 |
| Personal care | .22459978 | .45259229 |
The table above (Table 2) details the R-squared of the estimated time values regressed on the enumerators’ fixed effects. This is computed by running a regression which has the time estimate as a dependent variable and as independent a binary variable for each of the 15 enumerators (excluding one).
The R-squared value is interpreted as the proportion of the variation in time estimates explained by the individual enumerators. The higher the value, the more likely enumerator bias explains the variability in the data, resulting in higher the concerns about data quality.
Option A generally presents smaller R-squared values compared with Option B, suggesting that Option A presents smaller concerns on enumerators’ bias.
Conclusion
Results from this small experiment seem to suggest that, while taking on average longer, Option A is more reliable, at least in reducing enumerators’ bias. But more work and research is needed to understand which produces more accurate estimates of time use.
Please do get in touch, if you have ideas, experiences, and learning you would like to share with us as we continue investigating this fascinating measurement topic.
Related reading:
- UN’s Sustainable Development Goal 5.4
- Oxfam’s We-CARE programme
Author
