How do you Design a Study when Everything Changes? (Part 1)

Last week I gave you a quick update on what we have been doing related to Program Metrics.

I would like to brag about how we have been analyzing the data we have already gathered.  David and Rogonga have been doing some hard-core analysis of all of the data we gathered for Healthcare and Watsan in December, and they will have some values to share with us in about a month. They have done all the analysis and formula-building at this point, and what is left is just some data cleaning. It takes time because they have to reconcile hard copy surveys with what was entered into our data entry sheet. Jamie, as well, has been conducting complex analysis, in her case of the already extant and built-by-Jennifer Ag model. Last Friday she built a nested-if function that contained seven sub-functions. Those of you who love the puzzle-solving that working with Excel allows you to do will appreciate how fun that was for her. (I’m not kidding, it was fun! She told me.) I myself have gotten to do some excel modeling with the literacy-data we have gathered using the Uwezo tool. These values as well will be reportable once the data has been cleaned a bit.

We actually HAVE data. That is the great news. We are in very good shape, and we will have some to actually report to all of you in the near future.

The main thing that we have been spending the other half of our time on here on the team is Study Design. That is what we are calling determining the when, how, and who of the next and all subsequent iterations of data gathering for each Program Metric. Some things to take into account here are

1. How long it might take for an activity to actually affect a metric value
2. How much gathering the data related to the metric value will impact the community
3. How long it will take and how difficult it will be to gather the data
4. Whether any extenuating circumstances might affect the value of the metric within the timeframe that we have chosen for assessment

All of these points are important, and my team spent a little bit of time discussing this last point in some detail today.

Studying a community and how it changes and its members change is not like a perfect experiment that you might have learned how to run in High School science classes. There is no way to create a completely immutable environment when the subjects of your experiment are human. A couple of extenuating issues that have come into play with us so far are huge fluctuations in market maize prices, droughts (of course), Somali refugees coming into our communities, government mandates about school-closings and subsequent re-distributions of student populations, government rules about Community Health Workers, and many other things.

So the question we were faced with today was, how should we design our studies such that we are able to measure the impact of our interventions when we know that wildly varying extenuating circumstances are going to come into play for all potential subjects of our study, both comparison groups and standard groups?

Because this post is already a bit long (and because I don’t know the best answer to our question just yet), I’ll get back to you in two weeks with a follow-on post.

To be continued…