Some observations on what makes
a good case study
The most fundamental point is made well by Mosteller and Wallace in
their great study of disputed authorship of The Federalist Papers:
``The main value of our work does not depend much upon one's view of
the authorship question used as a vehicle for this study, although we
do add information there. Rather the value resides in the illustrative
use of the various techniques and in the generalizations that emerge from
their study. In retrospect, the methodolgical analysis could have
been restricted to sets of papers whose authors are known. Still, the
responsibility of making judgments about authorship in disputed cases
adds a little hard realism and promotes additional care that might otherwise
have been omitted.'' (Emphasis added)
Thus, in examining case studies, as in examining theory and methods,
we are still hoping to learn about statistical methods - which features
of particular methods are helpful, where the outstanding problems are
- but the context provides an essential backdrop. To maintain the emphasis
on practical utility we have to keep the context at the forefront of the
discussion.
Effective case studies should do the following:
- Be convincing scientifically.
That is, they should make it seem as if an interesting scientific
problem was being posed, and the work done went a long way toward
solving the problem, or at least advanced the state of knowledge somehow.
- Consider messy aspects of the problem
Interesting features of problems that precede formal analysis are
very important. These include variable definition, initial data display
or description, modeling, and prior specification among other things.
- Keep the solution of the scientific or technological problem in view.
This is really the Mosteller and Wallace point: we can not appreciate
applied statistics in the abstract, we must have a context for each
discussion. Analytical decisions have to lead toward practical answers,
and this is a big constraint we don't have to face directly when we
discuss new methodology in theoretical terms.
- Emphasize the link between the statistical problems and the substantive
issues
Perhaps the most important decisions a statistician makes concern
the way statistical problems are formulated. Yet, these steps are
generally presented in polished form rather than as the demanding
and difficult multi-faceted decision-problems that they usually are.
- Display intermediate decisions
As in the previous point, there are numerous decisions a data analyst
must make. It can be very instructive to have a record of these, or
at least of some of them, so that we can learn from each other what
characterizes the ``art'' of applied statistics.
- Assess methodology
When it's all done, it is very helpful to reflect on what approaches
have worked well, and which ones have not.
The last two items, in particular, are too often ignored. With case
studies it is especially important to consider a question that all research
reports should answer: ``What has this investigation taught us?"