Case Studies in Bayesian Statistics
Workshop 8 - 2005

The most fundamental point is made well by Mosteller and Wallace in their great study of disputed authorship of The Federalist Papers:

``The main value of our work does not depend much upon one's view of the authorship question used as a vehicle for this study, although we do add information there. Rather the value resides in the illustrative use of the various techniques and in the generalizations that emerge from their study. In retrospect, the methodolgical analysis could have been restricted to sets of papers whose authors are known. Still, the responsibility of making judgments about authorship in disputed cases adds a little hard realism and promotes additional care that might otherwise have been omitted.'' (Emphasis added)

Thus, in examining case studies, as in examining theory and methods, we are still hoping to learn about statistical methods - which features of particular methods are helpful, where the outstanding problems are - but the context provides an essential backdrop. To maintain the emphasis on practical utility we have to keep the context at the forefront of the discussion.

Effective case studies should do the following:

1. Be convincing scientifically.

   That is, they should make it seem as if an interesting scientific problem was being posed, and the work done went a long way toward solving the problem, or at least advanced the state of knowledge somehow.

2. Consider messy aspects of the problem

   Interesting features of problems that precede formal analysis are very important. These include variable definition, initial data display or description, modeling, and prior specification among other things.

3. Keep the solution of the scientific or technological problem in view.

   This is really the Mosteller and Wallace point: we can not appreciate applied statistics in the abstract, we must have a context for each discussion. Analytical decisions have to lead toward practical answers, and this is a big constraint we don't have to face directly when we discuss new methodology in theoretical terms.

4. Emphasize the link between the statistical problems and the substantive issues

   Perhaps the most important decisions a statistician makes concern the way statistical problems are formulated. Yet, these steps are generally presented in polished form rather than as the demanding and difficult multi-faceted decision-problems that they usually are.

5. Display intermediate decisions

   As in the previous point, there are numerous decisions a data analyst must make. It can be very instructive to have a record of these, or at least of some of them, so that we can learn from each other what characterizes the ``art'' of applied statistics.

6. Assess methodology

   When it's all done, it is very helpful to reflect on what approaches have worked well, and which ones have not.

The last two items, in particular, are too often ignored. With case studies it is especially important to consider a question that all research reports should answer: ``What has this investigation taught us?"