Out of sample predictive validation

The following table gives percent change in MSE for the hold-out sample compared with the predictions of the individual LS models.

I'll split the data set into two parts, so I take first half and the second half, I do an equal split. I estimate the model on the first half and then I go out and based on the prior, how well can I predict in each of these outer sample data sets. So in the first case, if I go back and do the individual least-square model I'm going to measure everything relative to that. And I'm going to just think about what's the average mean square area. The following table gives percentage change in MSE for the hold-out sample compared with the predictions of the individual LS models. So if take a pool model, a pool model would bring it down by 10%, a pool and a cluster model create similar, the cluster does it a little better, but not much. Now, If I do a Bayes approach with a v prior, so if I said k=5 I've essentially said that there's not a lot of differences in my prior between this and the initial least-square estimates. So it makes sense that these things are close. Now if I start saying that my Bayes is k parameter is moderate, if I say that k=1 what happens here is now I start coming up with a 17% decrease in auto sample least squared. Where if I finally go to like - strong Bayes prior, say it's equal to .1, , what happens here is a big decrease in least squared error. Like a 25% decrease. So, the point isn't to go back and say I'm trying to chose my k to minimize the outer sample mean-squared area. All I'm trying to do is say, look, my prior notions before going into this is to say that the demographics are important. You know, if I had to do this I would say, I'd like to go to a pool model, but I know a pool model is wrong, so what the data is essentially telling me is that there is some support for either this moderate or strong prior. The problem is that the data isn't really formally telling me which one should I go to. It's almost saying more is better but it's going to be difficult to discriminate. For more detail see here. Yes?

Question from Robert E. Kass:

Question from Robert McCulloch:

Question from Robert E. Kass:

Back