In this post I’m going to try to tie a pizza utensil, Bill Murray, and Charlie Blackmon together all while trying to help you avoid a pitfall I think many are making in their fantasy baseball research.
Read enough fantasy baseball advice and you’re bound to come across something like this:
We can now trust player x’s <insert rate statistic here> because he’s reached the number of plate appearances for the stat to become reliable.
Or maybe this:
We’ve reached the point of the season where <insert rate statistic here> starts to stabilize.
Maybe you even clicked on a link near the comment, saw some fancy tables with a lot of other stats and when they “stabilize”, references to r-squared, and then concluded, “Seems legit to me.”
These comments are usually followed by some kind of analysis that uses the stat in order to project into the future. This is the problem! More on that in a bit.
Not So Fast My Friend
I’ve long been a victim of this. I’m not a statistician, so if I someone makes claims like this and links to a study that looks legitimate at a quick glance, I’ll buy into it. This seemed even more reliable because the study is quoted at a lot of reputable sites like Fangraphs, Beyond the Box Score, and more.
But I’m also a regular listener of Fangraphs’ “Sleeper and the Bust Podcast” with Eno Sarris and Jason Collette. I’ve heard Sarris mention a disclaimer several times when referring to sample size stabilization points that has always left me a little unsettled. So I decided to investigate.
A Little History
The original study was performed in 2007 by a man writing under the name “Pizza Cutter”. It’s a heavy load of information to consume, but I do recommend it so you can understand how he performed the test. Plus, it’s proven to be a very popular piece of reference material, so it wouldn’t hurt to familiarize yourself with it.
You can’t really tell from the original study, but it turns out that the research has been misused and misinterpreted by many people. So much so that Pizza Cutter himself has since written several times that his work is being misused.
These Stats Are Not Predictive
Russel A. Carleton, who ditched the Pizza Cutter nickname (except on Twitter), is the man behind the stabilization points research. He has this to say about the predictive value of stabilization points:
…they are not nearly as powerful in predicting the future as people seem to believe that they are.
And it makes sense. When developing projections before the season starts, the typical projection system uses at least three years of data. So then why are we so quick to believe that three weeks of April at bats are meaningful at predicting the rest of the season!?!?!
When referring to an example of how his study is used to say, “this new strikeout rate that we’ve seen is what we can start to expect”, Carleton writes,
That’s not what the study was actually about.
If you haven’t gone and read the article yet, I do recommend it. You can just sense the angst in Carleton’s writing. The title of the article, “It Happens Every May”, speaks volumes. I can just see Carleton surfing the web as we speak reading countless articles inappropriately referencing his work and thinking to himself, “Every season I have to put up with this $#!_”.
It Helps To Understand What Carleton Was Trying To Do
Carleton wasn’t trying to develop a projection methodology in doing this research. He states that one of his favorite things to do is to make up his own statistics and study if they correlate to other metrics we already use in baseball research.
It doesn’t make sense to do advanced baseball research on small sample sizes. So all he wanted to know was how soon into a season, or with how small of a sample size, could he begin conducting these studies of his.
What Stabilization Points Really Mean
Continue reading “Cautionary Notes About Sample Size Stabilization Points”
