• Deadly Accurate

    by  • January 8, 2013 • Uncategorized • 4 Comments

    As I write this, I’m looking at my bookcase, which has ten issues of Baseball Prospectus sitting in a row, 2001-2010. Truthfully, I doubt I read much of the later editions once my fantasy drafting was complete and, with comparable, freely available projection systems available, it kind of seemed silly to buy them if I wasn’t getting value out of the books other than that.

    They’re fantastically marketed books though. If you’ve followed Nate Silver’s career at all, it’s kind of interesting to see how he’s marketed himself since leaving Baseball Prospectus. Silver first gained public prominence through the work that he did with Baseball Prospectus in developing a projection system that he called PECOTA. I just pulled the 2006 edition off my bookshelf and the back cover tells me that the book includes “Baseball Prospectus’s exclusive (and deadly accurate) PECOTA projection system, forecasting the chances that a player will break out, improve or collapse.” Beside that, there’s a box telling me that the 2005 edition of Baseball Prospectus predicted all sorts of things that happened. Deadly accurate! You can use individual projections to create team projections, which the Baseball Prospectus do. Here’s Silver talking about the White Sox prior to the 2005 season:

    That leaves the White Sox, who will not be the worst team in the division. The question remains, however, whether it’s better to have a 100-loss season here and there, or to be doomed forever to mediocrity, like the Dominique Wilkins-era Atlanta Hawks. It’s fitting that the White Sox are projected to finish almost exactly at par, because their moves this winter give every appearance of having been influenced by some bizarre, gravitational pull exerted by the .500 mark:

    Perhaps Kenny Williams has some form of Munchausen’s Syndrome by Proxy, and needs to ruin the team in order to save it? Perhaps it’s something in the White Sox’ ill-conceived lease with the Illinois Sports Facilities Authority, which doesn’t require the White Sox to pay rent until the 1.5 millionth customer has passed through the U.S. Cellular turnstiles? In any event, this was looking like the year in which the Sox might finally have done poorly enough to trigger some necessary, long-overdue front office changes, but the flurry of constructive late-winter signings ought to be just enough to pull them back into their self-built purgatory.

    Chicago won the World Series. So PECOTA wasn’t entirely deadly accurate. I looked and this did not make the back cover of the 2006 edition. I’m sure if I were to ask him, Silver would tell me that he didn’t write the back cover and, to his credit, he’s pretty up front about the limitations of what he does. Very few things are certainties (Khabibulin being a bad idea was a rare exception) and if you predict enough things, you’ll get some things right and some things wrong. He’s no longer involved with Prospectus Entertainment Ventures, or whatever they call themselves these days, so it’s not his fault anymore. It’s a shame that he didn’t have more of a voice on that stuff when he was there though, or that they haven’t broken this particular bad habit.

    Which brings me to the Hockey Prospectus projections for the season to come. As in days of BPro yore, they tout their credentials:

    VUKOTA gained credibility by correctly tabbing the up-and-coming Chicago Blackhawks as 2010 Western Conference champions. And as far as individual projections go, David Staples of the Edmonton Journal concluded that VUKOTA was the best for “predicting the point totals of Edmonton Oilers players” for the past two seasons.

    I’m not entirely sure how appropriate it is to be using VUKOTA to generate team predictions or how they go about doing it. I’ve never seen it explained anywhere. In typical Prospectus Entertainment fashion, we have little idea what goes into generating VUK – oh to hell with it, this is stupid, it’s not even an acronym, it’s just a stupid reference to the fact that the baseball ones are called PECOTA rankings, which was just a fake acronym anyway – into generating their projections.

    In last year’s annual, Tom Awad (who I respect as a hockey analytics guy) made the following comment:

    One thing you will notice is how tightly bunched up our projections for teams are: all 30 teams project between 81 points (Minnesota) and 107 points (Washington). To be clear, if you asked me if I think the Wild will finish last overall this season, I would answer “probably not”. They simply have the lowest projection of our 30 teams, but this is only one point less than Ottawa and Edmonton. There is still a significant amount of luck involved even over the course of an entire season of hockey. Even if we are right that 81 points is the Wild’s “true talent”, this just means that they will likely finish somewhere between 71 and 91 points. 71 points will lead to a lottery pick, while 91 points would leave them in the playoff race until the last weeks of the season.

    You can say the same thing about this year’s projections, which are even more tightly bunched, between 81 and 104 points. I’m not the math guy that Awad is, but this seems obviously wrong to me. I’ve looked before at how terrible teams tend to perform the following season, which is a topic of interest to Oilers fans, for obvious reasons. Key paragraph and chart:

    I’ve taken each season from 1995-96 to present and converted the point totals such that there are 2.24 points awarded per game, as there was in 2011-12. I’ve then created two groups of teams and looked at how they performed in their next five years. The first group is teams getting between 57 and 67 points in a season. I’ve summarized how those teams progressed in the table below.

    Hockey Prospectus has Columbus finishing last this year with 81 points, just as they predicted for Minnesota last year. It seems to me that they’ve somehow created a world in which the spread of talent is smaller than the real world. If they were right, I wouldn’t expect that teams coming off 57-67 point seasons averaged 73 the following year – I’d expect it to be more like 80 or 81, as the teams regressed towards the mean.

    Vegas would seem to disagree with Hockey Prospectus too. When Vegas set the over/under for the 2011-12 season, they ranged from 74.5 to 107.5, a spread of 33 points to the 26 point spread that Hockey Prospectus had. That seems to me to be a more likely true talent spread than the one that Hockey Prospectus is suggesting – I don’t disagree with the idea that the true talent difference between teams is smaller than that observed over 82 games, I just think that you’d have a hell of a time arguing that it’s as small as Hockey Prospectus does.

    I can’t say how or why Hockey Prospectus generates such a small true talent difference. It’s black box stuff, presented to the world at large as being valuable because it comes from a system and because David Staples credits them and they totally called the 2010 Blackhawks. It’s probably a better business model than an open source system – I tend to lay out the guts of my reasoning when I’m arguing a point, which means I have nothing proprietary to sell – but it’s hard to take it seriously when the system seems to have the talent spread between teams so far off from what the real world and Las Vegas suggest.

    Email Tyler Dellow at tyler@mc79hockey.com


    4 Responses to Deadly Accurate

    1. TigerUnderGlass
      January 8, 2013 at

      Just seems like good business to have a system that tells fans of bad teams they aren’t as bad as they think.

    2. Tom Benjamin
      January 11, 2013 at

      I agree that any system that expects to be taken seriously when the implied claim is an ability to predict the future had better tell us how the system works. Black box systems don’t cut it.

      It is possible to use probabilities to make good predictions, but there are inherent problems with the approach. I’d like to know what probability VUKOTA assigned to a Hawk championship in 2010. Surely it was only slightly more probable than the liklihood of the team they tabbed to finish second. If the system gave Chicago, say, a 20% or 25% chance of winning the West in 2010, did they make an accurate prediction?

    Leave a Reply

    Your email address will not be published. Required fields are marked *