David Staples has responded to the criticism of his work contained in “Uh, about Neilson Numbers…” Given that it’s buried in the comments in a post that’s a few days old, I figured I’d give it some prominence and rebut.
I will give you one thing, Tyler: you make a damn fine prosecutor.
Nonetheless, I don’t see you as a credible critique of my work. You’re so quick to judge, you miss out on necessary investigation and deliberation.
For example, the other night, in a lengthy debate on Twitter, I mentioned that I used the official NHL methodology for reporting plus-minus in my own work.
This evidently struck you as a very bad idea — given this post you just wrote — but before launching your attack you didn’t think it wise to ask a few basic questions, starting with:, “OK, you use the NHL methodology, which I think is crap, but why do you do that?”
Of course, tthe answer is that from the start of my own work in this area, my primary focus has to been to examine whether or not the NHL’s official plus-minus system is fair, and if it’s not fair, to ascertain just how unfair it is.
David Staples, December 4, 2011:
In 2007, not knowing about Neilson’s approach, but with the goal of coming up with a better way to rate the two-way players of hockey, I started to do this same work on all goals for and against the Edmonton Oilers.
In 2010, I realized that it was best to also study scoring chances for and against if I wanted to get create a better of map of each player’s two-way play. I started to use the Neilson method to study scoring chances as well.
“…A better way to rate the two-way players of hockey.” “…A better map of each player’s two-way play.”
Look, I’m all for academic investigation and such and lord knows that I do enough diddling with numbers myself but I would submit that “my primary focus has been to examine whether or not the NHL’s official plus-minus system is fair” is not synonymous with “I wanted to create a better map of each player’s two play.” Both are laudable goals but they aren’t necessarily the same thing. For instance, in one, you might use the NHL’s plus/minus methodology. In the other, you might conclude that that’s an asinine thing to do because you’re going to prevent yourself from being able to make inter-player comparisons.
In any event, what David says he is trying to do now is not what he has previously claimed to be doing. It’s not the way in which he’s presented his findings. Take this, for example, from October 7, 2011:
At the same time, Corsi focuses on shots at net but doesn’t zero in on the more important event in a hockey game, the scoring chance. There’s little doubt that if you looked at Crosby based on his real contribution to scoring chances for and against his team, you would find he created more chances than anyone else, while more than holding his own in his own zone.
Certainly when we look at Whitney and Vandermeer’s contribution to scoring chances for and against, that’s what we see. Whitney helped create far more chances for the Oilers per 60 minutes of play than did Vandermeer, and Whitney made mistakes on far fewer chances against than Vandermeer.
I would suggest that, if one’s intent was to make some sort of forensic analysis of the fallibility of the plus/minus stat, one wouldn’t be using his research like this. My understanding has always been that he was trying to find a better way to rate the two-way play of players. I took this from actually reading the words that he wrote.
I have written about this extensively for years, and often mentioned that to compare apples to apples, I was using the NHL’s methodology on reporting plus-minus. This wasn’t some deep dark secret of my work. It was the essence of my work, to make this offiical plus-minus to Neilson plus-minus comparison.
The December 4, 2011 article that I’m quoting from there is called “Frequently Asked Questions on Neilson Numbers.” Actual piece of that FAQ:
Do you assign errors to players killing off a penalty?
Yes. The player who takes the penalty and is off the ice when the power play goal against is scored will be assigned a primary error, as he is the one who created the uneven man situation. Penalty killers who make mistakes on the goal-against play will be assigned secondaries. But these errors aren’t included in the true plus/minus calculation, as they’re not committed at even strength.
What does David say in his review of the Oilers’ defencemen this year?
The Edmonton Oilers defencemen can be broken into three groups based on their even strength play in 2011-12…By this plus/minus system, here are the rankings of the Oilers defencemen in 2011-12 on their even strength play.
What does he say in the review of the Oilers’ centres?
By this plus/minus system, here are the rankings of the Oilers centres in 2011-12 on their even strength play.
If there was a review of wingers, I can’t find it. In any event, whatever the Neilson project started out as (I’m inclined to accept what David was saying before I called attention to this, but to each his own), David’s clearly become all about using it to rate two way play at some point.
In the end, after four years, I can report that the NHL’s official plus-minus gives false positive and negatives 35 per cent of the time (I’ve reason to believe the false positive and negative percentage is higher on scoring chances and higher yet on shots on net, but that’s a different debate).
This conclusion doesn’t make sense, even if that is the question one is trying to answer. My review of his results suggests that it will depend on the specific player at issue. Some, like Andy Sutton, are being wildly overrated by traditional plus/minus according to him, while others are getting blamed unfairly.
Moreover, if the intent of this project is to examine the NHL’s traditional plus/minus system, shouldn’t the analysis be limited to goals for and against only? Nobody’s getting pluses or minuses on scoring chances that don’t turn into goals. It seems that he’s done a ton of extra work that doesn’t have any thing to do with the purpose he’s now identified: examining whether the NHL’s system of plus/minus is fair.
Or, in the alternative, that I perfectly understood what he was trying to do and his objection is nonsense.
In your article, you report that GASP! I use this methodology and GASP! it’s a dumb, dumb methodology and GASP! I didn’t mention it in my F.A.Q. on Neilson numbers. The implication is “Dave’s not so bright and he’s also shady.”
Now, I might well have mentioned it in the F.A.Q. (and will do so now to avoid further accusation and confusion), but I can’t say this was a frequently asked question about my work. I repeatedly stated at the outset what I was doing it and have made no effort to hide it.
Why would I?
Perhaps Dave should change the part of the FAQ where he explicitly says the opposite? Yes, that should probably be changed. Also, if he’s making no effort to hide it, he might not want to say even-strength play when you mean something else when you report your results.
Now, if you had asked that first question, instead of donning the old prosecutorial robes, you might have found out more about my work that you don’t now, such as any data supporting my contention that there’s 35 per cent false positives and negatives on goals.
I…I’m not entirely sure what this sentence means. I’m not surprised that there are false positives and negatives on goals; I’ve always taken it is a given. I don’t criticize anyone for looking to prove that; it’s good to challenge assumptions, but I don’t think that’s close to the biggest problem with traditional plus/minus. A much bigger problem is that it’s not a stat that can be directly compared amongst players because some players play on the PP and others play on the PK and some do both and some do neither and all of those components count. Some guys, who are out on the ice with the goalie pulled, get tagged with minuses on empty net goals while others get pluses that don’t really represent hockey in the real sense. Sometimes guys leap over the boards as a puck goes in. In addition, there are the PDO issues. So yeah, +/- has a lot of problems.
Like I say, there’s nothing wrong with developing evidence to confirm something but I’m not sure what I’ve ever said to make it appear that I think the NHL’s traditional +/- scheme is a good one. Indeed, I’m laughing at the silliness of using it as the basis of a system for comparison precisely because I think it’s so bad.
I get it that many bloggers don’t want to accept that 35 per cent number, that it is a problematic number for all who push team-plus minus numbers as a useful tool in rating individual players.
No. David is imagining an objection that doesn’t exist. Most serious stats people accept that any team based system is going to result in a player getting credited or blamed for events for which he deserves no credit or blame. Truthfully, I can’t think of any who don’t – I’m saying “most” to be cautious. I don’t have a firm opinion on what the specific number is – I suspect it’d vary from player to player but nobody disagrees with that. This is why people do things like WOWYs and such, to try and suss out who the difference makers, good and bad, are. People do stuff like this because they recognize the limitation of the stat.
Of course, one of the strengths of a +/- system like Corsi is that it can help you identify guys who do things that keep the puck headed the right direction that doesn’t show up in scoring chances or Nielson numbers. There is stuff that happens between the scoring chances that matters. Nielson analysis doesn’t catch this.
But to be a credible critic of my conclusion, I’d suggest you maybe look at least one game yourself, track and break down the goals and scoring chances, see who is involved and isn’t. Better yet, do the work for five or 10 games.
You may question the import of my work at that point, you may still think that two different raters won’t come up with the same plus and minus marks for individual players on individual goals or chances.
But I’m confident you’ll accept one thing, that there’s a boat load of false negatives and positives on average on goals and chances. If you do the work long enough, you’ll find that false positive/negative marks on goals is at least one third of all marks handed out.
He’s flailing about here, arguing points that aren’t in dispute. To start with, I’m reasonably convinced that most people who’ve watched a lot of hockey will come up with pretty similar chance counts when they count chances in a hockey game. Most of the scoring chance counters are working off similar definitions and most of you have watched a lot of hockey. I’ve seen the discussions in the comments here and on Twitter when two teams with people counting chances are playing. They generally have similar results. This isn’t a critique that I have. I’ll even go so far as to say I expect that two people would come up with reasonably similar assessments of credit/blame on goals and chances.
I don’t know what would have prompted David to raise this as an issue. It’s like he doesn’t read the criticism of the past two posts, or he has some sort of rote response to any perceived criticism.
Now a hockey can can extrapolate – and I have — to make an apples to slightly-different-kind-of-apples comparison, putting up Neilson against Corsi or King numbers. I don’t think it’s unreasonable to do so, given that they are all plus-minus systems, purporting to be of value in rating individual players.
I’m also confident if you do that, you’ll find the Neilsons plus-minus gives the best inidcator, the best starting point, to rate individuals, because it helps negate Quality of Teammate, something that is inextricably woven into Corsi and King numbers, to their detriment.
This is baffling. For one, you can’t do any sort of a test or investigation to substantiate because of the way in which he collects his data. Nobody can put together a complete list of chances and the Neilsons awarded and look to see whether Nielson numbers negate Quality of Teammate or not. This is purely an argument of reason and poor reasoning at that.
On the one hand, I can see the rationale. By crediting/blaming those involved, you’re theoretically helping to minimize the extent to which guys get credited/blamed for stuff that they don’t have any involvement with. I get it. I hope that whenever David is tempted to say I don’t understand the rationale, he comes back and reads this paragraph.
On the other hand, I fundamentally disagree with the implicit statement there about how NHL hockey works. He is essentially saying that scoring chances are just something that happen based on the skill or lack thereof at making/preventing scoring chances of the individuals on the ice. Taylor Hall’s on the ice, he’s good at scoring chances so scoring chances will happen.
I disagree with this. I think if other guys on the ice are good at getting the puck going the right way, Taylor Hall’s scoring chance numbers will be better than if they weren’t. Even things that aren’t scoring chances, like breaking the cycle in your own end and getting the puck heading the right direction, or keeping a puck in at the blueline, or consistently winning battles…even if these things don’t lead directly to scoring chances, they create a climate in which scoring chances are more likely to happen. Drop Taylor Hall in my league, fine, he’s creating scoring chances on his own. I don’t think he is in the NHL. His Neilson numbers are built on a platform created by the five guys on the ice as a whole.
Of course, my rationale is as much a matter of logic as Dave’s. Except that Dave sees cases where Nielson numbers diverge significantly from other indicators (Corsi and scoring chances) and declares that his case is proven. Whereas I (and a lot of other people) would like to see some explanation for why, if guys like Jones and Sutton are so good and bad respectively, the team level stats don’t seem to show it, even once you make some efforts to account for quality of teammates and competition. This is (quite literally) the million dollar question.
David assumes that because guys like Jones and Sutton have good/bad Nielson numbers, it shows that Corsi and scoring chance data is incorrect. If he’s right, there has to be some explanation for why that disconnect exists. In effect, it means that the other players on the ice become worse/better respectively when those guys are out, because the problems Nielsons identify don’t show up in their team level numbers. That doesn’t seem a particularly satisfactory explanation to me.
Most of this other stuff is just stuff that Staples is pretending is my objection, because it’s easier turf to fight on. Well, except in the case of the asinine manner in which he’s adopted the Roth/Irvin stuff and pretended that it’s even strength and is now saying that this project is about something entirely different than he said on Cult of Hockey. I’ve no idea why he’d choose to fight on that turf because it just provides his critics with many, many hilarious examples for easy laughs in putting together a lengthy post about what is otherwise some dry stuff but I won’t tell him how to argue his case.
That said, there isn’t a person who takes stats seriously who wouldn’t love to have some sort of a system that accurately meted out individual contribution. It is exactly what you want to find if you’re interested in poking around with this stuff. Personally, I’d love it if Nielson numbers were the silver bullet that identified the guys who are really driving things. For the reasons laid out above, it seems irrational to think they are and, in any case, Staples certainly hasn’t tried to explain the tough cases.
If Staples really cares about what he’s doing though, he might eventually to try to back up the claim that they have some value in terms of predicting future success, rather than simply asserting over and over that this is how Roger Neilson did it. Right it now, it sure looks like he’s a guy who’s trying to carve out a niche for himself and doesn’t particularly care if the work has any merit, just like the PowerMax guys and guys with formulas to figure out which is the best goalie based on dividing wins by shutouts and multiplying by shootout save percentage or something. Exact same thing. There’s good money in snake oil, I guess.