Andy Sutton had an awfully interesting season last year. He had the best Corsi amongst Oilers defencemen, although his quality of competition was pretty low. He does well by the scoring chance metric as well, coming in fourth out of the thirteen guys who played defence for the Oilers this year and second out of the seven guys who played more than forty games (Tom Gilbert, Cory Potter, Ladislav Smid, Theo Peckham, Jeff Petry and Ryan Whitney are the other six guys who meet that threshold; only Gilbert had a better scoring chance share.)
David Staples prefers to evaluate players by Neilson numbers. The other night on Twitter, he made the following claim:
Andy Sutton was #Oilers most effective dman this year. That’s what Corsi plus-minus indicates.
Schultz/Petry/Smid/Potter/Sutton. #Oilers Top 5 d-men Neilson scoring chances. Sutton/Petry/Potter/Smid/Teubert. That’s Corsi. Of course, you then factor in QualComp and ZoneStarts. But I’m suggesting with Neilson’s, you start analysis in right spot.
Now, that’s a grossly misleading claim about Corsi – you shouldn’t really use Corsi in the way he’s suggesting but I’ll discuss that in more detail below. For now, I want to dig a little more into Andy Sutton’s season. There’s a lot of research that shows that Corsi is a good proxy for scoring chances, as is addressed most recently here. I’m going to use scoring chances here as a proxy for Corsi – Sutton does well by the scoring chance metric.
Since we’re doing this by way of a contrast with Neilson numbers, let’s take a quick look and see what they say. (Aside: if you haven’t read my most recent post about Neilson numbers and why you can’t compare them to Corsi, you should. Short version: despite explicitly making this comparison, David has collected his data in the old plus/minus way, which renders it not really appropriate to make the comparison he’s been touting all year.) David did a helpful post in April in which he looked at the Oilers defencemen. Here’s the money chart from that table:
(Actual quote introducing this chart: “By this plus/minus system, here are the rankings of the Oilers defencemen in 2011-12 on their even strength play.” We now know that what David means when he says “even strength” is “even strength + SH pluses + PP minuses.” Of course, as of a few days ago, his position is: “Never hid this. That you’re discovering it now, well, good on you.” I guess “Never hid this” means “Never hid this…so long as we exist in a world in which words have no meaning.”)
In any event, you see that Andy Sutton comes out rather poorly. We don’t know how badly David’s data is polluted by the SH+ and PP- but there it is. Sutton kind of stinks.
OK. Let’s look at the scoring chance data. Here’s a chart of the scoring chance and Corsi percentages for the Oilers defencemen in 2011-12.
No news there. It is worth mentioning that while Sutton does lead in Corsi, it’s not a large lead. It’s not particularly significant. As I alluded to above, there’s still the issue of QualComp. The problem with QualComp is that, while it’s great for sorting players, it’s less intuitive in terms of how significant those impacts are. While we don’t have a Corsi database to break down, we do have a complete set of scoring chance numbers for the Oilers.
I went through the Oilers’ 5v5 scoring chances for and against and sorted them by whether or not the other team’s top six was on the ice or not. I determined whether the top six was on the ice by identifying two top six players on each team who didn’t play together very much. I think I did a reasonable job of this. The Oilers were 956 CF and 1057 CA against at 5v5 this year, giving them 47.5% of the 5v5 chances. (To the point about using Corsi as a proxy for scoring chances: a href=”http://timeonice.com/mplayershots1112.php?team=EDM&first=20001&last=21230″>Edmonton had a .471 Fenwick and a .476 Corsi as a whole this year, virtually bang on.)
If I have any idea what I’m doing at identifying the other team’s best players (full disclosure: I cheated and asked the coaches by looking at who they gave the most TOI to), the Oilers will do worse against top six competition. So they did, getting 590 chances and giving up 703, a 45.6% rate. By definition, they therefore did better than their average against bottom six competition, outchancing them slightly 366-354 (50.8%). By inference, playing lots against bottom six forwards is going to help make your scoring chance numbers (and Corsi) look better.
The Oilers defence who played a lot against top six forwards are exactly the ones you’d expect. Gilbert, Smid and Petry are all north of 70% of their chances against top six competition. Schultz and Peckham (?) are at about 65%. Potter, Sutton and Whitney are all between 57% and 60%. The real flotsam (Peckham excepted) tend to be about 50% or less against the oppositon’s top six.
Let’s go a step deeper into this. How did our thirteen Oilers’ defencemen do against the opposition’s top six and bottom six forwards respectively?
You can see that it’s something of a blood bath when the other team’s top six are on the ice. Only four defencemen manage to post scoring chance percentages above 47% – Gilbert, Smid, Schultz and Petry. No surprise with that. Everyone except Smid does better against bottom six forwards too. No surprise with that. Notably, Sutton got torched against top tier competition.
I wanted to push a little further into this, so I sorted the chances by whether or the Oilers had two or more top six forwards on the ice in addition to the breakdown into top six and bottom six chances for the other team. I used RNH, Taylor Hall, Jordan Eberle, Ales Hemsky, Shawn Horcoff and Ryan Smyth as my top six forwards. This gave me four groups of chances: TT (Oilers’ top six against opposition top six), BT (Oilers’ bottom six against opposition top six), TB (Oilers’ top six against opposition bottom six) and BB (Oilers’ bottom six against opposition bottom six).
Quick sanity check; also this is a fun little graph. If I’m doing this in any sort of a sensible fashion, the other team’s top six should do better than the Oilers bottom six and the Oilers top six should be better than the opposition’s bottom six. The graph:
It passes the sniff test, I think. The Oilers’ top six gets barely outdone by the opposition’s top six when they go to head to head. The opposition’s top six puts the Oilers bottom six to the sword. The Oilers top six does reasonably well against the opposition bottom six and the opposition bottom six trump the Oilers’ bottom six. The results seem to be internally consistent. Also: the bottom six stinks. They’re all back next year, you say?
Let’s turn back to the defencemen and look at them through this prism. First, we’ll look at which of the four types of ice time they were most likely to play.
Unsurprisingly, Gilbert, Smid, Whitney, Schultz and Petry all played saw more than 40% of the chances they were on-ice for in TT ice time – Oilers top six on the ice and the other team’s top six on the ice. Somewhat surprisingly, Peckham’s also north of 40%. For all the talk that he hasn’t been the same since his concussion, it’d be interesting to see a similar breakdown of his minutes from 2010-11. I kind of wonder if he got a push this year into a quality of minutes that he just can’t handle.
The guy who got the most ice time in situations where the Oilers’ top six were on the ice along with the opposition’s bottom six? Cam Barker. Some pigs, you just can’t put lipstick on. Sutton and Potter played healthy shares against bottom six players as well, sort of split equally between playing with the Oilers’ top six on the ice against them and with the Oilers’ bottom six forwards.
OK. Up until now, I think I’ve got a pretty good bead on what’s going on with Sutton: the soft ice time made him look good. To an extent, I still think that’s true – when you start nicking for quality of opponents, he takes a bigger cut than other guys. Check out this graph though:
This is the percentage of the scoring chances that went the Oilers way when a given defenceman was on the ice, in the four situations outlined above. Sutton actually gives a pretty good account of himself. He’s north of 50% in three of the four situations, although his score when out with the bottom six against top six competition is pretty brutal. I don’t know that you can be entirely fair to him and say that his scoring chance rates (and, by inference, Corsi) was entirely a product of soft competition last year, because he did just fine when up against top six competition if the Oilers had their top six forwards out on the ice.
(Aside: it jumps out at me that the Oilers defencemen with mobility issues – Sutton, Peckham, Potter, Barker and Whitney – were the ones who really struggled in BT situations.)
When the chance percentages are presented in the order in which the Oilers experienced the most success (TB, TT, BB, BT), I think you can see a sort of downward progression in most cases; where a guy has a number out of step with that progression, it’s probably fair to wonder if he has some variance working for or against him. Gilbert’s a nice example of the measured progression I’m talking about. If, for example, you look at Cory Potter, you see his chance number goes 61.6%, 42.3%, 43% and then 36.9%. I don’t think I’d bet on his on-ice chance number in TB situations being 61.6% again. Similarly, Sutton did amazingly well in BB situations from a chance perspective. Petry’s got a bit of a goofy thing going on; he saw a greater chance share in BT situations than he did in TT situations.
It is, I think, important to maintain some perspective on these numbers. The samples aren’t massive when you break them down like this. For the sake of completeness, I’ve put together a graph showing the number of chances each defenceman was on-ice for in each category.
Let’s go one last step into Sutton’s season. We’ve sliced and diced it a lot of different ways but is it possible that the defensive partners that he had were somehow carrying his slow, frequently suspended behind all year? There is, I think, a general consensus that the Oilers had four defencemen who didn’t stink last year: Gilbert, Schultz, Petry and Smid. I split up Sutton’s performance in the four areas identified above into situations when he was on the ice with the “good” defencemen and the “bad” defencemen. I’m hardpressed to find much of a difference; he did slightly better with the bad ones, although it’s entirely possible that there’s a degree of granularity that my breakdown doesn’t catch.
Taken as a whole, what does all of this mean? Well if you go by Neilson numbers, you see a big divergence between the Oilers “good” four of last year and Andy Sutton. For whatever reason though, that’s not really reflected in the scoring chance numbers (and, by inference, the Corsi) when he’s on the ice. I am all for individual stats that break down the components of the game and attribute credit to individuals in a way that makes decision making better for teams in general and the Oilers in particular but the individual stats have to correlate with the results that occur on the ice.
This analysis would seem to suggest that there’s a significant disconnect between Sutton’s Neilson numbers last year (assuming they’re worth anything, given the pollution) and the scoring chance/Corsi numbers. If Sutton was markedly worse than the good defencemen group, you’d think his numbers would be terrible when you control for competition and teammates and just look at how he did when partnered with a bad defenceman. That’s not the case.
Was Sutton awful last year? I didn’t particularly like how he looked on the ice. Slicing the data a bunch of different ways though, I’m hard pressed to come up with anything that suggests that he wasn’t a useful player to have around or that he wouldn’t be a fine seventh defenceman for the 2012-13 Oilers.
* * *
(If you’re just interested in the Oilers stuff, feel free to skip to the comments now.)
This leads me to what really irritates me with what Staples is doing with Neilson numbers. It’s a variant of what the PowerScout guys are doing; what all the guys who are trying to sell snake oil to people about hockey stats are doing, overstating the certainty of their stat and misrepresenting other ways of doing things. If they don’t know better, it’s just dumb. If they do, it’s intellectually dishonest.
Rightly or wrongly, I’ve got a bit of a thing with people who advance intellectually dishonest arguments. David’s advocacy of Neilson numbers has been rife with it. He hasn’t done the work necessary to prove the concept (indeed, his numbers don’t really permit proof of concept because of the silly manner in which they were collected) but instead seems to rest on two points: a) Roger Neilson did this and b) because it attributes blame/credit to individuals, it’s better.
As to his first point, sports history is rife with examples of people who worked within the game, even smart ones, not understanding something that they were doing. Usually, it involves cases where they didn’t test the validity of their ideas against other options. Stolen bases or sacrifice bunts in baseball are a great example. As to his second point, it’s an attractive one. Of course, you’d want to test his data and ideas to see, first of all, if they’re even telling us anything different than Corsi. You’d also want to see if the data he’s collecting passes a sniff test. As we found out the other night, he’s mashing multiple game states together, something he now acknowledges is not sensible, and prevents any sensible examination of Neilson numbers on the numbers he’s gathered to date.
Dumb as those arguments may be, what really kills me is the misrepresentation of what other stats are intended to do. He either doesn’t understand what he’s talking about or he’s deliberately making dishonest argument. There’s no other way. If it’s rude to point that out, well, I don’t know what to say. Nobody forced him to get involved in the discussion or to take the position he takes or to not bother doing the work necessary to prove the merit of his concept.
According to David, “Andy Sutton was #Oilers most effective dman this year. That’s what Corsi plus-minus indicates.” He proceeded to make a similar claim about Cody Franson of the Leafs that had the exact same problem. If you read David’s site or follow him on Twitter, you’ve been seeing stuff like this all year as he’s tried to sell his stat.
There’s a problem. That’s not what Corsi indicates. You might as well say “Andy Sutton was the #Oilers most effective dman this year. That’s what his height indicates.” Height sometimes correlates with being the best defenceman – think Zdeno Chara – but not necessarily. If you know that one defenceman is 6’4″ and five other defencemen are 4’0″, you’re probably justified in making that assumption. Otherwise, you’re misusing a statistic to make an argument.
Outside of circumstances in which someone has a really massive Corsi edge over his teammates (which is not the case with the 2011-12 Oilers), it’s not appropriate to draw such a conclusion from Corsi alone. If you’re just interested in making an argument though, as opposed to answering a question, that’s the sort of thing you say.
There’s a cool Bill James piece in one of the Baseball Abstracts where he talks about the difference between sportswriters and sabermetricians. I can’t find it in my collection but some googling produced this, from a talk he gave in 2008. I’ve excerpted part of it; you should really read the whole thing. He characterizes sabermetricians as “persons who begin with a question” and sportswriters as “persons who begin with a position on the issue.”
When a person begins with the question itself he inevitably winds up confronting his own ignorance, and trying to find ways to fill in the gaps in his knowledge. The person who begins with a position on the issue never sees his own ignorance, and, in fact, deliberately avoids seeing his own ignorance. The person who begins with a position on the issue and argues for that position naturally tries to hide his ignorance of the other internal issues, since the things that he doesn’t know are a weakness in his argument. The person who begins with the question itself, on the other hand, inevitably winds up reveling in his own ignorance, celebrating his ignorance, and sharing it freely with the world at large.
But the person who begins with a position on the issue, by this process, becomes a borrower from the Bank of Knowledge. He borrows from the things that he knows, and uses them to construct an argument.
The person who begins with the issue itself, on the other hand, eventually becomes a contributor to the Bank of Knowledge. Forced to confront his own ignorance, he is forced to find ways to figure out the information that he is missing—ways to count things that haven’t been counted, or ways to estimate the parameters of things that are unknown. Through this process, he winds up knowing things that were not known before.
This is essentially what we do: We try to construct knowledge to fill in some of the spaces in our massive ignorance. We are not people who know things. We are people who are honest enough to admit that we don’t understand things, and frankly, we don’t believe that you understand them, either.
This site is written with that sort of thinking in mind – there are things I don’t know, or things that I wonder about, and I try to answer them by examining the evidence that’s available. Sometimes I’m more successful than others. I try to approach these issues with an open mind though, even when I think that I know the answer. To think, “If I’m right, what other things would be true?” Then I go and look to see whether those things seem to be true.
If David is right about Nielsen numbers, we should, whenever he gets around to doing his data properly, be able to see the impact of Nielsen duds and studs in the data. Guys should do worse or better in the chance count depending on whether they’re on the ice with them, all other things being equal. It will be the controversial ones, guys like Sutton and Ryan Jones, who will make his case because, if the stat means anything, teams will do worse and better respectively with them on the ice. That’s the acid test of whether the data he’s collecting has any meaning – the counts alone don’t mean a thing, no matter what he insists about what Roger Neilson was doing in 1975.
I am, to say the least, highly skeptical that this is the case. It makes more sense to me to think that Sutton was contributing in way that Nielsen numbers don’t catch and that Jones was screwing up in ways that they don’t catch than it does that Sutton was terrible and Jones really good but that chance counts and Corsi, looked at with an eye towards adjusting the context, don’t catch it.
If Neilson numbers are right and Sutton and Jones are so bad and good respectively, it should be reflected in other ways. David (and all the other guys with a stat to sell) would do well to ask “If this is true, what else would be true?” rather than shouting “THIS IS TRUE BECAUSE MISREPRESENTING ARGUMENT.”