• Asking Questions About Andy Sutton

    by Tyler Dellow • July 3, 2012 • Hockey • 18 Comments

    Andy Sutton had an awfully interesting season last year. He had the best Corsi amongst Oilers defencemen, although his quality of competition was pretty low. He does well by the scoring chance metric as well, coming in fourth out of the thirteen guys who played defence for the Oilers this year and second out of the seven guys who played more than forty games (Tom Gilbert, Cory Potter, Ladislav Smid, Theo Peckham, Jeff Petry and Ryan Whitney are the other six guys who meet that threshold; only Gilbert had a better scoring chance share.)

    David Staples prefers to evaluate players by Neilson numbers. The other night on Twitter, he made the following claim:

    Andy Sutton was ‪#Oilers‬ most effective dman this year. That’s what Corsi plus-minus indicates.

    Schultz/Petry/Smid/Potter/Sutton. ‪#Oilers‬ Top 5 d-men Neilson scoring chances. Sutton/Petry/Potter/Smid/Teubert. That’s Corsi. Of course, you then factor in QualComp and ZoneStarts. But I’m suggesting with Neilson’s, you start analysis in right spot.

    Now, that’s a grossly misleading claim about Corsi – you shouldn’t really use Corsi in the way he’s suggesting but I’ll discuss that in more detail below. For now, I want to dig a little more into Andy Sutton’s season. There’s a lot of research that shows that Corsi is a good proxy for scoring chances, as is addressed most recently here. I’m going to use scoring chances here as a proxy for Corsi – Sutton does well by the scoring chance metric.

    Since we’re doing this by way of a contrast with Neilson numbers, let’s take a quick look and see what they say. (Aside: if you haven’t read my most recent post about Neilson numbers and why you can’t compare them to Corsi, you should. Short version: despite explicitly making this comparison, David has collected his data in the old plus/minus way, which renders it not really appropriate to make the comparison he’s been touting all year.) David did a helpful post in April in which he looked at the Oilers defencemen. Here’s the money chart from that table:

    (Actual quote introducing this chart: “By this plus/minus system, here are the rankings of the Oilers defencemen in 2011-12 on their even strength play.” We now know that what David means when he says “even strength” is “even strength + SH pluses + PP minuses.” Of course, as of a few days ago, his position is: “Never hid this. That you’re discovering it now, well, good on you.” I guess “Never hid this” means “Never hid this…so long as we exist in a world in which words have no meaning.”)

    In any event, you see that Andy Sutton comes out rather poorly. We don’t know how badly David’s data is polluted by the SH+ and PP- but there it is. Sutton kind of stinks.

    OK. Let’s look at the scoring chance data. Here’s a chart of the scoring chance and Corsi percentages for the Oilers defencemen in 2011-12.

    No news there. It is worth mentioning that while Sutton does lead in Corsi, it’s not a large lead. It’s not particularly significant. As I alluded to above, there’s still the issue of QualComp. The problem with QualComp is that, while it’s great for sorting players, it’s less intuitive in terms of how significant those impacts are. While we don’t have a Corsi database to break down, we do have a complete set of scoring chance numbers for the Oilers.

    I went through the Oilers’ 5v5 scoring chances for and against and sorted them by whether or not the other team’s top six was on the ice or not. I determined whether the top six was on the ice by identifying two top six players on each team who didn’t play together very much. I think I did a reasonable job of this. The Oilers were 956 CF and 1057 CA against at 5v5 this year, giving them 47.5% of the 5v5 chances. (To the point about using Corsi as a proxy for scoring chances: a href=”http://timeonice.com/mplayershots1112.php?team=EDM&first=20001&last=21230″>Edmonton had a .471 Fenwick and a .476 Corsi as a whole this year, virtually bang on.)

    If I have any idea what I’m doing at identifying the other team’s best players (full disclosure: I cheated and asked the coaches by looking at who they gave the most TOI to), the Oilers will do worse against top six competition. So they did, getting 590 chances and giving up 703, a 45.6% rate. By definition, they therefore did better than their average against bottom six competition, outchancing them slightly 366-354 (50.8%). By inference, playing lots against bottom six forwards is going to help make your scoring chance numbers (and Corsi) look better.

    The Oilers defence who played a lot against top six forwards are exactly the ones you’d expect. Gilbert, Smid and Petry are all north of 70% of their chances against top six competition. Schultz and Peckham (?) are at about 65%. Potter, Sutton and Whitney are all between 57% and 60%. The real flotsam (Peckham excepted) tend to be about 50% or less against the oppositon’s top six.

    Let’s go a step deeper into this. How did our thirteen Oilers’ defencemen do against the opposition’s top six and bottom six forwards respectively?

    You can see that it’s something of a blood bath when the other team’s top six are on the ice. Only four defencemen manage to post scoring chance percentages above 47% – Gilbert, Smid, Schultz and Petry. No surprise with that. Everyone except Smid does better against bottom six forwards too. No surprise with that. Notably, Sutton got torched against top tier competition.

    I wanted to push a little further into this, so I sorted the chances by whether or the Oilers had two or more top six forwards on the ice in addition to the breakdown into top six and bottom six chances for the other team. I used RNH, Taylor Hall, Jordan Eberle, Ales Hemsky, Shawn Horcoff and Ryan Smyth as my top six forwards. This gave me four groups of chances: TT (Oilers’ top six against opposition top six), BT (Oilers’ bottom six against opposition top six), TB (Oilers’ top six against opposition bottom six) and BB (Oilers’ bottom six against opposition bottom six).

    Quick sanity check; also this is a fun little graph. If I’m doing this in any sort of a sensible fashion, the other team’s top six should do better than the Oilers bottom six and the Oilers top six should be better than the opposition’s bottom six. The graph:

    It passes the sniff test, I think. The Oilers’ top six gets barely outdone by the opposition’s top six when they go to head to head. The opposition’s top six puts the Oilers bottom six to the sword. The Oilers top six does reasonably well against the opposition bottom six and the opposition bottom six trump the Oilers’ bottom six. The results seem to be internally consistent. Also: the bottom six stinks. They’re all back next year, you say?

    Let’s turn back to the defencemen and look at them through this prism. First, we’ll look at which of the four types of ice time they were most likely to play.

    Unsurprisingly, Gilbert, Smid, Whitney, Schultz and Petry all played saw more than 40% of the chances they were on-ice for in TT ice time – Oilers top six on the ice and the other team’s top six on the ice. Somewhat surprisingly, Peckham’s also north of 40%. For all the talk that he hasn’t been the same since his concussion, it’d be interesting to see a similar breakdown of his minutes from 2010-11. I kind of wonder if he got a push this year into a quality of minutes that he just can’t handle.

    The guy who got the most ice time in situations where the Oilers’ top six were on the ice along with the opposition’s bottom six? Cam Barker. Some pigs, you just can’t put lipstick on. Sutton and Potter played healthy shares against bottom six players as well, sort of split equally between playing with the Oilers’ top six on the ice against them and with the Oilers’ bottom six forwards.

    OK. Up until now, I think I’ve got a pretty good bead on what’s going on with Sutton: the soft ice time made him look good. To an extent, I still think that’s true – when you start nicking for quality of opponents, he takes a bigger cut than other guys. Check out this graph though:

    This is the percentage of the scoring chances that went the Oilers way when a given defenceman was on the ice, in the four situations outlined above. Sutton actually gives a pretty good account of himself. He’s north of 50% in three of the four situations, although his score when out with the bottom six against top six competition is pretty brutal. I don’t know that you can be entirely fair to him and say that his scoring chance rates (and, by inference, Corsi) was entirely a product of soft competition last year, because he did just fine when up against top six competition if the Oilers had their top six forwards out on the ice.

    (Aside: it jumps out at me that the Oilers defencemen with mobility issues – Sutton, Peckham, Potter, Barker and Whitney – were the ones who really struggled in BT situations.)

    When the chance percentages are presented in the order in which the Oilers experienced the most success (TB, TT, BB, BT), I think you can see a sort of downward progression in most cases; where a guy has a number out of step with that progression, it’s probably fair to wonder if he has some variance working for or against him. Gilbert’s a nice example of the measured progression I’m talking about. If, for example, you look at Cory Potter, you see his chance number goes 61.6%, 42.3%, 43% and then 36.9%. I don’t think I’d bet on his on-ice chance number in TB situations being 61.6% again. Similarly, Sutton did amazingly well in BB situations from a chance perspective. Petry’s got a bit of a goofy thing going on; he saw a greater chance share in BT situations than he did in TT situations.

    It is, I think, important to maintain some perspective on these numbers. The samples aren’t massive when you break them down like this. For the sake of completeness, I’ve put together a graph showing the number of chances each defenceman was on-ice for in each category.

    Let’s go one last step into Sutton’s season. We’ve sliced and diced it a lot of different ways but is it possible that the defensive partners that he had were somehow carrying his slow, frequently suspended behind all year? There is, I think, a general consensus that the Oilers had four defencemen who didn’t stink last year: Gilbert, Schultz, Petry and Smid. I split up Sutton’s performance in the four areas identified above into situations when he was on the ice with the “good” defencemen and the “bad” defencemen. I’m hardpressed to find much of a difference; he did slightly better with the bad ones, although it’s entirely possible that there’s a degree of granularity that my breakdown doesn’t catch.

    Taken as a whole, what does all of this mean? Well if you go by Neilson numbers, you see a big divergence between the Oilers “good” four of last year and Andy Sutton. For whatever reason though, that’s not really reflected in the scoring chance numbers (and, by inference, the Corsi) when he’s on the ice. I am all for individual stats that break down the components of the game and attribute credit to individuals in a way that makes decision making better for teams in general and the Oilers in particular but the individual stats have to correlate with the results that occur on the ice.

    This analysis would seem to suggest that there’s a significant disconnect between Sutton’s Neilson numbers last year (assuming they’re worth anything, given the pollution) and the scoring chance/Corsi numbers. If Sutton was markedly worse than the good defencemen group, you’d think his numbers would be terrible when you control for competition and teammates and just look at how he did when partnered with a bad defenceman. That’s not the case.

    Was Sutton awful last year? I didn’t particularly like how he looked on the ice. Slicing the data a bunch of different ways though, I’m hard pressed to come up with anything that suggests that he wasn’t a useful player to have around or that he wouldn’t be a fine seventh defenceman for the 2012-13 Oilers.

    * * *

    (If you’re just interested in the Oilers stuff, feel free to skip to the comments now.)

    This leads me to what really irritates me with what Staples is doing with Neilson numbers. It’s a variant of what the PowerScout guys are doing; what all the guys who are trying to sell snake oil to people about hockey stats are doing, overstating the certainty of their stat and misrepresenting other ways of doing things. If they don’t know better, it’s just dumb. If they do, it’s intellectually dishonest.

    Rightly or wrongly, I’ve got a bit of a thing with people who advance intellectually dishonest arguments. David’s advocacy of Neilson numbers has been rife with it. He hasn’t done the work necessary to prove the concept (indeed, his numbers don’t really permit proof of concept because of the silly manner in which they were collected) but instead seems to rest on two points: a) Roger Neilson did this and b) because it attributes blame/credit to individuals, it’s better.

    As to his first point, sports history is rife with examples of people who worked within the game, even smart ones, not understanding something that they were doing. Usually, it involves cases where they didn’t test the validity of their ideas against other options. Stolen bases or sacrifice bunts in baseball are a great example. As to his second point, it’s an attractive one. Of course, you’d want to test his data and ideas to see, first of all, if they’re even telling us anything different than Corsi. You’d also want to see if the data he’s collecting passes a sniff test. As we found out the other night, he’s mashing multiple game states together, something he now acknowledges is not sensible, and prevents any sensible examination of Neilson numbers on the numbers he’s gathered to date.

    Dumb as those arguments may be, what really kills me is the misrepresentation of what other stats are intended to do. He either doesn’t understand what he’s talking about or he’s deliberately making dishonest argument. There’s no other way. If it’s rude to point that out, well, I don’t know what to say. Nobody forced him to get involved in the discussion or to take the position he takes or to not bother doing the work necessary to prove the merit of his concept.

    According to David, “Andy Sutton was ‪#Oilers‬ most effective dman this year. That’s what Corsi plus-minus indicates.” He proceeded to make a similar claim about Cody Franson of the Leafs that had the exact same problem. If you read David’s site or follow him on Twitter, you’ve been seeing stuff like this all year as he’s tried to sell his stat.

    There’s a problem. That’s not what Corsi indicates. You might as well say “Andy Sutton was the #Oilers most effective dman this year. That’s what his height indicates.” Height sometimes correlates with being the best defenceman – think Zdeno Chara – but not necessarily. If you know that one defenceman is 6’4″ and five other defencemen are 4’0″, you’re probably justified in making that assumption. Otherwise, you’re misusing a statistic to make an argument.

    Outside of circumstances in which someone has a really massive Corsi edge over his teammates (which is not the case with the 2011-12 Oilers), it’s not appropriate to draw such a conclusion from Corsi alone. If you’re just interested in making an argument though, as opposed to answering a question, that’s the sort of thing you say.

    There’s a cool Bill James piece in one of the Baseball Abstracts where he talks about the difference between sportswriters and sabermetricians. I can’t find it in my collection but some googling produced this, from a talk he gave in 2008. I’ve excerpted part of it; you should really read the whole thing. He characterizes sabermetricians as “persons who begin with a question” and sportswriters as “persons who begin with a position on the issue.”

    When a person begins with the question itself he inevitably winds up confronting his own ignorance, and trying to find ways to fill in the gaps in his knowledge. The person who begins with a position on the issue never sees his own ignorance, and, in fact, deliberately avoids seeing his own ignorance. The person who begins with a position on the issue and argues for that position naturally tries to hide his ignorance of the other internal issues, since the things that he doesn’t know are a weakness in his argument. The person who begins with the question itself, on the other hand, inevitably winds up reveling in his own ignorance, celebrating his ignorance, and sharing it freely with the world at large.

    But the person who begins with a position on the issue, by this process, becomes a borrower from the Bank of Knowledge. He borrows from the things that he knows, and uses them to construct an argument.

    The person who begins with the issue itself, on the other hand, eventually becomes a contributor to the Bank of Knowledge. Forced to confront his own ignorance, he is forced to find ways to figure out the information that he is missing—ways to count things that haven’t been counted, or ways to estimate the parameters of things that are unknown. Through this process, he winds up knowing things that were not known before.

    This is essentially what we do: We try to construct knowledge to fill in some of the spaces in our massive ignorance. We are not people who know things. We are people who are honest enough to admit that we don’t understand things, and frankly, we don’t believe that you understand them, either.

    This site is written with that sort of thinking in mind – there are things I don’t know, or things that I wonder about, and I try to answer them by examining the evidence that’s available. Sometimes I’m more successful than others. I try to approach these issues with an open mind though, even when I think that I know the answer. To think, “If I’m right, what other things would be true?” Then I go and look to see whether those things seem to be true.

    If David is right about Nielsen numbers, we should, whenever he gets around to doing his data properly, be able to see the impact of Nielsen duds and studs in the data. Guys should do worse or better in the chance count depending on whether they’re on the ice with them, all other things being equal. It will be the controversial ones, guys like Sutton and Ryan Jones, who will make his case because, if the stat means anything, teams will do worse and better respectively with them on the ice. That’s the acid test of whether the data he’s collecting has any meaning – the counts alone don’t mean a thing, no matter what he insists about what Roger Neilson was doing in 1975.

    I am, to say the least, highly skeptical that this is the case. It makes more sense to me to think that Sutton was contributing in way that Nielsen numbers don’t catch and that Jones was screwing up in ways that they don’t catch than it does that Sutton was terrible and Jones really good but that chance counts and Corsi, looked at with an eye towards adjusting the context, don’t catch it.

    If Neilson numbers are right and Sutton and Jones are so bad and good respectively, it should be reflected in other ways. David (and all the other guys with a stat to sell) would do well to ask “If this is true, what else would be true?” rather than shouting “THIS IS TRUE BECAUSE MISREPRESENTING ARGUMENT.”

    About Tyler Dellow

    18 Responses to Asking Questions About Andy Sutton

    1. Woodguy
      July 3, 2012 at

      I still remain convinced that Jones does better by David’s metric becuase he is often not in position to “make an error”

      Flying the Dzone before possesion is established or being the last one back into the Dzone takes him out of the picture to make an obvious mistake.

      His teammates pay for his absence by basically playing 4v5 for 1/3 of his shift.

      Decent PKers though.

      Good hair for the Edmonton market as well.

      • Colby Cosh
        July 3, 2012 at

        I share this suspicion. Is Jonesy even in camera range when these whoopsies are being counted, or is he off sneaking a sip of Happy Pop at the bench and hoping somebody springs him loose on a takeaway? It was at live games I noticed how reluctant he is to backcheck.

        • Darren
          July 4, 2012 at

          Don’t forget about looking for that ‘huge hit’ 5 seconds after the puck has left a defenceman’s stick that the crowd loves, but in reality just takes him out of position be be, you know, useful.

          • dawgbone
            July 6, 2012 at

            Plus he’s usually lying on the ice after he throws one of them, which puts him even further back.

            There was good Ryan Jones last year, the one who had 2 goals in his first 15 or 16 games, then there was the one who sacrificed his team for his goal total the last 3/4 of the season.

      • Hammers
        July 3, 2012 at

        I think Mr Jones may be in trouble with Ralph as coach . Maybe they should trade him while he has decent numbers as I think they go down this year.

    2. DerekZ
      July 3, 2012 at

      “This site is written with that sort of thinking in mind – there are things I don’t know, or things that I wonder about, and I try to answer them by examining the evidence that’s available. Sometimes I’m more successful than others. I try to approach these issues with an open mind though, even when I think that I know the answer. To think, “If I’m right, what other things would be true?” Then I go and look to see whether those things seem to be true.”

      This paragraph says so much towards why a certain segment of people do what they do. The conversations I have on IM with guys like Eric T. are so far ahead of what the snake oil guys are selling that it’s mind-boggling. Each time the open source guys (you included) find something new, they explore it, study it, compare it.

      They certainly don’t hide their methodologies and deliberately mislead those who read their work.

    3. Stocc
      July 3, 2012 at

      Reading the Bill James bit at the end felt astoundingly familiar to me. If you didn’t think “persons who begin with a question” vs. “persons who begin with a position on the issue” felt a lot like any classic science vs. religion debate then you probably haven’t paid attention to any classic science vs. religion debate.

    4. speeds
      July 3, 2012 at

      If Corsi% correlates really well chance%, over the long term, is it fair to say that Peckham is much closer to Schultz than it looks from the chance numbers alone?

      Peckham: corsi% 45.5, chance% 41.1

      Schultz: corsi% 45.7, chance% 52.2

      • Tyler Dellow
        July 3, 2012 at

        That’s a great question speeds and one I’m sort of wondering myself. It seems to me that what we would want to know whether Corsi moves towards chance percentage over time or the other way around. Instinctively, I’d think that chances would track towards Corsi – if we know that they tend to be similar in the long run and that chances are more likely to happen the more you’re in the right end of the rink, it seems rational to think that Corsi is a better predictor of chance share going forward where it is running at a significantly higher or lower rate.

        • Tach
          July 3, 2012 at

          This would also make sense to me (Chances track to corsi) as Corsi would have the larger set of data per game so variance would flush out in fewer games. Plus, it seems to me that scoring chances would have the larger tracking error (harder to distinguish a scoring chance from a shot directed towards the net) so it would further be subject to increased variance in lower sample sizes.

    5. Kris
      July 3, 2012 at

      Great post Tyler.

      Corsi requires context to be a useful predictor of player performance or a useful judge of the impact the player will have helping a team at ES. Staples doesn’t like this. He implies that any stat that requires context is worse than one that doesn’t. But of course lots of useful stats require context. For example, a player’s goal totals for a single year are not such a good predictor for future performance. With the context of TOI, SH%, ZoneStart,etc. then you have a better predictor and a better measure of the player’s performance.

      I mean it’s obvious that team effects create statistical noise even in stats that observe just the individual. Would Staples agree to the following: The goals scored stat once showed that Cheechoo was close to the best player in the league. He was not. Therefore, goals is a useless stat.” That would be absurd. Goals scored tell us a lot, but we need context. (This is even more true of evaluating players at the major junior level.)

      In fact Staples reasonning is so absurd that you can no longer treat him charitably or expect a discussion with him to be fruitful. The only way to persuade him will be via an authority he can’t dismiss.

      As I tried to say in the last post, you need an academic statistician, or a well-respected baseball-stats guy to call him out on this. Hopefully, if he’s called out publically by the experts he’ll quiet down.

      Think of the global warming debate or debates over the Kennedy conspiracy theory. You can’t win an argument with people who aren’t interpreting you charitably and arguing honestly and openly. He best you can do is show that everyone who is well respected on this issue disagrees with the quack theory in question. Then you move one.

    6. July 3, 2012 at

      With reference to the Bill James quote, I think it is important to acknowledge the spectrum that runs from say Don Cherry to say Dellow. The vast, vast majority of MSM sit down near Cherry. While Staples isn’t up near Dellow, he is one of the only MSM guys making an attempt to move across the spectrum.

      I don’t see anything intellectually dishonest in what Staples is doing. He seems to be progressing up the learning curve, making mistakes, and trying to better understand the game. I assume the stuff on this site is helpful to him. I certainly find it all very interesting to read and think about.

      In terms of ‘project Sutton is vastly overpaid’, does this new information have any bearing?

      • Tyler Dellow
        July 3, 2012 at

        In terms of ‘project Sutton is vastly overpaid’, does this new information have any bearing?

        I think it’s fair to wonder whether or not players of his ilk are accurately paid. That being said, I stand by the point I made about how guys like him tend to be paid.

        I don’t see anything intellectually dishonest in what Staples is doing. He seems to be progressing up the learning curve, making mistakes, and trying to better understand the game. I assume the stuff on this site is helpful to him. I certainly find it all very interesting to read and think about.

        I’ve taken an interest in soccer over the past few years. I started from a position of having little knowledge about the tactical stuff, although I think it’s interesting and want to learn more about it. That being said, I haven’t started making claims about tactics and data with very little evidence to support me. We went through this initially with the error stat with David; he was adamant about it’s worth. That seems to have disappeared. Now we’re into this.

        It is absolutely intellectually dishonest to denigrate, for lack of a better word, a competing stat, by misusing it. If you don’t know better fine. He’s been told a number of times though but he can’t seem to be bothered to start dealing with the stat as it’s supposed to be used.

      • Kris
        July 3, 2012 at

        Staples is also better than Satan (not Miroslav, but the guy with the red tail who isn’t Hellboy). Kudos to him for that, but it’s irrelevant. Global warming deniers are better than Nazis, but they’re still wrong and either intellectually dishonest or incapable. He’s arguing something absurd and without foundation. And he’s not admitting the flaws in his own position.

        • July 3, 2012 at

          He’s also better than any other MSM writing about the Oilers. He believes in global warming but holds onto an inferior explanation as the cause. He also makes mistakes, some stubbornly, trying to use a better explanation.

          I am not sure why I am somewhat attempting to defend the guy. I suppose I think it worth suggesting that it might be too early to write him off as a lost cause. I think Tyler is simply debating the issue, but some of the early comments are suggesting that Staples draws little Hitler mustaches on sleeping babies.

    7. art vandelay
      July 4, 2012 at

      Shorter version (aka reality): Andy Sutton is a barely competent NHL defencemen who barely dresses half the time for a perennial last-place club where the back end is a chronic problem; sorta like colon cancer.

    8. Matt K
      July 4, 2012 at

      I kind of gave up following the Oilers closely for a number of years (was in France for the cup run) but when I started to follow things again I started with Staples. I’ve always respected what he’s trying to do, and have usually been one of his defenders–particularly because he gets attacked quite frequently, and sometimes maliciously, on Twitter. It’s getting harder, however, to defend some of what he does. Tyler’s absolutely right about Staples’ misuse of corsi, and that makes me wonder what else Staples is misusing. He was one of the leading figures in the Souray lynch mob, as well. I used to think Tyler was just doing a character assassination on Staples because of Dennis’s scoring chance thing, but there’s definitely more to it than that. There’s something of a Russian novel in all of this.

      Beyond that, this whole post makes me think that Sutton certainly isn’t the problem. He’s far from being the solution, but there are other things that are much worse. Most frightening is the chart that shows how wretched the bottom six forwards, who are all coming back, have been and the fact that the Minnesota golden boy Tom Gilbert, who heads up almost every chart, is no longer on the team.

    9. JJ
      July 4, 2012 at

      Tyler,

      Have you ever looked at Fedor Tyutin , he might be a player Edmonton could trade for.

    Leave a Reply

    Your email address will not be published. Required fields are marked *