Colby Cosh had a good line a while back about the NHL and statistics (he was specifically talking about high event versus low event players, but the quote is of general application):
This will be the next stat revolution in hockey when the league finally realizes it already has the data–it just needs to present them to the fan. (Ideally, in some intelligible form that encourages number-crunching, as opposed to making it a bigger pain in the ass than rectal cancer.)
The NHL has presented more data to fans this year than ever before and they should be commended for that. They should be vehemently criticized though, for the fact that it’s harder to compile than ever. I used to have a spreadsheet with a VBA script that did most of the dirty work for me; it’s been rendered useless by the changes to the site. One of my favourite developments is an apparently new policy whereby they present different information on the event sheets and game sheets from game to game. For instance, they started presenting shots broken down by game state this year. It’s not compiled anywhere on their site (FYI: they seem to have difficulty compiling information correctly as well) so if you’re interested, you need to scrape it. Of course scraping it is a pain in the ass because you don’t know where it will be from game to game. As a source of anything more than general statistics, their site is borderline useless. I can’t believe that this would be that difficult to fix; hire a computer science student for a summer and turn him loose. They would do well to pay attention to MLB.com, who are currently making the Pitch f/x data available free of charge almost instantaneously, in an easy to grab format.
Anyway, as a result, I’ve been just grabbing the Oilers PBP files as the season went along. There’s a ton of interesting stuff in there, fodder for more than a few posts over the course of the summer. Today, there’s something that’s proven popular as we’ve gone along throughout the season: ES SF/SA data for all Oilers. I’ve split it into the two halves of the season for the sake of discussion.
If that’s too small for you, pop over to Flickr and look at the large version.
I’m just going to offer a series of stream of consciousness thoughts on the whole thing:
- I’d be leery of signing GlenX to a contract on the basis of last year’s numbers and thinking that he can make the jump to being a second or third line guy. I made the point a few times last year when people were talking about him replacing Torres that he was doing it against nobodies and it looks like he was having a big year as far as the percentages go while he did so, with a 13% shooting percentage when he was on the ice. That’s completely unsustainable. I like him a lot but if I was running the show, I suspect he wouldn’t end up coming back because I’m probably not willing to pay him what somebody else is.
- It’s going to be a few years before we have any sort of consensus on the question of player impact on save percentage but I continue to think that it’s going to be pretty small. I fiddled around with looking for a relationship between first and second half save percentages when a given player was on the ice but was unable to find anything. It’s going to be a tricky question to answer because roles and teammates change over the course of the season but there was nothing there with the Oilers this year; if anything, the relationship between save percentage in the first and second halves was a negative one.
- Accordingly, trumpet Stortini at your own risk (although those few who did have been out on a limb all along and are presumably fine with that). The Oilers fourth line was absolute dynamite down the stretch; it was really all percentages.
- The Pensky duo really fell off in the second half. I’m guessing that a lot of the blame for this can be heaped on the Horcoff injury.
- It’s not surprising that Pouliot’s EV+10 EV-4 in the second half was percentage aided. The small sample +21 in shot differential is nice though. I’m getting to the point where I can’t remember who was playing with who but I don’t think he was playing with anyone who would have been driving that number for him.
- I’ve seen Nilsson get a lot of credit in some places for his fine +/-; interesting to see that he was riding the save percentage bus all year long, right beside Stortini.
- I know that he was coming off a broken leg in the second half of the season but how long can Matt Greene be a bad third pairing defenceman for without losing his spot on the team? Some day in the future, it’d be interesting to know whether the Oilers braintrust thinks that the development of a (hopefully) middling third pairing defenceman justified the critically reviled Spring 2006 Matt Greene Festival of Penalties.
- Dennis has made this point before, I think, but while Pisani was back on the ice, he wasn’t back.
- Gagner’s improvement in the second half looks more likely to be repeated than Cogliano’s to me.
My early thoughts on next year’s team is that it could be better than this year’s team and not do as well. While I think that there was some real improvement in terms of the puck spending more time in the right end of the ice, I also tend to think that there was a big jump in terms of percentages that won’t necessarily repeat next year in the second half. It’s interesting looking at this list, trying to figure out where the lines that are going to dominate are going to be for the Oilers next year. I’m thinking that, at best, they can hope that the first and fourth lines can play with the opposition on roughly equal terms. The second line, likely Gagner/Nilsson/Cogliano, looks like it’s going to probably get outshot to me, although they might be able to get by on shooting percentage; I think Gagner is probably one of those guys who raises the shooting percentage when he’s on the ice. In order for them to experience any real ES success, the third line, say Pisani/Torres/Stoll, is going to have to handle their opposition and come out on top. That’s an awful lot to heap on a guy coming off colitis, a guy who missed half a year with a knee injury and a guy coming off an atrocious year who still looks like he’s suffering from a concussion. Of course, if Mathieu Garon isn’t playing 60 games in 2007-08 form or better, none of it will matter.
EDIT: I’m just fiddling around with some of the cooler things that you can do with the shot data. Here’s something that I think is interesting, with the possibility of developing useful information - you can figure out things like the ratio of shots for/against for a forward with a given defenceman on the ice and get closer to isolating the impact of certain players. For example: Horcoff in 07-08 with Tom Gilbert 157 SF/153 SA for a 1.026 ratio. With Staios: 124/133 (0.932). With Pitkanen: 115/107 (1.075). With Smid: 72/92 (0.783). With Souray: 81/72 (1.125) With Greene 30/40 (0.75). It’s interesting to me how consistent he is with Gilbert, Staios, Pitkanen and Souray and how bad the ratio is when Smid and/or Greene are on the ice, particularly when you consider that Smid and/or Greene were presumably playing softer minutes than the aforementioned four.
By way of contrast, let’s look at Tom Gilbert with the various centres. With Horc: 157/153 (1.026). With Stoll 157/199 (0.789). With Gagner: 151/199 (0.759). With Brodziak: 134/155 (0.865). It’s intriguing to me because those four guys probably played dramatically different levels of competition, with Stoll/Horcoff at the top (pretty amazing difference between the two with Gilbert on the ice), with Brodziak following and Gagner bringing up the rear and yet Gilbert posted his best results with Horcoff. One thinks that you could do some pretty cool stuff with this.
In the second half, when Matt Greene played his best, it looks like the team had a high save percentage when he was on the ice. He was second only to another guy I regard as a defensive stalwart, Stortini.
So we could say Greene (and Stortini) just got lucky, and all this will even out, and the team will let in more goals when he is on the ice next year. This is a very real possibility.
Or we could say his superior save percentage in the second half might indicate that Greene (and Stortini) turned a corner and did a much better job of limiting strong scoring opportunities, that he didn’t wander away from guys in the slot, but knocked them over.
His high save percentage lines up with my observation that when he came back from injury, Greene made very few crucial mistakes that directly contributed to goals against the Oilers, and he made such mistakes far less often (on a per minute basis) than a player like Pitkanen.
Plus he hits the hell out of the bad guys (Kesler), especially when they pick on the good guys (Hemsky) . . .
Yet so many Oiler fans are down on Matt Greene (and Stortini). Must be seeing something that I’m missing.
P.S. Excellent and provocative work, these last few posts.
Outstanding stuff, MC, lots to chew on here.
Accordingly, trumpet Stortini at your own risk (although those few who did have been out on a limb all along and are presumably fine with that). The Oilers fourth line was absolute dynamite down the stretch; it was really all percentages.
I’ve been on that limb from the start, and I’m very fine with that. In fact I feel like I’ve crawled up from that branch from the most fragile new growth to a much thicker and more stable perch. Stortini more than rewarded my vehement early defence of his particular “skill set” (for lack of a better term) by developing into a decent player much faster than the most wild-eyed optimist — me, presumably — could possibly anticipate.
And I disagree that it’s all percentages. Those certainly improved dramatically over the course of the season, but check out those shots totals. The Oilers were outshot 119:60 with Zack on the ice in the first half of the season; by the second half, Oilers were actually doing the outshooting with 46 on the ice, albeit by a more moderate margin of 158:148. Still, that’s a stunning turnaround no matter how you slice it. (Maybe Zack just needed to get away from Stoll and Torres?
)
Even more staggering was the change in goal differential, esp. GF which soared from just 2 in the first half to 19 in the second. This was a product of both the increased shots and the way better percentages. While I agree that Sh% of 12.0% is likely too good to sustain long term, at the same time that horrifying 3.3% from the first half is unsustainably bad. The season-long net Sh% of 9.6 is a lot more realistic than that of either half, not far from the NHL average in fact.
It’s going to be a few years before we have any sort of consensus on the question of player impact on save percentage but I continue to think that it’s going to be pretty small.
I agree that will be an extremely interesting question as data continues to accumulate, but I’m less sure the effect will be small. The range may be relatively narrow, but I expect the top defenders to consistently outperform the weaker ones by this metric. It will, however, be hard to separate the figure(s) from the ground, as the best defensive players will often draw the top opposition snipers — the same problem that cause headaches in “traditional” data such as +/-.
Again, let’s use Stortini as an example, since the team Sv% of .933 behind him was quite exceptional and no doubt unsustainable. But he was/is a responsible defensive player, almost always was between his man and the net, rarely allowed odd-man rushes. His biggest defensive weakness was to be sucked down low and to open up his point man at times, but outside shots into a wall of defenders are the way pf the new NHL, esp. as envisioned by Craig MacTavish, and a good Sv% is to be anticipated. Not to be discounted is that Zack’s opponents were often plumbers.
So certainly we can expect the Sv% behind 46 to normalize next year, but perhaps still be a little better than team average due to those factors. That said, as I’m sure Vic Ferrari will agree, a bad bounce or five over the course of a season can make a mockery of such micro-analysis of stats. This past season it’s fair to say Zack got more good bounces than bad, but down the stretch he was more than hitting his weight by any metric.
I fiddled around with looking for a relationship between first and second half save percentages when a given player was on the ice but was unable to find anything. It’s going to be a tricky question to answer because roles and teammates change over the course of the season but there was nothing there with the Oilers this year; if anything, the relationship between save percentage in the first and second halves was a negative one.
Just glancing at the defencemen the numbers are indeed erratic, but maybe there’s a kernel of truth in there nonetheless. The two defencemen who showed a huge improvement were Greene (+.038) and Smid (+.036), and that matches what I saw. Greene played a much stronger game after his second-half return from the busted ankle, while Smid stopped losing his man for goalmouth tap-ins (a.k.a. high percentage shots). Staios (+.018) also showed a nice improvement, in part because he too was victimized by many of those backdoor plays that burned his frequent first-half partner Smid. Grebeshkov (+.015) significantly reduced his high-risk blundering after New Year’s. Meanwhile Gilbert (-.016) and Pitkanen (-.018) had their defensive struggles in the second half.
(And all of that has to be taken with the huge grain of salt that my own observation of their defensive performance was results-driven to a significant extent. Those same GA that had me cursing their performance at times would also be reflected in those Sv% stats.)
Perhaps a more impartial method would be to compare the changes in Sv% for the individual defencemen against Staples’ assignment of defensive errors. I recognize that the fatal flaw is that he too only examined actual goals against — if the goalie made the save everybody is off the hook. But he tried to single out goal-causing mistakes, as opposed to merely being on the ice when a goal was scored, whereas on-ice Sv% makes no such distinction.
Best I could find for a mid-season error summary on Cult of Hockey was after 45 games. Here’s a breakdown of errors per game in those 45 games vs. the last 37; followed in parentheses by the change in per-game error rates (+ indicating an improvement, not an increase in errors!) and the change in Sv%.
Pitkanen: 0.47; 0.70 (-0.23; -.018)
Gilbert: 0.27; 0.62 (-0.35; -.016)
Staios: 0.33; 0.32 (+0.01; +.018)
Grebeshkov: 0.37; 0.33 (+0.04; +.015)
Smid: 0.43; 0.27 (+0.16; +.036)
Greene: 0.23; 0.12 (+0.11; +0.38)
Pitkanen and Gilbert committed far more errors in the second half, and the Sv% behind them went down significantly. Staios and Grebeshkov cut down on their errors a little bit, and the Sv% went up. Smid and Greene cut down on their errors by a lot, and the Sv% behind them went way up. Seems like a pretty decent correlation to me.
For example: Horcoff in 07-08 with Tom Gilbert 157 SF/153 SA for a 1.026 ratio. With Staios: 124/133 (0.932). With Pitkanen: 115/107 (1.075). With Smid: 72/92 (0.783). With Souray: 81/72 (1.125) With Greene 30/40 (0.75). It’s interesting to me how consistent he is with Gilbert, Staios, Pitkanen and Souray and how bad the ratio is when Smid and/or Greene are on the ice, particularly when you consider that Smid and/or Greene were presumably playing softer minutes than the aforementioned four.
Those numbers speak for themselves: Horc had way more trouble with Smid and Greene. But your last comment is piling on; since all the minutes being studied are with Horcoff on the ice, they likely aren’t very “soft”. The results may explain why MacT tried to soften the rest of their minutes, cuz 2 and 5 didn’t quite measure up with the big guns. (Bear in mind too that Horc missed the last two months when Greene and Smid played their best hockey of the season; I’d be interested to see similar numbers aligned with forwards who played the whole season.)
Does your data also include TOI of each guy with Horcoff? SF/SA per 60 would be interesting and might indicate whether the shortcomings were at the defensive or the offensive end of the ice. With Smid and Greene it’s not difficult to hazard a guess.
And all that said, I totally agree with your larger point in the edit that there’s lots of cool possibilities. It’s a statistical gold mine … when it isn’t a mine field. Thanks for this, and please keep us posted with your further results.
Ha! only after I submitted my reply did I see Staples beat me to the punch on the possible relationship between on-ice Sv% and errors. My own observations on this subject were derived completely independently from his comment immediately above, which to my mind strengthens the point.
I especially agree with David’s concluding remark:
P.S. Excellent and provocative work, these last few posts.
This is great Tyler. I’m rereading Moneyball, and I especially like the part where the amateur (Mccracken?) determines that pitchers have little to no effect at all what happens to balls once they’re put in play.
I’d love to see what sort of impact players have on the quality of shots on net, though that might be asking too much right now.
Also… interesting to note that Gilbert’s sv% behind him went from .925 to .909 between the halves. I know it was the second when people started to think he struggled, makes me wonder if that was aided by the amount of goals going in the net in the second half.
One of the reasons I appreciate this sort of statistical analysis so much is that I’m well aware of how much inherent bias we have in our own human minds, how we remember things incorrectly and things that suit our own predermined biases. I’m not saying that criticism of Gilbert in the 2nd half in unfounded, but I do like the way stats can prove how utterly wrong minds-eye observations are.
Aye, Julian, there’s the rub. Were Gilbert’s defensive struggles the result of poorer goaltending behind him, or did the opposition’s Sh% rise because Gilbert’s unit gave up more ten-bell chances? There’s room on both sides of that discussion, and it may be difficult to find the middle ground. The inherent biases we all have pertain not just to how we view the game, but how we interpret the statistics themselves.
His high save percentage lines up with my observation that when he came back from injury, Greene made very few crucial mistakes that directly contributed to goals against the Oilers, and he made such mistakes far less often (on a per minute basis) than a player like Pitkanen.
The only problem I have with this is that you’re only assigning errors when there’s a goal scored. I’d think that on any sequence when the opposition gets a chance in the Oilers end, errors are made. I appreciate what you guys are trying to do with the error stat and accept that there are limits on what can be done but I have a hard time seeing how you can find any causation here.
For what it’s worth, I wasn’t wildly impressed with Greene in the second half, although, as one of you or Bruce noted, it’s all subjective. My three defining plays for him in the second half are that shot to Kesler’s face, which felt good but should have been a penalty, seeing Ryan Shannon of all people drop a move on him and seeing him get beat wide by a nobody. Again, we need more years of data and only time will address that.
Yeah, I realize that Bruce, it’s a bit of a chicken or the egg thing. It’s why I wish we had some idea of how much impact a player has on shot quality against.
Best I could find for a mid-season error summary on Cult of Hockey was after 45 games. Here’s a breakdown of errors per game in those 45 games vs. the last 37; followed in parentheses by the change in per-game error rates (+ indicating an improvement, not an increase in errors!) and the change in Sv%.
Are thoes errors per game or per EV60, because if they’re per/game, the icetime leaders (ie Pitkanen, Gilbert, Staios) are going to get unfairly dinged, while guys lower on the totem pole (ie Smid, Greene) are going to be unfairly advantaged.
Are thoes errors per game or per EV60, because if they’re per/game, the icetime leaders (ie Pitkanen, Gilbert, Staios) are going to get unfairly dinged, while guys lower on the totem pole (ie Smid, Greene) are going to be unfairly advantaged.
Jonathan: They are per-game. I had no way of finding ES minutes after 45 games. The point wasn’t to compare the players against each other but against themselves, on the assumption being they are each playing about the same amount throughout. Both the errors and team Sv% metrics indicate Gilbert and Pitkanen got worse in the second half, Staios and Grebeshkov got better, and Greene and Smid got much better.
However, since you brought it up, here are the full-season error rates per EV 60 of the Oilers defenders:
Pitkanen 1.98
Grebeshkov 1.55
Gilbert 1.47
Smid 1.40
Staios 1.02
Souray 0.86
Greene 0.66
Tyler,
You are right — errors are constantly made that lead to excellent scoring chances, and they are not defined as “errors” by my system because the goalie makes the save.
I define only a few defensive blunders as “errors: — as “primaries” and “secondaries” — only the ones that lead to the puck going into the net. And, of course, this is a limitation of this particular stat.
That said, on first glance, there does seem to be some odd relationship here between “errors and save percentage. Over a stretch of games, a player who makes few errors also has a high defensive save percentage, it appears.
Upon reflection, I don’t know what to make of it, if anything (and I’m going to look more closely to see if, in fact, this is true).
But as a starting point, I’m still trying to figure out what offensive shooting percentages and defensive save percentages say about a specific player. Perhaps you can assist here.
It’s generally accepted that a goalie with a high save percentage is doing a good job.
But does the same hold true for defencemen and forwards with strong save percentages? Is this an indication that they are excellent defenders, or are they simply lucky, perhaps out there on the ice for a disproportionate amount of chances when the goalie happens to make great saves?
Here is a hypothetical . . .
Let’s say that when Defenceman A is on the ice, year after year, the save percentage is around .925, while for Defenceman B, the save percentage is around .905 year after year.
Can we conclude that Defenceman A is doing more defensively (such as clearing the front of the net) than Defenceman B to help his team prevent quality scoring chances, the kind that lead to goals against?
I suppose, if we want to rate the two players, we’d have to know the total number of shots being allowed as well, whether Defenceman A is also the ice for a lot fewer shots than Defenceman B.
Do we see top defenders, such as Lidstrom and Pronger, post high save percentages year after year?
P.S. And thanks again for providing this information about the Oilers players. I’m gathering from your comments that it’s not readily available for other teams, so it’s no easy thing to find the save percentages for other NHLers.
Jonathan: In my errors vs points plus/minus standings, I’ve broken down the stat on an errors per even strength minute basis.
Here are the numbers for the first 45 games.
1. Allan Rourke, one error every 137 minutes (13 games played, 1 error, 137 even strength minutes played).
2. Sheldon Souray, one error every 75 minutes (19 games played, 4 errors, 300 even strength minutes played).
3. Tom Gilbert, one error every 67 minutes (45 games played, 12 errors, 800 even strength minutes played).
4. Matt Greene, one error every 57 minutes (13 games, 3 errors, 172 even strength minutes played).
5. Steve Staios, one error every 53 minutes (45 games, 15 errors, 798 even strength minutes played).
6. Dick Tarnstrom, one error every 50 minutes played (26 games, 8 errors, 396 even strength minutes played).
7. Joni Pitkanen, one error every 39 minutes played (30 games, 14 errors, 543 minutes played).
8. Mathieu Roy, one error every 38 minutes played (four games, one error, 38 even strength minutes played).
9. Ladislav Smid, one error every 37 minutes played (35 games, 15 errors, 553 even strength minutes played).
10. Denis Grebeshkov, one error every 35 minutes played (38 games, 14 errors, 490 even strength minutes played).
As the season went on, Pitkanen and Gilbert’s errors per minutes crashed low, while Greene, Smid, Staios and Greb all improved.
As the season went on, Pitkanen and Gilbert’s errors per minutes crashed low, while Greene, Smid, Staios and Greb all improved.
Thanks David. Combining the two lists above we can find error rates for both halves (actually 45/37) of the season. I prefer the errors per 60 method over minutes per error, so here goes:
Pitkanen: 1.55 + 2.38 = 1.98
Grebeshkov 1.71 + 1.38 = 1.55
Gilbert: 0.90 + 2.20 = 1.47
Smid: 1.63 + 1.11 = 1.40
Staios: 1.13 + 0.90 = 1.02
Souray: 0.80 + 1.02 = 0.86
Greene: 1.05 + 0.52 = 0.66
No stat is an island, and errors are no exception. No doubt some of those variations are due to QUALCOMP: a top opponent is more likely to make you pay for a mistake. Gilbert had tougher minutes in the second half of the season, but the trend from 0.90 E/60 to 2.20 is worrisome to say the least. It also doesn’t explain why Pitkanen and Gilbert had over four times the error rate of Greene in the second half.
Bruce
For sure . . . the “error” stat doesn’t take into account Quality of Competition, a huge factor when you’re trying to determine what a player is doing on the ice.
And I’ve also yet to establish “rater reliability” when it comes to the error, any real proof that various experts trained on the same set of criteria for rating errors can look at the same goal against a team and generally assign the same errors to the same players.
But I’m planning on doing a study with other willing fans on this next season, to see if “rater reliability” can be established.
I’ve seen other areas where “rater reliabity” is crucial.
In these areas, a set of criteria are used to make observations and determinations, such as whether or not a criminal is a psychopath.
Pyschologists use something call the PCL-R to rate people on a psychopathy checklist. Subjects are studied to see if they have certain personality traits that psychopaths are known to have, based on decades of close study.
Psychologists have come to believe that if you properly train a psychologist, that person will rate a criminal on the PCL-R with generally the same score as other trained psychologists.
This is how “rater reliability” is established, and the results are deemed to be valid enough that they are accepted as evidence in criminal court sentencing hearings.
All that said, I’m not sure if rater reliability can be established when it comes to assigning “errors.” It will take some work, trial and error . . .
Ty: To be fair, errors only count when a goal is scored. I mean, a guy might throw away the puck and cause a breakaway but if Garon makes the save, it’s forgotten about, right?
I heard that some people actually look at ALL plays where the percentage to score rises — I believe they’re called scoring chances — but those guys are probably big-picturing idiots.
I mean, seriously, Staples is trying to re-invent the wheel; but without admitting the wheel even exists in the first place.
Staples seems like a super hard-working cat but he’s taken a lazy approach to this exercise. Would it take a lot of time to count scoring chances for/against every game and detail all the players on the ice for/against? Damn right it would.
But would it provide a much clearer question?
Damn right it would.
And because of what I’ve listed above, that’s why I pay close-to-zero heed to any talk of errors.
As for the rest, Pouliot played with a few different guys and on some different lines but, perhaps strangely, whenever he wasn’t on with 83 and Co, he was, at least to my eye, outchanching the opposition.
And I think I saw someone talk about Gagner and Cogs’s shooting pcts before so Cogs looks like the early favorite for counting stats regression come ‘09.
Dennis:
Now, you know I actually counted scoring chances one game this year, and posted about it, and believe that “scoring chances” is a useful stat.
So I certainly do admit there are other wheels, but I don’t admit there is any one wheel, any one stat that tells us all we need to know.
Different stats give us some small amount of insight into hockey, though all of them have their obvious pitfalls, and the “error” is certainly no exception.
As for counting scoring chances, the pitfalls are:
* Defining exactly what a scoring chance is. Of course, we all know when we see them, we can all hear the TV announcer’s voice rise higher, but for this to be a useful exercise, someone would have to draw up some rules about what is and what isn’t a scoring chance. We have to know what we’re counting before we can count.
Not to hard to do that, though, I suspect.
* Who should get credit for the scoring chance? Who should be blame for the scoring chance? Everyone on the ice, or just the players who really deserve the credit or the blame?
If we’re going to go the latter route — and it’s only by following that route that counting scoring chances becomes a more optimal stat, I would argue — then we would have to come up with some criteria to figure out who should get credit and blame.
So I’m not at all against people counting scoring chances, and would participate in such work if others were interested in it. But I won’t lead that effort because I am lazy.
I’m in the process of picking the lower hanging fruit of the new world of statistics, the easier stuff to grab, which in my mind is figuring out who is to blame for goals against and keeping track of that.
All that said, Dennis, it’s perfectly OK if you’re bored to tears with talk of the “error” and don’t think it’s worth much. As for me, I didn’t intend to sidetrack the discussion here about Tyler’s work on save and shooting percentages. There is enough to talk about on that subject, such as the strengths and pitfalls of looking at this particular stat and trying to figure out what it tells us about the merit of a particular hockey player, and what it does not tell us.
As I said, I’m still trying to get my head around all of that, though, of course, that could just be me
.
The only problem I have with this is that you’re only assigning errors when there’s a goal scored. I’d think that on any sequence when the opposition gets a chance in the Oilers end, errors are made.
To be fair, errors only count when a goal is scored. I mean, a guy might throw away the puck and cause a breakaway but if Garon makes the save, it’s forgotten about, right?
No argument, Tyler and Dennis. As I put it in the above comment: “I recognize that the fatal flaw is that he too only examined actual goals against — if the goalie made the save everybody is off the hook.” Staples has himself acknowledged this limitation to the method all along. Errors are assigned on actual goals against only, just like points. A guy can make 8 great offensive plays in a game and have nothing to show for it. Similarly a defensive player can blunder all night long only to be bailed out by teammates and goalie, and all we are left with is impressions and memories. Most of ‘em don’t even make the highlight reel, let alone the scoresheet.
Staples seems like a super hard-working cat but he’s taken a lazy approach to this exercise. Would it take a lot of time to count scoring chances for/against every game and detail all the players on the ice for/against? Damn right it would. But would it provide a much clearer question? Damn right it would. And because of what I’ve listed above, that’s why I pay close-to-zero heed to any talk of errors.
Maybe you should ignore the scoring race because it only counts points and not scoring chances. Seriously, Dennis, you seemingly focus on what the error stat isn’t instead of what it is. What it isn’t: a measurement of who was responsible for scoring chances that didn’t become goals. What it is: an assignment of who was responsible for scoring chances that did become goals, and by extension, an exoneration of sorts for those players who were on the ice but did nothing wrong. It’s a refinement of the existing minus stat; I see it is a complementary stat, and a damned interesting one at that. By limiting the sample size to just ES goals rather than 60 minutes of hockey per night, Staples has limited the review process to a manageable size, and I don’t blame him.
It would take an exhaustive research of game film to count all the little mistakes … in the little stretch of games where I “shadowed” Staples on his project I once spent two hours watching just five replays over and over from Team Breakdown Night in Nashville, an ugly 5-4 regulation loss which hurt Oilers playoff chances as much as any single game all year. This is a good example of the process: I assigned the maximum 15 errors and would’ve tacked on a few more if the concept weren’t, at the time, limited to a maximum three per goal. (And David, if you’re reading, I agree with your recent move to remove that restriction.) Besides 8 errors to forwards and 1 to a goaltender, I assigned 6 to the defence on this particular night. Pitkanen made 3 defensive errors in my view including the primary error on the game winner with two minutes left; Gilbert, Staios, and Grebs 1 error each, and Smid and Greene a noteworthy 0, cuz it wasn’t them creating the chaos on this night.
Educational or not, this was such a discouraging process to this Oiler fan that I didn’t last more than a month. Props to Staples for sticking with the program and exhaustively compiling data for the entire season, despite a fairly constant stream of vitriol from some quarters. From this amount of data, there is worthwhile information to be distilled. Anecdotal evidence from this game or that (see: Nashville, above) starts to pile up into a sortable data set. While nobody seemingly counts all the dangerous errors in a game, goal-causing errors are probably a pretty stable fraction of that. From a strictly defensive perspective, anybody want to argue that Pitkanen wasn’t the most mistake-prone Oiler defenceman this year? Or that Gilbert had far more own-zone struggles in the second half than the first? Or that Smid had far more own-zone struggles in the first half than the second?
Did Garon and Roli make more of their best saves behind Smid, say, than Gilbert? No way of knowing from the error stat; that’s not what it’s designed to do. What Staples’ method does do, however, is filter out GA that were not the fault of the player. If the goalie gives in a soft one, the defenders don’t get charged. If the forwards go off on a line change and hang the defencemen out to dry on a well-executed 3-on-2, the defenders don’t get charged. The two metrics, errors and team Sv%, are hardly independent, but they’re not identical either. And when they seem to corroborate one another as in the current case, the two stats combined are stronger than either of them by itself. (Which is why I raised the issue of the error stat now, not to hijack the thread but because Staples’ data and Mudcrutch’s are complementary.)
Whether it was due to their own improvement or improved play in front of them, both Oiler goaltenders had better Sv% numbers in the second half than the first. Four of the six regular defenders saw their on-ice Sv% numbers improve. The two who didn’t were seen to commit significantly more goal-causing errors in the second half than the first. Circumstantial evidence perhaps, but surely it’s fair to conclude the arrows are pointing in the same direction.
Dennis: RE Cogliano shooting % That would probably be here
Personally, I like the error stat, although I don’t think it’s the be-all and end-all of defensive play (and I doubt any of its proponents would advocate that it is).
The numbers on Pitkanen and Gilbert mesh well with what I saw throughout the year, and I wonder if Gilbert has a history of fading as the year goes on (sometimes it takes college players a while to get out of that).
The biggest problem with the error stat, IMO, is that we can’t compare it league wide to see how our defenders compare with other teams; we can only compare amongst Oilers. Work harder, Staples! Errors for every game! And minor leagues as well!
Getting back to save percentages behind a player, I just don’t see it as being significant, because the correlation between the players actions and the stat is so minimal. The goaltender is the primary control of the statistic, and each of the other four defending players on the ice has an effect as well, and since we’re discussing it, the quality of opposition is probably more important than the quality of the defenders.
Whom did Pouliot play with?
Here are the ES minutes for forwards and +/-:
MARTY REASONER 62:38 1
FERNANDO PISANI 61:36 1
ANDREW COGLIANO 54:13 -4
JARRET STOLL 46:46 -1
SAM GAGNER 39:25 -1
DUSTIN PENNER 38:12 0
ZACHERY STORTINI 35:44 1
GEOFF SANDERSON 29:41 -4
CURTIS GLENCROSS 28:55 3
KYLE BRODZIAK 23:18 4
In his second coming, Pouliot played mostly, I think, with Reasoner/Pisani and Brodziak/Glencross.
Staples: I think your tallying of errors is an useful endeavour. However, I’m not sure that you fully understand the nature of your stat.
Conceptually, I don’t agree that your error stats is the opposite of points.
Scoring a point is an offense creation event. It measures how an attacker proactively creates offense. Therefore, the opposite should be an offense destruction event. It should measure how a defending player proactively destroys offense (of his opponents, of course). For instance, breaking up a 2-on-1, preventing a puck from going into his unguarded net, etc.
I think that the application of the error stat actually lies at the other end of the scoring spectrum. That is, it should be used negate (perhaps partially) a scorer’s “undeserved” points that were scored not so much due to his own efforts but more so as a result of the defender’s mistake.
For instance, in soccer, let’s say that Beckham takes a corner and a member of the defending team mistakenly heads the ball into his own net, Beckham would not get credit for the goal. Rather, it would be classified as an “own goal”.
I think a similar process can be applied in hockey.
I agree, errors have something in common with own goals.
If they keep track of offensive linemen who blow assignments, and allow the quarterback to be pressured or sacked, it would be like that, too.
Or, in the NBA, if they keep track of players who get scored on, there would be some similarities.
Soccer has a number of interesting stats, such as passing percentage and metres run during a game that don’t get much discussion in the general press or on the blogs, and I haven’t found any blogger for my favourite team, Manchested United, that keeps track of errors in the way I do. In soccer, you could actually count errors for every major scoring chance, since there are only four or five of them every game.
If I could get Manchester United games regularly on TV, I might well do it myself.
For instance, I can tell you that the Chelsea goal by Lampard would definitely be an error for Rio Ferdinand.
There are only about five scoring chances per game per team for soccer, Staples.
So perhaps you could “score” those games correctly;)
I’ve done a full post on this at my site, but here is what I found in terms of a relationship between defensive errors and defensive save percentage.
Of the 18 Oilers who played at least 10 games in both parts of the season, seven players — Smid, Staios, Greene, Nilsson, Pisani, Stortini and Reasoner — saw their save percentage move up as their errors per game went down.
Another six players — Pitkanen, Gilbert, Penner, Hemsky, Sanderson and Gagner — saw their save percentage go down as their errors per game increased.
So there was this interesting relationship between save percentage and errors per game with 13 out of 18 Oilers regulars.
Only five players defied this trend, Grebeshkov, Cogliano, Brodziak, Stoll and Horcoff.
[i]”…saw their save percentage go down as their errors per game increased.”
“…saw their save percentage move up as their errors per game went down.”[/i]
By a show of hands, who thinks that the reduction in errors drove the save percentage up?
Conversely, who’s of the belief that the higher save percentage drove the ammount of errors down?
Based simply on the fact that errors are only accounted for when a puck gets behind the goalie, I feel this is an easy, but I dunno…
And could somebody tell me how to write in italics so I can join the cool kids club? Please.
You’ve almost got it - < and > around the tags instead of [ and ].
I agree with your analysis on the error and save percentage.
Before we deal with the issue of “cause,” I’m still trying to figure out exactly what is going on . . .
Can we agree on the following?
If a player is on the ice for more goals against, he is going to have more errors assigned to him, most likely.
If a player is on the ice for more goals against, his save percentage is going to go down, most likely.
So perhaps the relationship between the two stats is that they are describing the same thing: being on the ice for more or for fewer goals against.
If you’re out there for more goals, your errors go up, your save percentage goes down. If you’re out there for fewer goals, your errors go down, your save percentage goes up.
This statement doesn’t deal with cause. It just describes what we’re observing.
As for what “causes” what, for what causes a player to be on the ice for fewer goals against, that is a complex issue.
He could be out with weaker teammates. He could be out against stronger opposition. He could be in a slump defensively, or on a hot streak defensively. Or, over time, his overall defensive skill could be weakening or getting stronger.
Questions, questions, questions . . . I have more of them than I do certain answers . . .
Does an increase in defensive skill “cause” an increase in save percentage? With goalies it certainly seems to, and with teams overall, it certainly seems to, so perhaps it does for individual players as well. I suspect it does.
Does a slump on defence “cause” an increase in save percentage? I think it might well. That is what I saw with Gilbert in the second half of the year, a guy facing tougher opposition and a guy who was slowing down, and made many more errors per minute.
Does an increase in defensive skill “cause” an decrease in errors per minute? On that last point, I would say it certainly does, that players who get their defensive game together will make fewer errors than they do when they are struggling defensively. That’s how I saw it, but I was admittedly just one observer.
If defensive save percentage really is a strong indicator of sound defensive play, I’d be thrilled. I’m all in favour of picking the low hanging fruit of hockey statistics, and defensive save percentage could well be a superior stat to “errors” in that regard, especially if they’re telling us the same thing.
As Bruce has said, it’s difficult to assign errors. It takes a lot of time and some hockey expertise. But if we can simply look at defensive save percentage for players, and have a pretty good idea about their defensive skill, that would be a good thing. Of course, we’d also have to keep in mind the quality of their competition and of their teammates, namely their goalie in this case.
It’s going to be a few years before we have any sort of consensus on the question of player impact on save percentage but I continue to think that it’s going to be pretty small. I fiddled around with looking for a relationship between first and second half save percentages when a given player was on the ice but was unable to find anything. It’s going to be a tricky question to answer because roles and teammates change over the course of the season but there was nothing there with the Oilers this year; if anything, the relationship between save percentage in the first and second halves was a negative one.
I just re-read your post, and now see your position was made clear, that there is likely little relationship between the defensive skill of a player and his defensive save percentage.
An interesting position to take, and I’m sure you have your reasons, that you are basing your conclusion on some data or theory. I’d like to know more . . .
Ooops, didn’t master that italics thing. I’ll try again.
It’s going to be a few years before we have any sort of consensus on the question of player impact on save percentage but I continue to think that it’s going to be pretty small. I fiddled around with looking for a relationship between first and second half save percentages when a given player was on the ice but was unable to find anything. It’s going to be a tricky question to answer because roles and teammates change over the course of the season but there was nothing there with the Oilers this year; if anything, the relationship between save percentage in the first and second halves was a negative one.
I just re-read your post, and now see your position was made clear, that there is likely little relationship between the defensive skill of a player and his defensive save percentage.
An interesting position to take, and I’m sure you have your reasons, that you are basing your conclusion on some data or theory. I’d like to know more . . .
Ok, how do you end italics?
As for what “causes” what, for what causes a player to be on the ice for fewer goals against, that is a complex issue.
Definitely agree with pinkslippered on this one.
Also, I would argue that if you’re looking for cause, it goes something like this.
Player A (let’s say, Gilbert) plays above expectations against soft opposition. The error statistic, among others, shows this, and of course the coaching staff recognize it. The save percentage is primarily an effect of playing against opponents with weak scoring ability.
Thus, they bump Gilbert up to play tougher opposition. It’s more difficult, and he has trouble adjusting. The error statistic, among others, shows this. The save percentage drops, not because of the increase in errors, but because now Gilbert is playing against opponents who have much greater scoring ability.
There was a post over at IOF (this one!) where the sv% correlation to forwards based on ice time is shown. It’s pretty damning and I think supports the scenario I just outlined.
To make it a little clearer; EV SV% is primarily a result of opposition, and quality goaltending, and is only affected by defending players in a secondary way.
By a show of hands, who thinks that the reduction in errors drove the save percentage up?
Conversely, who’s of the belief that the higher save percentage drove the ammount of errors down?
That’s me with both hands in the air, Pinkslipper. A former goalie myself, I’ll be the first to say a hot goalie can make a lot of defensive problems disappear. The errors that got counted were only a fraction of the mistakes that were committed. But the plays that I counted as “errors” generally resulted in high-percentage shots, and it stands to reason that more of those are going to find twine.
Smid for example had a problem particularly in the first half of the season where he’d lose position on his man around the crease and the guy would slip behind him for a tap-in. I don’t care who the goalie is, those kind of shots have a high chance of going in. In the second half, Smid improved significantly in this area, his number of errors went down and the save percentage behind him went up. Seems like a pretty logical connection to me.
I do think that it’s going to take a few years of stats before we can begin to make real sense of individual Sv%. But we gotta start somewhere.
Jonathan:
No doubt, save percentage can drop and errors can go up partly because a player faces tougher opposition. But surely that is only part of the equation.
Players also go into slumps, get injured, loose confidence, and this actual drop in performance contributes to their drop in save percentage and rise in errors, does it not? This is the Gilbert scenario.
And, on the other side, a player like Greene (or Reasoner/Stortini/Staios) faced pretty much the same opposition all year, but his save percentage went up and his errors went down as the year went on, IMO, partly because he simply improved as a player.
A player like Reasoner faced similar opposition all year, but he broke out of a defensive slump about 30 games into the season, played better after that, and saw his errors go way down and his save percentage go up.
Staios started out struggling, made many errors and had a lower save percentage, then gained strength as the year went on, and his defensive stats showed it.
So it’s not just outside factors — luck, tough opposition — that lead to changes in save percentage in errors. These are major factors. But performance is also a factor, and, I would argue based on observation, a major factor, perhaps the dominant one. But that has yet to be proved.
David:
You misunderstand me. I believe that save percentage to some degree drives the error statistic, but that the error statistic has a negligible effect on save percentage.
Players also go into slumps, get injured, loose confidence, and this actual drop in performance contributes to their drop in save percentage and rise in errors, does it not? This is the Gilbert scenario.
Rise in errors, yes, drop in SV%, no.
Players also go into slumps, get injured, loose confidence, and this actual drop in performance contributes to their drop in save percentage and rise in errors, does it not? This is the Gilbert scenario.
Or, the goaltending behind him improved as Garon replaced Roli for the majority of games, causing a jump in SV% and a corresponding reduction of errors as outlined above. That’s my belief.
Jonathan: While it’s true Garon played more games in the second half and he was the better goalie all season long, both goalies improved from the first half to the second. The team as a whole improved from .910 to .915, significant but hardly an overwhelming difference.
Btw, while the errors stat doesn’t measure great saves, it does measure soft goals. Errors get charged to the goalies as well. Over the season Roloson committed 41 errors, 31 of those just in the first half of the season; while Garon committed just 10+14=24. Some were primary errors and some were even “unassisted” errors, a softie where the teammates got saddled with an undeserved minus and a lower Sv%. Perhaps an uneven distribution of these might explain some of the Sv% differences. Or perhaps over the course of the season they tend to balance out.
To further paraphrase what, atleast I believe, Jonathon and I are echoing here: a player’s mistake may result in a scoring chance against, but it has negligble effect on a goalie’s save percentage.
Save Percentage is an expression of the ammount of saves a goalie make from a given ammount of shots- of shots that actually make it on net.
So, once the gaffe is made that leads to the opportunity against, the non-goalie player has zero impact on whether the save is made. Say a player blocks the opportunity- then the shot doesn’t make it on net, and therefore it is not registered, and has no effect on the save percentage equation.
For the most part, skaters don’t make saves, and even under the rare event that they do get in front of a puck near the goal mouth, it won’t be registered as a shot against in the traditonal sense. It isn’t expressed by save percentage.
Ha-ha, italics!
You all will come to regret this day!
And no, Bruce, of course I wouldn’t argue that the quality of the scoring chance greatly affects the goalie’s chance at making the save. Yet, I feel that it is imperative to maintain a distinction, because no matter the quality of the opportunity, after a certain point it remains, for the most part, all on the goalie on whether or not the pucks passes the line.
I referred to a chart from IOF in an earlier comment. What Vic did was take the top three forwards in TOI for each team, than the next three, the next three, etc., and compared the effect these players had on EV SV% of their own and other goalies.
Here’s what he found:
Line 1 -
Own Goalie: -.001%
Opposing Goalie: -.010%
Line 2 -
Own: +.001%
Opponent: -.001%
Line 3 -
Own: 000%
Opponent: +.006%
Line 4 -
Own: 000%
Opponent: +.012%
What does that tell us? It tells us that SV% is a statistic driven by the opposing player, not the defender. To drive the point home, he posted the same numbers, but for defenders.
Pair 1
Own goalie: -.001%
Opponent: +.003%
Pair 2
Own: +.003%
Opponent: -.003%
Pair 3
Own: .000%
Opponent: -.003%
This means that the defensemen in the league, regardless of quality, have close to no impact on SV%.
Thanks Jonathan.
You have done some excellent work answering my questions here.
As I said when I posted my questions to Tyler about save percentage earlier in this thread, this stat is new to me, when it comes to its implications for positional players at least, so I’m still trying to figure out its significance. But, clearly, you guys who have been working on this, and checking out this stat, are convinced it’s not related to defensive performance. I’ll leave it at that for now.
No, I do have one more question
You say that defensive SV% is driven by the opposing player, not the defender.
Let’s take this as a fact then.
But what if we turn this around . . .
If we look at each Oiler player on offence, and the save percentage of the opposing team when each Oiler is out there, if save percentage is driven by the attacking player, can we look at the save percentages of the opposing team, see which Oilers really put the offensive boots to the opposition and drive down the opposition’s save percentage, then conclude these fellows are the dominant offensive players on the Oilers?
In other words, is shooting percentage an indicator of offensive dominance? That would make Cogliano the most dominant offensive player on the Oilers last year, not taking into account quality of opposition and teammates.
I just don’t see the point of trying to plant your flag on a new mountain when you refuse to tackle a much bigger once.
Count scoring chances and list players who were on the ice for/against.
I’ve done it for a couple of games and it’s tiring but it paints a true picture, IMO.
With all due respecgt, Dennis, “scoring chances” is your mountain, not my mountain.
I don’t have the time to count scoring chances by myself, let alone assess who should get credit or blame for each scoring chance. This isn’t my day job, just my obsession.
So even if I was convinced it was better to count scoring chances than errors, I’d be in a pickle. Plus I don’t do Pay Per View, so how would I score those games?
Now, if you want to organize a large team of folks to count scoring chances, count me in. We could all do one in ten games, that kind of thing. That wouldn’t be too onerous.
Hell, I would even organize it, and post results on my blog, but I’d need ten volunteers willing to do the work, and I doubt I could find ten willing fanatics.
You see, I can do this work on the “error” by myself, which is a good thing, because the idea hasn’t exactly caught fire. As Bruce mentioned, it’s time-consuming to assign errors, and most people have better things to do.
That said, I’m always happy to accept volunteers, especially for my upcoming 10-game study on “rater reliability” for the error, should that be your fancy
If we look at each Oiler player on offence, and the save percentage of the opposing team when each Oiler is out there, if save percentage is driven by the attacking player, can we look at the save percentages of the opposing team, see which Oilers really put the offensive boots to the opposition and drive down the opposition’s save percentage, then conclude these fellows are the dominant offensive players on the Oilers?
In other words, is shooting percentage an indicator of offensive dominance? That would make Cogliano the most dominant offensive player on the Oilers last year, not taking into account quality of opposition and teammates.
Well, Im just spitballing here, because Im nowhere near the statistical equal of the IOF gang or mc79, but I would argue that repeatable shooting percentage would indicate an offensively dominant player; ie shooting percentage over several seasons, and might even be a better indicator than straight goals- but like I said Im far and away from being the most qualified guy.
Oh, and I think Id be game for that error study, if youre looking.
This means that the defensemen in the league, regardless of quality, have close to no impact on SV%.
Jonathan: Do you really believe that statement? Does it seem plausible?
Colour me skeptical. Due respect to what Vic is trying to do (and thanks for the link), but all that he posted was his end results, not the underlying numbers or a clear explanation of the methodology w.r.t. Sv%. Those end results seem astonishingly uniform when it comes to own-team Sv%. In 45 years of being a stats buff,I don’t think I’ve ever seen a line of numbers quite that regular, no matter how general the category.
Jonathan: Do you really believe that statement? Does it seem plausible?
To quote Vic:
It’s flat, and doesn’t tell you much, because the best defenders play a lot against good forwards, so their results get pounded down to a level that’s within spitting distance of the terrible defenders in this league.
It does actually seem plausible to me because:
a) This statistic has all the negative of +/-; ie it is a catchall, not taking into account quality of teammates or quality of opposition and crediting or discrediting a player based on actions that arent his own- I think we see the same curve that we would see if we used straight +/-. And, looking at some of the guys in the NHL top ten in +/- (Johnny Oduya, Kent Huskins, Matt Niskanen, Jassen Cullimore and Doug Murray) I think we can reasonably see that whatever impact a defenseman has needs to be viewed thru the lens of situation.
b) In addition to having all of the negatives of +/-, this statistic has even more. If, say, Matt Greene plays most of his time against guys like James Sheppard, Aaron Voros and Todd Fedoruk, he will have a wicked save percentage behind him. His +/- will at least be evened out by whatever his offensive production against these guys is.
Bottom line, although defensemen have some impact, it is beaten down by quality of matchup, and the statistic is really driven by the forwards. Where I see the statistic as having promise is as a possible filter for straight +/-.