A lot of people, myself included, have referenced Gabriel Desjardins’ quality of competition numbers lately. While it’s great that there seems to be an increased awareness of this sort of thing, his methodology, found here, warrants some discussion, as I think that it has a few problems. I don’t mean to take unfair shots at the guy – I’ve done stuff like this before and it’s not easy, but the methodology would benefit from some improvement, I think. Desjardins explains Quality of Competition as follows:
How is the ‘Strength of Opponents’ or ‘Quality of Competition’ statistic calculated?
It is the average On/Off-Ice +/- of the opposing players a player faces. For example, if you lined up against Anaheim’s top line, you’d get:
Name Pos Team # On/Off +/- KUNITZ F ANA 14 +1.97 SELANNE F ANA 8 +1.65 PRONGER D ANA 25 +1.61 MCDONALD F ANA 19 +0.94 NIEDERMAYER D ANA 27 -0.31
The strength of opponent would be the average of 1.97, 1.65, 1.61, 0.94 and -0.31, which is 5.86/5 = +1.17. In general, if a player matches up against the other team’s first line, he’ll face a high strength of competition.
You can see from this that On/Off-Ice +/- is a crucial part of the numbers that he’s cranking out and, accordingly, you’re going to need to know how that’s defined before you can really evaluate what he’s doing here. On/Off-Ice +/- is defined as follows:
What is On-Ice/Off-Ice +/-?
It’s the difference between the team’s plus/minus rate when a player is on the ice and when he’s off the ice.
Why subtract the Off-Ice +/- from regular +/-?
Hockey’s a team game; a good two-way player on a bad team (Peter Forsberg, Ryan Smyth in 2006-07) will have a lower +/- than most players on a good team like Detroit or Nashville. That’s obviously not an accurate reflection of Forsberg and Smyth’s performance because they’d have a much higher +/- over the course of a season with a good team. Plus/minus relative to the rest of the team’s performance is a more accurate reflection of a player’s ability to score and prevent goals.
I’ve got some problems with this and, because he uses these numbers as a basis for his quality of competition numbers, some problems with that. My first problem lies with this statement: “Plus/minus relative to the rest of the team’s performance is a more accurate reflection of a player’s ability to score and prevent goals.” I don’t think that this is necessarily true. Take the Oilers as an example this year. Ryan Smyth wasn’t on the ice for a single one of JF Jacques’ 0 ESGF and 11 ESGA. Other than the extent to which coaching decisions relating to their usage were driven by the desire to minimize the usage of Jacques’, I can’t figure out how Jacques’ performance is relevant in our evaluation of Smyth’s. To keep things clean, we’ll assume that they didn’t play together and that’s why there are no events with the two of them on the ice together. What happens on the ice when Smyth isn’t there is irrelevant to how easy or how difficult it is for him to build a good +/-. Smyth is being credited here for something that he can’t control and that apparently didn’t impact him. Obviously, since the Quality of Competition numbers are based on these numbers, they suffer from this flawed assumption.
There’s another issue here as well. Desjardins is effectively assuming that each player’s offensive contribution is equivalent when he does this – he adds together the five players On ice/off ice +/- and divides by five. I doubt that this accurately reflects the contribution – I suspect that forwards, at the very least, impact the offensive side of the +/- equation more than the defenceman. Treating Pronger as a +1.61 player…well, he probably isn’t really, that number is probably driven by the forwards. The whole thing needs to be weighted differently. I’ve got some ideas on how, which I’ll get into at some point in the future.
A third issue. Consider a player who played only against terrible players and therefore had good numbers. The players who played against him would look like they were playing hard minutes when judged by this player’s numbers when, in actuality, they weren’t. There’s a bit a of a cyclical thing at work here. My understanding is that you can resolve this by some mathematical trick or another; I just can’t remember the name. It’s something that needs to be brought into this equation.
Despite my criticisms, I think that what Desjardins is doing here has real value, it just needs some sort of sensible rejigging. It’s a great first step and, from the perspective of someone who would like to see this sort of stuff – hockey stats that are supposed to mean something - catch on, I’m happy to see it. I just think that it needs some work from the logic/hockey side of things.