• Why Stats Guys Conflate PDO With Luck

    by  • April 4, 2014 • Hockey • 6 Comments

    Someone asked me a very good question on Twitter the other day.

    It’s a very good question.

    For those who are unfamiliar with PDO, it’s the sum of the shooting percentage and save percentage in a given situation for a given team or player. It has a huge impact on the results that are achieved when a player is on the ice. Imagine a player who is on the ice for 500 shots for and 500 shots against over the course of a season. Consider two situations. In the first situation, his team shoots 10% with him on the ice and posts a .940 save percentage. That’s a PDO of 104. In the second situation, his team shoots 6% with him on the ice and posts a save percentage of .900. That’s a PDO of 96.

    In the first situation, the player will be on the ice for 50 GF and 30 GA. In the second situation, he’ll be on the ice for 30 GF and 50 GA. That’s a massive swing. There’s a catch though. PDO tends to regress quite strongly towards 100.

    We can see this if we look at the data. There were 16 players in 2012-13 who met the following criteria: a) appeared in at least 36 games, b) played at least 10 minutes a night, c) played at least 75% of games and 10 minutes a night this year and d) posted a PDO of 104 or better in 2012-13. In 2012-13, those players’ teams scored 65.9% of the goals when they were on the ice and averaged a PDO of 105. This year, 50.1% and a PDO of 100.

    We see something similar in the bottom end of the table as well. Eight players a) appeared in at least 36 games, b) played at least 10 minutes a night, c) played at least 75% of games and 10 minutes a night this year and d) posted a PDO of 96 or worse in 2012-13. Their teams scored 33.7% of the goals when they were on the ice in 2012-13 and posted a PDO of 95. This year, their teams are scoring 48% of the goals when they’re on the ice and posting a 99.6 PDO.

    So why are stats guys hesitant to say that the 104+ PDO players played well and that the 96- PDO players played poorly? It has to do with differentiating between skill and chance. I can basically choose any cutoff that I want – 10 games, 60 games, two years – and identify the players with high and low PDO. From that point forward, my best guess for the average PDO of the group is basically 100.

    If I can choose any cutoff point that I want and achieve this effect, doesn’t that suggest that it’s not about players happening to play well? Wouldn’t we expect that players who are playing well, as opposed to just getting lucky/unlucky to have that continue? We might think that it would cool off in the future but everyone regressing to 100 from any cutoff I pick?

    It’s analogous to someone flipping coins. Imagine that I had 100 coin flippers flipping coins once every ten seconds. Every so often, I stop them and look at who’s good at turning up heads and who’s bad at it. Say I stop them after ten minutes or 60 flips. The odds are, I’d have about 8 guys who’d flipped heads less than 40% of the time and about 8 guys who’d flipped 60% of the time or more.

    If you were to guess what percentage of the time the guys who flipped a lot of heads were going to get heads going forward, you’d guess 50%. Your guess for the guys who flipped tails a lot would be the same. It’s very much like PDO in this way – your best guess going forward for groups of players who have extreme PDOs is generally right around 100. You wouldn’t say that the people who flipped a lot of heads or very few heads were flipping well or poorly. If groups of players with high or low PDO collectively behave a lot like groups of people flipping a lot of heads or very few heads, it’s probably most sensible to think of it as luck.

    That’s the abstract explanation. There’s a more practical explanation for why it’s best to think of it as luck. What’s in the past is in the past. What we as people interested in hockey are generally more interested in is the future – you can’t change the past. To the extent that the past interests us, it interests us because of what it can tell us about the future. If we know that really good or really bad PDO is likely to regress hard towards 100, it’s best not to put too much weight on it. The word we tend to use to describe something that isn’t likely to repeat in the future is luck.

    It doesn’t really matter whether it was luck in the way we tend to think of it – Tyler Bozak scoring a goal off his chest on a puck he couldn’t react to – or “luck” in the sense of a player making a bunch of shots. It doesn’t tell us much about the future. So it’s luck.

    Email Tyler Dellow at tyler@mc79hockey.com


    6 Responses to Why Stats Guys Conflate PDO With Luck

    1. Trentent Tye
      April 4, 2014 at

      If you look at the PDO of line combinations, can you extrapolate that shuffling players between groups of “two” (for forwards) produces better results for the 3rd player? For instance, Nuge and Eberle played quite a bit together and Hall was moved onto Perron/Gagner’s line. What was Nuge and Eberle’s PDO during that time, and then the last few games when Hall was moved onto it did their PDO change in a significant way? In other words, could PDO help further define who drives the bus on a line to help push coaches/arm-chair coaches to better ideal lines?

    2. Flips
      April 4, 2014 at

      I would guess that consistently playing with players that are at the outliers of certain “skilled” stats would have a noticeable affect on PDO. For example, if you played with Stamkos and Hasek your PDO would naturally be higher than playing with POS and Steve Mason. However, since significantly below average goaltenders don’t keep playing (unless you played for Maclean) in the long run you would see that number increase.

      • Tyler Dellow
        April 4, 2014 at

        I don’t disagree. You need to bring some common sense into it. That said, even the elites are subject to regression, although maybe not as far as back as regular joes.

        • Murat
          April 4, 2014 at

          Well said. I think this is an important qualifier, especially when you’re talking to people new to PDO.

    3. tony
      April 5, 2014 at

      Thank you SO much for this post. I’ve been going through your blog like crazy, having gotten very interested in hockey analytics as of late, but I could never find a source explaining PDO better than your (literally) first two paragraphs here just did.

      Keep up the amazing work, looking forward to new posts every day!

    4. jeffgm
      April 5, 2014 at

      If you are flipping PDO coins, the league (or group) average is mathematically constrained by 1000. Any one coin can be unfair (skilled or unskilled) and have a distinct mean. Take a single coin of Team A (heads) with a PDO of 1010 and Team B (tails) with a PDO of 990 odds. What is the average – exactly 1000 and even though each team has a distinct non-1000 PDO. The challenge we face statistically is differentiating when a non-1000 PDO is “significant” from the case of a “random chance” or a hot streak of heads or tails was the reason to explain the difference from league mean of 1000. And in small samples of 20 or 40 game it is difficult to statistically differentiate skill from luck.

      Our bias to “choose” 1000 as the point that teams or players regress to is an inherent stereotype we make assuming that each individuals mean is the same as the group mean. We have statistical tools available to define the confidence we have that a team will have a PDO of 1000 in the remaining games going forward (Tango, Snark D PDO work etc). This type of work can be complex but more complete then saying player x or team x will regress to 1000 because of a false group mean attribution (stereotype). I know you understand this but from twitter I don’t think everyone graps the distinction.

    Leave a Reply

    Your email address will not be published. Required fields are marked *