• Skepticism I

    by Tyler Dellow • January 8, 2012 • Uncategorized • 15 Comments

    There’s something unfortunate that happens when people with an interest in sports stats become simultaneously interested in turning a profit from that interest. The desire to generate a profit creates an incentive not to publish the guts of your work, so as to create a body of proprietary knowledge that people will pay you for. This prevents the sort of informal peer review process that has developed from grinding into gear. It also encourages you to market yourself, overstating your confidence and not identifying any weaknesses in your work. The feckless media fuel this, as they lack either the ability or desire to check claims.

    Parenthetical: this may well be an ability problem. The Toronto Star’s Damien Cox and the Boston Globe’s Kevin Paul Dupont, while not amongst the most power and influential people in hockey as determined by The Hockey News (smirks), both have awfully large bully pulpits. They both chimed in on Twitter this morning on the NHLPA’s refusal to acquiesce to the NHL’s planned re-alignment.

    Cox, in a series of tweets: “After decades of travel across North America, the NHLPA is suddenly concerned about travel. Joke. Huge one. Only bigger joke is belief its harder to make playoffs in an 8-team conference than 7-team. Amazing how some will believe anything. Supposed playoff “unfairness” in proposed alignment is same nonsense as saying going to bigger rink takes out most expensive seats. People, the fight for the fourth playoff spot in any of the new conferences won’t involve the 7th or 8th team. It’s not a factor.”

    Dupont: “All play 82 games. Top 4 teams in ea conference make dance. Red herring.”

    You would not, I think, need to be particularly gifted in mathematics to realize that teams in a seven team conference have a 4/7 chance at a playoff spot in a given year and teams in an eight team conference have a 4/8 chance at a playoff spot. Cox seems to inexplicably think there will be an equal dispersion of skill levels in the various conferences, with the eight team conferences just having an extra terrible team tacked on. It’s amazing.

    Even if division isn’t your bag, you can simply look at the last time that the NHL had a four division setup, with divisional playoffs first and the problem becomes obvious. In 1990-91, Philadelphia finished fifth in the Patrick with 76 points. That would have seen them into the playoffs in every other division. In 1987-88, the Rangers and Isles missed with 82 and 81 points respectively, which would have seen them through in every other division. In 1985-86 and 1986-87, Pittsburgh missed with 76 and 72 points, respectively, totals that would have seen them make it in the Smythe or Norris Divisions.

    While you can’t just say that those teams would have made it in any other division, what those seasons and, in particular, 1987-88, show us is that you aren’t guaranteed to divisions of equal strength. The number of teams in your division matters because it reduces the odds that you’ll run into four 100 point teams while you’re the guy with 95 points. This is obvious, but there it is.

    End parenthetical.

    There was a study that had a certain segment of the hockey world all heated up last week as Marc Appleby of Powerscout (“…known as the ‘Moneyball’ of the NHL” although by whom is not clear) tweeted out a link to a press release about a study that his company has done. I asked him for a link to something that sets out the research a little more clearly; no luck. This is what we get. Key paragraphs:

    The results will surprise many NHL hockey fans because the PowerScout advanced graphs do show that some fights can have a significant impact on a game, even be game-changers in some cases. To evaluate this impact, they used a graph called the Momentum Meter™.

    To gain a larger scope of the influence of fights on NHL games, the company analyzed 1,563 fights going back to the 09/10 season using their game archives. They have determined that a fight has a positive effect on at least one team’s momentum in 76% of fights and increases the momentum for both teams about 1 out of every 4 fights (23%). This research was done using the Momentum Meter, looking specifically at the increase in team momentum in the three minutes following the fight.

    Appleby agrees that the controversy is far from over and believes that PowerScout’s unique momentum perspective and research will likely add to the controversy as fans can now clearly see how fights impact games.

    The paid media got them a pile of free media. I believe one of the principals popped up on Oilers Now. There was a lengthy story on the Associated Press wire. Here’s how that article opens:

    This might be the last thing fans and critics on one side of the longest-running debate in hockey want to hear, especially in light of the rising number of concussions:

    Fighting works.

    The first-of-its-kind statistical analysis of the sport confirms the dirty little secret coaches and players have known since the dawn of the NHL. There’s no more readily available, sure-fire way to shift the momentum of a game than to send a player out to start a fight.

    I can already hear those guys saying that they didn’t write the story. Fine. But it’s consistent with the tone of their press releases, so they can hardly complain.

    EDMTOR

    It’s almost impossible to evaluate their claims on the basis of what’s been published. As I understand things, what they’re basically saying is that shots increase in the three minutes after a fight relative to the thirty seconds before. This is awfully interesting – as I’ve previously documented (without a PR blitz), there’s a marked decrease in shooting volume as you go from first liners to second liners, second liners to third liners and third liners to fourth liners. We know that the majority of fights involve players who aren’t first liners and probably not second liners – it wouldn’t surprise me to learn that 70 or 80% of hockey fights involved guys who are bottom six, bottom defensive pairing types.

    Now, there’s a logic argument here. If we know that most fights involve guys who get less ice time and we know that guys who play less tend to see fewer shots for and against when they are on the ice, we’d expect that shots would increase after a fight because the fight probably involved guys who are less likely to see shots taken by either team when they were on the ice, meaning that it’s more likely that guys who see more shots when they are on the ice will be coming over the boards in the next three minutes.

    Is that what these guys have found? I don’t know, because they haven’t presented their data in such a way that their claims can be evaluated. I’d bet some money on that being the case though. Until they show how they’ve controlled for these and various other confounding effects, this is what lawyers call a bald claim.

    This sort of thing is all the more aggravating because statistical analysis in hockey is a pretty new field, with lots of people who doubt that it can add anything of value. It’s irresponsible for people interested in this stuff to make sensational claims that aren’t properly documented and able to be reviewed and challenged by others but this stuff is rife in sports analysis – Baseball Prospectus was particularly bad for it. It’s terrible because, in the long run, it leads people to conclude that there’s nothing you can learn from this stuff.

    I’ll bet it sells better than an Excel generated graph on a website showing that there are fewer shots when lesser players are on the ice though.

    About Tyler Dellow

    15 Responses to Skepticism I

    1. Hawerchuk
      January 8, 2012 at

      Normally a business that’s about to spend $60M in a year would have some underlying financial model for asset valuation. Financial firms have been doing it forever and lots of people have made huge amounts of money by having better automated models. Insurance companies do huge amounts of risk management and are populated by lots of people with extensive stats backgrounds. Plenty of companies have used A/B experiments to statistically evaluate decision-making. And of course MLB, NFL and NBA teams have extensive stats teams.

      The NHL occupies a unique spot because “everyone knows” there’s something to be learned from “stats”, but nobody knows how to evaluate the validity of any particular stat. (This is where we were in baseball circa 1986 – there were plenty of player evaluation metrics that “competed” with Runs Created, but they were usually just meaningless sums of other stats, like RBIs.) So a guy with a slick-looking site and a lot of vapor behind him can sell his numbers to a sports website or a team.

      When I see something goofy like this, I usually like to explain the stats and give away for free the data or ratings that are somehow being sold. But I can’t figure out WTF momentum is.

    2. Bruce McCurdy
      January 8, 2012 at

      “Momentum meter” puh-leeeeze.

      As I understand things, what they’re basically saying is that shots increase in the three minutes after a fight relative to the thirty seconds before.

      Talk about a method that doesn’t pass the smell test. Your objection was my original response as well – “but but but the low-event guys are apt to be out there in the ~30 seconds before the fight, and they’re almost certainly nailed to the bench (or the penalty box) for the next faceoff and at least a full rotation of the 1-2-3 lines.

      Also, your interpretation doesn’t say anything about whether their method is restricted to the next three minutes having to be at even strength or whether it includes anything that happens in those three minutes, whereas the 30 seconds preceding most fights is typically played at even strength. I haven’t got stats but you don’t usually see fights on special teams. Most of the thugs don’t play special teams for starters.

      And of course some fights actually lead to odd-man situations. Again I don’t have stats, but I’d be willing to bet that the instigator’s team would be more inclined to give up the next goal than to score it.

      I was going to suggest a counter study to show how much “energy” gets infused into the game when fourth-liners are out there measured in “shots per 60″, but by the time I reached the last paragraph I realized you were way ahead of me.

    3. January 8, 2012 at

      I don’t think there’s anything inherently wrong with wanting to profit from sports stats, but perverse incentives do arise. The challenge lies in maintaining integrity (e.g. opening up proprietary calculations to outside inspection) while still being able to turn a profit.

      In ‘actual science’ (as opposed to sports science), few if any scientists profits off of formulas or calculations alone; instead, they earn their living from being sponsored (or hired) by a company or institution that can product something of value from the formula. In sports analysis, who’s going to sponsor an amateur stathead? Maybe one day there’ll be 30 NHL statistician positions, similar to how most (all?) MLB clubs have somebody like that, but even then an NHL job is a slim possibility that is probably better realized through other means. So what about producing something based on your analysis that people will pay for? There are some for-pay analysis packages, but in that situation your reputation and integrity again rely on being open to inspection and verification.

      My conclusion is that there’s not much you can do besides put ads on your website and give your insights away for free. Sports analysis, and in particular hockey analysis, isn’t a highly lucrative field and trying to make it one by keeping your methods proprietary is not the way to break that mould.

      • Hawerchuk
        January 9, 2012 at

        Heh, yeah, I don’t think anyone’s quitting their day job aside from the inventors of Powermax – it’s not in the least bit ironic that the original Powersauce was made out of apple cores and chinese newspapers. The big money in this field was already made by Mike Smith selling proprietary stats – it must be in the millions – and everything else is chump change going forward.

      • Triumph
        January 9, 2012 at

        I think it’s one thing to sell proprietary stats to a team or organization, but quite another to attempt to sell them to the general public via articles and blogs. There’s been some awfully dubious work in that field over the last ten years as Tyler pointed out.

        If someone spent a lot of time and came up with something really descriptive/useful and couldn’t sell it to an organization, then fine, try to sell it, but the mixture of overconfidence, hucksterism, and lack of reflection in the for-profit sports stats world is more than a little disappointing when they’re peddling clarity and objectivity.

    4. January 9, 2012 at

      Here is the fighting research findings on momentum you’ve been waiting to chew on…I respect your skepticism and questions as Momentum is a new concept to everyone – thanks for the honest feedback. We believe over time you’ll find as we have that it is an excellent proxy to replicate the ups and downs of each team’s momentum swings as you would see and feel if you were watching the game live. A momentum meter is essentially designed to display an entire detailed box score (with context) in one simple picture.

      Measuring the Impacts of NHL Fights on Momentum:

      http://powerscouthockey.com/blog/measuring-impact-nhl-fights-momentum-and-scoring-well-potential-coachs-strategy

      • dawgbone
        January 9, 2012 at

        My problem is this wording here:

        However, momentum in hockey has a peculiar characteristic to it. It is fleeting. Teams do not possess it completely throughout a game. Rather it seems to appear spontaneously after events in the game when a goal, fight or successful penalty kill trigger a team to play with more determination.

        It also just happens without a triggering event. People continually look for events that might have caused a change. Often times in Hockey it’s not a goal or a hit that swings “momentum” (which is just another word for carrying the play), but rather the natural progression of a game featuring 2 teams who are at similar level in terms of ability.

        When you are on the defensive and the other team constantly has the puck, what often happens is that you get several lines in a row where you can’t progress the puck forward and you are constantly making changes while playing defensive hockey. This is similar to the field position battle in football. One team is constantly starting at their own 10 yard line and find it difficult to progress forward.

        What often ends up happening isn’t a fight or a goal or a big hit, but a break where early on in one of these shifts, the team under pressure is able to move the puck up the ice and gain the zone rather than just dumping the puck in after another exhausting defensive zone shift. The area where the puck starts begins to shift. Instead of just being able to get the puck to the top of the other teams zone and changing, the puck is going deeper into the zone, which allows you to not just change and set up defensively, but change and set up your forecheck. It goes back to the similarity with football where a team starts one possession on the 10, gets 5 yards and punts. Starts on their 10 and gets 10-15 yards and punts. Starts on their 20 and gets 10 yards and punts. Starts on their 20 and the running back gains 15 yards on a play and suddenly they are now punting from their 45 instead of their 20 and are no longer getting hammered in terms of starting position.

        Hockey has natural ebbs and flows when played by teams with similar ability. They exist without fights and hits because even in non-contact rec leagues that sort of play exists.

      • Hawerchuk
        January 9, 2012 at

        Marc,

        In simple terms, what is momentum? Derivative of shots/minute?

    5. Tach
      January 9, 2012 at

      Am I crazy here, or is this not a fair summary of what these guys are saying (and I will assume the premise that “momentum” matters and this is not just Tyler’s no goons on the ice effect at play):

      1. 25% of the time, a fight makes no difference to the play of the game, or both teams experience a detrimental effect from fighting.
      2. 25% of the time, both teams experience a positive effect from fighting.
      3. 25% of the time, your team gets a net positive effect from fighting.
      4. 25% of the time, your team get a net negative effect from fighting.

      So, 1/4 of the time a fight can help your team. 1/4 of the time it helps the other team. 1/2 the time it makes no difference whatsoever.

      So the best advice they can give a coach is “If you tell your fighter to go get in a fight, it is equally likely it will help your team as the other team.” How is this in any way helpful. I certainly can’t see how it can be used in any way to support fighting as a strategic tool like the second article proposed.

      • Passive Voice
        January 9, 2012 at

        Yeah, that was my initial reaction too. Sounds a whole heck of a lot like the result of two coin flips, doesn’t it? :)

        I guess (and this is assuming the math is right and that momentum is actually a real and desirable thing and all that) the application might be when down a goal late in the game. It’d amount to basically the same logic as pulling the goalie; there’s a chance you’ll get more shots/chances to tie it, at the cost of increasing the odds you lose by more than one (which, hey, whatever).

        But yeah, unless the distribution in the probably purposefully vague “at least one team” is wildly skewed towards one side in particular (the trailing team? the home team?), this seems pretty piffling.

        • Hawerchuk
          January 10, 2012 at

          Except nobody fights except when the game is out of reach!

      • Keith
        January 9, 2012 at

        Also, does it even indicate whether the momentum goes toward the fight “winner” or “loser”?

        Home and Away seems like a lousy way of doing it especially if you’re talking about the home team’s player having to start a fight and get his teeth knocked out to generate momentum.

    6. Axeman
      January 10, 2012 at

      Part of the problem is a lack of clear understanding of momentum. PS has decided that shots / 5 minutes quantifies mo. On what basis? Momentum has a number of components. We can discuss seasonal momentum – teams get the feeling they are destined to win, and play with a level of confidence that spills over to increased chances, increased scoring, goalies stopping everything. Watching a team on a one month run of 13-1-1 exemplifies this – players believe they can’t lose, and they don’t, for all kinds of reason (including, of course, dumb ass luck). Do fights affect that?

      There is an in-game momentum, which shows up most obviously when a good team trails a bad team in the third period, and pulls out all the stops to tie it up, while the bad (leading) team continually ices the puck – they know they suck, and are hanging on for dear life. There is in-shift momentum, where Sidney Crosby or Henrik Sedin or someone takes the puck and won’t give it to the other team. Because they are good enough to do so.

      Momentum is the most misunderstood aspect of any sport, and the comment earlier that indicated it can change without a triggering event is spot on. I’d say this is more often than not the case. Fights rarely, in my mind, trigger a shift. Anecdotal evidence, of course – I’ve played for a half century, at a very high level once upon a time, and rarely saw a fight change the course of not just the game, but even the next shift. By the time officials get everyone escorted and equipment picked up, a few ,minutes have passed, and the stick banging from the bench has concluded.

      There’s a cliche in baseball that momentum is your next starting pitcher. In hockey, momentum is having the puck.

      Great piece, great comments …..

      • PopsTwitTar
        January 10, 2012 at

        The problem I am having with all this is “momentum” is the result, not the underlying event. Certain things happen – hits, passes, shots, chances, whatever – and the mind jumbles all that together and say a team has “momentum”. So what? Probably a dumb question, but has anyone shown that having this “momentum” wins hockey games?

        • Hawerchuk
          January 10, 2012 at

          Ironically, just the opposite is true. A team that loses by a goal will have tremendous “momentum” in the 3rd period as they take shot after shot and fail to score, culminating in the desperation that is leaving the net empty.

          On the other hand, a team that scores to tie the game in the third period will see its “momentum” sapped as it now begins to play more defensively. The team that allowed the tying goal will see its “momentum” rise as it no longer concedes territory to prevent goals.

    Leave a Reply

    Your email address will not be published. Required fields are marked *