• Coleman Analytics

    by  • May 14, 2009 • Uncategorized • 15 Comments

    Lowetide’s linked to a cool Mike Smith post over at The Hockey News in which Smith talks some more about his company and what they have for sale. It’s interesting stuff, although I’ve got some doubts about the value of what he’s selling, unless he’s not even mentioning the good stuff.

    In the fall of 2005 I helped a business consultant, Richard Coleman, put together a company, Coleman Analytics, which provides hockey analytics to NHL GMs and coaches. It’s pretty much hush-hush. We limit the number of franchises because we believe the information is too valuable to let every team have it.

    The clients that use it keep that fact confidential and we do as well. This is easier than you would assume since most GMs have little interest in our data. Every coach who has seen it has wanted it, but unfortunately some of their GMs weren’t so keen on the idea. It’s called teamwork – or is it the lack thereof?

    But let’s start at the end. This season, 2008-09, we have five clients. All five made the playoffs. Last season, we had six clients; five made the playoffs. (That is 10 of 11, if you’re counting). The first two seasons, 2005-06 and ’06-07, we had a total of nine clients and seven made the playoffs. (That is 17 of 20, if you’re counting).

    If you assume that the client list hasn’t changed, only grown and then declined this year – it seems to me that after the initial pitch, it’s probably only going to change if a GM changes – five of the following seem most likely to be the current clients: Calgary, Detroit, Anaheim, San Jose, Pittsburgh, Philadelphia, New Jersey, Washington, New York Rangers, Boston and Montreal.

    Only one of Columbus, Carolina, St. Louis, Chicago and Vancouver could have been clients last year and this year. Given his claimed success rate, the Jackets, Blues and Hawks are probably out, if you assume that teams are more likely to maintain subscriptions to this. Canucks seem possible though.

    I am not surprised that this stuff isn’t of interest to the Oilers.

    Essentially, we break the game into several distinct games. We then track player and team performance during each of these “new games.”

    What makes up these different games? One is when a game is on the line. Another is when the game is out of reach, i.e., a team has a three goal lead or greater. Two others are shorthanded and power play situations. Yes, you can probably think of others: How about power play situations when the game is on the line, or when it is out of reach?

    As someone who’s looked at this stuff, I’m seriously doubtful that the information he’s generating could be of much use. To start with, a hockey game is pretty much always on the line. Matt Fenwick’s made the point: most goals in hockey are big goals. Most moments are big moments. Without having seen Smith’s definition, it’s tough to say definitively but I’ve got serious doubts that he’s capturing anything useful. For example, in 2007-08, about 38% of ES shots occurred with the game tied, another 35% of ES shots occurred in one goal games and

    The three goal lead or greater stuff strikes me as even dodgier, just because it happens so infrequently. I don’t know what kind of conclusions Smith can draw.

    There is one player, who shall remain nameless, who has never been above the bottom third in scoring when the game is on the line and his scoring rate decreases as the game situations become tougher – and he has never made less than $5 million. No, it’s not Alex Ovechkin, nor Teemu Selanne nor Sidney Crosby.

    Because all have earned less than $5MM at some point in the last four years, let alone ever? The bit about never making less than $5MM almost has to be hyperbole, given the NHL’s salary structure. In any event, if you assume that the guy is a forward (as seems reasonable) and made more than $5MM this year, you come up with the following list of candidates: Heatley, Crosby, Ovechkin, Sundin, Briere, Spezza, Gomez, Vanek, Richards, Gaborik, Kovalchuk, Hossa, Smyth, Thornton, Lecavalier, Drury, Iginla, Datsyuk, Nash, Marleau, Elias, Mike Fisher (apparently made $6MM last year; I must have missed all the stories raging about that), Havlat, Kariya, Malone, Sakic, Legwand, Nylander, Alfredsson, Mike Richards, Gagne, Jokinen, Rolston, Bergeron, Naslund, Ribeiro, St. Louis and Staal. Given that it’s presumably someone towards, the upper end of that list, I’ve got no idea who he’s talking about.

    Another example: How do you rate the best goalie? Is it save percentage? How about save percentage on close shots? Or maybe save percentage in the third period when the game is on the line or save percentage in shorthanded situations on close shots when the game is on the line…You get the point. More likely, it is a combination of several analytics.

    I almost think he has to be kidding with this stuff. “Save percentage in shorthanded situations on close shots when the game is on the line”? Depending on how tightly he defines on the line, he could be talking about 30 or 40 shots annually. I’m not sure how a combination of several analytics matters, unless he’s discovered that some parts of save percentage are exceedingly repeatable while others are exceedingly random. I’ve got significant doubts that he has.

    Hey, maybe there is no need for the GM to know that the player he just traded for, who makes more than $5 million, scores in the 27th percentile when the game is on the line. Really, why would a GM want to know this? His owner might. But then, maybe it is valuable for the GM to know that the defenseman he can get for a fourth round pick at the trade deadline, who makes less than $1.25 million and has another year on his contract, is in the 81st percentile for plus-minus in the third period when the game is tied.

    You almost think, reading this, that Smith hopes it’s going to be read by someone who shunned him and proceeded to trade for a player who makes more than $5MM who scores in the 27th percentile when the game is on the line, however that’s defined. Fun fact: the only forward moved at the trade deadline this year who made more than $5MM was Olli Jokinen. Brad Richards, Sergei Fedorov and Marian Hossa might fit the description from the year before. Maybe Tkachuk, Bertuzzi or Smyth from the year before that.

    In any event, while I think that it’s good that better ideas are coming to hockey, it seems to me that we’re in the game winning RBI phase. In twenty years, it might get interesting. Until then, this needs some open sourcing.


    15 Responses to Coleman Analytics

    1. Vic Ferrari
      May 15, 2009 at

      Yeah, the clutchness stuff seems like snake oil to me. I mean I’m sure there is more to it than that, and he has given no indication as to what is being provided in the scouting stats package for the coaching staffs.

      It would take no more than three hours for me to prove him right or (far more likely) wrong. And if I could do it in that time, what’s to stop the team’s from doing it? They have even more NHL data, and available in convenient database format.

      I know he’s only charging $50k or so per year. And a lot of these GMs are old friends. Still, looks like a nonstarter to me.

    2. Vic Ferrari
      May 15, 2009 at

      Also, in a recent article in a BC paper, he claimed to be able to separate a players contribution from his linemates. That’s a hell of a trick if he really can do it.

    3. Vic Ferrari
      May 15, 2009 at

      To add: If Dennis franchised the scoring chance thing, and could show in a mathy but imminently understandable way that there was consistency in marking chances across the board. There would be a terrific market for that I’m sure. If he included blue line turnovers at the same time, then even moreso.

    4. Olivier
      May 15, 2009 at

      I was quite surprised by it while watching the Pittsburgh/Was game #7 on RDS… They used the Versus feed and, ‘lo and behold! They were showing scoring chances! And their count didn’t seem absurd either…

      Another point: “Matt Fenwick’s made the point: most goals in hockey are big goals. Most moments are big moments.”

      Where did he write about that? I’d love to read some more about this…

    5. Robert Cleave
      May 15, 2009 at

      Here’s one from the archives at BoA, Olivier.

    6. Vic Ferrari
      May 15, 2009 at

      And to add one more thing:

      I suspect that Mike Smith is targeting NHL owners with this article, in an effort to sell his product.

      That’s the theory that ticks all the vague boxes in my head, anyways.

    7. MattM
      May 15, 2009 at

      I’m going to go with Olli Jokinen as the mystery player. Smith almost certainly means “has never made less thatn 5mil since I started selling these numbers”, which from memory coincides with when Jokinen signed the deal he’s on now.

      Also, how useful would it be to have the NHL game sheet data in a database? If I could build that, what would you want in it and how would you want it organized? I’m a pretty lazy guy, so no promises, but if you guys spec out for me what you’d actually want, I could see about doing it. One caveat is that I’m mostly working with MS tech right now so it would end up in a MS friendly format, but I have some handy web scraping tech at my disposal that might allow me to do this.

    8. May 15, 2009 at

      Sounds like somebody is paying for useless “clutch scoring” tables to me.

      Two years ago the NHL team I blog about was approached about a NHL draft optimization program that someone was selling. That’s about all I can say about it, but I’m of the opinion that such a program would be much more valuable than what Smith is peddling (assuming there isn’t something else more amazing that comes with the package).

    9. Hawerchuk
      May 17, 2009 at

      Never underestimate what teams are willing to pay for; and what ideas they’ll reject out-of-hand. Lots of people get excited about an abstruse regression even if they don’t understand it…

      Through a convoluted sequence of events back in 2004, I ended up talking to one of the many Sharks owners about leverage index in hockey (essentially what Smith is selling). Paraphrasing the guy’s response: “What a bunch of crap. We don’t use it.”

      Funny thing about Dennis’ work – which I personally think is the right way to go – I talked to a guy from the Kings and their coaching staff counts scoring chances like Dennis does – but they want to switch to a feed-based system. I pointed him in your direction to see the counter-point…

    10. May 18, 2009 at

      Re: Smith’s success claims. Consider the selection bias problems.

      1) More innovative teams are more likely to take a chance on buying something. Their on ice success may be the product of being more open minded in the first place.
      2) It is also possible that high revenue teams are more disposed to throw some money at this. When you consider that 53% of all 30 teams make the playoffs, and roughly two thirds of high revenue teams qualify–well the track record claims look a bit less impressive.

    11. mc79hockey
      May 19, 2009 at

      Through a convoluted sequence of events back in 2004, I ended up talking to one of the many Sharks owners about leverage index in hockey (essentially what Smith is selling). Paraphrasing the guy’s response: “What a bunch of crap. We don’t use it.”

      I’m open to the clutch stuff, I’d just like to see it have a rational basis. The problem is, there really aren’t a lot of minutes in a hockey game that are REALLY high leverage. Hockey is a pretty generally high leverage game. A single goal means a lot.

      If you’re going to go down the road that Smith has done, it seems to me that you need to quantify what you’re doing. I don’t think that if he tried, he’d find that there’s much of a difference, leverage wise between an awful lot of minutes in a hockey game.

    12. May 19, 2009 at

      re: clutch scoring.

      If it exists the correct way to measure it is as follows. Does player X consistently score points that raise his team’s win probability more than an average NHL player.

      To solve this you’d need to calculate and assign the win probability to every goal/point over a couple of seasons to show if someone had a consistently ability.

      Years ago I looked at Game Winning Goals (GWG) to see if the stat was just complete garbage or even slightly useful. For guys who score a lot of goals (Sakic, Lemieux, etc) the GWG was essentially random over their career given that they score a lot of goals and played for teams than won many games.

      However, I did find that set up men like Adam Oates had a higher than expected % of their goals were GW goals. My interpretation is that a guy like Oates prefers to pass, but is more likely to shoot in close and late situations and therefore ended up with a slighter higher % of GW goals (as a share of all goals that he scored). I could be wrong about that, but that’s my story.

    13. Hawerchuk
      May 21, 2009 at

      MC/11: I was paraphrasing the Sharks guy, but he essentially agreed with you – a one-goal lead, any time, was something to aggressively protect, even if the leverage (win% for an additional goal) is much higher in the last minute of the 3rd period. You see this in scoring rates – they are much lower with a one-goal lead than the average GPG.

      I just did a piece on which players get the most icetime in the last two minutes of tied or one-goal games. It correlates pretty well with who sees the toughest minutes in general. Coaches and GMs know who they can rely on in these situations (eg – Brian Campbell sits on the bench, regardless of his PP skills), and I wonder how much Smith adds to this with his analysis?

      If he identified one player in the entire league who’s over-matched against the other team’s top line, is that not something a lot of people could figure out for less than $50,000?

    14. May 21, 2009 at

      You see this in scoring rates – they are much lower with a one-goal lead than the average GPG.

      Hawerchuk: How do scoring rates with a one-goal lead compare to rates with a one-goal deficit? Or, with score tied? My guess is that the latter situation produces the lowest scoring rates towards the end of games, at least regular season games.

      Brian Campbell sits on the bench, regardless of his PP skills

      Hard to believe the guy who just signed a $50+ MM contract isn’t an automatic choice to be out there. That contract will be a millstone in two dimensions — cap hit, duration — that costs Chicago one or two younger, better defencemen (James Wisniewski being candidate #1).

      One of the more interesting moments of these playoffs occurred when the camera caught Scotty Bowman reacting to an ENGA that sealed a Hawks’ loss. It was easy to read the animated, angry Bowman’s lips as he snarled: “Campbell! What the hell was he doing in the corner?!” And, as usual, the legend was absolutely right.

      Campbell’s recent blueline turnover that resulted in a 3-on-1 against and the GWGA in OT, and his insistence that he would try it again, is more evidence in a bulging file of a guy who doesn’t have the firmest grasp of cost v. benefit. Fun to watch, but …

    15. Hawerchuk
      May 22, 2009 at

      Bruce – it’s been a while since I ran these numbers but what I mostly remember was that the likelihood of tying the game given a one-goal deficit was much lower in reality than if you assumed scoring was uniform for all leads and time remaining. I should run these numbers again – it’s an interesting effect.

    Leave a Reply

    Your email address will not be published. Required fields are marked *