I recently read a great book by a guy named Dan Gardner, a writer with the Ottawa Citizen, called Risk: Why We Fear Things We Shouldn’t – and Put Ourselves in Greater Danger. I think of him as being sort of a less-pop, more substantive Malcolm Gladwell type – I’m not a huge Gladwell fan, but I do support the idea of critically examining ideas.
He’s got an interesting background, having worked for a few years for Mike Harris’ government in Ontario. If you take some time and read through the archives of his blog or his seemingly defunct website, which I recommend doing, you kind of a get a sense that he’s a guy who has, if not a libertarian bent, at the very least a strong bias towrads evidence, and skeptical of government policy in a lot of areas, particularly with respect to drugs, prostitution and law and order issues.
It’s a bit surprising to me that he lasted as long as he did in the Harris government, given that it was sort of at the forefront of the Americanization of the Canadian conservative movement and strikes me as being closely tied to our current federal government, which seems to govern on the basis that the only thing more important than actually solving a problem is being seen to have solved a problem by a sufficiently broad segment of the electorate that you can ensure your re-election. It’s a variant of the Oilers’ modus operandi.
There’s a lot of great stuff in Gardner’s book and I’ll probably touch on some of it from time to time in the future. His thesis is basically that people are lousy at correctly perceiving and responding to risk, something that I agreed with before and agree with more strongly now that I’ve read his book. The point I’m interested in right now though is a point that he made about science. Gardner wrote:
Unfortunately, the language of science is the opposite of the simple, definitive statements the media want. In science, all knowledge is tentative, every fact open to challenge. Science never delivers absolute certainty. Insteda, facts are said to be known with degrees of confidence…Uncertainty is so central to the nature of science that it provides a handy way of distinguishing between a scientist tlking as a scientist and a scientist who is using the prestige of his white lab coat to support political activism: Look at the language. If a scientist delivers teh simple, unconditional, absolutely certain statements that politicians and journalists want, he is talking as an activist, not a scientist.
Now, I’m not going to pretend that what I do here is anything approaching science, but I’m probably guilty of this from time to time, although perhaps not as often as I’m accused of, complete with supporting misrepresentations of what I said. I have a suspicion that if Nikolai Khabibulin plays 60 games of .915 hockey, I’ll hear that I said that was impossible, which is not at all what I’m saying. With that said, one of the things I’m struck by with the Khabibulin signing is the complete absence of any evidence that says, “Yeah, this is a good idea” that doesn’t involve looking at things that are heavily team influenced, like Stanley Cups and playoff success.
In light of some of the comments to my previous post, as well as the inevitable signing of a guy with a lot of success to a complete steal of a contract, I thought it’d be interesting to explore the contrast between Martin Biron and Nikolai Khabibulin a bit more. In a previous post, I noted that they’ve both faced a lot of shots since the lockout and that there was a substantial gap in their performance, with Biron having faced 5605 shots with a .912 save percentage, while Khabibulin faced 5628 shots and stopped them at a .904 clip. A 47 goal difference or so over five years, which is a lot – you’re basically talking about two wins a year.
Save percentage is fickle enough that I wondered whether that’s a difference that actually tells us anything about a goalie. I’m not the type of guy who gets all that worked up about save percentage differences over the course of a single season, just because of the randomness involved. If we were talking about a single season with a difference like that, I wouldn’t really care – I’d be interested in the long term.
I set up a study to look at this. It involved going through each year for which there’s at least four years of old save percentage data and identifying pairs of players where a) both goalies have faced at least 5,500 shots over the preceding four years, b) there’s a gap of between .005 and .010 between them and c) they both faced at least 2,000 shots over the next four years. For those interested, the scope of the study covers players between 1983-84 and 2003-04 – obviously, I don’t have the four years I need at the front end to go beyond that.
I was able to identify 247 pairs of goalies who met the criteria. One of my favourite points here is that Dominik Hasek only made two pairs – his save percentage was so much better than the second best guy throughout his time in the NHL that he simply didn’t have any comparators for much of the study. When THN publishes their next “Best Hockey Players of All-Time” magazine, he won’t rank inside the top twenty and that’s criminal. He should, in my opinion, be part of the discussion with Gretzky, Orr, Lemieux and Howe.
Anyway, after I identified my pairs that met the criteria, I proceeded to score the thing. Basically, if the “after” save percentages were within +/- .004 of each other, I scored it a tie. If the goalie with the better save percentage coming in posted a save percentage of .005 or more above that of the fellow with a lesser save percentage, I called it a win. Otherwise, it was a loss. The players identified as having the better save percentage posted a record of 104 wins, 65 losses and 78 ties.
In the circumstances specifically faced by the Oilers, they were considering a player who is five years older than another player. Given that the 5,500 shot pre-requisite is going to knock a lot of players out of the running, I’m not going to be dealing with a lot of particularly young guys. So I took a look at only those pairs in which the player with the lesser numbers was at least five years older than the other player. Again, pretty decisive outcome in favour of the player with the better numbers 15 wins, 7 losses and 17 ties. Sean Burke is actually responsible for four of the losses – he had an atrocious .876 save percentage in 1992-93 and was then below .906 just once in the next eleven seasons.
Looking at this, I don’t see a lot that suggests to me that Khabibulin is the better bet in terms of save percentage for the next four years. It’s another slice of data, another different way of looking at the problem but again, it doesn’t suggest that the Oilers made a smart bet here. The fact that they felt the need to put so much on what looks like a poor bet is all the more perplexing, particularly when the guy with better odds was available for less.
Khabibulin may well be in the minority who ends up beating the odds. I don’t see any soft stuff (injury record, save percentage trends, extraneous circumstances, etc.) that suggests that he will be but that’s not the point I’ve been driving at since they signed him: no matter how you look at it, this just looks like a terrible bet. The Oilers (the hockey ops people anyway) just don’t seem to be very good at identifying and managing risk. Cheering for them is not unlike having an emotional investment in a guy at a blackjack table who hits on 19 with the dealer showing 6. You’re left hoping that he pulls a 2 which, even if it works out, is kind of depressing to cheer for.