What is This Castrol Index Crap?

As regular readers know, I'm not a big fan of Atiba Harris. I think he's a limited but underwhelming winger who's being played at forward for no immediately obvious reason. In fact, I even ran a post called "The Bench Atiba Harris Club" before the poor guy underwent knee surgery and granted me my wish in a rather extreme, unwelcome manner.

The reason I mention this is that, today, Major League Soccer unveiled something called their "Castrol Index": something that aims to be an index computing the value of each player in Major League Soccer. And, as Harris's smiling defenders did not hesitate to point out to me as soon as the index came out, Atiba Harris is ranked very well: he's coming in fifth in the league, behind only Thierry Henry, Steve Zakuani, David Ferreira, and Omar Gonzalez. Pretty illustrious company.

So will I now admit that I was wrong and that Harris, ranked here as one of the five best players in Major League Soccer regardless of position, is a valuable part of the Whitecaps organization? Of course I won't. I'm a bit of a statistics maven and this particular statistic reeks to high heaven for so many reasons I can't even begin to comprehend why anybody could possibly take it seriously.

Listen. I'm not writing this article to attack Harris again; that's not the point. The point is that this statistic plainly cannot be trusted. And yes, I will tell you why.

A good rule of thumb for any statistic is to see if the results it generates pass the smell test. Obviously a statistic is meant to show us what our eyes don't see, but if it's clearly ranking players in a completely mental manner then it should be viewed with suspicion.

The Castrol Index are collated through the end of April. So close your eyes and remember every Vancouver Whitecaps game played in March and April. Think about the players you saw there and approximate where you'd rank them if you had to place them in order of effectiveness. Write it down, even. Give it your best, most serious effort. Then compare your results to what the Castrol Index tells you is correct. Ready?

All right. Now, everybody who thinks Camilo was a better forward than Eric Hassli, raise your hand. How about everyone who thought Michael Boxall was a more useful defender than Alain Rochat? Everybody who thought Blake Wagner was better than Gershon Koffie and Davide Chiumiento? Everyone who thought that Russell Teibert wasn't one of the six best midfielders on this team and was comprehensively outplayed by Jeb Brovsky? How about everyone who thought that John Thorrington, who was very good in his brief appearance in the New England Revolution match, was significantly less effective than Kevin Harmse?

This index bombs the smell test. We probably should have taken the hint when it ranked Atiba Harris as the fifth-best player in Major League Soccer but sometimes it's good to be thorough. Whether it's just because of poor sample sizes a month and a half into the season or because the statistic is just calculated wrong, this isn't even a defensible order of the most valuable players on the Vancouver Whitecaps. If you try to stick up for that ordering of players you are blind and mad and I want nothing more to do with you.

Now, to be fair, I should look at how the Castrol Index is calculated. And I'd love to be able to do that; surf on over to MLS's Castrol Index FAQ page to see all the information they're willing to give us. It "uses the latest technology to objectively analyze player performance to help you see soccer from a completely different perspective", they tell us: that perspective presumably being that of a demented five-year-old. There's a whole lot of text in the soothing lime green of a motor oil company, and they actually ask the question "How is the Castrol Index calculated?", which seems helpful. But they don't actually answer it.

2)    How is the Castrol Index calculated?

The Castrol Index tracks every move on the field and assesses whether it has a positive or negative impact on a team’s ability to score or concede a goal.

A key factor for all areas of performance in the Castrol Index is in which zone on the pitch the action takes place. Players receive points for each successful pass they complete, but the number of points awarded depends on which zones the ball is passed from and received in. Similarly, misplaced or intercepted passes are penalised depending on how much trouble the mistake is likely to land the team in.

The Castrol Index is also able to split up the rewards of a goal between penalising the goalkeeper for letting in a shot he should have saved and rewarding the attacker for scoring a goal. The number of Castrol Index points awarded for tackles, interceptions and blocked shots also depends on which zone they are made. Successfully taking the ball from a striker near the penalty spot will earn more points than a tackle out on the wing. Conceding free kicks and penalties will result in deductions.

Seriously? That's your answer? Are you huffing glue? Read that block quote again if it didn't sink in the first time. Not only do they not tell you how to calculate the index, they quite courteously explain what it is, in any statistical sense, completely worthless.

They don't actually tell you how the index is calculated in the section titled "how is the Castrol Index calculated"! They are courteous enough to tell us that it's loaded with subjective factors like "long balls are worth more" (well, that explains Blake Wagner's gigantic ranking) and "we measure interceptions based on how bad we think the resulting scoring chance is". This is, quite literally, not a statistic. If you can't calculate it yourself because it's based on proprietary information and loads of subjective factors computed by random guys who are asking you to take your word for it, what you have isn't a statistic. It's a chance to hawk motor oil to suckers.

But even if this was calculated intelligently, it still wouldn't work. Statisticians in all sports have learned to be wary of "one-size-fits-all" statistics. The best analogue is VORP in baseball; Value Over Replacement Player. It's a single number that tells you how a given player compares to a "replacement player" (a meaningful concept in baseball). I would be digressing if I explained it here, but most baseball fans agree it's at least a reasonably useful statistic. And yet most baseball fans are also fully aware that it's not the be-all and end-all. Just because a player is fifth in VORP doesn't mean he's the fifth-best player in baseball; you'd use other statistics to determine where he succeeds, where he struggles. They also separate pitcher VORP and non-pitcher VORP, because pitchers and other positions do radically different jobs. They'd never do anything as mad as put pitchers and hitters on one big "Castrol VORP Index" and expect people to take it seriously.

If this were a good statistic, it might be a useful comparative tool but certainly not as the definitive, valuable list that Major League Soccer is hyping it up as. But this isn't a good statistic. It's a lump of subjective data made by people who won't show their work. It's a thing. It's a waste of time and space and electrons. If you see anybody taking the Castrol Index seriously, hit them with a rolled-up newspaper. This is worthless.