Thursday 19 February 2015

The trouble with Net Run Rate

In any competition in which there is pool or round-robin play to rank teams before playoff rounds, there needs to be some method of deciding the relative ranking of teams who finish equal on wins and losses. Ideally, this method will reward the teams that have performed best, and also not create any perverse incentives for teams to do anything other than act in a way to maximise their probability of winning.

A nice example of perverse incentives came in the 1999 Cricket World Cup. Only two teams out of New Zealand, Australia, and West Indies were going to carry on from their group into the next round. The rules were such that teams carried through only their results against other teams that made it to the next round. Prior to the match between the West Indies and Australia, New Zealand had beaten Australia but had lost to the West Indies. Australia therefore needed to beat the West Indies, but also wanted WI to be the team that carried through with them so that their loss against NZ didn't matter. As is traditional in the Cricket World Cup,the method used to rank teams with equal numbers of wins and losses, was net-run-rate (NRR)--the difference in a team's average runs scored per over faced and its average runs conceded per over bowled. Batting second, Australia therefore did a deliberate go-slow in order to win, with their 5th wicket partnership taking an extraordinary 127 balls to score the 49 remaining runs needed for a win. This was designed to elevate the West Indies' NRR above New Zealand's. As it turned out, the strategy was not successful, as New Zealand still had a match against the lowly ranked Scotland, and took extraordinary risks to not only win that match but win it by a sufficient margin for their NRR to overtake the West Indies'.

In the current World Cup, there isn't the same "super 6" 2nd stage where teams only carry through some of their points from the first round, but NRR is still used as the tie-breaker. This system is still flawed, as exemplified by Tuesday's match between New Zealand and Scotland. Anyone looking at the two innings scored could be mistaken for thinking that the match was close. It wasn't. What happened was that New Zealand bowled Scotland out for a very low total, and was almost guaranteed a win. When it was New Zealand's turn to bat, they strove to win the match in a few overs as possible, in order to maximise their runs-per-over figure. The fact that they lost 7 wickets in the attempt meant that they did present Scotland with the sniff of a chance of an upset, but the 7 wickets will have no bearing on their eventual NRR.

This exemplifies three problems with NRR:
  1. The effect of a large win against a lower-ranked team on NRR depends on which team bats first, since the team batting second only bats until it has overtaken the other team's score, meaning that that innings gets a lesser weight in the runs-per-over calculation than an innings where all 50 overs are faced. 
  2. The magnitude of a victory when the team batting second wins is a function not only of how many balls it took the team to amass the winning total but also the number of wickets lost in the process. NRR only takes the former into account. This creates the perverse incentive where New Zealand put their win (slightly) at risk by worrying only about how many overs they used and not how many wickets they lost. 
  3. The ranking of two or more teams should not depend on which one beat up the most on a team ranked well below them. If, as could easily happen, three teams (say, Australia, New Zealand and Sri Lanka), finish in a tie for first place in their group, the determination on goes through the quarter finals ranked 1st, 2nd, 3rd, should not come down to which team beat Scotland b the biggest margin.
So with these flaws in mind, here is a sequence of proposals to replace NRR with a different tie-breaking rule.

Adjustment 1: To deal with the first problem above, use the average margin of victory/loss rather than NRR: If the team batting second loses, its margin is its score divided by the score required to tie the match. This will be less than 1. The winning team's margin is the reciprocal of this--the target score divided by the chasing team's score. If the team batting second wins, its margin is the number of balls available to it + 1 divided by the number of balls actually used. The losing team's margin is again the reciprocal of this. In the case of a tie, the margin is 1.0 for both teams.

Adjustment 2: To deal with the problem of teams sacrificing wickets for the sake of fast scoring, amend Adjustment 1 in the case where the team batting second wins, by dividing the predicted score at the end of 50 overs by the score required to tie (the implicit score predictor in Duckworth-Lewis would work for this, although I'd prefer to use WASP due to its adjustment to conditions).

Adjustment 3: Make the calculations iteratively. Let there be n teams in a pool. Construct the table at the end of pool play using points scored, and using Adjustments 1 and 2 to rank teams otherwise tied. Then remove the bottom-ranked team and give them a rank of n. Now reconstruct the table using only games played amongst the remaining n-1 teams, and again find the lowest ranked team. Give it rank n-1, remove it and reconstruct the table with the remaining n-2 teams, etc. As an example of how this could be beneficial, imagine that in the current world cup, Sri Lanka beat Australia, Australia beat NZ, and all three beat England and the other three teams except that the game between Australia and Scotland is rained out. Under the system in place for this competition, Sri Lanka and NZ would finish ahead of Australia simply because Australia were denied to opportunity to play Scotland. Under Adjustment 3, the games against Scotland would be irrelevant for deciding the relative ranking of the top three teams. *

Adjustment 4: O.K. now I am getting well out of the realm of feasible rules into the kind of competition we would have if the ICC comprised exclusively economists, but it is fun to speculate. My adjustment 2 still does not properly align incentives because maximising the expected margin of victory is not the same thing as maximising the probability of victory. So instead, let's define the margin of victory in the following way. Draw the WASP-worm graph of the percentage probability of winning for the second innings as a function of the number of balls bowled. This is a graph  is contained within a rectangle that has a length of 300 and a height of 100. The value for the team batting second would be the area under the graph divided by the area above it. The value for the team batting first would be the reciprocal. Using this method, it would be possible for the winning team to have a lower score than the losing team, but no matter: this scheme means that the way to maximise your team's tie-break variable would be to maximise your probability of winning.

Adjustment 4 tries to align incentives with the only thing that should ever matter in sport--trying to win--but it doesn't deal with the situations like Australia's go-slow against the West Indies in 1999 (or NZ's go slow against South Africa three year's later that shut Australia out of their own tri-series final). The format used in this year's World Cup does not contain the possibility of such strange incentives, but Adjustment 3 would add that. With an obvious nod to the Gibbard-Satherwaite Theorem and Arrow's Impossibility Theorem, then, let me suggest throwing all of these out the window and instead using the following manipulation-proof tie-breaking formula:

Adjustment 5: Rank all teams leading into the tournament based on recent performances. In the event of two or more teams being tied on points at the conclusion of pool play, their relative ranking will be according to their pre-tournament ranking, fully independent of play during the tournament.

* The ICC might argue that they have addressed the problem in a simpler way by restricting the next tournament to only 10 teams. But the results to date in this World Cup suggest that there will still be some very weak teams and non-competitive matches given the non-competitive process for selecting the 10 teams.

10 comments:

  1. Nicely outlined Seamus. Can I please be in the room when you explain Adjustment 4 to the ICC? If I understand you correctly, a team that is losing all the way and then pulls off a miracle at the end will do very badly under Adjustment 4, despite winning. You are effectively rewarding teams who dominate the match, but give very little attention to the end result. In that case, why not include the first innings as well (convert the predicted score to a probability)? Another option, which I think we have discussed before, is to base the margin on the number of balls remaining when the game is "won", and we define that as the moment where the probability gets over (say) 95% for the final time.


    In any case, despite all the interesting options, my favourite is Adjustment 5. It would also give teams the incentive to take every tour seriously. A caveat is that I'd want to see multiple ranking systems go against each other to see which one is the best predictor of the higher ranked team winning.

    ReplyDelete
  2. Yes, it is possible by my metric to win a match an get a lower contribution to the margin-of-victory measure than the losing team. But a), it would hardly ever happen. (Another coat of varnish for Malinga to bowl Langeveldt in the 2997 WC and get 5 wickets in 5 balls would have done it, but it would be rare), and b), who cares: the incentive is still to maximise the probability of winning at all times, and the points for the win always matter more than the points for the tiebreak variable. More weight can be put on the end result by weighting the graph at the end of the innings more than the start without changing the incentive compatibility of the metric. And yes, it would make sense to include the first innings as well.


    I was going to suggest the 95% metric, but then decided to go with the Gini-coefficient-inspired one instead, partly because of its incentive compatibility, but mostly because I like its complexity!


    But like you, if I had to choose one, it would be #5.

    ReplyDelete
  3. Doesn't Quidditch have a similar problem? If getting the flying thing wins you the Quidditch Cup, but scoring points gets you house points towards the big house cup thing, and there's no time limit to the game, and if both houses in the Quidditch Cup are behind some other house for the house cup.....


    [See latest chapters of HPMOR....]

    ReplyDelete
  4. The analogy (for a non-incentive-compatible version), would be if a quidditch team were ahead by more than the negative of the value of the snitch (so they would win if they caught it), but need to win by a larger margin to go through to the final, and so deliberately don't catch the snitch till they have scored more points.

    ReplyDelete
  5. To be fair, supply of police to some bar at 4am is clearly more difficult than to an exciting tournament during business hours, so its not totally inconsistent of them
    That said, it is certainly dumb.
    In addition, the trespass argument provided elsewhere falls down too. For a person to be trespassed, they must first be asked to leave and refuse. You cant bake-in trespass to ticket T&Cs, it doesn't work that way.

    ReplyDelete
  6. Somehow, this all reminds me of a newspaper cartoon during the Springbok tour of 1981. At every game, there was a ring of police around the edge of the field, facing the crowd not the game. In the cartoon, one policeman had dressed himself up with back-to-front clothes so that he could appear to be observing the crowd while actually watching the game.

    ReplyDelete
  7. As far as examples of rules that incentivise anything other than winning go, I think this one is tough to beat. If you don't know the story then read the link below, but the TL;DR version is that in this particular football match Barbados needed a draw so that the game would go into extra time while Grenada was happy with either a one-goal win or a one-goal loss, so with the match locked at 2-2 late in the game Grenada were trying to score in either goal while Barbados defended both goals. http://en.wikipedia.org/wiki/Barbados_4%E2%80%932_Grenada_(1994_Caribbean_Cup_qualification)

    ReplyDelete
  8. I think Seamus told me about that one years ago; had forgotten. Great story.

    ReplyDelete
  9. For bizarre incentives in sport, I don't think that one will ever be beaten, but Lee Germon's 70-run over might run it close: http://en.wikipedia.org/wiki/Lee_Germon#Most_runs_in_a_first_class_over

    ReplyDelete