Tuesday, October 9, 2012

Difference in Difference and the Alcohol Purchase Age: A Stillman Preview

Reducing the alcohol purchase age to 18 really doesn't seem to have increased youth drinking, or at least not on the aggregate statistics. I couldn't see any case for worsened outcomes on the aggregate trends. Steve Stillman kindly forwards the tables from a paper he's now finalising using proper difference-in-difference and regression discontinuity design techniques to look at the effect of the alcohol purchase age change. 

Difference-in-difference method compares changes in a control group's behaviour with those in the treatment group. The counterfactual then is that changes in drinking among youths would have mirrored changes in drinking among 22-23 year olds but for that the alcohol purchase age dropped to 18.

In Table 3, below, Stillman reports difference-in-difference results on drinking behaviours. And I'll apologise here for pasting in an image of Stillman's tables; when he has the full paper up, I'll link it.

In the models with statistical controls for income, ethnicity, deprivation index and the like, Stillman finds evidence of relative reductions in drinking among younger cohorts. Fifteen to seventeen year olds reduce their binge drinking rates, relative to the change among 22-23 year olds, and also reduce their likelihood of suffering alcohol-related injury.  By 2006/2007, that cohort also was consuming fewer drinks per drinking session and had reductions in the log of total consumption, again relative to consumption among 22-23 year olds. 


Looking at Part B of the table, which reports the results from specifications controlling for other covariates, we find some evidence of improved outcomes among 15-17 year olds and no evidence of worsened outcomes. 

The binge drinking rate among 15-17 year olds was 13.3% in 1996/1997. Steve kindly, and patiently, corrected my initial misreading of the results: the -0.15 coefficient suggests a 15 percentage point reduction from 13.3 rather than a 15% reduction. I had incorrectly assumed that the tables presented transformed results as the fifteen percentage point reduction seemed implausibly large - binge drinking rates obviously cannot be negative. The results then tell us that binge drinking must have increased among the 22-23 year old control group. 

Similarly, the rate of alcohol-related injury dropped substantially relative to that experienced among 22-23 year olds. 12.7% of kids in that age cohort reported ever having had an injury subsequent to drinking in 1996/1997. The 8.8 percentage point reduction in the injury report rate would seem to have dropped reported injuries to a third of their rate prior to the change in the purchase age but, again, the counterfactual here is that the injury rate among 15-17 year olds would have increased in line with the increase among 22-23 year olds. 

Relative to the counterfactual that 15-17 year olds' results would have mirrored those among 22-23 year olds, the difference-in-difference estimates tell us that 
more than 7,300 fewer kids presented with alcohol related injuries than we would have otherwise expected and that more than 12,000 fewer kids participated in binge drinking.

The model cannot distinguish between changes caused by the reduction in the purchase age and anything else that might have simultaneously affected binge drinking rates among kids but not among the comparison cohort, or that might have caused young adults in the 22-23 year old reference group to start binge drinking. But it’s hard to point to other policies that would have had the age-specific effects.

Results on reductions in drinks-per-session are also substantial: a one drink reduction where the mean otherwise was just over three. The rest of the table shows no significant effects of the reduction in the alcohol purchase age: no increase in the likelihood of drinking or in the frequency of drinking. I couldn't see anything in the aggregate stats suggesting increases in youth drinking after the reduction in the purchase age; Stillman can't find it using difference-in-difference either.

Doug Sellman had suggested that binge drinking rates could mask what's going on in the upper tail of that distribution: those who binge drink might be consuming a lot more harmfully consequent to the change. I suppose we can't rule it out, but per-session consumption and the log of total consumption are both down; the maths then require that group is fairly small or that everyone else's consumption is down by more than is suggested above.

Stillman finds evidence of increased hospital admissions for alcohol use disorder. But, the effects are not large. Figure 1, below, uses regression discontinuity design (RDD) with a break at the time of the law change. RDD isn't possible for binge drinking because we there only have three waves of annual data. The advantage of RDD over difference-in-difference is that we can look at the period right around the change in the purchase age, making it unlikely that other changes have affected outcomes. But, if it takes a while for behaviours to change subsequent to policy changes, RDD will have a harder time picking that up.

Yup, there's a break at the time of the law change. The biggest effect was among 18-19 year olds. Alcohol-related hospital admissions rose from about 6 per 100,000 to about 10 per 100,000 per month. There were about 55,000 kids in that age cohort at the time of the law change. So the RDD design suggests that the law change resulted in perhaps two additional 18-19 year olds per month showing up for treatment. It's a relatively large proportional change, so regressions will show large percentage effects. But again, the base rate is tiny. In difference-in-difference estimation, the largest estimated effect was a 49.6% increase in hospital admissions related to an alcohol use disorder among 18-19 year olds three years after the law change. That corresponds to about three people per year.

What about traffic accidents? Stillman finds a small initial decrease in alcohol-related car accidents among 18-19 year olds after the purchase age declined, followed by an increase. 


The regression discontinuity design finds no significant effect. When Stillman runs the difference-in-difference estimates, he finds no effect of the change in alcohol purchase age on alcohol-related traffic accidents in models with a full range of demographic controls that allowed for age-specific time trends. A model with a more limited set of control variables suggested no effects on traffic accidents until three years after the law change. At that point, accident rates increased by 13.7% among 15-17 year olds and by 25.3% among 18-19 year olds. While the percentage increase sounds large, again, the base rate is tiny: the estimates correspond to an increase from 0.5 to 0.57 car accidents per month for 15-17 year olds, or an increase of about one car accident per year. For 18 and 19 year olds, we went from 1.2 accidents per month before the change to 1.46 afterwards: 3.5 additional accidents per year. And, again, those effects disappear with the addition of age-specific time trends, which might just matter once we get to three or more years from the time of the policy change.

In short, it’s pretty hard to build a case for substantially worsened outcomes consequent to the drop in the alcohol purchase age; doing so requires relying on estimates without the full set of desired controls and putting a lot of weight on a pretty small absolute change in the number of accidents. Further, since negative effects on accidents really only seemed evident from three years subsequent to the alcohol purchase age, we have to worry whether anything else happened that differentially affected youth versus adult drink driving rates. And, the results from difference-in-difference estimation suggested pretty substantial reductions in youth binge drinking subsequent to the reduction in the alcohol purchase age, relative to the older comparison cohort.

We are very lucky that clever herestheticians prevented the drinking age from rising. It would have imposed substantial costs on 18-20 year olds for no real benefit. 

No comments:

Post a Comment

As I randomly delete anonymous comments, please use a handle. Comments on older posts go into moderation; sorry if there are delays in hoisting your comment from the moderation pool.