About me

nice_port_selfie

Postdoc at UC Berkeley School of Information.

PhD in Statistics and Public Policy.

One-time semi-professional passista.

These days I study semi-parametric causal inference.  I work on developing estimators and techniques which draw on machine learning tools and expert knowledge to carefully answer policy questions.

Email: jacqueline dot mauro at berkeley dot edu

More info: jacquelinemauro.com

Advertisements
About me

Million dollar idea

OK hear me out. Instead of digging tunnels for trains, let’s just make a new ground floor for all our cities one story up and run all the trains and cars and delivery trucks underneath it.

Transit costs? Solved.

Pedestrian deaths? Solved.

Sea level issues? Imma say solved.

Carbon capture? Yea why not.

I see no downside.

Million dollar idea

SF data blog (aka justifying my podcast addiction)

Episode 1: presumption laws

[IN PROGRESS]

OK so I’m gonna try something new. I have started listening to the SF Chronicle’s podcast and I am loving it. But also theoretically I should probably not be listening to podcasts quite so much. But listen, sometimes code is slow.

In any case, they bring up a lot of interesting topics about my beloved city and so I’m thinking I’ll pick up on some of these and go wild on the data side. We’ll see what happens.

The first post I wanna dive into is this idea of presumption laws. This came up in the August 13 episode of City Insider, which interviewed (bamf) Fire Chief Jeanine Nicholson. Chief Nicholson had previously battled aggressive breast cancer, and mentioned in the interview that the laws around workers compensation for firefighters are unusual.

In most cases, a worker has to prove that something about their job caused their injury or illness. With a presumption law, the employer has to prove that the job was not the cause of the injury/illness. Now,

{Job caused illness} = {Job didn’t cause illness}^C

meaning the “event” that the job caused the illness is just the converse of the “event” that the job didn’t cause the illness. This implies that the probability that the job caused the illness is just,

P(Job caused illness) = 1-P(Job didn’t cause illness)

This sort of makes it look like there isn’t a huge difference between what the employer has to prove under presumption laws and what the employee has to prove in the usual case, and so the number of cases that go for the employee might be about the same.

That is, for example, a trial without presumption laws might find that the probability the job caused the illness was .4 and the employee would not get worker’s comp. The same case tried without the presumption laws would find the probability the job didn’t cause the illness was .6, and the outcome would be the same.

Not so. Let’s do this.

merciless maya rudolph GIF by NETFLIX

Firefighting and cancer

First, the numbers. Firefighters get a lot of cancer. In Boston, the rates are estimated to be about twice the overall average [1]. A CDC report [2] found increased risk of cancer, especially mesothelioma.

But is the cancer caused by the firefighting? On that point, we get a lot more hemming and hawing. See,

If you are a fire fighter and have cancer this study does not mean that your service caused your cancer. This study cannot determine if an individual’s specific cancer is service-related. In addition to exposures that you may have encountered as a fire fighter there are other factors that may influence whether or not you developed a particular cancer, and this study was not able to address many of these factors. [2]

And the classic, “may be/association/suggests” language, which gives authors a good way to avoid making statements about causes,

We report that firefighting may be associated with increased risk of solid cancers. Furthermore, we report a new finding of excess malignant mesothelioma among firefighters, suggesting the presence of an occupational disease from asbestos hazards in the workplace. [3]

Proving Causation

The classic and most trusted way to prove one thing causes another is to run an experiment. Experiments should have the exposure randomly assigned and compare the exposed people to a control group who weren’t exposed, which is why we call these sorts of experiments “Randomized Controlled Trials” (RCT’s). In this case, an ideal experiment would involve randomly making some people firefighters. Or randomly sending some firefighters out to inhale/roll around in more toxins than others. In other words, not a workable solution. But in the absence of an experiment, proving causation gets way, way harder. Some even think it’s impossible [4]. Some will trot out the old “correlation is not causation” apple. Sigh. I know.

Image result for correlation is not causation spongebob

In these cases, add the fact that not only does the worker have to prove firefighting can cause cancer in general, they have to prove firefighting caused their own cancer. Statistics is generally much better at looking at large samples than small samples, and a sample of one is about as small as it gets.

So hopefully we’re starting to get a sense of why proving cancer is caused by the job is way harder than keeping your employer from proving the cancer wasn’t caused by the job. That is, suing your employer when the presumption is that the cancer was caused by the job makes it the employers job to prove causation and not yours. And that makes all the difference.

Impact of the law

Unfortunately, we don’t have a lot of data readily available about how many more cases are won in states with presumption laws than without.

In Pennsylvania, the number of claims has increased since their presumption law was passed.

Before Pennsylvania enacted its presumption law, in July 2011, Kachline said, Philadelphia firefighters with cancer filed only about two or three cases a year. By December 2013, 62 had filed claims. [5]

In Texas, the law seems to be giving firefighters very little of a leg up,

Of 117 workers comp cancer claims filed by firefighters in the state since 2012, 91 percent have been denied, according to the Texas Department of Insurance. [6]

Probability of Causation

Alright so we’ve sort of seen that proving that something caused your cancer is hard, but I want to talk about why that’s true mathematically. Turns out, it’s a pretty interesting question. Here, I’m going to lean heavily on the work of my brilliant, generous heroine Prof. Maria Cuellar of Penn.

Anyone who has had the (mis)fortune to take a stats 101 course has heard of hypothesis testing. We set our “null” hypothesis as sort of what conventional knowledge says about the world. We call the null hypothesis H0 (“H naught”). The alternative Ha (…”H A”) is the new theory we’re trying out. With a presumption law, our null is exactly what we “presume” to be true–that the job caused the cancer:

H0: job caused cancer; Ha: job did not cause cancer

Without the presumption law, this is flipped:

H0: job did not cause cancer; Ha: job caused cancer

As many of us have heard (and some of us have repeated) a hundred times, we can’t prove the null, we can only reject it in favor of the alternative. So in the presumption law setting, we might reject that the job didn’t cause the cancer in favor of the idea that the job did cause the cancer, based on the evidence presented. In the usual setting, we reject the employer’s claim that the job didn’t cause the cancer in favor of the claim that the job did cause the cancer, and we give the employee some money. Whew.

The reason these two settings are so different is that we need sufficient evidence in order to reject the null. In order to fail to reject the null, there just has to be not enough evidence to reject it. The problem is not symmetric. It’s much easier to not have enough evidence than it is to have enough evidence.

Now that we’ve established our null and alternative hypotheses, we have to figure out what we mean by “enough evidence.” Often in hypothesis testing we’ll have a question like, “is the mean of group 1 bigger than the mean of group 2?” This looks like:

H0: E(X1) = E(X2) ; Ha: E(X1) > E(X2)

We can estimate the true means E(X1), E(X2) using the average of the data we have from each group. We reject the null hypothesis if the sample average of group 1 is big enough compared to the sample average from group 2 to make us suspicious that group 1 and group 2 have the same means.

In our case, we aren’t interested in the means of groups. We want to know the probability of cancer for an individual if they hadn’t been exposed. We can’t use sample averages anymore, instead we have to look at something a bit more delicate.

Under the usual laws, you have to prove that the cancer you got was from the exposure. Meaning, if you had not been exposed, but everything about you was the same, you would not have gotten cancer. In causal inference these types of “had X, would have Y” statements are called potential outcomes. So for example,

Y(A=1) = 1 and Y(A=0) = 0

would mean that if I’m exposed (A=1) I get cancer (Y=1) and if I’m not exposed (A=0), I don’t get cancer (Y=0). The employee has to show that the probability of causation (PC),

PC = P( Y(A=0) = 0| Y=1, A=1, X ) 

is high. That is, they need to show that the chances are good that the firefighter would not have gotten cancer if they hadn’t been exposed (Y(A=0) = 0) knowing that this firefighter was exposed (A=1), did get cancer (Y=1) and has some other characteristics (X) like their age, their smoking habits, gender, etc. So our new hypothesis test looks like:

(i) H0: PC < t1 ; Ha: PC > t1

Where t1  is some legally determined threshold for “likely enough.” For example, if t1=.5, the employee wins their case if it seems any more likely that the exposure caused the cancer than that something else did. If t1=.75, it means we rule for the employee if there is at least a 75% chance that the cancer is due to the exposure and not something else.

The reason estimating this is so difficult is that for a person who was exposed (A=1), we can’t see what would have happened if they hadn’t been. Aforementioned heroine Prof. Cuellar has some cool estimands for this problem [7].

In the presumption law case, the employer wants to show that the employee would probably have gotten cancer anyway. Meaning, they want to show

P( Y(A=0) = 1| Y=1, A=1, X )  = 1-PC

is high. That’s what they want to prove, which means we start with the assumption of the opposite: the probability they would have gotten sick anyway is small. This gets translated into a hypothesis test of,

(ii) H0: 1-PC < t2 ; Ha: 1-PC > t2

Which we can just move around terms to make:

H0: PC > 1-t2; Ha: PC < 1-t2

Let’s say t1 = 0.75 and t2 = 0.25. That means case (i)–no presumption law–is:

H0: PC < 0.75 ; Ha: PC > 0.75

And case (ii)–with a presumption law–becomes:

H0: PC > 0.75; Ha: PC < 0.75

confused disney animation GIFJust to reiterate (because I’m confusing even myself here, there’s a reason we’re told to stay away from double negatives) case (i) now means: “I assume the probability that this person would have stayed healthy if they hadn’t been exposed is less than 75%.” To win the case, the employee has to show enough evidence that the chances they would have stayed healthy if they hadn’t been exposed are high.

Case (ii) says, “I assume the probability this person would have stayed healthy if they hadn’t been exposed is at least 75%.” To win the case, the employer has to show that the chances they would have stayed healthy without exposure are relatively low. In some sense these seem symmetric, but statistics are always measured with uncertainty.

So let’s say we Do Math and come up with a best guess of the probability of causation that’s 0.7, and we’re pretty sure the truth somewhere between 0.6 and 0.9. We have this range because 0.7 is just a guess, an estimate. We used modeling and sampled data and whatever else to get to it, so we can’t be totally sure it’s exact. But based on probability distributions, we can get a good idea of what a likely range is. So we’re not sure it’s exactly 0.7, but we’re pretty sure it’s not anything higher than 0.9 or lower than 0.6.

In case (i), this means I don’t have enough evidence to reject the null that the probability of causation is less than 75%, because I said there’s a pretty healthy chance it goes all the way down to 0.6. The employer wins. In case (ii), it means I don’t have enough evidence to reject the null that the probability is greater than 0.75%–it could be as high as 0.9 and I wouldn’t be surprised. The EMPLOYEE wins! As long as the range we estimate isn’t totally outside the null hypothesis, the null wins.

It’s harder to have enough evidence than it is to not have enough evidence.

pew pew finger guns GIF by SpongeBob SquarePants

Hopefully this made sense. The long and short of it is, things like presumption laws should theoretically make it way, way easier for employees to win these kinds of cases. That’s because proving you wouldn’t have gotten sick in the alternate universe where you weren’t exposed is much harder than sitting back and letting the other guy try to prove there’s an alternate universe out there where you got sick anyway.

Thanks for listening, all. Hopefully now there’s an army out there rooting for me to keep listening to podcasts and so I’m obliged to continue.


Sources (I’m being lazy with citations, I’m sorry):

[1] https://www.nbcnews.com/health/cancer/cancer-biggest-killer-america-s-firefighters-n813411

[2] https://www.cdc.gov/niosh/pgms/worknotify/pdfs/ff-cancer-factsheet-final-508.pdf

[3] https://oem.bmj.com/content/71/6/388.full

[4] My mother

[5] Daniels RDKubale TLYiin JH, et al Mortality and cancer incidence in a pooled cohort of US firefighters from San Francisco, Chicago and Philadelphia (1950–2009) 

[6] https://www.pewtrusts.org/en/research-and-analysis/blogs/stateline/2015/12/07/special-treatment-for-firefighters-with-cancer-some-states-say-yes

[7] https://www.houstonchronicle.com/news/houston-texas/houston/article/Despite-Texas-law-nine-in-10-firefighters-with-13144635.php

[8] https://arxiv.org/pdf/1810.00767.pdf

Some other useful info:

https://www.safetynational.com/conferencechronicles/presumption-laws/

Presumptive Legislation for Firefighter Cancer

SF data blog (aka justifying my podcast addiction)

Prediction v Inference v Causal Inference

Maria Cuellar and I were on a long drive back from a conference recently, and to keep ourselves entertained we had a wide-ranging argument about the difference between prediction, inference and causal inference. Yea, this really is how statisticians have fun.

I was confused about where inference fit in the whole story. I figured, prediction is just fitting a model to get the best \hat{Y}, regardless of the “truth” of the model. If I find some coefficients \hat{Y} = \hat{\beta_0} + \hat{\beta_1}X + \hat{\epsilon}, I’m only saying that if I plug in some new X, I’ll predict a new \hat{Y} according to this model. Easy.

If I care what the real relationship is between variables, I’m doing inference, right? That is, I claim Y = \beta_0 + \beta_1X + \epsilon because I think that every increase in X really implies a \beta_1 increase in Y, with some normal error. In other words, I think that when Y was being generated, it really was  generated from a normal distribution with mean \mu = \beta_0 + \beta_1X and some variance. I’ll get confidence intervals around my coefficients and say that I’m 95% sure about my conclusions.

But I’m playing fast and loose with language here. When I say “implies” do I mean “causes”? Most people will quickly and firmly say no to that can of worms. But! when people talk about regression, they will often say that X affects Y–affects is just a different word for causes so… what’s the deal? How is this not (poor) causal inference?

Well, it’s sort of still my impression that it is. But that doesn’t mean there isn’t such a thing as inference that’s totally separate from causal inference.

Inference asks the question — from this sample, what can I learn about a parameter of the entire population? So if I estimate the median of a sample, I can have some idea of what the median is in the whole population. I can put a confidence interval around it and be totally happy. This isn’t the same as prediction and prediction intervals, because I’m not asking about the median for some future sample and how sure I am that I’ll my guess of the median will be in the right range. I’m asking about the real, true, underlying median in the population.

So what about that regression example? Well, inference will say, there is a true \beta_1 in the population, such that if I took (X^TX)^{-1}X^TY I would get back \beta_1. Does that mean that \beta_1 has any real meaning? No. It’s some number that exists and I can get a confidence interval around. But if my model is wrong, the coefficients don’t say anything particularly interpretable about the relationship between X and Y.

All that to say, Maria was right and I’m sorry.

Prediction v Inference v Causal Inference

Not good science

I often read The American Conservative, a conservative outlet which I think is generally careful, smart and honest. I recommend it, especially if you’re a liberal who is looking for another viewpoint.

With that said, this article fails in its interpretation of data. Spectacularly. The author presents this figure:

He then concludes from it: “for communities who wish for their children to remain heterosexual, to form heterosexual marital unions, traditional families, etc., neutrality on the matter of sexuality will result in five to eight times as many people claiming homosexuality or bisexuality as would have otherwise been the case.”

Slow down.

This leap is not warranted. Setting aside any ideological disagreements, the scientific argument being made has a number of statistical issues that anyone who has dealt with data should identify at a glance. They are:

  1. The figure has no confidence intervals — we have no way to know if the trends we are looking at would be wiped out by randomness and/or missingness.
  2. We have no information on missingness, coverage errors or the many other issues that arise with survey taking.
  3. We have no idea how these lines were generated (splines? linear smoothers?)
  4. The figure shows the share identifying as LGB by age, not the number who are LGB. If older people are more likely to call themselves straight regardless of their underlying orientation, we would see the same pattern.
  5. This figure tells us nothing about the cause of the trend. To assert that this figure tells us that “neutrality on the matter of sexuality” is the reason behind any trend shown here is way premature.

The author looked at a figure and jumped to a conclusion he likely already believed, because it seemed to lend some support to his beliefs. I think we are all vulnerable to this kind of thinking. Luckily downer statisticians are here to remind you that a scatterplot of a survey can only tell you so much. And that so much is really not that much.

Not good science

Even stats 101 is better with gifs

Everything is better with a gif.

I made some figures for TA’ing last fall, and I like them. Basically frequentist statistics can seem weird, but computers can sample from the same distribution/population over and over again, so I think that’s a handy way to think of it when people talk about repeated experiments.

In this first one you sample from a distribution that is not Normal (meaning it doesn’t look like a bell curve) a bunch of times and each time calculate the sample average. If you keep track of your sample averages in the figure on the right, they start to look Normal. Magic.

 

CLT_from_pois

 

In the second one, you sample from a distribution a bunch of times and each time you calculate the sample average and the 95% confidence interval. The line turns red each time the confidence interval doesn’t contain the true mean (which is 10 in this case). The confidence interval misses about 5% of the time. MAGIC.

 

ConfInt.gif

Even stats 101 is better with gifs

Pubs

I’ve recently gone from having very few publications, to having a couple publications so I’m posting them up here.

In the first, we studied stressors on the US ICBM (Inter-Continental Ballistic Missile) force. The second looked at the Los Angeles Fire Department’s hiring practices, which had come under considerable… fire. In the third I lent a small hand looking at publishing trends in China and the last few are some articles I wrote as a fresh-faced college student.

Enjoy!

  1. Hardison, C. M., Rhodes, C., Mauro, J. A., Daugherty, L., Gerbec, E. N., Ramsey, C. (2014). Identifying Key Workplace Stressors Affecting Twentieth Air Force: Analyses Conducted from December 2012 Through February 2013. Santa Monica, CA: RAND Corporation, RR-592-AF.
  1. Chaitra M. Hardison, Nelson Lim, Kirsten M. Keller, Jefferson P. Marquis, Leslie Adrienne Payne, Robert Bozick, Louis T. Mariano, Jacqueline A. Mauro, Lisa Miyashiro, Gillian S. Oak, Lisa Saum-Manning. (2015) Recommendations for Improving the Recruiting and Hiring of Los Angeles Firefighters.  Santa Monica, CA: RAND Corporation, RR-687-LAFD. (http://www.rand.org/pubs/research_reports/RR687.html)
  1. Xin S, Mauro J, Mauro T, Elias P, Man M (2013). Ten-year publication trends in dermatology in mainland China. Report: International Journal of Dermatology, 1-5.
  1. Columbia Political Review: “Seeing Through the Fog: San Francisco Provides a Model for Health Care that Works” (http://goo.gl/Fas4t) and “Empowe(red): Ethical Consumerism and the Choices We Make” (http://goo.gl/g7GzF)
Pubs