I've recently been reminded that it is good to do policy pieces, in addition to "zomg graphs". This one has been waiting a while to go out, and as I'm giving a talk this week on privacy to at least partially non-technical folks with policy opinions, it seemed good to dust off.
This post is fundamentally about the role privacy technology might play in the world, whether it is a force for good that improves peoples lives, or it provides cover for bad behavior? What properties should privacy technology have if they want to be a force for good, rather than just a source of papers? What analogies will help us and others understand the role privacy technology can play, either for better or worse?
Phillip Rogaway has a paper on "The Moral Character of Cryptographic Work", in which he argues that scientists doing politically consequential work have a moral obligation to shepherd the application of that work. The article is about cryptography, but it applies equally well to shaping political discourse, using machine learning to "optimize" people's lives, and---somewhat removed---to people melting icebergs so that they can repeatedly count words in Java. I recommend everyone read it.
Rogaway doesn't actually make the analogy that privacy technology is like bombs, but his paper is largely basen on the analogy to the atomic scientists: smart, academic types performing what could be seen as fundamental research, but which has much more serious consequences that one should not ignore. The concern that privacy technology could be like bombs, at least in the sense that it could have serious unevaluated downsides, seems totally reasonable.
If you want, you can leap to Page 20 of the article for Rogaway's assessment of differential privacy (a roughly one page read).
Rogaway starts out by introducing differential privacy, with a quote from Cynthia Dwork ("Differential privacy: a survey of results"). Differential privacy "addresses concerns that any participant might have about the leakage of her personal information: even if the participant removed her data from the data set, no outputs ... would become significantly more or less likely."
Rogaway presents his alternate view in a fairly long paragraph, one I will fully quote, but in parts:
At some level, this sounds great: don’t we want to protect individuals from privacy-compromising disclosures from corporate or governmental datasets? But a more critical and less institutionally friendly perspective makes this definitional line seem off.
I don't know Phillip, but I do know Cynthia, and "institutionally friendly" isn't a description I would ever produce. Admittedly, she doesn't spend her time wearing black and smashing Starbucks windows.
Rogaway has three specific complaints:
Most basically, the model implicitly paints the database owner (the curator) as the good guy, and the users querying it, the adversary. If power would just agree to fudge with the answers in the right way, it would be fine for it to hold massive amounts of personal data about each of us. But the history of data-privacy breaches suggests that the principal threat to us is from the database owner itself, and those that gain wholesale access to the data (for example, by theft or secret government programs).
I can't speak to the narrative that Rogaway has constructed for himself, other than to say that I and most people I know strongly disagree with it. I'm not familiar with anyone who thinks DP means we should have more government surveillance, and I'm pretty sure Cynthia in particular doesn't believe that. The implication that we are ignorant of or complicit in data breaches is some serious poo-slinging.
Second, the harm differential privacy seeks to avoid is conceived of in entirely individualistic terms. But privacy violations harm entire communities. The individualistic focus presupposes a narrow conception of privacy’s value.
I don't understand this complaint. I agree that the harm differential privacy seeks to avoid is in terms of the disclosure of individual data, but nothing about differential privacy restricts its guarantees to individual harm.
Consider the recent Equifax debacle (ed: this has been in the outbox for a while), which we all expect will cause substantial harm to a broad collection of people, both people in the collections whose data are disclosed and people not in the collections who nonetheless have to deal with the fallout in their communities. Had differential privacy been respected none of the individual disclosures would have happened, and we are forced to speculate about the community privacy harms that Rogaway fears.
It is fair to worry about e.g. the disclosure of statistical information about communities, e.g. creditworthiness in a subpopulation, something that differential privacy might casually reveal. To the extent that you want to suppress statistics about communities, you want something other than differential privacy. I would probably say "something other than 'privacy' at all", because for me the word "privacy" describes individual rights (note: not individual harm), but it is reasonable to disagree about the word.
Finally, differential privacy implicitly presupposes that the data collection serves some public good. But, routinely, this is a highly contestable claim. The alternative of less data collection, or no data collection at all, is rarely even mentioned.
Rogaway can contest the claim until he is routinely blue in the face. It won't erase the cases where access to data does have social value.
The US Census bureau is tasked with collecting data that is used in no small part (as I understand it) for the enforcement of the Civil Rights Act. This sensitive information contains personal demographic information, as well as several indicators of "how is life working out for you?", and the results reveal whether the US is failing various subpopulations. The proposed alternative of "just don't track access to education by race" or "just don't collect salary information by gender" sounds like a fine solution only to people who are not well-positioned to see what goes wrong when we ignore this data.
And, the reason that differential privacy papers don't discuss less or no data collection is the same reason that cryptography papers don't start with the suggestion "maybe don't have so god-damned many secrets!" The work is presented under the premise that it could be useful, but whether it should be used is an excellent question that just about all privacy researchers I know have opinions on (both pro and con).
In the end, one must compare the reduction in harm actually afforded by using differential privacy with the increase in harm afforded by corporations having another means of whitewash, and policy-makers believing, quite wrongly, that there is some sort of crypto-magic to protect people from data misuse.
Absolutely. One must compare these two things, ideally on a case-by-case basis. Should Google collect your medical history to advertise at you better; probably !#@$ing not. Should Google release statistical information about pay inequality by gender; probably !@#$ing yes. We can reach different conclusions about different problems.
Differential privacy, and privacy tech generally, is about enabling opportunities that might not otherwise exist. In the same vein, atomic energy enables various opportunities, some of which are terrible. Like most atomic scientists, I have serious (occasionally public) opinions on what you should be doing with the stuff I've had a part in making, which I think is what Rogaway actually wants.
Here is a better analogy. Privacy technology is a lot like condoms. Seriously.
Both are designed so that two consenting parties can safely participate in what they otherwise believe to be a mutually beneficial encounter, without the concern of unpleasant unintended consequences down the road. Both are meant to provide the essence of the experience, and while there may be disagreement on how well this is achieved---one party may really wish they didn't have to use them---the risks of unprotected interaction may outweigh the benefits to the point that it isn't going to happen otherwise.
I think that condoms are generally viewed as a good thing. At least, I'm not aware of the counter-arguments, and I don't much agree with the folks I see on television who don't want to make them broadly available.
Let's see if we could frame three complaints about condoms, unfairly pinning these opinions on Rogaway. I also apologize for using vanilla gender roles; any number of any people should use any strictly positive number of condoms in whichever ways they mutually agree upon.
At some level, this sounds great: don’t we want to protect individuals from unintended pregnancies and sexually transmitted infections? But a more critical and less dude-friendly perspective makes this definitional line seem off.
Most basically, condoms implicitly paint the dude as the responsible one, and their partner as the liability. If dudes would just wear condoms, it would be fine to have sex with whomever they desire. But the history of unplanned pregnancies and sexually transmitted infections reveals that most involve some irresponsible dude.
Second, the protection condoms afford are about the interaction of two individuals, but unplanned pregnancies and sexually transmitted infections harm entire communities. The individualistic focus presupposes a narrow conception of safe sex's value.
Finally, condoms implicitly presuppose that sex is worth having. But, routinely, this is a highly contestable claim. The alternative of less sex, or no sex at all, is rarely even mentioned.
In the end, one must compare the reduction in harm actually afforded by using condoms with the increase in harm afforded by dudes having another means of getting in your pants, and sexual-health proponents believing, quite wrongly, that there is some sexo-magic to protect people from unplanned pregnancies and other STIs.
Ideally you agree that these arguments aren't exactly top tier. That third one is down-right conservative.
So what is the difference? Privacy technology is meant to be similarly liberating. A hot company wants your data, but doesn't want to use privacy technology? Tell them to !#@$ off unless they do, because you know that they have the option. "Trust us, we'll anonymize your data later"; !#@$ that, you can apply the randomization yourself. Or whatever, decide for yourself because you are a grown-up and have that right.
Rogaway's argument seems to be that differential privacy empowers authoritarians, giving them cover to do what they want. Of course, you could make the same argument against condoms, that they empower and provide cover for sexual predators, but somehow as a society we've figured out that such an argument is bullshit. Condoms, and understanding sexual health generally, have been incredibly valuable for a substantial number of interactions that were going to happen one way or the other.
You could make the argument that, rather than condoms, privacy technology is like Rohypnol. I'd love to see the argument, but the gap here is consent. My understanding is that there are many consenting parties who would like to safely exchange data, for their mutual benefit. Not everyone wants to exchange data, and absolutely no part of differential privacy is about removing consent (and "opt-out because dp" is a bullshit position). Nonetheless, educating people about privacy technology seems better than just advocating data abstinance.
I don't know, I see privacy and condoms as analogous.
Just FYI.
I think the main difference between the two analogies is that like condoms differential privacy still involves consent, in a way that having an atomic bomb dropped on your town does not. Differential privacy is a way to facilitate two consenting parties who want to do something that isn't fundamentally zero-sum, but which done badly has many risks. Bombs are a way for one party to get what they want, and are not generally employed between consenting parties.
What are better analogies? How should we decide whether what we've chosen to work on (by volume, quite a lot of data-Rohypnol) is ethical? How should we direct the future course of work we've done to improve society?