In 2014 the White House commissioned a 90-day study that culminated in a report (pdf) on the state of “big data” and related technologies. The authors give many recommendations, including this central warning.
Warning: algorithms can facilitate illegal discrimination!
Here’s a not-so-imaginary example of the problem. A bank wants people to take loans with high interest rates, and it also serves ads for these loans. A modern idea is to use an algorithm to decide, based on the sliver of known information about a user visiting a website, which advertisement to present that gives the largest chance of the user clicking on it. There’s one problem: these algorithms are trained on historical data, and poor uneducated people (often racial minorities) have a historical trend of being more likely to succumb to predatory loan advertisements than the general population. So an algorithm that is “just” trying to maximize clickthrough may also be targeting black people, de facto denying them opportunities for fair loans. Such behavior is illegal.
On the other hand, even if algorithms are not making illegal decisions, by training algorithms on data produced by humans, we naturally reinforce prejudices of the majority. This can have negative effects, like Google’s autocomplete finishing “Are transgenders” with “going to hell?” Even if this is the most common question being asked on Google, and even if the majority think it’s morally acceptable to display this to users, this shows that algorithms do in fact encode our prejudices. People are slowly coming to realize this, to the point where it was recently covered in the New York Times.
There are many facets to the algorithm fairness problem one that has not even been widely acknowledged as a problem, despite the Times article. The message has been echoed by machine learning researchers but mostly ignored by practitioners. In particular, “experts” continually make ignorant claims such as, “equations can’t be racist,” and the following quote from the above linked article about how the Chicago Police Department has been using algorithms to do predictive policing.
Wernick denies that [the predictive policing] algorithm uses “any racial, neighborhood, or other such information” to assist in compiling the heat list [of potential repeat offenders].
Why is this ignorant? Because of the well-known fact that removing explicit racial features from data does not eliminate an algorithm’s ability to learn race. If racial features disproportionately correlate with crime (as they do in the US), then an algorithm which learns race is actually doing exactly what it is designed to do! One needs to be very thorough to say that an algorithm does not “use race” in its computations. Algorithms are not designed in a vacuum, but rather in conjunction with the designer’s analysis of their data. There are two points of failure here: the designer can unwittingly encode biases into the algorithm based on a biased exploration of the data, and the data itself can encode biases due to human decisions made to create it. Because of this, the burden of proof is (or should be!) on the practitioner to guarantee they are not violating discrimination law. Wernick should instead prove mathematically that the policing algorithm does not discriminate.
While that viewpoint is idealistic, it’s a bit naive because there is no accepted definition of what it means for an algorithm to be fair. In fact, from a precise mathematical standpoint, there isn’t even a precise legal definition of what it means for any practice to be fair. In the US the existing legal theory is called disparate impact, which states that a practice can be considered illegal discrimination if it has a “disproportionately adverse” effect on members of a protected group. Here “disproportionate” is precisely defined by the 80% rule, but this is somehow not enforced as stated. As with many legal issues, laws are broad assertions that are challenged on a case-by-case basis. In the case of fairness, the legal decision usually hinges on whether an individual was treated unfairly, because the individual is the one who files the lawsuit. Our understanding of the law is cobbled together, essentially through anecdotes slanted by political agendas. A mathematician can’t make progress with that. We want the mathematical essence of fairness, not something that can be interpreted depending on the court majority.
The problem is exacerbated for data mining because the practitioners often demonstrate a poor understanding of statistics, the management doesn’t understand algorithms, and almost everyone is lulled into a false sense of security via abstraction (remember, “equations can’t be racist”). Experts in discrimination law aren’t trained to audit algorithms, and engineers aren’t trained in social science or law. The speed with which research becomes practice far outpaces the speed at which anyone can keep up. This is especially true at places like Google and Facebook, where teams of in-house mathematicians and algorithm designers bypass the delay between academia and industry.
And perhaps the worst part is that even the world’s best mathematicians and computer scientists don’t know how to interpret the output of many popular learning algorithms. This isn’t just a problem that stupid people aren’t listening to smart people, it’s that everyone is “stupid.” A more politically correct way to say it: transparency in machine learning is a wide open problem. Take, for example, deep learning. A far-removed adaptation of neuroscience to data mining, deep learning has become the flagship technique spearheading modern advances in image tagging, speech recognition, and other classification problems.
A typical example of how a deep neural network learns to tag images. Image source: http://engineering.flipboard.com/2015/05/scaling-convnets/
The picture above shows how low level “features” (which essentially boil down to simple numerical combinations of pixel values) are combined in a “neural network” to more complicated image-like structures. The claim that these features represent natural concepts like “cat” and “horse” have fueled the public attention on deep learning for years. But looking at the above, is there any reasonable way to say whether these are encoding “discriminatory information”? Not only is this an open question, but we don’t even know what kinds of problems deep learning can solve! How can we understand to what extent neural networks can encode discrimination if we don’t have a deep understanding of why a neural network is good at what it does?
What makes this worse is that there are only about ten people in the world who understand the practical aspects of deep learning well enough to achieve record results for deep learning. This means they spent a ton of time tinkering the model to make it domain-specific, and nobody really knows whether the subtle differences between the top models correspond to genuine advances or slight overfitting or luck. Who is to say whether the fiasco with Google tagging images of black people as apes was caused by the data or the deep learning algorithm or by some obscure tweak made by the designer? I doubt even the designer could tell you with any certainty.
Opacity and a lack of interpretability is the rule more than the exception in machine learning. Celebrated techniques like Support Vector Machines, Boosting, and recent popular “tensor methods” are all highly opaque. This means that even if we knew what fairness meant, it is still a challenge (though one we’d be suited for) to modify existing algorithms to become fair. But with recent success stories in theoretical computer science connecting security, trust, and privacy, computer scientists have started to take up the call of nailing down what fairness means, and how to measure and enforce fairness in algorithms. There is now a yearly workshop called Fairness, Accountability, and Transparency in Machine Learning (FAT-ML, an awesome acronym), and some famous theory researchers are starting to get involved, as are social scientists and legal experts. Full disclosure, two days ago I gave a talk as part of this workshop on modifications to AdaBoost that seem to make it more fair. More on that in a future post.
From our perspective, we the computer scientists and mathematicians, the central obstacle is still that we don’t have a good definition of fairness.
In the next post I want to get a bit more technical. I’ll describe the parts of the fairness literature I like (which will be biased), I’ll hypothesize about the tension between statistical fairness and individual fairness, and I’ll entertain ideas on how someone designing a controversial algorithm (such as a predictive policing algorithm) could maintain transparency and accountability over its discriminatory impact. In subsequent posts I want to explain in more detail why it seems so difficult to come up with a useful definition of fairness, and to describe some of the ideas I and my coauthors have worked on.
Until then!
You should be careful regarding disparate impact law. Hiring practices that have a disparate impact are not always illegal. For instance, if you are hiring a math professor, requiring an advanced degree in mathematics has a highly disparate impact. Women, blacks, hispanics, etc are much less likely to have advanced degrees in mathematics than men, asians or whites. But it is highly unlikely that requiring a math PhD for a math professor job would be ruled illegal.
This is also not academic. If your algorithm for credit rating disfavors loan applicants with a history of failing at repayment or lower income, it will have a disparate impact on blacks for instance. But it will likely be legal. And I would argue your algorithm is not unfair. It is life which has unfairly discriminated against blacks. Your algorithm is merely noticing this fact.
I don’t think I said anything to imply this, but I think a more interesting example that has not yet seem to come up in a high profile court case is hiring practices for ethnic restaurants. There it is a difficult question what counts as a relevant qualification, because it is well known that people enjoy ethnic food more when it is served by a person of perceived matching ethnicity. But this clearly has a disparate impact.
All that being said, the question is not whether the algorithm is fair by “nature,” but instead what is the property we *want* to impose on an algorithm to fit the fairness demand? Legal disparate impact seems not to be enough.
You wrote “In the US the existing legal theory is called disparate impact, which states that a practice can be considered illegal discrimination if it has a “disproportionately adverse” effect on members of a protected group.” That is technically true, but as stated it would imply that you can’t act in ways that disproportionately negatively impact people based on protected classifications even if you have a good reason to. (e.g. the math professor example above)
(Less important, but still worth thinking about. Discrimination laws do not create protected groups. They create protected classifications. e.g. African-Americans aren’t protected. Everyone is protected from being discriminated against based on race. It is a bit pedantic, but the “protected group”/”protected class” terminology causes people to adopt the harmful myth that anti-discrimination legislation only protected minorities.)
These are good points, and I admittedly skipped over that part of the law in my writeup (though I read enough about it!). In any case, from my perspective I’m still asking the more basic question of what it even means for an algorithm to be fair, which seems unrelated to the question of whether you can justify a discriminatory practice.
Yuck. Didn’t think I was going to be reading about social justice in a math blog, especially not the nebulous claim that data about a group is racist. Bonus points for conflating classism with racism, which is just the modern american way of demonizing any view on social structure by tying it to bigotry. I’d rather not get into the political, which is why I’m reading a math blog and not a politics blog – casual navel-gazing like this isn’t why I’m here!
Would you read about the mathematics of privacy or security? These all have the same flavor of motivations. Even something like PageRank started as an idea about social values. If you don’t like talking about the motivation then come back when I start laying down definitions and writing programs. I just felt like the topic needed some motivation.
“Would you read about the mathematics of privacy or security? These all have the same flavor of motivations. ”
How about the mathematics of pyschometrics, the “g” factor, factor analysis, PCA, bell curves, variance, heritability statistics, causal links, and so on? Somehow I think the (disturbing to some) different means of different population groups might make that territory you’d rather not tread on.
Better stick to how algorithms can be “racist” or something.
As a math+CS major, I’m curious why you think exploring/questioning the ways in which the practices of our field(s) could lead to discrimination is “navel gazing”? Clearly most of the posts on this blog are about super fascinating mathematical topics, so I feel like a few posts like this every now and then that go into the social justice realm are actually very important, especially given the racial and gender disparities that obviously exist in our fields.
Interesting concept in the article, thanks for writing!
I think that there’s a rather large logical error though. The error stems from a difference in proving causation vs. showing correlation. To quote, “here’s one problem: these algorithms are trained on historical data, and poor uneducated people (often racial minorities) have a historical trend of being more likely to succumb to predatory loan advertisements than the general population.” The algorithm would be targeting poor, uneducated people. Not people of color. It’s correlating race with uneducated behavior. It’s not learning directly how races plays a factor. (Entirely different conversation if we, as a society, are okay with this being legal behavior).
If you had a dataset that did not contain any attributes that directly showed an applicant’s race — e.g. having a “race” attribute — then I postulate that it is impossible for a learning algorithm (or any other statistical procedure) to be racist (racism: defined as prejudice on perceived or actual race). Reason being that racism is having the direct relationship: “let me look at your race and make a decision.”
Everyone is continuing to use the word “racism” incorrectly. Racism is the idea that one race is SUPERIOR over another. It isn’t just generalizations about race based on stereotypes. The way these 2 different, arguably VERY different definitions have been conflated is very disturbing.
Apparently the law doesn’t care if it’s correlation or causation. The point is that even if a practice is not racist by intention, it can have an disproportionate adverse impact on a particular racial class, and so it’s still considered illegal. Maybe it’s not called racism, but that’s why I don’t use the word racist in the article, rather discrimination which may or may not be intentional.
Your comment reminded me of a story I read a while back about an algorithm learning to discriminate on ethnicity and gender pretty explicitly.
http://www.theguardian.com/news/datablog/2013/aug/14/problem-with-algorithms-magnifying-misbehaviour
tl;dr: An automated med school admissions program was trained on past admissions decisions and tended to screen out females and people with non-European looking names.
The people who made this algorithm did not intend for this to happen, but I think it’s fair to say that the algorithm was “making decisions” based on gender and/or perceived ethnicity.
I don’t quite understand this article. 2 + 2 will always be 4, whether or not we are talking about 2 black guys, an asian and white dude or 4 hispanics. You would rather add a “weight” field to these “races” is what you are saying I think. The reason there is no exact science to this is because there is no exact science to this and should be avoided. I get that our government has been doing this kind of weighting for like 60 years but I never thought it made anything more “fair”. In fact I’d argue that black people for example would be better off today if the Fed hadn’t intervened so heavily in their lives way back when. The same holds for algorithms, imho.
One might easily accept the claim that “2+2=4” is not discriminatory, but how can you convincingly make the same claim about the behavior of a neural network or a support vector machine, or worse a black-box algorithm? My main claim is that no definition of fairness is clear enough to do this in a useful or convincing way. There is evidence that algorithms can discriminate and that it causes problems, and it is an interesting mathematical modeling problem. So I won’t try to avoid it, and my job is to try to poke at it with science.
I think what you’re saying is not that algorithms are “discriminating” but rather that the data that we have is biased and that bias is reflected in the output of the algorithm?
I’m sure there’s a bad joke about bias and variance waiting just around the corner, btw :/
One problem is bias within the data, but another is bias of an analyst who (by doing a biased analysis of potentially unbiased data) designs a model that encodes these biases directly.
Well Philosophically PrometheeFeu has a point, if quite arguable…
[the “protected group”/”protected class” terminology causes people to adopt the harmful myth that anti-discrimination legislation only protected minorities.)]
But in practice, this is exactly so. You see, I’m nor a “user” of those services. In everyday life this “legislation” indeed protects only minorities. And, on another viewpoint, “Ad hoc” legislation is not a good idea.
I don’t think that’s always reflected in the case law. For example, I seem to remember a group of white male firefighters suing their station for refusing to promote them when the exam they took appeared to be biased against the black firefighters (and hence none of the black firefighters would have been promoted). They won the suit because their management did not have enough evidence that the exam had adverse impact against the black firefighters. (Or something like that; the point is that the law favored the white males)
Well, yes, I see. But this “opera buffa” situation was due to those gauche laws. A distorted thesis, this cause of multiple and unforeseeable effects.
My impression is that you do not separate cleanly between (using ad hoc terminology) direct discrimination and discrimination by correlation (the latter not being true discrimination).
To take a simple example: If the area A has a White–Black proportion of 9–1 and area B one of 1–9, then a decision based on area simply does not discriminate by race. (Direct discrimination regarding area; “descrimination” by correlation regarding race. The consequences might be similar, but the situation is very different. Notably, the minority in each area is treated just like the majority, and someone who moves to the other area will be treated according to the new area.
Failure to make this distinction has done the U.S. a whole lot of harm, notably in form of unduely far-going laws regarding e.g. the “disparate impact” discussed above. Notably, IQ tests, that would be very valuable tools when hiring, are (barring recent changes that I might be unaware of) in most cases illegal and preference given to more topic relevant tests—even when the latter are worse predictors through a too low “g-loading”.
Taken to an extreme, such reasoning would make almost any reasonable means of discrimination impossible. Hiring an engineering graduate for an engineering position or a trained nurse for a nursing position would be impossible, because there are proportionally fewer women than men who have engineering degrees and vice versa.
That said, there are many areas where it could be justified to restrict what criteria an algorithm may use. Not because e.g. certain ethnical groups would be disadvantaged, but because the judgment is made in a too blanket manner, disregarding individual variation, or because the criteria are too secondary: Should a credit card application from someone with a reasonable salary and a good previous record be rejected because he happens to live in an area where disproportionally many others fail to pay their debts? Hardly fair.
Why should discrimination be “untrue” if it is indirect? Tons of discrimination is indirect! Not only would a mathematical definition that only models direct discrimination probably be useless, but it would also likely be uninteresting.
Do you understand what discrimination actually is? Or are you just misusing the word to signify differences in outcome correlating to e.g. sex, race, religion, …?
Discrimination implies that one is able to discern and (optionally) act on a difference, e.g. noting that one job applicant has a useful degree and that the other is a high-school dropout. Most cases of discrimination are not merely legitimate but positive and useful. The unfortunate habit to say “discrimination” and e.g. “sexual discrimination” (or, worse, any disapproved of behaviour, say using politically incorrect words) is wide-spread, but is also ultimately very, very incorrect and ignorant.
Now: If you find it unfortunate that someone has less chances through indirect mechanisms, this may be a worthy topic—but it is not a matter of e.g. sexual or racial discrimination.
Furthermore, a mathematical model which is not developed using correct and strict terminology and concepts, that does not differ between direct and indirect effects or correlation and causation, etc., is damned to be misleading, prone to be abused, incorrect, and/or uninteresting.
Under US law, a practice can be called discriminatory even in the absence of intent. E.g. if people in poorer neighborhoods are disproportionately shown ads for predatory loans because an algorithm decided it has higher clickthrough. They have fewer chances for a fair loan treatment because they aren’t exposed to normal loan offers. Is that direct or indirect? In either case, it’s illegal.
It is also quite silly to say that racial discrimination does not play a huge role in minorities having fewer “chances” due to indirect mechanisms. That’s demonstrably false; just look at school districts in the US.
I am not an expert on U.S. law and it is conceivable that your statement contains too little context or too few details. However, my first impression is that the law is formulated using incorrect terminology, which is just as bad as mathematical models using incorrect terminology. Copying that error is not a good idea.
Nowhere do I say that “racial discrimination does not play a huge role in minorities having fewer “chances””: It is quite conceivable that there are areas, e.g. schooling, where the ultimate cause is based on racial discrimination, hypothetically because good schools or scholarships are more prone to accept applicants with English ancestors. However, this does not alter the point I make in any way. Furthermore, even looking at e.g. schooling it would be simplistic to assume racial discrimination as the (sole, largest, or a major) cause without further investigation, seeing that there are at least two other possibilities to consider, namely discrimination by other factors (e.g. income or social status of the parents) and actual differences in accomplishment (e.g. through different parental attitudes towards schooling, genetic differences in aptitude, a “too cool for school” vs. a “goody two-shoes” attitude, …)
The doctrine of “adverse impact” as part of the 1964 Civil Rights Act (Title VII) codifies it clearly: illegal discrimination need not be direct or intentional. You may think the law should be different, but it’s certainly correct terminology. A benefit of a mathematical approach is that the mathematics can stand for itself regardless of word choice.
Quite the contrary: The law may well be correct—but the terminology most certainly is not.
“Discrimination” has a clear meaning. Incorrect use by others, even by law-makers, should not be taken as a pretext to use it in a different and incompatible meaning.
Petrified with fear, he ran away…
I see that I have a major misformulation above:
“discrimination” and e.g. “sexual discrimination”
should be
“discrimination” and !meaning! e.g. “sexual discrimination”
That’s when you decide what’s fair for society as a whole. Set max interest rates in to law either by the legislature or by an administrative body doing it through regulation. But then what would you do when businesses stopped offering loans to high risk borrowers and that turns out to have a disparate impact on minorities? Do you force the businesses to offer loans that they can say with a high degree of certainty they will lose money on so everybody has to pay higher interest rates to make up for the people that will default but they legally have to loan to? As in there’s now a defacto “floor” to what I can get no matter how good my credit is. It’s easy to say,”well, if that’s what it takes to get people on to an equal footing, then that’s what’s gotta happen”.
How about if someone came up with an algorithm that was pretty accurate on forecasting whether a person up for parole would violate it? Now it turns into,”well, poorer. less educated people that don’t have too many family ties are more likely to re-offend” which unfairly impacts minorities. So now you gotta tell people,”well, there’s this algorithm and we did this 5 year study that shows it works, but the parole board can’t take it into consideration, so even though it says you probably won’t re-offend, your arse is staying in jail”. Yes, parole boards already do stuff like that by taking in to consideration the type of environment the parolee would return to. And a lot of advocates decry that as unfair. But if we could have hard empirical data showing that a certain algorithm works, how is it fair to not use it when it can get people out of incarceration? Should we just go to mandatory sentences for everything without the possibility of parole? A lot of people would call that racist too.
I recently read a somewhat amusing blog post about deep learning: http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html
tl;dr : learning textures on the picture doesn’t constitute for actual image recongnition.
Thank you so so much for this article – as a CS and Math student (and longtime reader of the blog!) who also cares very deeply about social justice, I had never really thought of the links between the two until I read this piece. I’m going to try my best to keep up with the literature on this and hopefully attend the next FAT-ML 🙂
All this, based on the premise that “disparity” is evil, of course. Why? Matter for Math students and logic-loving philosophers (and some economists -bad money displaces-, etc): What could happen in an interaction where population A is concerned and enforces equal rights and diversity and population B that is conservative?
This is a very interesting topic and brilliant and amusing models can be built.
While arguing with the wrong side of the Right, I will frequently argue that races do not exist. A corollary of that, of course, is that races cannot be discriminated against. In other words, any claim of disparate impact is vacuous.
Related: Rational discrimination, The Logic of Life by Tim Harford.
Thanks for the article on what seems to be a topic that has recently grown in interest.
I thought you might enjoy some history where science made a huge error in diagnosis based on bias in the latent variables of the data (the latent variable being economic status), this error is reported to have caused the death of over 10,000 babies during the end of the 18th to the beginning of the 19th century.
http://www.orderofthegooddeath.com/poverty-the-thymus-and-sudden-infant-death-syndrome#.VgkdKrMYz7A
And a fantastic podcast where this is discussed: http://www.radiolab.org/story/91671-how-to-cure-what-ails-you/
Wow, I will take a look at this for sure!
There is a new breed of racists that insist people are different and that some(i.e whites) are naturally more intelligent than others. They insist strongly that the science is on their side.
They tend to be white males, anti immigrant, anti feminist, anti social justice warrior anti Muslim/Arab/black and demonstate a strong sense of victimhood.
I noticed how prominent they were in the comments section of a recent DNews video about race. Very “pro science” in insisting that other races are simply not as intelligent as whites. Lots of thumbs up for it.
Magusjanus reminded me they exist and in significant numbers and are probably growing.
I like the way you handle those who disagree with you. It is an inspiration to me. Keep up the good work.
Your comment gives every impression of trying to artificially discredit a group of people based on their opinions rather their arguments or methods. This is the more unfortunate, since this is an area where the scientific opionion is indeed very far from that of e.g. the average journalist or politician. To boot, it gives the impression of a holier-than-thou attitude that is utterly unwarranted. (I do not know whether this is your intention, but I found it quite common for e.g. PC groups, U.S. Democrats, the “Cultural Creative”, …, to try to gain new adherents by painting themselves as morally superior to their opponents—either you are with us or you are morally deficient. In this they remind me strongly of some religious groups.)
> There is a new breed of racists that insist people are different
That different racial groups can be different is not racism. Racism is considering members of one racial group inherently better than members of an another based, ipso facto, on group membership.
> and that some(i.e whites) are naturally more intelligent than others.
In this wide world of different opinions, I cannot speak for what every person you have encountered believes, especially not if you have been focusing on small extremist groups, but the scientifically accepted view and that which reasonable debaters claim (often shortly before being called “racist”) is that Ashkenazi and/or East Asians have the highest average IQs, followed at some distance by Caukasians, followed by other groups. (Indeed, it is almost funny how many allegedly White supremacists texts and authors, say “The Bell-Curve”, put another group than Caukasians at the top of their IQ listings…)
Take note of the “average”: All to often, detractors treat a claim about averages as a claim about each and every individual group member, be it through poor reading comprehension, a presumption of guilt, or out of intellectual dishonesty.
Furthermore, the “naturally” is misleading in as far as even a “genetically” has less ramifications. (Notably, a major point of genetics, which the anti-race propagandists and self-proclaimed anti-racists, as well as most actual racists, invariably overlook is that genetics make us moldable: If one group has a higher average IQ today, it could be the other way around at some point in the future. The common conclusion from the PC corner, that proponents of a genetic explanation for e.g. IQ differences also see these differences as set in stome for all eternity, is very, very wrong.)
All in all, your claim sounds like a strawman.
> They insist strongly that the science is on their side.
If you knew the science, you would know that they are absolutely correct if we refer to the actual claims that I have commonly seen (as opposed to the odd rant from a small extremist minority). Even going by your likely strawman, they would be correct in having very strong scientific support if we compare the common U.S. groupings of Whites, Blacks, and “non-White Hispanics”, with some minor reservations for the cause of the well-established differences.
> They tend to be white males, anti immigrant, anti feminist, anti social justice warrior anti Muslim/Arab/black and demonstate a strong sense of victimhood.
Not only does this seem like “guilt by association”, but the associated beliefs are to some degree distortion of opinion, to some degree beliefs that are not negative:
Being anti-feminist is only a logical conclusion of what feminism has proved to be—and contrary to what feminists like to claim, being in favour of equality between the sexes does not make someone a feminist, or an anti-feminist someone who wants to send the women back to the kitchen. (No more so than being in favour of equal rights indepent of family makes one a socialist, or a non-/anti-socialist someone who wants a feudal society.)
Social justice usually means that equal opportunities should be set aside in favour of equal outcomes—and I cannot fault anyone for finding that bad. (The days when the money or the title of the family one
was born into was the main determinant of success in life are long gone. Back then, demands for social justice, be it by that phrase or some other, had very different implications—and most of those who object to cries for the oxymoron social justice today would have been far more sympathetic then.)
Most people alleged to be anti-immigrant are on the outside anti-immigration—a very different thing. Moreover, many are not even anti-immigration but merely suggesting caution or expressing fears about immigration getting out of hand. Similarly, most alleged anti-Muslims/-Arabs are in fact anti-Islam—or more likely anti-Islamism. As for actual anti-Blacks, they are very rare in comparision—by what I have seen so far, likely rarer than the anti-Whites…
Throwing in the added slur of victimhood seems like an artificial ad hominem and, quite frankly, a very weird connection. If you want to find people with a strong sense of victimhood, go to the nearest Feminist or Black-Rights forum.
Sorry, do not take me by the bad side. I do not have in mind to offend os diminish anyone or any group. Believe me, PLEASE. I only am tempted to abandon sophystication for a while and use raw, stick-in-the-mud, bar-beer philosophy, gross, vulgar, numbers.
Lets us begin by Beethoven followed (if not chronologically) by Bach, passing by Galileo, Leonardo, Aristoteles and…Pitagoras! Archimedes! Leonardo. This reminds me of Gauss, Lagrange, Peano, Cardano. Einstein. The engineers at Boeing (the 747, what a machine!) Bill Windows, iStevephone. And millions of others.
All this, seems to me a product of one main “culture”. The European culture. Even the multi-nobel-winners Jews are product of European (or American, same thing, you know) Universities. (Kaiser Willem, Milan, Sorbonne & more) I fancy if ALL relevant art and science, except the zero and some algebra, belongs and is due to this culture. ALL (Well, yes, the Taittinger blanc-des-blancs too. & St. Emilion)
Now, how do you all feel about this simple “statistics”, this gross “statement”? Discriminatory? Racist? As an european, am I SAYING that I am “superior”? This is the meaning of the statement? I swear this is noto so!
No, please, do not call me “racist” I am not, I swear for all sacred things I believe in.
I am only the best Pouilly-Fuissé producer in the world. Discriminating? Offensive? Why?
That statistic is skewed by the survivor’s bias. For all you know, there were just as many fantastic amazing things invented in China and Africa and Incan Peru that were lost to history.
A real statistic would be: count up all the significant inventions ever invented (impossible because we can’t be sure we got them all), and see which fraction were invented purely in Europe without any outside influence. Then count up all the inventions we’ve agreed are *mistakes* and see how many came from Europe alone.
Don’t believe me that great inventions can come from non-Europe? Consider the printing press, *the* primary example of European technology changing the world. It existed both in China and Korea some hundreds of years before Gutenberg came along. Other examples: the wheel and the sailboat were probably invented in Mesopotamia. The idea for medicine and surgery probably came from Egypt, while for the next four millennia Europeans were still trying to pray and blood-let their sicknesses away.
Now, that’s going a bit too far. Of course, we can accept that ALL art and science Created in Europe had origin in other areas. Even alien science. WE know that Inacs had a gorgeous time recording tablets. That GUNPOWDER! is chinese (Marco Polo brought it to Europe). Columbus did not discover America. A detail piched here, a detail picked there and you will find that the Wrigtht Brothers DID NOT invent the heavy than air gizmo. It Was Alberto Sanros Dumont, a Brazilian millionaire (take of WITHOUT a rail. Jusat like today).
Now, putting all in a nutshell: If we theorize that an idea oe a little gizmo invented by Lucy is the seed of, let’s say, the grammophone then, OK. Nothinh is European.
But, as a “de facto” sistuation, we live in a world where history recognizes (wrongly?) all culture (tecnical or not) as European.
Nope, I’d guess this is now your personal bias showing through because you don’t know anything about other cultures. You can damn well bet that the history books of India and China (amassing a population bigger than Europe) don’t focus on just European culture. Again, you cannot make general claims about life if you don’t know anything about the lives of even half the world!
No. Just don’t say that. I’ve been working and teaching in most of Europe Latin America and Russia.
I Know very well (if not deeply) the story of all Orient.
See, you, as a (young probably) scientist cannot assume that “I don’t know anything” about other cultures. I lived even in Africa!
I imagine that you must take care of your personal bias. As a scientist you may know the “law of first publishing”. Well… Orientals may very well have invented the (movable type?) print, but they failed balatntly in speading the new technology. As for medicine, read your story. Dark ages had a regression. But read Esculapius. In Roman and Greek times Europeans whwre ligth years ahead of ANY oriental country. And look at the speed of the SPREAD of knowledge. By the time of Pasteur, all the world was behind Europe in health care.
My point was only:
Using a very simple “algorithm” if I say “European culture is superior and is very pleasanto to live in this culture”, am I saying, “ipso facto” that I DESPISE other cultures?
And, well, let’s clarify why i entered the discussion. My Company (owner and CEO) has been producing, for the last 35 years, very successful scoring and Rating models. I am also an old (33 years) Gold trader (Options & futures).
This discussion is becoming dishonest, because you’re not being clear on what claims you’re making and what claims are hypothetical. I can only address what you write, and I don’t know how you can expect to get logical, reasonable answers without being precise. I can try to address the one question you seem to be asking, which is: does claim (1) always imply claim (2), where claim (1) is “European culture is superior to every other culture” and claim (2) is “I despise all other cultures.” I’d argue to some extent yes, but more importantly I argue claim (1) is false, which makes the implication vacuous. You also make a slightly different claim that I’ll call (3): “European culture is the ‘de facto’ culture of the world.” The only way I can interpret this is “If you pick a person uniformly at random from the world, with the largest probability they will identify primarily with European culture than with any other fixed culture.” This claim is easily seen to be false because the population of China alone is much larger than Europe, as well as India. It should also be clear that claims (1) and (3) are not equivalent, though you have been identifying them as so. Moreover, the “algorithm” which operates by taking the most commonly observed thing/behavior X and outputting that X is the best thing/behavior is very biased. Finally, it does not matter whether the results of an algorithm are correlated with “hatred” or feelings of humans toward other humans. If the algorithm still has a negative impact on the lives of the discriminated group, it is a problem.
You are right.
The question of what it means for an algorithm to be fair is a potentially interesting one, but the SJW bent of this article is off-putting. In some comments, you backpedal and claim to just be raising the topic of what it might mean for an algorithm to be unfair, but right away in the article, in the first few paragraphs, you mention that people are “coming to realize” that equations can be racist, and that it’s ignorant to not believe so. Starting out an article with “it’s ignorant to disagree with this position,” especially when you haven’t even clearly spelled out the position, is pretty exasperating.
Let me try for you. I think one of your claims is this: let S represent the population, where we identify people with their set of feature values, let p : S -> [0,1] be a predictive function trained on data from a subset of people using a model explicitly designed to ignore some given (binary, for simplicity) feature, and let f : S -> {0,1} be the function saying whether said feature is true or false of a given person (e.g. S is all Americans, p is the probability they will take the high-interest loan, and f is whether they are black). Assume that higher values of p(s) imply some negative impact on s (if p(s) is high, banks will target s for a high-interest loan, and let’s pretend this denies opportunities for fair loans, which itself is a highly dubious claim). Your claim is that if p and f are “highly” correlated, then p, or the model used to generate p, is (morally) unfair, or discriminatory, or has unlawfully disparate negative impact?
We should be clear to distinguish intentional things like racism, intentional discrimination, and disparate treatment, from unintentional, circumstantial things like unintentional discrimination and disparate impact. Let’s take the holier-than-thou SJW tainting of this discussion out of the picture, because it’s just a distraction. The burden of proof would be on you to prove intentional discrimination anyways. Let’s focus on unintentional disparate impact. The first question is, in such a situation, might unintentional disparate impact occur? Yes, if the predictions are applied faithfully (without some other human or non-human intermediation deciding whether to actually trust the prediction), almost by definition. That’s not really an interesting question. The harder questions are whether the disparate impact is immoral, and whether something should be done about it. Note that there’s some independence between those two questions. Sometimes, the disparate impact may not be immoral, but we may want to do something about it anyways. Natural disasters are not immoral, but they’re still a problem and we still want to protect people from them, and help people affected by them. Of course, both these questions can only be answered on a case-by-case basis, and are not interesting in the general case. Expressing outrage about how immoral and racist equations can be in general will win you lots of Internet brownie points, but isn’t actually substantial or worth discussing.
Also, what exactly are you trying to get at when talking about algorithms “encoding” our prejudices. I would make a more neutral statement, that algorithms can reveal our prejudices. But they don’t “encode” them in the sense that they “codify” them, that is, give them some sort of legal or moral authority.
You may dislike the slant of the article, but it’s my personal blog, not the NY times.
But let me clarify two things. There are two big picture claims: (1) Math and algorithms cannot perpetuate human bias, even informally and (2) the question of what it means for an algorithm to be fair is interesting and open.
My claim is that (1) is the thing that ignorant belief people are just now coming to realize is hogwash. I can claim it’s ignorant not because of an ideological belief that discrimination is bad, but simply because it’s an unscientific and illogical claim. It’s an appeal to a mystical, misunderstood force that turns the argument into a thinly veiled tautology (“Math can’t be racist because it’s math!”) that no educated person can take seriously.
Part of the misunderstanding is that people don’t realize machine learning relies on data which is labeled (at the end of some long chain) by humans making decisions. And a machine learning algorithm is designed to find majority trends in data and spit them out as hypotheses. So anyone with a basic understanding of machine learning already knows that algorithms can encode (this is what I mean by encode, perpetuate) human bias. I don’t mean to be neutral because it’s a fact independent of slant.
Now let’s put aside the legality of using algorithms that you know perpetuate discriminatory biases. The legal issues here are so muddled and screwed up that we have no hope of resolving them before it actually comes up in court (for example, the way disparate impact is currently phrased, US hiring law actually encourages the use of discriminatory algorithms, even if you know the algorithm you’re using is biased and would be illegal if you carried out its steps manually!). Putting that aside, if you give me an algorithm, how can I tell you whether it’s fair or not? This is the primary research question, which is unrelated to (1) above. I think this is a much harder question than whether using a biased algorithm is immoral or justifiable.
> But let me clarify two things. There are two big picture claims: (1) Math and algorithms cannot perpetuate human bias, even informally and (2) the question of what it means for an algorithm to be fair is
interesting and open.
> My claim is that (1) is the thing that ignorant belief people are just now coming to realize is hogwash.
We can look at this two ways, depending on your exact meaning:
Either you are building and attacking a straw man, because the claim that math and algorithms cannot perpetuate human bias is an extremely unlikely one. There may or may not be some very ignorant people with no understanding of math and algorithms who has this opinion. Among even semi-educated and -intelligent people it is simply not a regularly occuring opinion. Furthermore, your claim that “people are just now coming to realize is hogwash” is it self hogwash, because they are not. They have known this all along and have not claimed the opposite to be the case.
Or you are confusing the tool with its wielder: Math and algorithms are not capable of being biased in the sense humans are, but are only biased (in a slightly different sense) to the degree that their makers were biased or commited mistakes. (A claim which incidentally does _not_ imply that they cannot perpetuate bias!) That guns don’t kill people, people kill people, may not be a very convincing argument against gun control—but it is still true and the gun remains only a tool of the killer. (Exepting freak accidents involving a gun, obviously.)
> […] no educated person can take seriously.
Precisely: no educated person can take, takes, or has until now taken. You reveal your own strawmanning. (Disclaimer: Education is not the same as intelligence and there are always rare exceptions.
I use “no” only to parallel your claim. The main point, however, is that any such educated people would be a small fringe group, and that you are strawmanning.)
> Now let’s put aside the legality of using algorithms that you know perpetuate discriminatory biases. The legal issues here are so muddled and screwed up that we have no hope of resolving them before it actually comes up in court (for example, the way disparate impact is currently phrased, US hiring law actually encourages the use of discriminatory algorithms, even if you know the algorithm you’re using is biased and would be illegal if you carried out its steps manually!). Putting that aside, if you give me an algorithm, how can I tell you whether it’s fair or not? This is the primary research question, which is unrelated to (1) above. I think this is a much harder question than whether using a biased algorithm is immoral or justifiable.
I have the very distinct impression that you have made up your mind that any algorithm that is to the disadvantage of one group is unfair, ipso facto and per definitionem. This, however, is the first question to address. I, for instance, happen to be of the very different opinion that differences in outcome are only unfair to the degree that they reflect differences in opportunity. Now: Are there differences in opportunity? If so, to what degree are they related to the individual and relevant characteristics of that individual? To what degree are they incidental, irrelevant, relating to a group he belongs to (but from which he deviates in this particular aspect)? To what degree do they represent pragmatically acceptable and to what degree inacceptable heuristics? Etc. Unfairness of this kind does certainly exist—but it does not appear to be the topic of your writings.
> There may or may not be some very ignorant people with no understanding of math and algorithms who has this opinion.
The designer of the current algorithm that aids the Chicago Police Department in predictive policing is quoted on record saying that their algorithm is not racist because it is “only” about multivariable equations. I don’t have statistics about what the general populace thinks (I wish I did), but it’s clearly not a straw man because somebody who is in a position of authority (and influence!) made this claim, and nobody corrected them. I find this highly unlikely if, as you claim, nobody believes this opinion. And this is not only an educated person, but a person who is supposed to be educated both in mathematical modeling and civil rights!
> I have the very distinct impression that you have made up your mind that any algorithm that is to the disadvantage of one group is unfair, ipso facto and per definitionem
While disadvantaging a particular group is a good guiding informal notion, I really want to find a rigorous definition that can remove the need to appeal to the real world. I have not made my mind up on a definition.
> Are there differences in opportunity?…
My problem with this is it’s not well-defined. What do you mean by incidental, irrelevant, “relating to a group,” pragmatic, acceptable, and unacceptable? What do you even mean by opportunity when an algorithm (say, one that serves ads for loans) might control the opportunity? I think if you could give me rigorous mathematical definitions of all of these things you would have yourself a nice theory of fairness.
And for what it’s worth, the follow-up post to this does measure differences relative to opportunity, in a sense. Read on and find out 🙂
http://jeremykun.com/2015/10/19/one-definition-of-algorithmic-fairness-statistical-parity/
I apologize for the late answer. Due to a hard-drive crash, I have had a bit of a timeout.
> > There may or may not be some very ignorant people with no understanding of math and algorithms who has this opinion.
>
> The designer of the current algorithm that aids the Chicago Police Department
> in predictive policing is quoted on record saying that their algorithm is not
> racist because it is “only” about multivariable equations. I don’t have
> statistics about what the general populace thinks (I wish I did), but it’s
> clearly not a straw man because somebody who is in a position of authority
> (and influence!) made this claim, and nobody corrected them.
If we below assume that the these claims were made and in made in a context where they mean what you think that they ment (I have not made any attempts to check on this) and assuming that they expressed a true opinion (as opposed to e.g. an attempt to artificially justify or “advertize” something):
Persons in positions of authority and influence are not automatically free from ignorance—not even close.
> I find this highly unlikely if, as you claim, nobody believes this opinion.
Pay attention: I did NOT claim that nobody has this belief. YOU did. I kept your formulation and added an explicit disclaimer concerning your and my use of that word.
> And this is not only an educated person, but a person who is supposed to be
> educated both in mathematical modeling and civil rights!
An education in mathematical modeling is an good sign, but by no means a guarantee that he understands math and algorithms.
I point you to e.g. http://www.aswedeingermany.de/50CompanyLife/50WorthOfFormalQualifications.html
Civil rights has no direct bearing on this understanding to begin with, although it might become highly relevant during e.g. a modelling attempt.
> > Are there differences in opportunity?…
>
> My problem with this is it’s not well-defined. What do you mean by incidental, irrelevant, “relating to a group,” pragmatic, acceptable, and unacceptable? What do you even mean by opportunity when an algorithm (say, one that serves ads for loans) might control the opportunity? I think if you could give me rigorous mathematical definitions of all of these things you would have yourself a nice theory of fairness.
>
You miss my point entirely: From what I have read on this page, YOU have failed to the appropriate leg-work and preparations. YOU do not know what the words you use actually mean. YOU seem to not have a clear mind on the unde
rlying concepts. YOU seem to lack a qualitative understanding of the issues. It is YOUR job as the proponent of a particular angle to do this job—not mine as a by-passer. I am trying to make YOU take a step back and do the thinking you appear to have skipped over.
(As an aside on qualitative understanding: Such is often gained from models, but it is also a pre-requisite for building models and any serious attempt at modelling requires equally serious attempts at building a qualitative understanding in advance.)
You should also beware that merely giving mathematical definitions, no matter how rigorous, does not result in a “nice theory”: It gives the ability to think, communicate, define, and reason using clear and well-defined con
cepts—but that is not the theory it self, it is what leads up to the theory.
https://sites.google.com/site/zliobaitefiles2/Zliobaite_fair_regression.pdf