|Making Hybrid Images||Neural Networks
|Bezier Curves and Picasso||Computing Homology||Probably Approximately
Correct – A Formal Theory
I’m making a few changes to the funding of Math ∩ Programming. First and foremost, I’m launching a Patreon campaign.
The way Patreon works is that you, dear reader, sign up for a monthly donation of any amount you please (as little as $1/month). There are at least three benefits for you doing this:
- You show your support of mathematics and programming. You provide documented evidence that you’re a good person. You help to ensure that Math ∩ Programming stays a high quality resource for everyone. You feel good.
- There are aggregate milestone goals that make Math ∩ Programming a better place. The first, for example, is that when Math ∩ Programming reaches $200/month I will permanently remove ads. See below for a detailed description of how ads currently support the blog (spoiler: it’s not much).
- There are individual benefits if you decide to pledge $5 or more per month. This includes a monthly Google hangout I’ll host, and a sneak peek for every new post and private discussion with me. I’m still thinking about how exactly I’ll implement the member preview, but my current plan is to have a private subreddit. But even if you don’t use reddit there’ll be another way to get access. The highest tier of rewards involves physical merchandise.
I’m excited about Patreon because it seems like an excellent platform. For example, the popular Numberphile YouTube channel makes almost $3k USD per month from patrons. To put that into perspective it’s $36k per year, and my graduate student stipend is only about $17k per year. Assuming Math ∩ Programming could get even half the success of numberphile, I could have potentially funded my entire graduate work just from blogging!
And channels like Numberphile are purely for entertainment’s sake. Math ∩ Programming has the additional benefit of providing working code for algorithms that are directly applicable to business. So if you have ever used the code at Math ∩ Programming as the start of a project or feature, or even if you just have fun reading about math and seeing cool applications, consider becoming a patron to say thank you!
Here are some other minor funding changes.
- One-time donations are now preferred through Square, due to the lower (1.9%) transaction fee. Square requires a debit card, so if you don’t have one or don’t want to use one you can still use PayPal to donate.
- People rarely buy merchandise. I made a total of $147 on merchandise since 2013. So I’m going to stop doing that for now. Maybe I just have to come up with better merchandise (comments are welcome).
- When I link to textbooks on Amazon I’m going to use Amazon Affiliate. Amazon pays me a little bit of money if you use the link and then end up buying something.
Funding so far
As of August 1, 2015 I have made a total of exactly $2,847.55 USD from ads and donations. About $320 of that is from 2015. It works out to about $70 per month since I first asked for money in 2013 and set up ads. It’s a nice little chunk of change, but nothing to get too excited about. Here is a chart of my ad income:
Donations have provided the rest of the funding, but donations appear to follow a Poisson distribution and the median monthly revenue is zero. By far most of the donations were in the few months after I first asked for donations.
I started my blog with pretty low expectations: I learned a lot of cool things and I wanted to share them, while understanding them better by writing code and filling in proof details. That’s still the core dream, and it will always be the core of Math ∩ Programming. So while it’s pretty cool that I can make any money at all from my blog, and I’m interested to see if I can grow it into a viable side business, you can rest assured that Math ∩ Programming will stay true to its core.
It’s about time we got back to computational topology. Previously in this series we endured a lightning tour of the fundamental group and homology, then we saw how to compute the homology of a simplicial complex using linear algebra.
What we really want to do is talk about the inherent shape of data. Homology allows us to compute some qualitative features of a given shape, i.e., find and count the number of connected components or a given shape, or the number of “2-dimensional holes” it has. This is great, but data doesn’t come in a form suitable for computing homology. Though they may have originated from some underlying process that follows nice rules, data points are just floating around in space with no obvious connection between them.
Here is a cool example of Thom Yorke, the lead singer of the band Radiohead, whose face was scanned with a laser scanner for their music video “House of Cards.”
Given a point cloud such as the one above, our long term goal (we’re just getting started in this post) is to algorithmically discover what the characteristic topological features are in the data. Since homology is pretty coarse, we might detect the fact that the point cloud above looks like a hollow sphere with some holes in it corresponding to nostrils, ears, and the like. The hope is that if the data set isn’t too corrupted by noise, then it’s a good approximation to the underlying space it is sampled from. By computing the topological features of a point cloud we can understand the process that generated it, and Science can proceed.
But it’s not always as simple as Thom Yorke’s face. It turns out the producers of the music video had to actually degrade the data to get what you see above, because their lasers were too precise and didn’t look artistic enough! But you can imagine that if your laser is mounted on a car on a bumpy road, or tracking some object in the sky, or your data comes from acoustic waves traveling through earth, you’re bound to get noise. Or more realistically, if your data comes from thousands of stock market prices then the process generating the data is super mysterious. It changes over time, it may not follow any discernible pattern (though speculators may hope it does), and you can’t hope to visualize the entire dataset in any useful way.
But with persistent homology, so the claim goes, you’d get a good qualitative understanding of the dataset. Your results would be resistant to noise inherent in the data. It also wouldn’t be sensitive to the details of your data cleaning process. And with a dash of ingenuity, you can come up with a reasonable mathematical model of the underlying generative process. You could use that model to design algorithms, make big bucks, discover new drugs, recognize pictures of cats, or whatever tickles your fancy.
But our first problem is to resolve the input data type error. We want to use homology to describe data, but our data is a point cloud and homology operates on simplicial complexes. In this post we’ll see two ways one can do this, and see how they’re related.
The Čech complex
Let’s start with the Čech complex. Given a point set in some metric space and a number , the Čech complex is the simplicial complex whose simplices are formed as follows. For each subset of points, form a -ball around each point in , and include as a simplex (of dimension ) if there is a common point contained in all of the balls in . This structure obviously satisfies the definition of a simplicial complex: any sub-subset of a simplex will be also be a simplex. Here is an example of the epsilon balls.Let me superscript the Čech complex to illustrate the pieces. Specifically, we’ll let denote all the simplices of dimension up to . In particular, is a graph where an edge is placed between if , and places triangles (2-simplices) on triples of points whose balls have a three-way intersection.
A topologist will have a minor protest here: the simplicial complex is supposed to resemble the structure inherent in the underlying points, but how do we know that this abstract simplicial complex (which is really hard to visualize!) resembles the topological space we used to make it? That is, was sitting in some metric space, and the union of these epsilon-balls forms some topological space that is close in structure to . But is the Čech complex close to ? Do they have the same topological structure? It’s not a trivial theorem to prove, but it turns out to be true.
The Nerve Theorem: The homotopy types of and are the same.
We won’t remind the readers about homotopy theory, but suffice it to say that when two topological spaces have the same homotopy type, then homology can’t distinguish them. In other words, if homotopy type is too coarse for a discriminator for our dataset, then persistent homology will fail us for sure.
So this theorem is a good sanity check. If we want to learn about our point cloud, we can pick a and study the topology of the corresponding Čech complex . The reason this is called the “Nerve Theorem” is because one can generalize it to an arbitrary family of convex sets. Given some family of convex sets, the nerve is the complex obtained by adding simplices for mutually overlapping subfamilies in the same way. The nerve theorem is actually more general, it says that with sufficient conditions on the family being “nice,” the resulting Čech complex has the same topological structure as .
The problem is that Čech complexes are tough to compute. To tell whether there are any 10-simplices (without additional knowledge) you have to inspect all subsets of size 10. In general computing the entire complex requires exponential time in the size of , which is extremely inefficient. So we need a different kind of complex, or at least a different representation to compensate.
The Vietoris-Rips complex
The Vietoris-Rips complex is essentially the same as the Čech complex, except instead of adding a -simplex when there is a common point of intersection of all the -balls, we just do so when all the balls have pairwise intersections. We’ll denote the Vietoris-Rips complex with parameter as .
Here is an example to illustrate: if you give me three points that are the vertices of an equilateral triangle of side length 1, and I draw -balls around each point, then they will have all three pairwise intersections but no common point of intersection.
So in this example the Vietoris-Rips complex is a graph with a 2-simplex, while the Čech complex is just a graph.
One obvious question is: do we still get the benefits of the nerve theorem with Vietoris-Rips complexes? The answer is no, obviously, because the Vietoris-Rips complex and Čech complex in this triangle example have totally different topology! But everything’s not lost. What we can do instead is compare Vietoris-Rips and Čech complexes with related parameters.
Theorem: For all , the following inclusions hold
So if the Čech complexes for both and are good approximations of the underlying data, then so is the Vietoris-Rips complex. In fact, you can make this chain of inclusions slightly tighter, and if you’re interested you can see Theorem 2.5 in this recent paper of de Silva and Ghrist.
Now your first objection should be that computing a Vietoris-Rips complex still requires exponential time, because you have to scan all subsets for the possibility that they form a simplex. It’s true, but one nice thing about the Vietoris-Rips complex is that it can be represented implicitly as a graph. You just include an edge between two points if their corresponding balls overlap. Once we want to compute the actual simplices in the complex we have to scan for cliques in the graph, so that sucks. But it turns out that computing the graph is the first step in other more efficient methods for computing (or approximating) the VR complex.
Let’s go ahead and write a (trivial) program that computes the graph representation of the Vietoris-Rips complex of a given data set.
import numpy def naiveVR(points, epsilon): points = [numpy.array(x) for x in points] vrComplex = [(x,y) for (x,y) in combinations(points, 2) if norm(x - y) < 2*epsilon] return numpy.array(vrComplex)
Let’s try running it on a modestly large example: the first frame of the Radiohead music video. It’s got about 12,000 points in (x,y,z,intensity), and sadly it takes about twenty minutes. There are a couple of ways to make it more efficient. One is to use specially-crafted data structures for computing threshold queries (i.e., find all points within of this point). But those are only useful for small thresholds, and we’re interested in sweeping over a range of thresholds. Another is to invoke approximations of the data structure which give rise to “approximate” Vietoris-Rips complexes.
In a future post we’ll implement a method for speeding up the computation of the Vietoris-Rips complex, since this is the primary bottleneck for topological data analysis. But for now the conceptual idea of how Čech complexes and Vietoris-Rips complexes can be used to turn point clouds into simplicial complexes in reasonable ways.
Before we close we should mention that there are other ways to do this. I’ve chosen the algebraic flavor of topological data analysis due to my familiarity with algebra and the work based on this approach. The other approaches have a more geometric flavor, and are based on the Delaunay triangulation, a hallmark of computational geometry algorithms. The two approaches I’ve heard of are called the alpha complex and the flow complex. The downside of these approaches is that, because they are based on the Delaunay triangulation, they have poor scaling in the dimension of the data. Because high dimensional data is crucial, many researchers have been spending their time figuring out how to speed up approximations of the V-R complex. See these slides of Afra Zomorodian for an example.
Until next time!
In 2014 the White House commissioned a 90-day study that culminated in a report (pdf) on the state of “big data” and related technologies. The authors give many recommendations, including this central warning.
Warning: algorithms can facilitate illegal discrimination!
Here’s a not-so-imaginary example of the problem. A bank wants people to take loans with high interest rates, and it also serves ads for these loans. A modern idea is to use an algorithm to decide, based on the sliver of known information about a user visiting a website, which advertisement to present that gives the largest chance of the user clicking on it. There’s one problem: these algorithms are trained on historical data, and poor uneducated people (often racial minorities) have a historical trend of being more likely to succumb to predatory loan advertisements than the general population. So an algorithm that is “just” trying to maximize clickthrough may also be targeting black people, de facto denying them opportunities for fair loans. Such behavior is illegal.
On the other hand, even if algorithms are not making illegal decisions, by training algorithms on data produced by humans, we naturally reinforce prejudices of the majority. This can have negative effects, like Google’s autocomplete finishing “Are transgenders” with “going to hell?” Even if this is the most common question being asked on Google, and even if the majority think it’s morally acceptable to display this to users, this shows that algorithms do in fact encode our prejudices. People are slowly coming to realize this, to the point where it was recently covered in the New York Times.
There are many facets to the algorithm fairness problem one that has not even been widely acknowledged as a problem, despite the Times article. The message has been echoed by machine learning researchers but mostly ignored by practitioners. In particular, “experts” continually make ignorant claims such as, “equations can’t be racist,” and the following quote from the above linked article about how the Chicago Police Department has been using algorithms to do predictive policing.
Wernick denies that [the predictive policing] algorithm uses “any racial, neighborhood, or other such information” to assist in compiling the heat list [of potential repeat offenders].
Why is this ignorant? Because of the well-known fact that removing explicit racial features from data does not eliminate an algorithm’s ability to learn race. If racial features disproportionately correlate with crime (as they do in the US), then an algorithm which learns race is actually doing exactly what it is designed to do! One needs to be very thorough to say that an algorithm does not “use race” in its computations. Algorithms are not designed in a vacuum, but rather in conjunction with the designer’s analysis of their data. There are two points of failure here: the designer can unwittingly encode biases into the algorithm based on a biased exploration of the data, and the data itself can encode biases due to human decisions made to create it. Because of this, the burden of proof is (or should be!) on the practitioner to guarantee they are not violating discrimination law. Wernick should instead prove mathematically that the policing algorithm does not discriminate.
While that viewpoint is idealistic, it’s a bit naive because there is no accepted definition of what it means for an algorithm to be fair. In fact, from a precise mathematical standpoint, there isn’t even a precise legal definition of what it means for any practice to be fair. In the US the existing legal theory is called disparate impact, which states that a practice can be considered illegal discrimination if it has a “disproportionately adverse” effect on members of a protected group. Here “disproportionate” is precisely defined by the 80% rule, but this is somehow not enforced as stated. As with many legal issues, laws are broad assertions that are challenged on a case-by-case basis. In the case of fairness, the legal decision usually hinges on whether an individual was treated unfairly, because the individual is the one who files the lawsuit. Our understanding of the law is cobbled together, essentially through anecdotes slanted by political agendas. A mathematician can’t make progress with that. We want the mathematical essence of fairness, not something that can be interpreted depending on the court majority.
The problem is exacerbated for data mining because the practitioners often demonstrate a poor understanding of statistics, the management doesn’t understand algorithms, and almost everyone is lulled into a false sense of security via abstraction (remember, “equations can’t be racist”). Experts in discrimination law aren’t trained to audit algorithms, and engineers aren’t trained in social science or law. The speed with which research becomes practice far outpaces the speed at which anyone can keep up. This is especially true at places like Google and Facebook, where teams of in-house mathematicians and algorithm designers bypass the delay between academia and industry.
And perhaps the worst part is that even the world’s best mathematicians and computer scientists don’t know how to interpret the output of many popular learning algorithms. This isn’t just a problem that stupid people aren’t listening to smart people, it’s that everyone is “stupid.” A more politically correct way to say it: transparency in machine learning is a wide open problem. Take, for example, deep learning. A far-removed adaptation of neuroscience to data mining, deep learning has become the flagship technique spearheading modern advances in image tagging, speech recognition, and other classification problems.
The picture above shows how low level “features” (which essentially boil down to simple numerical combinations of pixel values) are combined in a “neural network” to more complicated image-like structures. The claim that these features represent natural concepts like “cat” and “horse” have fueled the public attention on deep learning for years. But looking at the above, is there any reasonable way to say whether these are encoding “discriminatory information”? Not only is this an open question, but we don’t even know what kinds of problems deep learning can solve! How can we understand to what extent neural networks can encode discrimination if we don’t have a deep understanding of why a neural network is good at what it does?
What makes this worse is that there are only about ten people in the world who understand the practical aspects of deep learning well enough to achieve record results for deep learning. This means they spent a ton of time tinkering the model to make it domain-specific, and nobody really knows whether the subtle differences between the top models correspond to genuine advances or slight overfitting or luck. Who is to say whether the fiasco with Google tagging images of black people as apes was caused by the data or the deep learning algorithm or by some obscure tweak made by the designer? I doubt even the designer could tell you with any certainty.
Opacity and a lack of interpretability is the rule more than the exception in machine learning. Celebrated techniques like Support Vector Machines, Boosting, and recent popular “tensor methods” are all highly opaque. This means that even if we knew what fairness meant, it is still a challenge (though one we’d be suited for) to modify existing algorithms to become fair. But with recent success stories in theoretical computer science connecting security, trust, and privacy, computer scientists have started to take up the call of nailing down what fairness means, and how to measure and enforce fairness in algorithms. There is now a yearly workshop called Fairness, Accountability, and Transparency in Machine Learning (FAT-ML, an awesome acronym), and some famous theory researchers are starting to get involved, as are social scientists and legal experts. Full disclosure, two days ago I gave a talk as part of this workshop on modifications to AdaBoost that seem to make it more fair. More on that in a future post.
From our perspective, we the computer scientists and mathematicians, the central obstacle is still that we don’t have a good definition of fairness.
In the next post I want to get a bit more technical. I’ll describe the parts of the fairness literature I like (which will be biased), I’ll hypothesize about the tension between statistical fairness and individual fairness, and I’ll entertain ideas on how someone designing a controversial algorithm (such as a predictive policing algorithm) could maintain transparency and accountability over its discriminatory impact. In subsequent posts I want to explain in more detail why it seems so difficult to come up with a useful definition of fairness, and to describe some of the ideas I and my coauthors have worked on.
A while back we featured a post about why learning mathematics can be hard for programmers, and I claimed a major issue was not understanding the basic methods of proof (the lingua franca between intuition and rigorous mathematics). I boiled these down to the “basic four,” direct implication, contrapositive, contradiction, and induction. But in mathematics there is an ever growing supply of proof methods. There are books written about the “probabilistic method,” and I recently went to a lecture where the “linear algebra method” was displayed. There has been recent talk of a “quantum method” for proving theorems unrelated to quantum mechanics, and many more.
So in continuing our series of methods of proof, we’ll move up to some of the more advanced methods of proof. And in keeping with the spirit of the series, we’ll spend most of our time discussing the structural form of the proofs. This time, diagonalization.
Perhaps one of the most famous methods of proof after the basic four is proof by diagonalization. Why do they call it diagonalization? Because the idea behind diagonalization is to write out a table that describes how a collection of objects behaves, and then to manipulate the “diagonal” of that table to get a new object that you can prove isn’t in the table.
The simplest and most famous example of this is the proof that there is no bijection between the natural numbers and the real numbers. We defined injections, and surjections and bijections, in two earlier posts in this series, but for new readers a bijection is just a one-to-one mapping between two collections of things. For example, one can construct a bijection between all positive integers and all even positive integers by mapping to . If there is a bijection between two (perhaps infinite) sets, then we say they have the same size or cardinality. And so to say there is no bijection between the natural numbers and the real numbers is to say that one of these two sets (the real numbers) is somehow “larger” than the other, despite both being infinite in size. It’s deep, it used to be very controversial, and it made the method of diagonalization famous. Let’s see how it works.
Theorem: There is no bijection from the natural numbers to the real numbers .
Proof. Suppose to the contrary (i.e., we’re about to do proof by contradiction) that there is a bijection . That is, you give me a positive integer and I will spit out , with the property that different give different , and every real number is hit by some natural number (this is just what it means to be a one-to-one mapping).
First let me just do some setup. I claim that all we need to do is show that there is no bijection between and the real numbers between 0 and 1. In particular, I claim there is a bijection from to all real numbers, so if there is a bijection from then we could combine the two bijections. To show there is a bijection from , I can first make a bijection from the open interval to the interval by mapping to . With a little bit of extra work (read, messy details) you can extend this to all real numbers. Here’s a sketch: make a bijection from to by doubling; then make a bijection from to all real numbers by using the part to get , and use the part to get by subtracting 1 (almost! To be super rigorous you also have to argue that the missing number 1 doesn’t change the cardinality, or else write down a more complicated bijection; still, the idea should be clear).
Okay, setup is done. We just have to show there is no bijection between and the natural numbers.
The reason I did all that setup is so that I can use the fact that every real number in has an infinite binary decimal expansion whose only nonzero digits are after the decimal point. And so I’ll write down the expansion of as a row in a table (an infinite row), and below it I’ll write down the expansion of , below that , and so on, and the decimal points will line up. The table looks like this.
It’s a bit harder to read, but trust me the notation is helpful.
Now by the assumption that is a bijection, I’m assuming that every real number shows up as a number in this table, and no real number shows up twice. So if I could construct a number that I can prove is not in the table, I will arrive at a contradiction: the table couldn’t have had all real numbers to begin with! And that will prove there is no bijection between the natural numbers and the real numbers.
Here’s how I’ll come up with such a number (this is the diagonalization part). It starts with 0., and it’s first digit after the decimal is . That is, we flip the bit to get the first digit of . The second digit is , the third is , and so on. In general, digit is .
Now we show that isn’t in the table. If it were, then it would have to be for some , i.e. be the -th row in the table. Moreover, by the way we built the table, the -th digit of would be . But we defined so that it’s -th digit was actually . This is very embarrassing for (it’s a contradiction!). So isn’t in the table.
It’s the kind of proof that blows your mind the first time you see it, because it says that there is more than one kind of infinity. Not something you think about every day, right?
The Halting Problem
The second example we’ll show of a proof by diagonalization is the Halting Theorem, proved originally by Alan Turing, which says that there are some problems that computers can’t solve, even if given unbounded space and time to perform their computations. The formal mathematical model is called a Turing machine, but for simplicity you can think of “Turing machines” and “algorithms described in words” as the same thing. Or if you want it can be “programs written in programming language X.” So we’ll use the three words “Turing machine,” “algorithm,” and “program” interchangeably.
The proof works by actually defining a problem and proving it can’t be solved. The problem is called the halting problem, and it is the problem of deciding: given a program and an input to that program, will ever stop running when given as input? What I mean by “decide” is that any program that claims to solve the halting problem is itself required to halt for every possible input with the correct answer. A “halting problem solver” can’t loop infinitely!
So first we’ll give the standard proof that the halting problem can’t be solved, and then we’ll inspect the form of the proof more closely to see why it’s considered a diagonalization argument.
Theorem: The halting program cannot be solved by Turing machines.
Proof. Suppose to the contrary that is a program that solves the halting problem. We’ll use as a black box to come up with a new program I’ll call meta-, defined in pseudo-python as follows.
def metaT(P): run T on (P,P) if T says that P halts: loop infinitely else: halt and output "success!"
In words, meta- accepts as input the source code of a program , and then uses to tell if halts (when given its own source code as input). Based on the result, it behaves the opposite of ; if halts then meta- loops infinitely and vice versa. It’s a little meta, right?
Now let’s do something crazy: let’s run meta- on itself! That is, run
So meta. The question is what is the output of this call? The meta- program uses to determine whether meta- halts when given itself as input. So let’s say that the answer to this question is “yes, it does halt.” Then by the definition of meta-, the program proceeds to loop forever. But this is a problem, because it means that
metaT(metaT) (which is the original thing we ran) actually does not halt, contradicting ‘s answer! Likewise, if says that
metaT(metaT) should loop infinitely, that will cause meta- to halt, a contradiction. So cannot be correct, and the halting problem can’t be solved.
This theorem is deep because it says that you can’t possibly write a program to which can always detect bugs in other programs. Infinite loops are just one special kind of bug.
But let’s take a closer look and see why this is a proof by diagonalization. The first thing we need to convince ourselves is that the set of all programs is countable (that is, there is a bijection from to the set of all programs). This shouldn’t be so hard to see: you can list all programs in lexicographic order, since the set of all strings is countable, and then throw out any that are not syntactically valid programs. Likewise, the set of all inputs, really just all strings, is countable.
The second thing we need to convince ourselves of is that a problem corresponds to an infinite binary string. To do this, we’ll restrict our attention to problems with yes/no answers, that is where the goal of the program is to output a single bit corresponding to yes or no for a given input. Then if we list all possible inputs in increasing lexicographic order, a problem can be represented by the infinite list of bits that are the correct outputs to each input.
For example, if the problem is to determine whether a given binary input string corresponds to an even number, the representation might look like this:
Of course this all depends on the details of how one encodes inputs, but the point is that if you wanted to you could nail all this down precisely. More importantly for us we can represent the halting problem as an infinite table of bits. If the columns of the table are all programs (in lex order), and the rows of the table correspond to inputs (in lex order), then the table would have at entry a 1 if halts and a 0 otherwise.
here is 1 if halts and 0 otherwise. The table encodes the answers to the halting problem for all possible inputs.
Now we assume for contradiction sake that some program solves the halting problem, i.e. that every entry of the table is computable. Now we’ll construct the answers output by meta- by flipping each bit of the diagonal of the table. The point is that meta- corresponds to some row of the table, because there is some input string that is interpreted as the source code of meta-. Then we argue that the entry of the table for contradicts its definition, and we’re done!
So these are two of the most high-profile uses of the method of diagonalization. It’s a great tool for your proving repertoire.
Until next time!