Skip to main content
Loading Explore guides…
Unsupervised Machine Learning · Acedra
Home
Explore
Computer Science
Unsupervised Machine Learning
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
114 segments
Copy
Hide video
0:10
Good morning, everyone. This is Jack Van Horn from the BD2K Training Coordinating center at the University of Southern California, and I'd like to welcome you all to our Big Data to Knowledge Guide to the Fundamentals of Data Science. Today, we have the pleasure of Ali Shojaie from the University of Washington who's going to be talking on unsupervised machine learning. This follows on the heels from our talk last week by Daniella Witten on supervised statistical learning.
0:41
This one makes a nice compliment. Dr. Shojaie is an associate professor of biostatistics and an adjunct associate Professor of statistics at the University of Washington in Seattle. He originally trained in Industrial Systems Engineering and obtained his PhD in statistics from the University of Michigan while completing a master's degree in applied mathematics and human genetics. His current research lies at the intersection of machine learning for high dimensional data,
1:06
statistical network analysis and applications in biology and the social sciences. He teaches regular and short courses on statistical machine learning and network analysis. So we're very pleased to have Ali here to talk about his topic of unsupervised machine learning. Ali, thank you so much. Thank you, Jack, for that introduction. And thanks also to Crystal for organizing all these efforts. I also wanted to thank Michelle Dunn, who spearheaded many of the efforts on the BD2K
1:36
training activities and was key in the success of BD2K training activities. So as Jack mentioned, I'm going to talk about unsupervised machine learning techniques today. And this is one of the main two classes of statistical learning along with supervised machine learning, which was the topic of last week's webinar. In supervised machine learning, as some of you saw last week, we use a data set, x, to predict or detect associations with response, y.
2:08
Whereas in unsupervised machine learning, there is no response y. So we're only looking at a dataset, x, and we're trying to identify patterns or signal in that dataset. And that's the topic of today's webinar. I should also mention that there are other categories of statistical machine learning-semisupervised learning, which is somewhere between supervised and unsupervised, and reinforcement learning, but we will not talk about those today.
2:31
So to better understand what unsupervised machine learning tries to do, it's helpful to look at supervised learning, which is more familiar. This is a classical supervised learning task where we have observations in the space of features x1 x2. So we have two features here. These could be abundances of two proteins or activities measured at two different brain regions. And each dot in this plot corresponds to one subject, let's say.
3:02
So in addition to having the measurements of different subjects with these two features, we also have the labels associated with each subject. And so in this case, we have two labels, blue and orange. And so a classical supervised learning task would be to classify the observations based on the feature space. So an example would be to define a support vector classifier, which would be a hyperplane that would distinguish the blue and orange classes.
3:42
And then we could use this classifier that we learned, let's say, based on some training data in order to predict the outcome of future observations. The main difference from supervised learning in the setting of unsupervised learning is that we still have the same data, we still have the same features x1 and x2, but we no longer have the labels for the classes. So instead of having the setting where we have blue and orange points, we just have, let's say, gray points.
4:10
We don't know what these points correspond to. All we know is that we have measurements of x1 and x2. So we can't, for example, do classification anymore. However, we could still think about whether there are groupings in the data. So this would correspond to a clustering task. Or whether there are lower dimensional representations that could capture information in the data. Or whether there are associations between the features, for example, in this case, x1
4:33
and x2, that would be meaningful and would help us to identify some patterns in the data. So to summarize, the main difference between supervised learning and unsupervised learning is that, in supervised learning, we have a data matrix, x, which is an n by p array of data corresponding to p measurements for each of n subjects. In the previous example, p was 2. We had two measurements in general. We could have more.
4:58
And in biological applications, especially in biomedical big data, we usually have p in the order of thousands or hundreds of thousands. And y is a vector of outcome or response for each one of the subjects. In unsupervised learning, we still have the same data matrix, x, but we no longer have a vector of outcome or response. And all we are doing is to look at the data matrix in order to identify patterns or signal that is interesting.
5:31
Now, why do we use unsupervised learning given that there's no association with the outcome? Well, unsupervised learning is an important part of the data analysis pipeline. In fact, unsupervised learning is used commonly in the exploratory data analysis phase. Along with data wrangling and data visualization, a lot of times unsupervised learning techniques are used to aid data wrangling and data visualization. And sometimes, they are used to generate new hypotheses that we could then validate in
6:02
independent data. And this is when we basically identify interesting patterns and signal in our data in order to generate hypotheses. Unsupervised learning methods are also used in conjunction with supervised learning techniques and confirmatory data analysis. So Brian Caffo gave a really nice explanation of exploratory versus confirmatory data analysis a couple of weeks ago at one of these webinars. And as he pointed out, oftentimes analyses are somewhere in this spectrum of exploratory
6:36
data analysis to confirmatory data analyses. There are analyses that are purely exploratory. There are analyses that are purely confirmatory. And there analyses that fall somewhere in between the two. And for many of the analysis that falls somewhere between the two, unsupervised learning is used extensively. It is important to keep in mind that when we use data to generate hypotheses, and then later on test those hypotheses, if we are using the same data, the findings of our study
7:05
are still exploratory. We cannot make strong claims based on what we find partly because these might not be reproducible. We drive the hypotheses based on the same data, and we have, in essence, validated the same hypotheses using the same data. If we want to ensure the reproducibility of results, we should instead use, for example, a training test approach that was discussed in last week's webinar or a validation-centered approach. For example, we should use crossvalidation.
7:34
And we should do that on the whole pipeline. So all the steps from data wrangling, data visualization, unsupervised learning, which might generate some hypotheses, all the way to supervised learning that would verify these hypotheses or look for association with the outcome-- they should be validated in independent data or in held out dataset in order to make sure that the results are reproducible. And if this is not possible, then we should interpret the results as an exploration of the data.
8:03
We should not get potentially too excited about what we are observing in our data. So with that introduction, we'll focus on two of the main classes of unsupervised learning methods throughout today's webinar. We will focus on clustering techniques and dimension reduction. We will not talk about assocation rule mining and graphical modeling network analysis, which although interesting, we won't have time to cover them in this lecture.
8:30
Clustering, of course, is widely used in analysis of biomedical big data. Here's a very well-known example of one of the pioneering applications of clustering in analysis of gene expression data, where [INAUDIBLE] and others used clustering techniques to identify subtypes of breast cancer. And they later validated these clusterings and verified them-- these subtypes-- and they used them to define effective therapies. Now, in clustering, the main objective is somewhat vague.
9:06
The objective of clustering is to group the objects into meaningful clusters such that objects within each cluster are more similar to one another than objects in other clusters. Now, the reason that this is somewhat vague is that, depending on how we define meaningful clusters or meaningful subsets, we could have different number of clusters and we could get different clusterings in our data. Here's a simple example.
9:34
Depending on how we define what a meaningful cluster is, we might get different number of clusters. One might reasonably say that there are two clusters in this dataset, separating them into red and blue. Another person could also reasonably look at this dataset and say there are six different clusters. And this just depends on how we define clusters-- how we define similarity within groups and difference outside of these groups.
10:01
Even someone might look at this data and say, well, there are four clusters. All of this is dependent on how we define meaningful clusters. So this is one of the complications of clustering. Similarly, the way that we define similarity among objects has consequences in terms of what we mean by meaningful concepts. So here, for example, we could consider two clusters defined based on two curves in the data or two manifolds in the data. These would be the red and blue clusters.
10:28
Or someone might reasonably use a different notion of similarity in the observations and define different clusters. So these are issues that are important and need to be decided throughout different clustering tasks. And different choices would result in different clustering outcomes. In addition to deciding what we mean by meaningful cluster, we should also decide what we want to cluster. We have a couple of different options. We could cluster observations.
10:59
This is the most commonly used option. And for example, in the paper that I showed you, clustering was used in order to identify subtypes of breast cancer. The samples were clustered in order to identify subtypes of breast cancer. We could also cluster the features. This is used extensively when we want to find genes that have similar functions, for example, in order to identify proxies to biological pathways. In fact, cluster both-- we could cluster observations and features together.
11:35
And this is often referred to as bi-clustering. Throughout this lecture, given the similarities of these tasks, we will focus only on clustering of observations. But same techniques and same issues that we will discuss here will also come up in clustering of features or in biclusters. Cause So today, we will talk about two classes of clustering techniques, hierarchical methods and partition based methods. And we will cover an example of each one of these briefly.
12:04
We will talk with hierarchical clustering and we will talk about k-means clustering. Hierarchical clustering is basically a way to get a sequence of nested clusters that are organized in a tree structure, which is referred to as a dendrogram. And there are two classes of hierarchical clustering methods. We could build these trees in a bottom up fashion starting with n individual clusters and then merging them until we get 1 cluster that contains all the observations.
12:34
Or we could do it in a top down way. We could start with one cluster and use it in a measure of distance in order to separate the clusters further down and get n separate clusters. And these, again, are similar somewhat. We will focus on bottom up approaches here in this lecture. So to explain the main idea of hierarchical clustering, consider this simple example with nine observations plotted on the space of two features, feature one and feature two.
13:04
Now, when we use hierarchical clustering, we get a dendrogram, which is a tree on the left that explains how merging should occur between observations, how observations should be clustered together. And for example, we could easily see that in this dendrogram, observations 5 and 7 are merged together. And then they're merged with observation 8, and then merged with observation 2, and then later 9. And on the other hand, 6 and 1 are merged together. And they are merged with 4, merged with 3.
13:42
And then at the very end, the two groups are merged together. Now, how are these dendrograms formed? Basically, we should use some notion of distance or similarity in our dataset in order to form these dendrograms. So going back to our plot of data points, we see that 5 and 7 are very similar to each other-- very close to each other on this plot. So those are the first two that have merged to form a cluster. 6 and 1 are the next two closest points.
14:10
Then those are merged together and they form a cluster. The next cluster that is formed is by joining 8 to 5 and 7. So that's another cluster. And then the next cluster would be adding 4 to 6 and 1. So you see that as we go up the tree, you're forming-- on this vertical axis, we're forming-we're merging clusters to form bigger and bigger clusters. And at the very top of the tree, always we'll get one cluster that contains all the points. What do we need to make a dendrogram?
14:45
Well, we need at least two pieces of information. One, we need a similarity of distance between observations. As we discussed in the previous example, we need to know how far the observations are from each other. Now, we don't need to necessarily have a plot of observations, but we need to have some notion of distance or similarity between observations. But in addition to that, we also need a measure of similarity or distance between clusters.
15:10
As I mentioned, as we go up the tree, in a lot of cases, further up, we merge sets together. And in those cases, we have to think about similarities between sets or distances between sets. And this is what is known as a linkage function. So this simple example with two clusters with two and three observations in each explains the notion of a linkage function. There are three different choices that we will discuss.
15:43
One is that in order to define a distance between the two sets, we take the minimum distance between the observations in the two sets-- between the points of the two sets. And that is known as a single linkage function. The opposite is to take the maximum distance between the observations of two sets. And that would give us a complete linkage. And then we also have the option of taking the average of the distances between the points in all the points in the two sets.
16:11
And that would give us the average linkage function. There are, of course, a few other sources of linkage function, but we will not discuss those in today's lecture. So we mentioned that there are three different choices of linkage function. Which one of these should we use? The answer is that there is no best linkage function and that the choice of linkage function that is appropriate depends on characteristics of data that are unfortunately, a lot of times, unknown to us.
16:39
So to give you a summary, single linkage function works best when there aren't many outliers or the data is not very noisy. But it's capable of identifying clusters with diverse shapes. On the other hand, complete linkage and average linkage to some degree are more robust to presence of outlier and noise in the data, but they require clusters that are roughly comparable in size. And they work better with spherical distributions.
17:10
So if we have clusters that have the irregular shapes, we might not be able to distinguish them well with complete linkage or average linkage. And here are some examples that would explain hopefully some of these issues. So if we have outliers as is the case in this data set where we have observations that fall somewhere between the two main clusters, then single linkage clustering might have difficulty separating the two clusters from each other.
17:39
And instead, they might identify one point as one cluster and all the other observations as another cluster, whereas complete linkage and average linkage do a better job of separating the two clusters from each other because they're less prone to presence of outliers. And of course, in reality, you wouldn't know which one of the two clusterings is correct-- whether complete linkage is giving the right answer or average linkage.
18:08
The main difference is these few points, or in this case, this is simulated data, so I do know which one is correct, and complete linkage is doing the correct clustering. But in general, we wouldn't know the difference between. As I mentioned, single linkage is better at capturing clusters that have nonspherical shapes. Here's a good example of that. At the single linkage, especially if we don't have-- if we have some separation between
18:35
clusters, we could distinguish clusters that are embedded and, in this case, nested-- these embedded clusters within each other, whereas complete linkage and average linkage won't be able to separate these two clusters properly and distinguish them from each other. And the last example shows that the setting when we have clusters that have very different sizes. In this case, complete linkage could break down. It's the most sensitive to having clusters of roughly equal size.
19:04
In this case, as you see, it forces them-- two clusters that identify to have roughly equal size whereas single linkage and average linkage do a better job of separating these two clusters from each other. Now, all of these examples that I showed you were in the simple setting that we had two feature so we could plot the observations. Of course, in the analysis of biomedical big data, we often can't plot the observations. And we have no sense of what the shape of the true clusters are.
19:35
So finding the right choice of linkage function and, in fact, even finding the right choice of distance between or similarity between observations is often challenging. Now, going back to the dendrogram, as we discussed, at the bottom of the dendrogram, we have a leaf for each one of the observations. And as we go up the tree, these leaves get fused to form branches. Now, I want to emphasize the interpretation of the dendrogram here.
20:09
And there are a couple of important things that I want to point out. One is that the earlier the points are fused together, the earlier the fusions occur, the more similar the observations of the clusters are to each other. So the fusions at the bottom of the tree indicate that these observations are very similar to each other, whereas fusions that have happened later on indicate that these observations are not based on [INAUDIBLE].
20:33
So that's an important piece of information that the dendrogram provides us. One thing in that goes hand-in-hand with that is that the vertical axis in dendrogram is very informative. The height of the first fusion between clusters containing two different observations corresponds to how different they are from each other. So if you look at these two observations, for example, they are fused at the very bottom of the cluster.
21:04
So that indicates that the distance between these two observations is very small, whereas if you consider this leaf and another leaf in this branch, for example here, their corresponding clusters are merged much higher in the graph. So the distance between those two clusters is around 7. So there is a significant difference between points that are merged earlier in the dendrogram than points that are merged higher in the dendrogram. And the vertical axis provides us that information.
21:38
On the other hand, the horizontal axis is not informative at all in the dendrogram. So we should not interpret the horizontal axis in any informative way. And the prime example of that are these two leaves. Hopefully, you guys see my cursor. Consider this observation, this leaf, and the one that is very adjacent to it in this leaf. And even though these are adjacent to each other on the horizontal axis, they're in fact the last two observations to get merged at the very top of the tree.
22:08
So they can't be any more different from each other in this example. So the observation at the horizontal axis and the layout of the leaves on the horizontal axis is somewhat arbitrary, and you should not interpret that in an informative way. Now, depending on where we cut this dendrogram or where we cut this tree, we get different number of clusters. So for example, we could cut the tree somewhere around here. And then we will get two clusters, these red and green clusters.
22:51
So anywhere from the very top to the distance 7, we get two clusters. If we cut the dendrogram further towards the bottom, we get more and more clusters. So if we cut the dendrogram some where here, we get three clusters. And if we cut it further down, we get additional number of clusters identified. Now, one thing to keep in mind and is important is that, as we discussed, fusions that occur earlier in the dendrogram indicate more similar clusterings.
23:32
So if we cut the dendrogram further down, we are cutting into groups of observations that are more similar with each other. So ideally you want to look for gaps in where the fusions occur. So in this example, cutting it at 2 or 3 is sensible. 3 is probably the best choice. But cutting it to get 2 or 3 dendrograms is sensible. But cutting it to get more clusters is not very reasonable because then we're cutting into clusters that are somewhat homogeneous inside them.
24:07
Now, let's look at how these dendrograms would look like in a somewhat more realistic setting. And this is the clustering of NCI60 data. There are 64 cancer cell lines. And in each one of them, about 6,800 genes were measured. And if you use complete linkage, average linkage, and single linkage, dendrograms that we get are somewhat very different than dendrograms I showed you in that simpler example. For one, the clusterings in all of these dendrograms are less clear.
24:46
There is less separation between the leaves-between the clusters. And then all the leaves are somewhat combined towards the top of the tree. With complete linkage, we get a little bit more separation towards the top. But with average linkage and single linkage, basically, we don't have good separation between the clusters. Another feature of these high dimensional settings that we see in this example is the behavior of single linkage clustering.
25:16
And this is often due to the noise that is induced because of the high dimensionality. In a lot of high dimensional examples, single linkage is not capable of identifying meaningful clusters, and what we see is that clusters are formed by adding one observation at a time. So as you go higher on the tree, one observation is added to the previous cluster, and that cluster gets bigger. That's the dominant pattern that we see.
25:48
And this is often referred to as chaining behavior of single linkage clustering. So in a lot of high dimensional applications, single linkage, despite its appealing properties to detect clusters with diverse shapes, is not very effective in high dimensions. We discussed briefly hierarchical clustering. It's an easy to interpret method. It gives a family of nested clusters. It's computationally rather efficient. And it also works in high dimensions with the caveat that we just discussed.
26:21
The disadvantages of hierarchical clustering is that it doesn't really correspond to a formal optimization criteria. It doesn't achieve anything in any formal objective. And the final solution is chosen by the data scientists or the analysts. And that could be somewhat arbitrary. And as we discussed, different merging or splitting criteria could give rise to different solutions. In contrast to hierarchical clustering and hierarchical methods, k-means clustering,
26:59
which is an example of partition based methods, tries to find the best partition of the observations into a priori defined number of clusters. So we first consider a sort of-- that we want to partition our observation to k clusters. And k is given. And then k-means clustering tries to partition the observations in the best possible way into these number of clusters. It has a simple and intuitive objective function. And it's computationally really rather efficient.
27:26
It could be applied to data sets with a large number of observations. However, the main drawback is that there is no hierarchy among clusters. If you change k, for example, from 2 to 3-- that instead of wanting 2 clusters, now you want 3 clusters, there's no guarantee that we have a hierarchy among clusters. So the clustering memberships might completely change. To explain the k-means clustering, I briefly would like to discuss its algorithm partly
27:53
because the actual optimization function that is associated with k-means clustering is not tractable. Finding the exact solution to that is not tractable. On the other, hand, we could find very good approximate solutions using a simple algorithm. The idea is that we first randomly assign each one of the observations to one of k clusters. And then we iterate two steps until the clustering memberships don't change anymore. First we calculate cluster centroids.
28:21
This is the mean of the observations in each one of the clusters. And then we assign cluster memberships. We assign each point to the centroid that is closest to it. So here's an illustration of that procedure. This is a dataset. Suppose that we want to cluster it into three clusters. First, we randomly split the observations into one of three clusters, red, blue, and green. And then we just simply calculate the centroids, the mean of the red, that mean of the blue, and the mean of the green.
28:50
Of course, in this case, these means are not that different from each other. But nonetheless, we could now assign each one of the observation to a centroid that is closest to it. So now, we get these three clusters in the data. And in fact, if we continue this process, then you would recalculate the centroids. And then even after 10 iterations, we don't get any changes. So in this somewhat toy example, we get the partitions very quickly just by one pass of the k-means clustering algorithm.
29:22
And in many examples, this algorithm converges very quickly, and we get the results of a good partition of data into k clusters. Now, as we discussed, k-means clustering does not give us a hierarchy among clusters. If we change k, the memberships will change. k-means also works best when we have compact spherical clusters with somewhat comparable number of members. So in that regard, k-means behavior, a lot of times, is somewhat similar to complete linkage clustering.
29:59
And one of the issues that might occur with k-means clustering is that it may result in small empty clusters. And this could be potentially due to presence of outliers. A remedy for this is to remove those empty clusters and refit the partition with the algorithm. In a lot of cases, that improves the solution. But nonetheless, this could occur. Now, here are some examples of settings where k-meas could work well. In this case, k-means does a good job of partitioning the observation to 4 sets.
30:35
It's just a partition, whereas single linkage clustering gives just one big cluster, and then observations on the periphery that are just being distinguished from those. Similarly, k-means clustering would be able to separate these two clusters from each other, whereas single linkage clustering, because of this somewhat overlap between the two clusters, won't be able to separate them from each other. And it identifies these artificial clusters.
31:04
On the other hand, k-means, as we discussed, prefers spherical distributions. So it has difficulty identifying these embedded clusters, whereas single linkage does a good job of that. And also, again in this example, k-means can't distinguish between these two clusters, whereas single linkage can do a good job in that example. Now, in k-means, we discussed that we need to know the number of clusters, but that's also somewhat true about hierarchical clustering.
31:38
We need to decide where to cut the dendrogram and how many clusters to include. In general, this is a difficult question. Like many other unsupervised learning tasks, we need a validation set approach. And these approaches don't work directly in unsupervised learning techniques, unlike supervised learning where we could do cross-validation, for example, in order to choose the tuning parameter in k-means or hierarchical clustering if it's not trivial and it is somewhat difficult.
32:08
There are some guidelines. For example, as we discussed in hierarchical clustering, we could examine the dendrogram, see where the gaps are in the dendrogram, and use those in order to identify the right number of clusters. There are also some guidelines for k-means clustering. We could vary the number of clusters. We could vary k and then look at the value of the objective function over different k's. And then the guideline is that we should be looking at the, quote unquote, "elbow" in
32:39
the plot, meaning that, usually the value of the objective function drops down quickly and then it flattens out. And then the spot where we see that the flattening of the curve of the objective function usually corresponds to a right number of clusters for the data. But again, these are somewhat arbitrary, and they're not very rigorous guidelines. Now, probably the best possible way to find the number of clusters is again to go back
33:10
to the fact that hierarchical clustering, k-means clustering, general clustering, and unsupervised learning are parts of the data analysis pipeline. So we could, in fact, consider varying the value of k, or the number of clusters, and see the effect of it on the downstream analyses. Oftentimes, as we discussed, we use unsupervised learning to further to try to identify some patterns and then use supervised learning techniques later on in order to validate these
33:43
or in order to do some downstream analysis. And the best possible way to identify what is the right number of clusters for that dataset that you're analyzing is to use a validation set approach or a training test approach in conjunction with that supervised technique to, at the end, determine what is the appropriate number of clusters that you should consider. There are some challenges and caveats with clustering, as we discussed.
34:09
There is, of course, no one size fits all solution for clustering, or even a good definition of what clusterings should look like. There's not a single best criterion for obtaining a partition because, as we discussed, the definition is, by design, somewhat vague. In biomedical big data, clusters could have arbitrary shapes and sizes, and we don't have a way of identifying what are the shapes and sizes of clusters or understanding what the shapes and sizes of clusters.
34:46
So we won't be able to distinguish these from each other. And it is very hard to evaluate how well a clustering algorithm performs, especially when we are working with biomedical big data. So these are the issues that come up when we try to validate and invalidate clustering results. And we should keep in mind that, again, the best way to do this would be just to view clustering as part of the data analysis pipeline and validate it at that perspective.
35:17
The final point I want to emphasize on is that the preprocessing steps, the data wrangling steps, including centering and scaling for example, could affect the result of clustering. So again, we should see cautioning as a step in that whole process and think about the consequences for example whether we center or scale the observations or variables and the effect that that would have on the clustering algorithms. So now moving on from clustering to dimension reduction.
35:57
As we discussed, the goal of clustering is to identify homogeneous subgroups in the data. On the other hand, dimension reduction methods try to find low dimensional representation of the data that maintain important features of the day. Today, we will only talk about probably the most widely used dimension reduction technique, which is the Principal Components Analysis, or PCA. But there are many other choices out there.
36:21
We won't talk about mult-dimensional scaling, independent components analysis, self-organizing maps, and non-negative matrix factorization, and many variants of these methods. This is a growing and active area of research, but we will just talk about the basic concepts using PCA as an example. Now, why do we use dimension reduction? There are many reasons to do dimension reduction. For example, in high dimensions, data visualization becomes very difficult.
36:51
We can't really plot many two dimensional scatter plots if we have large number of features, p. We would want to reduce the dimensionality in order to aid visualization for example. Often, there's high degree of redundancy in the data. there's correlational among features, so we don't need to have all of the features in order to understand the properties of the data or characteristics of the data. There are many features in any study that may not be informative for that particular study.
37:26
You could think of these as somewhat noise features. And it would be helpful to somewhat reduce the effect of these features and come up with dimension reductions that would correspond to easier analyses. And dimension reduction techniques can retain important features of the data while reducing these noise features and also redundancy in our analysis, both to aid visualization and also to help with supervised learning techniques, as we will discuss later.
38:07
So the most well known dimension reduction method, dimension in principal components analysis, or PCA-- the idea is that we start with data matrix, x, the same data matrix we had at the beginning. This is an n by p array of value for n subjects and p features. And the idea is that not all these features are needed. You might be able to reduce the dimension. And the goal of PCA is to find a lower dimensional presentation of the data that captures most of the information in the data.
38:45
And PCA is widely used in biomedical big data. For example, in population genetics in genome-wide association studies, PCA is used to correct for stratification. It's used to study patterns of human migration. And also, it's used to identify and correct batch effects. In last week's webinar, it was discussed that batch effects would have severe consequences on the performance of supervised learning techniques.
39:14
And usually as a pre-processing step, we need to identify these artificial effects, or batch effects, or artificial nonbiological effects that are results of data acquisition or processing steps. And it would be important to identify these and try to correct for these so that we have a data set that truly represents important biological information. And one of the main tools and techniques for identifying and correcting batch effects is based on PCA.
39:48
So to explain the main idea of PCA, again, let's look at a simple example, with two features, x1 and x2, and some observations in this scatterplot of two features. Now, the goal of PCA, as we discussed, is whether we can find a lower dimensional representation. So for example, whether we could find a good one dimensional representation of the data in this example. Well, we have some choices. We could use one of the two variables. So we could use either x1 or x2 as our dimension reduction.
40:25
Probably a better choice to use a linear combination of the two. So we could form a new variable, z1, that is a linear combination of x1 and x2-- basically, a way to average the two variables. So this would correspond to, basically a new direction, depending on the choices of w1 and w2, we would get different lines. And then the question is, how, do we choose these weights, w1 and w2. And PCA's answer to this question is that the direction that is most informative is
40:57
a direction that captures the majority of the variance in the data. In PCA, we look to find the direction that captures most of the variance in the data or most of the variation in the data. So in this toy example, that corresponds to the direction of maximum correlation between x1 and x2. Now what if we needed another direction? Suppose that we wanted another direction, z2, that was another linear combination of x1 and x2.
41:32
Well, to find additional principal components-- and these directions are referred to as principal components-- we choose the subsequent linear combinations that are orthogonal or perpendicular to the previous ones. So once we found this direction-- again, it's called a principal component, or PC-- once we found this direction, we try to find future directions that are orthogonal to the previous ones. So what that means is that we will choose the vector v of weights that is orthogonal
42:03
to the previous vector w, but still explains the majority of variables in the data. Of course, in the two dimensional setting, once we find the vector, v, that is orthogonal to w, then we don't have any choice. That defines the vector. This is always the case for the last PC. The last PC is defined directly by the fact that it's perpendicular to all the previous ones. However, if p is larger than 2, then we have many choices for the second PC.
42:31
And then we would choose among those the one that explains the maximum variation in the data and is orthogonal to the previous ones. So in this example, again, this choice of second PC is somewhat clear. It's the one that is orthogonal to the first one. And then basically, when you look at this plot, based on the two principal components, PCA just defines a new coordinate system for the observations that now you could think of plotting the observation along this new co
43:04
ordinate system instead of the original coordinates system. And that's, a lot of times, how we use PCA in order to do data visualizations in high dimensions. If we had higher dimensions when we have p larger than 2, PCA again defines a lower dimensional representation. And in fact, it defines a lower dimensional representation that is the best approximation to the data. So we can define this two dimensional approximation of the data where we project each one of the observations into this plane.
43:43
And that would give us the PCA solution. So the PCA solution with the first principle component and the second principle component captures the majority of variability that could be captured by a two dimensional representation. This is the best approximation to he data that we can get. So here is a classical example of a application of PCA on a data set that was collected back in 1973. And these are crime statistics in arrests per 100,000 residents for solved murder rate
44:21
along with the percentage of urban population in the 50 United States. As we can see from the two dimensional scatter plots, there is some correlation between some of these crime statistics. Murder and assault, for example, are correlated with each other. There's some correlation between assault and rape and assault and murder. And urban population is less correlated with these two. But nonetheless, there is some correlation in the data.
44:48
Now, if we apply PCA-- I won't go into detail of how we get these. If we apply PCA and then plot the observation of the space of the first two PCs, we see some interesting patterns just using two principal components. So instead of now having four variables, we've reduced the dimensionality into two. And that allows us to do visualization, for example, and identification of interesting patterns. So in the space of the first two PCs for example, we have the observations plotted.
45:22
And in this particular plot, which is referred to as a biplot, we also have the variables or features overlaid in this spot using these arrows. So this shows that basically the first principal component is somewhat of an average of all four variables. As we saw, these variables were correlated with each other. So the first principle components in these settings usually captures an overall average of the variables.
45:46
So long as the first principle components has been moved to the right, we have higher levels of some combination of urban population, rape, assault and murder in the states that are on the right and a lower values of those in the states that are in the far left. Whereas the second principle component direction tells us something interesting about the contrast between assault and murder that have negative values on this second principle component
46:19
versus urban population and rape that have positive values in this second principle component. So basically, states that are at the bottom half of this second principle component are those that have higher rates of arrest due to assault and murder. And states that are in the top half of the plots were based on the second principal component are those that have higher rates of arrest based on rape and also have higher urban population in those states.
47:04
So already, this plot gives us some really interesting information about the data that we could not just see based on the 2D scatterplots. Just plotting everything in the space of the first two PCs in addition to dimension reduction gives us a way of identifying these interesting features. Now again, the question that comes up is that how many principal components should be used. As before, this is somewhat tricky question to answer.
47:39
On the one hand, by increasing the number of principal components, we would include more and more variation of the data. We explain more of the variation in the data. On the other hand, if we use too many PCs, we don't get the advantage and benefits of dimension reduction. Ideally, we want the smallest number of PCs that capture the important information of the data. Unfortunately, this is not easy, as we discussed.
48:10
And again, the best way to achieve this, although there are some guidelines and some ideas about how we could choose the number of PCs is to think about PCA as part of unsupervised learning in the data analysis pipeline, and try to identify the number of clusters that is best based on that. As one example, in last week's webinar, one of the methods that was discussed was principal component regression, which is basically applying PCA to first obtain a lower dimensional representation
48:46
of the data, and then apply either regression or classification on that space in order to predict the outcome or find associations with the features in that space. And this is a very powerful tool, especially for prediction. It's not as great for finding association because you are now in this lower dimensional space that is the combination of the previous variables. But nonetheless, a really good way of identifying how many principal components we should include
49:21
is to look at the outcome of that regression or classification at the end, and then try to use cross-validation, for example, in order to identify how many principal components should be used in order to get the best prediction in the test data. To summarize, I want to emphasize-- again, I know that I've mentioned this a couple times and I've repeated this over and over, but this is really important. And unsupervised learning is just one step in the data analysis pipeline.
49:52
So we should see it as part of the data analysis pipeline. And the results of unsupervised learning should be ideally validated in that pipeline. We widely use unsupervised learning techniques to preprocess data. So it's used in exploratory data analysis to aid data visualization, data wrangling. For example, as we discussed, its used to correct for batch effects. And then, its also used for data visualization, as we saw some examples of it.
50:24
It's used to extract features and discover patterns in the data. So all of this was part of exploratory data analysis, and we use it to generate new hypotheses. But it's also used in confirmatory data analysis as we just discussed in the example of a principal component regression. These unsupervised learning techniques are used to improve the performance of supervised techniques or in conjunction with supervised learning techniques in confirmatory data analysis.
50:49
The selection of tuning parameters and validation of the results of unsupervised learning techniques is often difficult. This is partly because we don't have a response-- we don't have an outcome to use, for example, in order to do cross-validation. However, if we see it again as part of the whole pipeline, we would be able to use the outcome of the analysis. If this is not feasible, if there is truly no outcome, then what you would want to do
51:21
is to try to validate what we find in an independent dataset. So let's say we get some clusterings, and we should repeat the clustering on an independent data to see if clusterings that we get are somewhat similar, or if we use that clustering from the previous study, whether those correspond to interesting patterns in the new dataset as well. And especially, when we are using unsupervised learning techniques purely as exploratory
51:50
data analysis tools, we should be cautious about interpreting the results of those analyses. We should keep in mind that without external validation, there is danger that we might not be able to produce the findings of our exploratory data analysis and what we are getting could be potentially just features that we have captured by looking at the data. And they might not be generalizable to other datasets and other studies. And with that, I want to thank you for listening.
52:28
And I'm happy to answer any questions that they might have. Thank you, so much Ali. That was a really great overview on unsupervised machine learning. I do want to encourage folks who are in our audience, if you have any questions to use the little question submission system to send in a few questions. And I'll try and get to as many as we can before the top of the hour. One question, for you, Ali, is you had made-- in your discussion on the creation of the
52:57
dendrograms in your clustering algorithm discussion, you mentioned measures of distance. Are there any caveats about the choice of the measure of distance, whether it's a Euclidean metric, whether it's like a city block metric, or perhaps even an information criteria that one needs to worry about at all? Absolutely. So there are many choices as you mentioned. Some of them-- there are many choices of distance or similarity. And the main caveat is that there is no best measure.
53:26
That different datasets and different depending on what are the clusterings within the data, different measures might work better in different situations. And unfortunately, that's another thing that we won't know. And the best way to do clustering is to often try different metrics and try different similarity or distance measures and see which one of them validates better. So again, I want to go back to this idea that unsupervised learning is part of the pipeline.
53:59
Which one of them gives us results that are more generalizable if we look at the test set. So ideally we should try a couple of different choices of distance or similarity to see which one of them gives us more generalizable results. OK. Great. Well, one other question was in principal components analysis. Are there any recommended axis rotation techniques which might be useful in helping to make your results more interpretable?
54:36
Well, so principal component is, by design, a way of rotating the axes. So basically, going back to the slide I had on principal components, basically what principal components does is that it gives you a new coordinate system. Basically, it's a rotation of the original data. Now, in addition to rotation, in a higher dimension, you're getting this rotation in a reduced dimension. So instead of having a three dimensional space, you have a two dimensional space.
55:13
But principal components already tries to find the optimal rotation that captures the information in the data. The feature of the principal component-- the property of principal component is that the two axes are orthogonal to each other. So that helps open up the correlation basically between observations. So instead of having observations plotted along the correlation, as we had x1 and x2, now, if we look at this new coordinate system-- in this new coordinate system, the observations
55:49
can be opened up from each other more easily. So I would say just looking at principal components usually gives us the right rotation of the data. A lot of times the question is how many of these PCs should be used or which one of the PCs should we look at. Should we look at PC1 1 and PC 2 or should we look at PC 1 and PC 3 if you are doing 2D scatterplots. So that's usually the more difficult question to answer. So you should inspect your data, I guess, is inspects the results-
56:22
Yes. --and that will tell you- Yes. --which kind of brings me to one of the questions from our audience is there seems to be a lot of supervision in unsupervised machine learning. How do you kind of strike a balance between how much inputs or trial and error that a user might input with letting the machine do what it does? So that's actually a really good question. In general, I would say that unsupervised learning is one of the areas
56:54
that supervision, if you will, or involvement of the data scientists is somewhat critical. Many of the unsupervised learning techniques are harder to put into an automatic framework. A lot of times, there is need for data scientists to be engaged. And this is a really good opportunity, for example, for interaction from data scientists with the data at see what are the interesting features. We don't want to do this to the extent that we are basically finding artificial patterns in the data.
57:36
And again, that goes back to finding patterns that are generalizable at a test dataset or in cross-validation. But a lot of times, unsupervised learning techniques do require some involvement by the data scientist. So how does that-- your interest is in building a model of the data, are there different approaches that you need to be considering between the use of a supervised approach versus an unsupervised learning approach? Well, of course.
58:14
We use unsupervised techniques, a lot of times, to gain insight into what are the characteristics of the data, what are the features of the data. And those will help us determine what are the right models-- what are the proper unsupervised learning techniques. So that's one way that these tools are used together. The other way, as I briefly discussed in the case of principal component regression is that sometimes unsupervised learning techniques are used in order to aid supervised learning
58:44
techniques-- in order to improve the performance of supervised learning techniques. So the final model is basically using unsupervised learning to first reduce the dimension. And then we build a prediction model or, let's say, a regression model. So we could use these in conjunction, and use them together in order to get a model that is most appropriate. But the answer, to some extent, depends on what is the goal of our analysis.
59:16
So if the goal is purely prediction, we could use principal component regression. And PCA together with regression or prediction. However, if the goal is to find association between the original features and the outcome, that technically doesn't work very well because each one of the PCs is a linear combination of all the original features. So we can't use principal component regression in order to identify which one of the features is associated with the outcome.
59:48
So the answer to this question somewhat depends on what is the goal of our analysis. Do we want to do a good job of prediction or do we want to find association between the features and the outcome? And once we answer that question, it will determine what are the right choices of combinations of supervised and unsupervised learning techniques. Well fantastic. Well, we've run out of time. And so we need to kind of sign off now. But I want to thank Ali for his talk today.
1:00:20
It was a really fantastic overview of unsupervised machine learning. I want to thank those of you who signed in. And thank you for your questions. And we'll look forward to talking to you again next week. Once again, thank you, Ali. Enjoy your weekend, everybody. Thank you.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat