Surveying scikit-learn usage — Part III: Clustering and Overall Popularity
As part of our ongoing focus on the Python open source ecosystem, we’ve been publishing a series on the use of scikit-learn. In our first post, we looked at the history of scikit-learn from a 10,000-foot perspective. In our second post, we then looked how tree-based ensemble methods, best exemplified by random forests, have been the most popular class of classifiers and regressors.
On today’s post, we’ll shift our focus to clustering and overall use case popularity.
Clustering
First, we’ll look specifically at sklearn’s clustering package. As always, all data comes from our licens.io software supply chain data products.
Figure 1 below shows the relative popularity — both recently and over all time — for the methods in sklearn.cluster
.
If you polled a room full of data scientists about what they thought were the most popular clustering methods, K-means and hierarchical clustering would almost certainly top the list. And unsurprisingly, KMeans
and AgglomerativeClustering
are #1 and #3 on the list.
KMeans
is clearly dominant in popularity, especially when you include the MiniBatchKMeans
implementation. In fact, KMeans
alone is more popular than all the other clustering methods put together.
Notably, both of these methods “suffer” from the requirement to set n_clusters
— the number of unique clusters or memberships to assign to the records. While there are many contexts where the number of clusters may be known or inferred through silhouette/elbow/scree plots, there are also many cases where the user may not have any prior for this parameter.
DBSCAN
— the #2 method on our list — is most likely so popular because it does not require n_clusters
to be set explicitly.
To be honest, I don’t use K-Means much for a variety of reasons; when I do, I tend to use the MiniBatch
implementation. Always averse to my priors “tainting” results or in-sample hyperparameter optimization, I default to using DBSCAN
or OPTICS
unless there’s a good reason not to.
Overall Use
We’ve looked at clustering, classification, and regression, where KMeans and Random Forests have come out on top overall. However, scikit-learn provides many methods and classes other than these supervised and unsupervised techniques.
In fact, as Figure 2 demonstrates, the most popular references to sklearn often don’t use any of these methods.
The #1 most common use for sklearn is…train_test_split
for splitting data. Not even cross-validation splits for complex folds. Just simple splits of aligned features and targets.
Next up in #2 is BaseEstimator
. This class, as its name implies, implements the scikit-learn-compatible “protocol” for any machine learning estimator. Whenever you see a package that claims to provide a sklearn-compatible interface, this is how they do it. For example, gensim
, dask
, bugbug
, giotto
, divik
, gordo
, and hdbscan
all use BaseEstimator
to provide natural fit
and predict
behavior. Even “next-generation” deep learning packages like keras
have implemented sklearn-compatible APIs.
After this, #3 StandardScaler
and #4 PCA
both implement simple preprocessing steps. To be honest, I’m not sure why someone would use scikit-learn instead of plain-old numpy
or scipy
for these methods unless they fit into a Pipeline
, but the data speaks.
Excluding the other supervised and unsupervised methods we’ve seen before, we also see accuracy_score
and confusion_matrix
— two assessment methods.
In summary, we find that scikit-learn’s extensibility and utility has created an ecosystem in and of itself. Many packages that do not use any supervised or unsupervised sklearn methods either utilize its “helper” methods or create their own compatible supervised or unsupervised methods.
For the many maintainers, contributors, and funders of scikit-learn, these data hopefully put a smile on their face. The positive impact of its accessible implementations and enormous adoption of its API will be felt for many years to come.
Originally published at https://licens.io.