Please stop making “simple” machine learning libraries!
Why many times simplicity does not mean ease to use
Introduction
The other day I was watching a lot of videos of “The Coding Train”. This is a really lovely YouTube channel with the extremely funny and enthusiastic host Daniel Shiffman. One of the main contents of this channel are “Coding Challenges” where Daniel uses mostly JavaScript frameworks to implement interesting small applications. One of these frameworks that he even made a series about is ml5.js. This library builds on top of Tensorflow.js and is, according to its own website,
A neighborly approach to creating and exploring artificial intelligence in the browser.
I tried ml5.js and I am disappointed. Here’s why.
An out-of-the-box model
From my understanding ml5.js wants to make developing and deploying machine learning models as easy and beginner-friendly as possible. That’s a good idea, so let us start with a simple model. We can include the framework very comfortably by including a cdn link into an HTML-header (listing 1)
Now we are already good to go! Fortunately, ml5.js already comes with some prebuild models so we can just use one of them. Furthermore, it is made to seamlessly integrate with p5.js, a framework to simplify working with media and graphics. So, using p5.js to grab a frame from the webcam and feeding it into an ml5.js model takes a mere 30 lines of code (listing 2).
This works really well and took no longer than 15 minutes to set up. So, what if we want to use our own custom model? That’s where my problems began.
A custom model
It is nice and handy that we can use pre-trained models in ml5.js and even train models inside the browser. But let’s be honest, these two use cases are probably not too common. It will happen more often that you have built and trained in Python and now want to serve this model in the browser. So, let’s try to do this in ml5.js.
I really don’t want to do anything fancy so I just implement a classificator for MNIST. Heading over to the reference they describe how to implement a convolutional neural net for image classification (image 1):
Well, ok, I get that ml5.js wants to be beginner-friendly, but this seems a little bit too short. A library should make things easier not hide all the behavior completely from the user. What is actually going on here? How many layers do we have? How many filters? Stride size? Activation function? We don’t know. Suggesting this as a way to create a neural network seems a little bit off.
But I don’t want to be too nit-picking because just a few lines below we are told how to define our own custom neural net (image 2):
The next thing that struck me right away was that there were hardly any references to Tensorflow. In fact, it was just mentioned on the landing page that ml5.js build on top of it, but in the actual documentation it is nowhere to be found. So if I define my own network like in image 2 I can use all available layers from Tensorflow? And they have the same name? The reference neither mentions this nor gives us a list of available layers, so we don’t know.
The interface of the neural network also differs from Tensorflow (image 3)
Instead of just .predict() we have four different methods: .predict(), .predictMultiple(), .classify() and classifyMultiple(). Also, we can’t hand the data in the .train() method, but have to call .addData() beforehand. That … seems overly complicated and not so user-friendly.
Ok, let us not dwell too much on this, because as mentioned above, it is more common to use an already trained model than train it in the browser. So, how can we load a pre-trained Tensorflow model in ml5.js? It is not in the reference. Think about this. This library builds on top of Tensorflow and we are given no clear instructions on how to actually load a Tensorflow model. After consulting the Tensorflow.js documentation and researching a GitHub repo that also uses ml5.js I figured out that there seems to be a Python package called tensorflowjs that converts a regular Tensorflow model into a .json-File with the architecture and .bin files with the model weights. A single explaining paragraph in the ml5.js documentation would have helped a lot here.
Anyways, let’s move on. After I read the image from the webcam I want to resize it to feed it into my neural net. In the ml5.js reference there is a tab called “Utils” where we should find such functionality. I click there and … there is one method listed. One. Method. The whole Utils section consists of the method .flipImage(). That’s all the Utils. Other image or sound manipulating methods? Nope. That’s not useful at all. Ok, so I have to change the resolution of my webcam, not too difficult.
Alright, now I can load my own model (listing 3)
When I execute this code I get an error message (image 4)
Um, that’s interesting. As I said above, I trained my model on the MNIST dataset so of course the expected input has the dimension (28, 28, 1). But why does my net expect an input of size (1, 224, 224)? I never specified that? I changed the resolution of the image I grab from the webcam to (28, 28) to match my net. After digging into the GitHub repo of ml5.js I found this line (listing 4):
So … the image size is hard-coded to 224? Surely there must an option to change this, right? Wrong! The ImageClassifier class has options we can set (image 5), but the image size is none of them.
To summarize: I have a trained model, but no matter how I resize my input images, this class will always resize it to (224, 224) and I have no option to change this. Why … why would I want this? Why would anyone want this? At this point I gave up. Like, what am I supposed to do? Derive my own class from the ImageClassifier base class and change this constant? Clone the whole ml5.js code just to change this one line? Questions over questions. Is it possible to deploy an MNIST classifier in ml5.js? Maybe, but I really don’t know how.
Summary: a model included in ml5.js was exceptionally easy to run, but deploying my own model is nearly impossible. The interface is strange in some point (in my opinion!), I am given no instructions on how to import a Tensorflow model, there are absolutely no utility functions to make things easier and a crucial parameter is hard-coded and apparently cannot be changed.
Conclusion
This whole article reads like a salty rant about this specific library, but that’s really not my intention. I know that a lot of open source projects are developed by volunteers in their free time and I am more than thankful for this. But lately I stumbled upon a lot of ML libraries that were advertised with slogans like “Code model xy in just 10 lines of code”, “5 lines of code for a computer vision SOTA model — use framework xy now!” or something similar. That’s not what I want. Sure, for ML novices it is really nice to have a model out of the box up and running in a few lines of code, but what people who really want to apply these frameworks need is customizability and flexibility — I’d rather write 50 lines of code than 5 if it gives me more power over my model.
