Re-Packaging Open Source Deep Learning Models for JS Developers
The popularity of TensorFlow.js, along with this recent announcement and the exciting performance gains of
We wanted a solution that would feel intuitive, both in the browser for the web and while doing Node.js development. To do so, we knew we’d need to create a module that was consumable in different ways and required the least possible amount of setup to get started with.
We looked at a variety of tools and methods, but in the end, we settled with a solution that should be very familiar to most JS developers —
npm. Read on to get more details about what’s contained in the module and how to use it, but if you’re eager to get started the MAX Image Segmenter model is available now.
Re-packaging models for JS developers
Some readers may be familiar with the MAX Image Segmenter model already, but for those of you who aren’t I’d recommend taking a look at the model’s page on the Model Asset eXchange to learn more. In short, the model accepts an image as input and then returns a segmentation map, which is a pixel-by-pixel object prediction for the entire image. It can look something like this:
After converting the model as described in va barbosa’s previous article, we bundled it with all the pre and post-processing logic required to process user-submitted images, the TF.js code needed to run the model, and then made that package publicly available on
npm, the largest registry for open-source software among JS developers.
Based on my experience, I see 3 major benefits to packaging and consuming a model this way:
- Less code needed to process user input or clean the model’s output.
- No need to worry about environment set-up, since everything is included.
- Model assets are downloaded and run directly on the client, rather than across the web
Usage for the web
Using this new model format for the web couldn’t be simpler. Thanks to jsDelivr, an awesome open-source CDN, all it takes are these two lines of code to import everything you need to use the model in a web app:
This loads the model, it’s dependencies, and even get things “warmed up” so users can get the fastest responses possible once they begin submitting input. For more on changing this default behavior and other options, see the module’s README.
For a more interactive look at how easily this model can be integrated into a web app, I whipped up a quick demo on CodePen that shows off the basic functionality of the model and how to visualize its output. Click the Run Pen button to run the demo:
Normally, making predictions on user data requires network calls, API’s, and working with remote servers. With this new method, your app and the model both live in the browser, so communication between the two naturally becomes much simpler.
Usage for Node.js
To test our method, we went ahead and refactored a Node.js CLI utility called
magicat to use this npm-installable model rather than the manually converted version it had been built with. The result was cleaner, more concise code with better performance. You can’t ask much for much more of an improvement than that!
To install the model, all it took was one command at the terminal to add it to my app’s dependencies and get everything set-up:
$ npm install @codait/max-image-segmenter
In my app, one
import statement was all I needed to gain access to the model’s original functionality plus some nice JS-only enhancements. These include things like building the image pre-processing into the module, so I don’t have to convert image sizes or do any manipulation of tensors myself -just pass an image to the
predict method. Even the prediction itself is enhanced, as there are now built-in convenience methods for output parsing as well.
magicat-type app should be even more straightforward to develop.