Image for post
Image for post

Using OpenCV.js in Browser based environment

Aswin Vijayakumar
Oct 31, 2019 · 5 min read

OpenCV is a powerful tool which does image manipulation and processing at a fast pace. To bring the OpenCV library in the browser the browser needs to support libc++ library which almost all browsers support.

Asm.js is written in C++ that browsers use in their native environment. The OpenCV.js source provided here has been compiled using Emscripten (EMSDK) which makes use of Asm.js and libc++ to transpile C++ code to the javascript code. Javascript runs in a single threaded environment, but Emscripten has something which is similar to tasks in C++ (Openmp) called Web Workers. With the WebAssembly library supported by recent browser versions multi-threading is possible. WebAssembly is commonly used in WebGL and benchmarking results are brilliant.

Computer Vision involves Pattern Recognition, Artificial Intelligence and Machine Learning. In a client server model, the server need not own the complete Sequence Diagram model of a given task. It is in some use cases, ideal to split the Sequence Diagram into two contexts: Client and Server contexts.

An OpenCV project that is reliant on Client-side and Server-side

The table shown below is from the post on Approaching any Machine Learning problem. It separates the phases of machine learning into :

  • Training and validation split,
  • Data Transformation
  • ML Model Selector
  • ML Entity Selector
  • Hyperparameter Selector
  • Feature Extraction
  • Producing output via Optimization

While training the model, the data can be transformed into a variety of ways. A simple method is normalizing the data by subtracting the mean of the data and dividing by standard deviation. OpenCV on the client side can transform the data and extract the features such as in face detection it could be to extract the face location.

Image for post
Image for post
Taken using https://aidemos.microsoft.com/face-recognition, @Courtesy https://thispersondoesnotexist.com/

OpenCV has many cascade algorithms such as Viola Jones, HOG (Historical Oriented Gradients). Viola Jones is an unsupervised algorithm that extracts features using proximal pixels. It does image integration of the image similar to what image convolution does but has a restriction that the image sizes need to be specified. Viola Jones is an Unsupervised learning algorithm whereas Convolution Neural networks use Supervised learning techniques unless it is a case of feature extraction using CNNs. They are classified usually as Haar Cascades in OpenCV Library. I believe we need better Service standards for AI, Machine Learning and Computer Vision. The standards that lead us to better performant models.

I’ve got a special affinity towards Design by Contracts in Object-Oriented Methodology. It goes by this technique. If a contract initiator has certain responsibilities, the client has certain obligations towards it. Also if the client has certain responsibilities, the contract initiator has certain obligations towards the client. They are documented in the table below:

OpenCV.js is best for thresholding on the client side as shown here. Other tasks that client side can do are:

  • Erosion
  • Dilation
  • Convolution
  • Laplacian
  • Image Gradients
  • Region Growing
  • Image Segmentation

Image segmentation using region growing and thresholding is especially a crude method, and that requires us to provide parameters to the model in order to get the work done. So this domain is very useful to be executed on the client-side.

Image for post
Image for post
Result of Image Thresholding

There are many libraries that make use of parallel computing, such as Emscripten to transpile JS code, SIMD.js, Asm.js. From the developers point of view, they make the tasks easy to execute and even do more by localising the computational power.

To setup OpenCV.js in Browser

Calling OpenCV methods in browser environment

While coding make sure you write to the same object because that way you can reduce the memory consumption. This is ECMAScript, that works in Chrome but not in every browser. You can use Webpack to compile ECMAScript to native JS.

Data formats and Datasets

A simple example could be that “the arrays are written using Javascript Array notation”

The data formats that any client side algorithm uses must comply with the formats used by the datasets. This could encapsulate the data transformation routines, shapes and sizes of images, classes that one image will map onto, etc. I like Microsoft’s CNTK BrainScript because it is a text-based format and very useful for capturing requirements from the client-side. A simple example could be that “the arrays are written using Javascript Array notation”. I believe CNTK’s BrainScript text readers are an art of invention because they fulfil the best needs that any algorithm wants. A machine learning framework that is mostly reliant on the client side must use data formats that BrainScript produces. This sort of a technique is similar to the React’s technique which uses JSX style syntax to capture almost all functionality that a user wants with almost close native implementation. Microsoft Azure uses diagramming based models to automate the process of delivering a Machine Learning solution, that is good and provides us the flexibility to design some ML tasks.

Featuretools

Featuretools (Python based) package is a feature synthesis package that creates relationships between data attributes. This is somewhat similar to Data Warehousing because it can generate a fact table, or even a star schema. The data warehousing standards say that we need to split Date into dimensions of Time, Hour, Day, Month, Year, etc. This is super important for featuretools as well because they work with temporal data often. For synthesis and workflow automation, temporal data is a must.

OpenCV and Feature Engineering

This is an excerpt of cascade haarcascade_smile.xml. See what metadata is stored in the cascade file. It looks at those Viola Jones proximal pixels and measures.

Conclusion

I have got to write more on the topic, because the topic on synthetic datasets and front-end excites me. Try writing some threading models in C++ and compile into JS code using openmp. You could see the close to native experience in Javascript. There is ample opportunity in using ONNX.js models in browser based environment. Some of the tools that do the work are:- CoreMLTools (from Apple), ONNX.js (from Microsoft), TensorFlow.js (from Google). The very task of creating a tensor meshgrid is difficult in any of these algorithms. With the client-side processing of vectorized data, there are many use cases of problem solving possible and reduces compute power.

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…

Sign up for Analytics Vidhya News Bytes

By Analytics Vidhya

Latest news from Analytics Vidhya on our Hackathons and some of our best articles! Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Aswin Vijayakumar

Written by

Inspirational developer. Intel Edge Scholar with @Udacity

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Aswin Vijayakumar

Written by

Inspirational developer. Intel Edge Scholar with @Udacity

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store