Add ML capabilities to your app with ML Kit
ML Kit beta brings Google’s machine learning expertise to mobile developers in a powerful and easy-to-use package. We can use base MLKit’s base APIs or run custom models to add online/offline ML capabilities to apps.
Base APIs are available (as on Aug 17, 2018 ) for following tasks:
- Image Labeling
Addresses use cases related to identifying objects, locations, activities, animal species and so on. On device usage is free and supports around 400+ commonly used labels. Cloud version provides free usage upto 1000 features/months and exposes 10k+ labels. Knowledge Graph Entity ID support is available on both on device and cloud version. See more on pricing here. Check out example labels and a sample example here
2. OCR or Text Recognition
Addresses use cases like scanning credit card numbers, business cards and so on. Apps can add this feature to automate many data entry tasks. Or we can build AR apps which will read text from live camera feed and show information by making an API call to database with pre-filled information. For instance, getting the flight or train schedule/arrival/departure information by scanning the paper ticket or the e-ticket. Cloud version offers identification of text in a broad range of languages and also text extraction from documents. Checkout sample code of recognizing text in images here
Lose It! — #12 app on iTunes in Health & Fitness category (Rated 4.7 with over 161.3k ratings) as on Aug 17, 2018 uses the API to power nutrition label scanning. The app has over 10 million downloads on Android.
3. Face Detection (not Face Recognition)
Using this Base API, we can address uses cases around detecting faces and facial landmarks (nose, eyes and so on). Imagine building a snapchat like filter of your own or developing an app which rates people on the basis of some metrics related to facial landmarks. Face detection also works for a live feed — create an application which opens up gate only if you smile!
4. Barcode Scanning
Barcode scanning API provides functionality to read data using most of the standard barcodes. Following barcodes are supported:
- Linear formats: Codabar, Code 39, Code 93, Code 128, EAN-8, EAN-13, ITF, UPC-A, UPC-E
- 2D formats: Aztec, Data Matrix, PDF417, QR Code
5. Landmark Detection
Using this API, we can detect well known landmarks in an image. In addition to name and geographical coordinates, we get a number of other details like similar images and son on. Below are details obtained by using a picture of Golden Gate Bridge
Here’s json response we get from the API
{
"landmarkAnnotations": [
{
"mid": "/m/035p3",
"description": "Golden Gate Bridge",
"score": 0.35074955,
"boundingPoly": {
"vertices": [
{
"x": 287,
"y": 169
},
{
"x": 607,
"y": 169
},
{
"x": 607,
"y": 318
},
{
"x": 287,
"y": 318
}
]
},
"locations": [
{
"latLng": {
"latitude": 37.810868,
"longitude": -122.476272
}
}
]
},
{
"mid": "/m/035p3",
"description": "Golden Gate Bridge",
"score": 0.3461872,
"boundingPoly": {
"vertices": [
{
"x": 79,
"y": 157
},
{
"x": 299,
"y": 157
},
{
"x": 299,
"y": 287
},
{
"x": 79,
"y": 287
}
]
},
"locations": [
{
"latLng": {
"latitude": 37.811013,
"longitude": -122.477801
}
}
]
}
],
"labelAnnotations": [
{
"mid": "/m/015kr",
"description": "bridge",
"score": 0.9573081,
"topicality": 0.9573081
},
{
"mid": "/m/01bqvp",
"description": "sky",
"score": 0.9408597,
"topicality": 0.9408597
},
{
"mid": "/m/0csby",
"description": "cloud",
"score": 0.88219845,
"topicality": 0.88219845
},
{
"mid": "/m/01b2w5",
"description": "sunset",
"score": 0.87522304,
"topicality": 0.87522304
},
{
"mid": "/m/0d1n2",
"description": "horizon",
"score": 0.8456816,
"topicality": 0.8456816
},
{
"mid": "/m/06npx",
"description": "sea",
"score": 0.84551364,
"topicality": 0.84551364
},
{
"mid": "/m/051gs3",
"description": "fixed link",
"score": 0.813437,
"topicality": 0.813437
},
{
"mid": "/m/0csy8",
"description": "suspension bridge",
"score": 0.7733133,
"topicality": 0.7733133
},
{
"mid": "/m/01b3l7",
"description": "dusk",
"score": 0.7567896,
"topicality": 0.7567896
},
{
"mid": "/m/06cnp",
"description": "river",
"score": 0.75497615,
"topicality": 0.75497615
},
{
"mid": "/m/01b3kg",
"description": "dawn",
"score": 0.7400379,
"topicality": 0.7400379
},
{
"mid": "/m/0838f",
"description": "water",
"score": 0.7339397,
"topicality": 0.7339397
},
{
"mid": "/m/02_ktj",
"description": "morning",
"score": 0.7337289,
"topicality": 0.7337289
},
{
"mid": "/m/01b2q6",
"description": "sunrise",
"score": 0.73126,
"topicality": 0.73126
},
{
"mid": "/m/02l215",
"description": "reflection",
"score": 0.713901,
"topicality": 0.713901
},
{
"mid": "/m/026fm63",
"description": "calm",
"score": 0.70837736,
"topicality": 0.70837736
},
{
"mid": "/m/04mx32",
"description": "evening",
"score": 0.7023143,
"topicality": 0.7023143
},
{
"mid": "/m/013vs",
"description": "afterglow",
"score": 0.6465814,
"topicality": 0.6465814
},
{
"mid": "/m/05kq4",
"description": "ocean",
"score": 0.6145469,
"topicality": 0.6145469
},
{
"mid": "/m/0157l1",
"description": "cable stayed bridge",
"score": 0.5945261,
"topicality": 0.5945261
},
{
"mid": "/g/11k2xz7mr",
"description": "meteorological phenomenon",
"score": 0.5603266,
"topicality": 0.5603266
},
{
"mid": "/m/04p25",
"description": "loch",
"score": 0.54516745,
"topicality": 0.54516745
},
{
"mid": "/m/09wdrl",
"description": "extradosed bridge",
"score": 0.5216861,
"topicality": 0.5216861
}
],
"safeSearchAnnotation": {
"adult": "VERY_UNLIKELY",
"spoof": "VERY_UNLIKELY",
"medical": "VERY_UNLIKELY",
"violence": "VERY_UNLIKELY",
"racy": "VERY_UNLIKELY"
},
"imagePropertiesAnnotation": {
"dominantColors": {
"colors": [
{
"color": {
"red": 55,
"green": 44,
"blue": 40
},
"score": 0.1882846,
"pixelFraction": 0.116575204
},
{
"color": {
"red": 131,
"green": 106,
"blue": 103
},
"score": 0.016361877,
"pixelFraction": 0.010496031
},
{
"color": {
"red": 218,
"green": 195,
"blue": 172
},
"score": 0.009196449,
"pixelFraction": 0.016569875
},
{
"color": {
"red": 253,
"green": 204,
"blue": 114
},
"score": 0.004897723,
"pixelFraction": 0.0041025095
},
{
"color": {
"red": 35,
"green": 27,
"blue": 23
},
"score": 0.15742785,
"pixelFraction": 0.10160371
},
{
"color": {
"red": 68,
"green": 49,
"blue": 36
},
"score": 0.1535307,
"pixelFraction": 0.0701689
},
{
"color": {
"red": 99,
"green": 86,
"blue": 90
},
"score": 0.1457985,
"pixelFraction": 0.08300922
},
{
"color": {
"red": 75,
"green": 46,
"blue": 32
},
"score": 0.06324315,
"pixelFraction": 0.032020885
},
{
"color": {
"red": 102,
"green": 72,
"blue": 55
},
"score": 0.049987912,
"pixelFraction": 0.09126752
},
{
"color": {
"red": 97,
"green": 75,
"blue": 59
},
"score": 0.036990125,
"pixelFraction": 0.023975704
}
]
}
},
"cropHintsAnnotation": {
"cropHints": [
{
"boundingPoly": {
"vertices": [
{
"x": 50
},
{
"x": 562
},
{
"x": 562,
"y": 639
},
{
"x": 50,
"y": 639
}
]
},
"confidence": 0.79999995,
"importanceFraction": 0.83
},
{
"boundingPoly": {
"vertices": [
{},
{
"x": 639
},
{
"x": 639,
"y": 639
},
{
"y": 639
}
]
},
"confidence": 0.79999995,
"importanceFraction": 1
},
{
"boundingPoly": {
"vertices": [
{},
{
"x": 639
},
{
"x": 639,
"y": 536
},
{
"y": 536
}
]
},
"confidence": 0.79999995,
"importanceFraction": 0.83
}
]
},
"webDetection": {
"webEntities": [
{
"entityId": "/m/051gs3",
"score": 0.8717412,
"description": "Bridge–tunnel"
},
{
"entityId": "/m/035p3",
"score": 0.83145,
"description": "Golden Gate Bridge"
},
{
"entityId": "/m/0157l1",
"score": 0.59367347,
"description": "Cable-stayed bridge"
},
{
"entityId": "/m/015kr",
"score": 0.5786,
"description": "Bridge"
},
{
"entityId": "/m/0csy8",
"score": 0.5603842,
"description": "Suspension bridge"
},
{
"entityId": "/m/06npx",
"score": 0.5181158,
"description": "Sea"
},
{
"entityId": "/m/09wdrl",
"score": 0.44899777,
"description": "Extradosed bridge"
},
{
"entityId": "/m/0bmz2",
"score": 0.2979,
"description": "Electrical cable"
},
{
"entityId": "/m/03p1p8r",
"score": 0.2936,
"description": "Sky plc"
}
],
"visuallySimilarImages": [
{
"url": "https://www.videvo.net/videvo_files/images/preview_161022_077_SanFrancisco_4K.jpg"
},
{
"url": "https://thumbs.dreamstime.com/b/sailboat-golden-gate-bridge-sunset-clouds-sailboat-golden-gate-bridge-sunset-colorful-wispy-clouds-107643862.jpg"
},
{
"url": "https://d2v9y0dukr6mq2.cloudfront.net/video/thumbnail/S_-0n06Vipfunqrp/san-francisco-bay-golden-gate-golden-gate-bridge-at-sunset-golden-gate-bridge-over-san-francisco-bay_rpeotwkcl__S0000.jpg"
},
{
"url": "https://i.pinimg.com/236x/41/26/89/412689698db67e9d6c970ff9afe2f7eb--golden-gate-bridge-vin.jpg"
},
{
"url": "http://goldengate.org/exhibits/images/GGB-exhibit2-4_5.jpg"
},
{
"url": "http://c1.peakpx.com/wallpaper/721/408/35/golden-gate-bridge-sunset-wallpaper.jpg"
},
{
"url": "https://media-cdn.tripadvisor.com/media/photo-s/05/5f/22/e0/blazing-saddles.jpg"
},
{
"url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-sunrise-san-francisco-beach-california-60924538.jpg"
},
{
"url": "https://d2v9y0dukr6mq2.cloudfront.net/video/thumbnail/pbJRSBz/the-golden-gate-bridge-sunset_4jiio98lg__S0000.jpg"
},
{
"url": "http://footage.framepool.com/shotimg/qf/909179899-san-francisco-bay-golden-gate-strait-golden-gate-bridge-pacific-coast.jpg"
},
{
"url": "https://us.123rf.com/450wm/kitleong/kitleong1509/kitleong150900270/45116349-golden-gate-bridge-sfo-sun-raise.jpg?ver=6"
},
{
"url": "https://i.dailymail.co.uk/i/pix/2014/03/16/article-0-1C585E9A00000578-390_634x422.jpg"
},
{
"url": "http://media6.trover.com/T/58c4309010b26810ab02f909/fixedw_large_4x.jpg"
},
{
"url": "https://n7tpw52zdt-flywheel.netdna-ssl.com/wp-content/uploads/2017/05/san-francisco-golden-gate-bridge-8.jpg"
},
{
"url": "http://rubinweddings.com/wp-content/uploads/2016/04/San-Francisco-and-Golden-Gate-Bridge-Engagement-Session-0016.jpg"
},
{
"url": "https://static1.squarespace.com/static/5935836e5016e13ecccd943e/5986791837c5815a82171acc/598679184c0dbf1460634378/1501985450661/IMG_1712-1.jpg"
}
],
"bestGuessLabels": [
{
"label": "golden gate bridge",
"languageCode": "en"
}
]
}
}And many more base APIs like Smart Reply (to provide suggested text snippet that fits context) are expected to be added to the Kit over next few years.
In addition to the Base APIs, we can upload our own TensorFlow Lite models which can then be exposed as an API via ML Kit.
About Author
Arpit is a seasoned technologist with vast experience in leading large cross-functional and cross-geography teams. Arpit also consults clients on competitive market analysis, defining MVPs, product ideation, product monetization and go live strategies.
Arpit believes we should all contribute back to society. He has set his goals for social work in five broad areas. You can read more about the same in his blog post “Do Good, Together” on Tumblr. Arpit is interested in working with people who want to contribute towards the same goals.
You can follow Arpit on Linkedin or on Twitter
ABC. Always be clappin’.
