Real time pose estimation and human body detection without package using PoseTracker API (+20 fps IOS, Android & Web)
After a lot of time spend at looking for the best options to create real time human body detection on mobile. A lot of package and AI tested on iOS and Android. I decided to create a way to help others dev to add this to there application without package and all the problems that come with them !
Easier and faster with PoseTracker API
PoseTracker is the first real-time posture detection API optimized for Web, iOS, and Android. There is no need to install a package, everything is provided by our API callable from a WebView 🚀.
This is how it’s work :
- You define your needs, such as exercises, reps, time between reps and then send them to our API.
- PoseTracker responds with a URL that will guide and analyze the user’s training.
- You can call the URL within a WebView in your mobile or web application.
- The WebView provides analysis, posture detection, and exercise recognition.
- We provide live feedback to your application, so that you can create your own design to display information to your users.
Creating a expo react native application with PoseTracker Free endpoint
For now, we will use the free PoseTracker endpoint to create a WebView that detects 17 points of the human body and returns them in order to display a human skeleton over the user’s camera. (So we are starting at step 3.)
1. Setup config and import required
Create an expo react native application and install these package :react-native-webview, expo-camera, react-native-svg
Define our App.js base and allow our app to access the user’s camera :
import { StyleSheet, Text, View} from 'react-native';
import { WebView } from 'react-native-webview'
import {useEffect, useState} from "react";
import { Camera } from 'expo-camera'
export default function App() {
const [hasPermission, setHasPermission] = useState(false)
//Our API need to have access to the devise camera
useEffect(() => {
;(async () => {
const { status } = await Camera.requestCameraPermissionsAsync()
setHasPermission(status === 'granted')
})()
}, [])
if (!hasPermission) return <View style={styles.container}><Text>The app needs access to your camera. Allow it in your device settings</Text></View>
return (
<View style={styles.container}>
<WebView
javaScriptEnabled={true}
domStorageEnabled={true}
allowsInlineMediaPlayback={true}
mediaPlaybackRequiresUserAction={false}
style={{
width: width,
height: height,
zIndex: 1,
}}
source={{
uri: `https://posture-detector-api.vercel.app/posture-detect?width=${width}&height=${height}`,
}}
originWhitelist={['*']}
injectedJavaScript={jsBridge}
onMessage={(event) => {
const info = JSON.parse(event.nativeEvent.data)
webViewCallback(info)
}}
/>
{renderPose(currentPoses)}
</View>
);
}2. Data exchange between PoseTracker WebView and our app :
We need to build a WebView and set the following source : https://posture-detector-api.vercel.app/posture-detect?width=${width}&height=${height}
The endpoint takes 2 params width & height for the width and height of the camera output.
import { WebView } from 'react-native-webview'
const width = 300
const height = 300
<WebView
javaScriptEnabled={true}
domStorageEnabled={true}
allowsInlineMediaPlayback={true}
mediaPlaybackRequiresUserAction={false}
style={{
width: width,
height: height,
zIndex: 1,
}}
source={{
uri: `https://posture-detector-api.vercel.app/posture-detect?width=${width}&height=${height}`,
}}
originWhitelist={['*']}
injectedJavaScript={jsBridge}
onMessage={(event) => {
const info = JSON.parse(event.nativeEvent.data)
webViewCallback(info)
}}
/>We handle WebView responses with what’s inside the onMessage param.
The endpoint use what we are sending throw injectedJavaScript to send back data to our App :
const jsBridge = `
(function() {
window.webViewCallback = function(info) {
window.ReactNativeWebView.postMessage(JSON.stringify(info));
}
})();
`This bridge allow us to receive data from the WebView in our app.
3. Handle data received
onMessage={(event) => {
const info = JSON.parse(event.nativeEvent.data)
webViewCallback(info)
}}With this WebView param we are parsing the data received and send them to webViewCallback :
//This func receive data from the WebView
const webViewCallback = (info) => {
switch (info.type) {
case 'body poses':
return handlePoses(info.poses)
default:
return handlePoses(info)
}
}Check out the API documentation to learn more about what’s inside the information received from WebView.
For this tutorial we will only handle one type of message info.type = 'body poses' , it look like :
Object {
"type": "body poses"
"poses": Object {
"keypoints": Array [
Object {
"name": "nose",
"score": 0.17302913963794708,
"x": 51.53513909486614,
"y": 292.0035267462864,
},
.....
Object {
"name": "right_ankle",
"score": 0.0781388059258461,
"x": 42.53210480412571,
"y": 278.1044364463628,
},
],
"score": 0.28030067682266235,
}
}4. Drawing the body skeleton over the camera output
First we need to store these poses :
const [currentPoses, setCurrentPoses] = useState()
const handlePoses = (poses) => {
//console.log('current body poses :', poses)
setCurrentPoses(poses)
}Then inside the main View we pass this state to {renderPose(currentPoses)} :
//Check if poses has keypoints to draw
export const renderPose = (poses) => {
if (poses && poses.keypoints?.length > 0) {
return drawSqueleton(poses)
} else {
return <View />
}
}Now we can draw the skeleton using react-native-svg :
//threshold accuracy model
const MIN_KEYPOINT_SCORE = 0.3
export const drawSqueleton = (poses) => {
const circles = drawCircles(poses)
const lines = drawLines(poses)
return (
<Svg
style={{
position: 'absolute',
left: -50,
zIndex: 30,
transform: [{ scaleX: -1 }]
}}>
{circles}
{lines}
</Svg>
)
}
export const drawCircles = (poses) => {
const circles = poses.keypoints
.filter((k) => (k.score ?? 0) > MIN_KEYPOINT_SCORE)
.map((k) => {
const x = k.x
const y = k.y
return (
<Circle
key={`skeletonkp_${k.name}`}
cx={x}
cy={y}
r='8'
strokeWidth='4'
fill='#cff532'
stroke='#FFC300'
/>
)
})
return circles
}
export const drawLines = (
poses,
showFacePoints = true
) => {
let lines = []
// key points by name
const points = new Map()
poses.keypoints.map((point) => points.set(point.name, point))
lines.push(drawLine(points.get('left_shoulder'),points.get('right_shoulder')))
lines.push(drawLine(points.get('left_hip'),points.get('right_hip')))
// left arm
lines.push(drawLine(points.get('left_shoulder'),points.get('left_elbow')))
lines.push(drawLine(points.get('left_elbow'),points.get('left_wrist')))
// left side
lines.push(drawLine(points.get('left_shoulder'),points.get('left_hip')))
// left leg
lines.push(drawLine(points.get('left_hip'),points.get('left_knee')))
lines.push(drawLine(points.get('left_knee'), points.get('left_ankle')))
// right arm
lines.push(drawLine(points.get('right_shoulder'),points.get('right_elbow')))
lines.push(drawLine(points.get('right_elbow'),points.get('right_wrist')))
// right side
lines.push(drawLine(points.get('right_shoulder'),points.get('right_hip')))
// right leg
lines.push(drawLine(points.get('right_hip'),points.get('right_knee')))
lines.push(drawLine(points.get('right_knee'), points.get('right_ankle')))
if (showFacePoints) {
lines.push(drawLine(points.get('right_ear'), points.get('right_eye')))
lines.push(drawLine(points.get('right_eye'), points.get('nose')))
lines.push(drawLine(points.get('nose'), points.get('left_eye')))
lines.push(drawLine(points.get('left_eye'), points.get('left_ear')))
}
return lines
}
function drawLine(pointA, pointB) {
if (pointA.score < MIN_KEYPOINT_SCORE || pointB.score < MIN_KEYPOINT_SCORE)
return
const x1 = pointA.x
const y1 = pointA.y
const x2 = pointB.x
const y2 = pointB.y
return (
<Line
key={`skeletonkp_line_${pointA.name}_${pointB.name}`}
x1={x1}
y1={y1}
x2={x2}
y2={y2}
stroke='#cff532'
strokeWidth={4}
/>
)
}We did it!
The next step : sign up for our API and start detecting movements 💪
Source code Github for the react native App :
Any needs for posturale detection application ? Visit www.movelytics.fr
