posenet-similarity
v0.4.8
Published
A package helps to quickly get the similarity (distance) between two poses estimated by tfjs Posenet
Downloads
69
Readme
PosenetSimilarity
A package helps to quickly get the similarity/ distance between two poses estimated by tfjs Posenet.
Installation
via script tag
<script src="https://cdn.jsdelivr.net/npm/posenet-similarity/dist/posenet-similarity.min.js"></script>via NPM
npm install posenet-similarityExample usages
NOTE: PosenetSimilarity doesn't need Posenet to be installed and work together. The examples just show how you might chain the outputs of Posenet with PosenetSimilarity.
When imported via script tag, PosenetSimilarity exposes pns globally for accessing the APIs.
<html>
<head>
<!-- Load TensorFlow.js -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"></script>
<!-- Load Posenet -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/posenet"></script>
<!-- Load posenet-similarity -->
<script src="https://cdn.jsdelivr.net/npm/posenet-similarity/dist/posenet-similarity.min.js"></script>
</head>
<body>
<img id='pose1' src='/images/pose1.jpg '/>
<img id='pose2' src='/images/pose2.jpg '/>
</body>
<script>
var pose1ImageElement = document.getElementById('pose1');
var pose2ImageElement = document.getElementById('pose2');
// For more detailed Posenet setup, please refer its own document.
// Load Posenet model
posenet.load().then(function(net) {
// Estimate the two poses
return Promise.all([
net.estimateSinglePose(pose1ImageElement),
net.estimateSinglePose(pose2ImageElement)
])
}).then(function(poses){
// Calculate the weighted distance between the two poses
var weightedDistance = pns.poseSimilarity(poses[0], poses[1]);
console.log(weightedDistance)
})
</script>
</html>via NPM
import * as posenet from '@tensorflow-models/posenet';
import { poseSimilarity } from 'posenet-similarity';
// For more detailed Posenet setup, please refer its own document.
async function estimatePoseOnImage(imageElement) {
// Load the posenet model from a checkpoint
const net = await posenet.load();
// Estimate the pose on the imageElement
const pose = await net.estimateSinglePose(imageElement);
return pose;
}
const pose1ImageElement = document.getElementById('pose1');
const pose2ImageElement = document.getElementById('pose2');
Promise.all([
estimatePoseOnImage(pose1ImageElement),
estimatePoseOnImage(pose2ImageElement)
]).then(poses => {
// Calculate the weighted distance between the two poses
const weightedDistance = poseSimilarity(poses[0], poses[1]);
console.log(weightedDistance)
});API
poseSimilarity(pose1, pose2[, options]): number
As the main and entry method of this package, it accepts 2 pose objects which were the predictive results from tfjs PoseNet and returns a numeric value of the similarity or distance between them.
import { poseSimilarity } from 'posenet-similarity';
// calculate with the default strategy
const weightedDistance = poseSimilarity(pose1, pose2);Some optional features can be enabled by setting the third parameter, options, and we will introduce them below.
options
{
strategy: string ('weightedDistance' | 'cosineDistance' | 'cosineSimilarity') | Function,
customWeight: {
mode: string ('multiply' | 'replace' | 'add'),
scores: Object | number[]
}
}strategy: string | Function
Without specifying a strategy, poseSimilarity uses weightedDistance to calculate the distance between pose1 and pose2 in default, but you can still choose to use other build-in or your own strategy by setting this option.
It accepts string and Function types of inputs. When using string, strategy accepts 'weightedDistance' (default), 'cosineDistance' or 'cosineSimilarity'.
// Use weightedDistance
const weightedDistance = poseSimilarity(pose1, pose2, { strategy: 'weightedDistance' });
// Use cosineDistance
const cosineDistance = poseSimilarity(pose1, pose2, { strategy: 'cosineDistance' });
// Use cosineSimilarity
const cosineSimilarity = poseSimilarity(pose1, pose2, { strategy: 'cosineSimilarity' });- Bigger the return value means more similar when using similarity based strategies (e.g. cosineSimilarity)
- Smaller the return value means more similar when using distance based strategies (e.g. cosineDistance and weightedDistance)
If none of above strategies suit your needs, you can pass your own function. A custom strategy function will receive 3 parameters from poseSimilarity.
- vectorPose1XY: An array
[p1_x1, p1_y1, p1_x2, p1_y2, ...]contains normalized positions of pose1. - vectorPose1XY: An array
[p2_x1, p2_y1, p2_x2, p2_y2, ...]contains normalized positions of pose2. - vectorPose1Scores: An array
[p1_s1, p1_s2, ...]contains the confident scores of pose1.
To keep this page short, please refer the Algorithms page for more details about what the 3 parameters are.
// Your own strategy function
function myStrategy(vectorPose1XY, vectorPose2XY, vectorPose1Scores) {
// ...
}
// Use it by assigning it to the strategy option
const myCalculation = poseSimilarity(pose1, pose2, { strategy: myStrategy });customWeight: Object
const options = {
customWeight: {
mode: string ('multiply' | 'replace' | 'add'),
scores: Object | number[]
}
}
const weightedDistance = poseSimilarity(pose1, pose2, options);When using weighted strategies, e.g., weightedDistance, the weights are the confident scores of each keypoint in default. However, you can manipulate the scores by setting customWeight option, and it can be useful when you want to emphasize some keypoints.
NOTE: Only weighted and your own strategies function will be affected by the changes made by customWeight.
customWeight requires mode and scores properties. mode accepts 'multiply', 'replace' or 'add' which decides how you manipulate the original scores, and you must specify one when setting the customWeight option.
- 'multiply' mode will multiply the original keypoint scores by your custom scores.
- 'replace' mode will replace the original keypoint scores with your custom scores.
- 'add' mode will sum up the original keypoint scores and your custom scores.
Your custom scores can be set in the scores property which accepts an Object or number[].
- When giving an Object, the keys are the part names of keypoints to be modified, and values are numbers to manipulate the original scores, e.g., { nose: 0.1, leftEye: 0.2 } will only modify the scores of nose and leftEye.
- When giving a number[], the elements are numbers to manipuate the original scores and their indexes are corresponded with ids of keypoints, e.g., [0.1, 0.2] will only modify the scores of nose and leftEye.
Let's take a look at a simple example. Assume we have a simplified set of keypoints as below.
[
{
"position": {
"y": 1,
"x": 1
},
"part": "nose",
"score": 0.1
},
{
"position": {
"y": 2,
"x": 2
},
"part": "leftEye",
"score": 0.2
},
{
"position": {
"y": 3,
"x": 3
},
"part": "rightEye",
"score": 0.3
}
]The results of modified scores in different modes will be:
/*
{
customWeight: { mode: 'multiply', scores: { leftEye: 2} }
or { mode: 'multiply', scores: [1, 2] }
}
*/
[
{
"position": {
"y": 1,
"x": 1
},
"part": "nose",
"score": 0.1
},
{
"position": {
"y": 2,
"x": 2
},
"part": "leftEye",
"score": 0.4 // 0.2 * 2
},
{
"position": {
"y": 3,
"x": 3
},
"part": "rightEye",
"score": 0.3
}
]/*
{
customWeight: { mode: 'replace', scores: { leftEye: 2} }
or { mode: 'replace', scores: [0.1, 2] }
}
*/
[
{
"position": {
"y": 1,
"x": 1
},
"part": "nose",
"score": 0.1
},
{
"position": {
"y": 2,
"x": 2
},
"part": "leftEye",
"score": 2 // replaced by 2
},
{
"position": {
"y": 3,
"x": 3
},
"part": "rightEye",
"score": 0.3
}
]/*
{
customWeight: { mode: 'add', scores: { leftEye: 2} }
or { mode: 'add', scores: [0, 2] }
}
*/
[
{
"position": {
"y": 1,
"x": 1
},
"part": "nose",
"score": 0.1
},
{
"position": {
"y": 2,
"x": 2
},
"part": "leftEye",
"score": 2.2 // 0.2 + 2
},
{
"position": {
"y": 3,
"x": 3
},
"part": "rightEye",
"score": 0.3
}
]