How can I get continuous landmarks from video for Mediapipe Handpose - mediapipe

Question

I am new to Javascript. I am trying to get output from MediaPipe Handpose. When I give an image into this model, I easily get an output. But when I try for a video, it doesn't work.
This is the head
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs-core"></script>
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs-converter"></script>
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs-backend-webgl"></script>
<script src="https://cdn.jsdelivr.net/npm/#tensorflow-models/handpose"></script>
My video ID details
Source<br>
<video id="video" src="o_new.mp4" width="640" height="480" controls>
</video>
<canvas id="canvas" style="overflow:auto"></canvas>
Inside the script
<script>
const video = document.getElementById("video");
async function main() {
// Load the MediaPipe handpose model.
const model = await handpose.load(maxContinuousChecks = 60);
console.log('Model Loaded')
// Pass in a video stream (or an image, canvas, or 3D tensor) to obtain a
// hand prediction from the MediaPipe graph.
const predictions = await model.estimateHands(video);
console.log('Estimated Hand')
console.log(predictions)
if (predictions.length > 0) {
/*
`predictions` is an array of objects describing each detected hand, for example:
[
{
handInViewConfidence: 1, // The probability of a hand being present.
boundingBox: { // The bounding box surrounding the hand.
topLeft: [162.91, -17.42],
bottomRight: [548.56, 368.23],
},
landmarks: [ // The 3D coordinates of each hand landmark.
[472.52, 298.59, 0.00],
[412.80, 315.64, -6.18],
...
],
annotations: { // Semantic groupings of the `landmarks` coordinates.
thumb: [
[412.80, 315.64, -6.18]
[350.02, 298.38, -7.14],
...
],
...
}
}
]
*/
for (let i = 0; i < predictions.length; i++) {
const keypoints = predictions[i].landmarks;
console.log('keypoints Loop')
// Log hand keypoints.
for (let i = 0; i < keypoints.length; i++) {
const [x, y, z] = keypoints[i];
console.log(`Keypoint ${i}: [${x}, ${y}, ${z}]`);
}
}
}
}
main();
</script>
How can I get continuous landmarks in an output object for video?
Here is the error.

Answer 1

I am updating my previous answer. You don't want to use set interval like I put in my former solution. When I used it for more than a few minutes it filled up my gpu memory and crashed webgl. I was able to comb through the developers demo.js file, and found a solution. In your js file replace your main() function with the below code:
const state = {
backend: 'webgl'
};
let model;
async function main() {
await tf.setBackend(state.backend);
model = await handpose.load();
landmarksRealTime(video);
}
const landmarksRealTime = async (video) => {
async function frameLandmarks() {
const predictions = await model.estimateHands(video);
if (predictions.length > 0) {
const result = predictions[0].landmarks;
console.log(result, predictions[0].annotations);
}
rafID = requestAnimationFrame(frameLandmarks);
};
frameLandmarks();
};
video.addEventListener("loadeddata", main);
This console logs continuous landmarks. It doesn't log anything if a hand is not detected. Also, it looks like the developers updated the recommended script tags a few days ago, so I would recommend updating your index.html file. They should be:
<!-- Require the peer dependencies of handpose. -->
<script src="https://unpkg.com/#tensorflow/tfjs-core#2.1.0/dist/tf-core.js"></script>
<script src="https://unpkg.com/#tensorflow/tfjs-converter#2.1.0/dist/tf-converter.js"></script>
<!-- You must explicitly require a TF.js backend if you're not using the tfs union bundle. -->
<script src="https://unpkg.com/#tensorflow/tfjs-backend-webgl#2.1.0/dist/tf-backend-webgl.js"></script>
<!-- Alternatively you can use the WASM backend: <script src="https://unpkg.com/#tensorflow/tfjs-backend-wasm#2.1.0/dist/tf-backend-wasm.js"></script> -->
<script src="https://unpkg.com/#tensorflow-models/handpose#0.0.6/dist/handpose.js"></script></head>