hand-pose model with TensorFlow.js returning `NaN` for scores - tensorflow

Hi there potential helpers, I've been following this guide https://blog.tensorflow.org/2021/11/3D-handpose.html
I seem to have everything setup vaguely correctly
But detector.estimateHands(video) is returning an object with NaN as the score and every coordinate:
[{
handedness: "Right",
score: NaN,
keypoints: [
{x: NaN, y: NaN, name: 'wrist'},
{x: NaN, y: NaN, name: 'thumb_cmc'},
etc...
],
keypoints3d: [
{x: NaN, y: NaN, z: NaN, name: 'wrist'},
etc...
]
}]
The thing that makes me think it's almost working is that it only returns that object if I actually have my hand in front of the webcam. And if I hold two hands up it returns two of these objects. And then if I hide my hands it returns an empty array.
So it is recognising hands, it's just not returning any usable scores.
Here is the essence of the code:
const model = handPose.SupportedModels.MediaPipeHands;
const config = {
modelType:"full",
runtime: "tfjs"
};
const detector = await handPose.createDetector(model, config);
and then in an requestAnimationFrame loop:
const video = shadowRoot.querySelector('video');
const hands = await detector.estimateHands(video);
console.log(hands);
console.log(hands[0]?.score);
I've got all the mentioned npm modules installed
#tensorflow/tfjs
#tensorflow-models/hand-pose-detection
#tensorflow/tfjs-core
#tensorflow/tfjs-converter
#tensorflow/tfjs-backend-webgl
And I've tried playing with config that can be passed into createDetector and estimateHands but with the typescript definitions there's not much choice.
Anyone seen this before and have any idea how to fix it?
Thank you 🙇

Related

GSAP with MotionPath and autoRotate not working

Im using GSAP with MotionPathPlugin and the PixiPlugin to make my square follow a path. I have set autoRotate to true but there seems to be a bug, as you can see in the GIF. Seems like my sprite is over rotating a little bit.
The path is an array of {x, y} values.
I also tried using the array method for autoRotate as such autoRotate: ['x', 'y', 'angle', 90, true], // 4th param is angle in rad
motionPath: {
autoRotate: ['x', 'y', 'angle', 90, true], // 4th param is angle in rad
alignOrigin: [0.5, 0.5],
// autoRotate: true,
align: values,
path: values,
curviness: 0,
},
Thank you for any input.
After a lot of searching I found I need to useRadians: true, in the MotionPath vars option.
gsap.to(sprite, {
motionPath: {
autoRotate: 1.5708,//in radians, offset rotation by 90 degree
path: values,
curviness: 0,
useRadians: true,
},
}

Exporting TFRecords training patches with Google Earth Engine (kernelSize issues)

I've been using GEE to export some training patches from Sentinel-2 to be used in Python.
I could make it work, by following the GEE guide https://developers.google.com/earth-engine/tfrecord, and using the Export.image.toDrive function and then I can parse the exported TFRecord file to reconstruct my tiles.
var image_export_options = {
'patchDimensions': [366, 366],
'maxFileSize': 104857600,
// 'kernelSize': [366, 366],
'compressed': true
}
Export.image.toDrive({
image: clipped_img.select(bands.concat(['classes'])),
description: 'PatchesExport',
fileNamePrefix: 'Oros_1',
scale: 10,
folder: 'myExportFolder',
fileFormat: 'TFRecord',
region: export_area,
formatOptions: image_export_options,
})
However, when I try to specify the kernelSize in the formatOptions (that was supposed to "overlaps adjacent tiles by [kernelSize[0]/2, kernelSize[1]/2]", according to the guide) the files are exported but the '*mixer.json' doesn't reflect the increased number of patches and I am not able to iterate through the patches afterwards. The following command crashes the google colab session:
image_dataset = tf.data.TFRecordDataset(str(path/(file_prefix+'-00000.tfrecord.gz')), compression_type='GZIP')
first = next(iter(image_dataset))
first
The weird is that the problem happens only when I add the kernelSize to the formatOptions.
After some time trying to overcome this issue, I realized a not well documented behavior when one uses the kernel size to export patches from GEE.
Bundled with the exported TFRecord, there exists one xml file called mixer.
It doesn't matter if we use:
'patchDimensions': [184, 184],
'kernelSize': [1, 1], #default for no overlapping
or
'patchDimensions': [184, 184],
'kernelSize': [184, 184], #half patch overlapping
The mixer file remains the same and no mention to the kernel/overlapping size:
{'patchDimensions': [184, 184],
'patchesPerRow': 8,
'projection': {'affine': {'doubleMatrix': [10.0,
0.0,
493460.0,
0.0,
-10.0,
9313540.0]},
'crs': 'EPSG:32724'},
'totalPatches': 40}
In the second case, if we try to parse the patches using tf.io.parse_single_example(example_proto, image_features_dict), where image_features_dict equals something like:
{'B2': FixedLenFeature(shape=[184, 184], dtype=tf.float32, default_value=None),
'B3': FixedLenFeature(shape=[184, 184], dtype=tf.float32, default_value=None),
'B4': FixedLenFeature(shape=[184, 184], dtype=tf.float32, default_value=None)}
it will raise the error:
_FallbackException: This function does not handle the case of the path where all inputs are not already EagerTensors.
Can't parse serialized Example. [Op:ParseExampleV2]
Instead, to parse these records which have kernelSize > 1, we have to consider patchDimentions + kernelSize as the resulting patch size, even though the mixer.xml file says on contraty. In this example, our patchSize would be 368 (original patch size + kernelSize). Be aware that for odd kernel sizes, the number to be added to the original patch size is kernelSize - 1.

Is it a bug in zingchart ( boxplot legend )

I am trying to draw a boxplot using zingchart. I need a legend for the chart, but the legend generated is wrong. is it a bug in zingchart?
Below is the json passed in:
{
"type": "boxplot",
"vertical-labels": true,
"legend":{},
"scale-x": {
"labels": []
},
"series": [
{
"data-box": [ [760, 801, 848, 895, 965 ],[733, 853, 939, 980, 1080 ]],
"text":"f1"
},
{
"data-box": [[733, 853, 939, 980, 1080 ],[733, 853, 939, 980, 1080 ]],
"text":"f2"
}
]
}
And this is the genrated boxplot, which should have two series, but it generated four series.
Is it a bug or I passed the wrong json data? Thanks in advance.
sinsin.
Sorry to see that you're having an issue with our boxplot module. However, this is an issue with the boxplot module itself, and not anything you did! I'm on the ZingChart support team, so I'll be sure to pass this on to our developers. In the meantime, you can create your own legend using shapes with labels on those shapes, as I've done here.
I'm a big fan of boxplot charts, but this is a relatively new ZingChart module, and it looks like it needs some more love.

Trouble setting up the SimpleVector encoder

Using the commits from breznak for the encoders (I wasn't able to figure out "git checkout ..." with GitHub, so I just carefully copied over the three files - base.py, multi.py, and multi_test.py).
I ran multi_test.py without any problems.
Then I adjusted my model parameters (MODEL_PARAMS), so that the encoders portion of 'sensorParams' looks like this:
'encoders': {
'frequency': {
'fieldname': u'frequency',
'type': 'SimpleVector',
'length': 5,
'minVal': 0,
'maxVal': 210
}
},
I also adjusted the modelInput portion of my code, so it looked like this:
model = ModelFactory.create(model_params.MODEL_PARAMS)
model.enableInference({'predictedField': 'frequency'})
y = [1,2,3,4,5]
modelInput = {"frequency": y}
result = model.run(modelInput)
But I get the final error, regardless if I instantiate 'y' as a list or a numpy.ndarray
File "nta/eng/lib/python2.7/site-packages/nupic/encoders/base.py", line 183, in _getInputValue
return getattr(obj, fieldname)
AttributeError: 'list' object has no attribute 'idx0'
I also tried initializing a SimpleVector encoder inline with my modelInput, directly encoding my array, then passing it through modelInput. That violated the input parameters of my SimpleVector, because I was now double encoding. So I removed the encoders portion of my model parameters dictionary. That caused a spit up, because some part of my model was looking for that portion of the dictionary.
Any suggestions on what I should do next?
Edit: Here're the files I'm using with the OPF.
sendAnArray.py
import numpy
from nupic.frameworks.opf.modelfactory import ModelFactory
import model_params
class sendAnArray():
def __init__(self):
self.model = ModelFactory.create(model_params.MODEL_PARAMS)
self.model.enableInference({'predictedField': 'frequency'})
for i in range(100):
self.run()
def run(self):
y = [1,2,3,4,5]
modelInput = {"frequency": y}
result = self.model.run(modelInput)
anomalyScore = result.inferences['anomalyScore']
print y, anomalyScore
sAA = sendAnArray()
model_params.py
MODEL_PARAMS = {
'model': "CLA",
'version': 1,
'predictAheadTime': None,
'modelParams': {
'inferenceType': 'TemporalAnomaly',
'sensorParams': {
'verbosity' : 0,
'encoders': {
'frequency': {
'fieldname': u'frequency',
'type': 'SimpleVector',
'length': 5,
'minVal': 0,
'maxVal': 210
}
},
'sensorAutoReset' : None,
},
'spEnable': True,
'spParams': {
'spVerbosity' : 0,
'globalInhibition': 1,
'columnCount': 2048,
'inputWidth': 5,
'numActivePerInhArea': 60,
'seed': 1956,
'coincInputPoolPct': 0.5,
'synPermConnected': 0.1,
'synPermActiveInc': 0.1,
'synPermInactiveDec': 0.01,
},
'tpEnable' : True,
'tpParams': {
'verbosity': 0,
'columnCount': 2048,
'cellsPerColumn': 32,
'inputWidth': 2048,
'seed': 1960,
'temporalImp': 'cpp',
'newSynapseCount': 20,
'maxSynapsesPerSegment': 32,
'maxSegmentsPerCell': 128,
'initialPerm': 0.21,
'permanenceInc': 0.1,
'permanenceDec' : 0.1,
'globalDecay': 0.0,
'maxAge': 0,
'minThreshold': 12,
'activationThreshold': 16,
'outputType': 'normal',
'pamLength': 1,
},
'clParams': {
'regionName' : 'CLAClassifierRegion',
'clVerbosity' : 0,
'alpha': 0.0001,
'steps': '5',
},
'anomalyParams': {
u'anomalyCacheRecords': None,
u'autoDetectThreshold': None,
u'autoDetectWaitRecords': 2184
},
'trainSPNetOnlyIfRequested': False,
},
}
The problem seems to be that the SimpleVector class is accepting an array instead of a dict as its input, and then reconstructs that internally as {'list': {'idx0': 1, 'idx1': 2, ...}} (ie as if this dict had been the input). This is fine if it is done consistently, but your error shows that it's broken down somewhere. Have a word with #breznak about this.
Working through the OPF was difficult. I wanted to input an array of indices into the temporal pooler, so I opted to interface directly with the algorithms (I relied heavy on hello_tp.py). I ignored SimpleVector all together, and instead worked through the BitmapArray encoder.
Subutai has a useful email on the nupic-discuss listserve, where he breaks down the three main areas of the NuPIC API: algorithms, networks/regions, & the OPF. That helped me understand my options better.

What is the purpose of the _classifierInput encoder field in the model parameters?

If you take a look at default model parameters as created by Cerebro, you see the following encoders:
{
'encoders': {
'_classifierInput': {
'classifierOnly': True,
'clipInput': True,
'fieldname': u'f',
'n': 100,
'name': '_classifierInput',
'type': 'AdaptiveScalarEncoder',
'w': 21
},
u'f': {
'clipInput': True,
'fieldname': u'f',
'n': 100,
'name': u'f',
'type': 'AdaptiveScalarEncoder',
'w': 21
}
}
}
What is the purpose of the _classifierInput encoder field? It looks like it just mirrors the encoder field that comes after it.
This is in clamodel.py:
def _getClassifierOnlyEncoder(self):
"""
Returns: sensor region's encoder that is sent only to the classifier,
not to the bottom of the network
"""
return self._getSensorRegion().getSelf().disabledEncoder
If you want the CLA to learn to predict (or "compute") a value, but not to use the value as input data, I think this is how you do it. For instance, you might have training data which includes the "answer" but this will be missing later (this is how a lot of the ML competitions work).