I am modeling an ML problem as a sum of several segments. For each segment, we construct an ML model to predict its median and calculate MAE as the metric. Now I would like to get the median and MAE of the sum. Is it possible? How?
If I switch to mean and RMSE, will the segments be easier to add up?
Related
Can someone explain to me what tfma.metrics.MeanLabe does and how it should be used and what is the difference between tfma.metrics.MeanLabe and tfma.metrics.MeanPredictio and tfma.metrics.MeanAttributions. I am not sure why there is no explanation about these functions and the job that they do? How I can understand the details about them?
I appreciate it if someone can explain the job of these metrics.
Thanks
TFMA provides support for calculating metrics that were used at training time (i.e. built-in metrics) as well metrics defined after the model was saved as part of the TFMA configuration settings.
tfma.metrics.* consists of Standard TFMA metrics and plots.
tfma.metrics.MeanLabel calculates mean label by calculating the ratio of total weighted labels and total weighted examples.
tfma.metrics.MeanPrediction calculates mean prediction by calculating the ratio of total weighted predictions and total weighted examples.
tfma.metrics.MeanAttributions calculates mean attributions by calculating contribution of each input feature to the prediction made by the model.
This metrics are provided in metrics_specs section of tfma.EvalConfig which holds specifications for the model, metrics, and slices that are to be evaluated. Please refer TFMA tutorial for better understanding on using these metrics.
Hope this helps. Thank you!
I have two datasets which are same shape: (576, 450, 5) where 576 is the number of examples, 450 is the time points and 5 is the number of channels.
I want to calculate entropy and KL-divergence between these two datas. But I know that the entropy and kl-divergence are calculated between probability distributions but the datas are just numerical values(not probability distributions). So how can I calculate these for my datas? Should I convert my data to probability distributions? If so how can I do it with my 3d data? Thank you.
You can use quantiles to derive the empirical distribution of each dataset if it were binned and use that to compute the entropy, mutual information, etc. (any measure or distance that relates to one or more probability distributions) between the binned distributions.
In tensorflow, this can be achieved by using tfp.stats.quantiles as follows tfp.stats.quantiles(x, num_quantiles=4, interpolation='nearest'), where you can replace x with a dataset and set num_quantiles to any reasonable number.
The crucial thing to be careful of here is that the cut points should be the same for the two datasets (i.e., both binned random variables must have the same support).
More generally, you need to train/estimate a statistical model of the two datasets and then use that model to compute these metrics. In the above, the statistical model is a categorical distribution.
In sum, you can either:
Call tfp.stats.quantiles with num_quantiles on one dataset and then re-use the cut_points to compute quantiles for the other dataset. To do so you will need tfp.stats.find_bins.
Decide on the cut_points based on some other metric (equal partitions of the support of the data?) and then call tfp.stats.find_bins on both datasets.
The alternative I would favour is a variant of option 2. You can use quantiles to get the cut_points that correspond to both datasets if the datasets were concatenated together. You can then use those cut_points for binning both datasets.
Once you have the quantiles and/or the bins, you have a categorical probability distribution describing each dataset and from there these measures/distances can be computed easily.
When I used tensorflow 1 (estimator), there is a tfevent file recording events per log_step_count_steps, including the global_step/sec. I used that to calculate the average training speed.
I wonder what's the counterpart in tensorflow2?
Thanks,
Muyang
Mean Average Precision Confusion. (What happen if the model make no predictiona at all)
So , I am trying to understand what mean Average Precision. What I understand so far. First it will check if the predicted box and the ground box overlapped by certain IOU Threshold. Then , it will check the predicted box confidence score and sort them in the confidence order. Then it will calculate the AP at different recall value such as the precision-recall curve. So basically, what I understand is that of all the prediction that the model made, it will sort the prediction box based on the confidence score and calculate the PR curve.
I am using this as a reference to understand mAP (https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52)
But what will happen if the model make no prediction at all? or If the model makes only one prediction and that prediction is correct. Then is is the mAP one in this case? Because we will sort them based on model predicted box and since the model get the predicted the box right the mAP will be one.
Can I extract the confidence values or variance in prediction error from a tensorflow regressor? e.g. if the model gives a prediction x, then can I know the confidence band, like is x in +-25% range of the actual value?
I'm afraid it's not as easy as when using sofmax in the output layer. As said in here you can use the MSE of the NN on the validation as an estimate for variance, then use your desired value of confidence. Be aware that this approach assumes a lot of things (ie. distribution of errors is allways the same which may not be true) so if you really need those confidence intervals, a regression NN is not the best fit for you.