Standard Halcon example gives abormal program termination (4 or 11) with HDevelop 22.11 - halcon

I am trying to run the following template script in HDevelop 22.11 Progress.
But I keep getting either the following error:
Abnormal program termination: Received signal 11
and
Abnormal program termination: Received signal 4
I have been unable to find a good way to debug the issue.
I have been able to find out that it may be a mismatch in CUDA version, but from what I can tell 22.11 should support CUDA 11 and up and the graphcis card is using cuda 12.
* Deep learning anomaly detection example.
*
* This example demonstrates the general workflow for anomaly detection
* based on deep learning.
*
* The workflow typically consists of four steps which are described here:
* 1. Dataset preprocessing.
* 2. Training of the model and threshold estimation on 'ok' images.
* 3. Qualitative evaluation of the trained model.
* 4. Inference on new images.
*
dev_update_off ()
dev_close_window ()
set_system ('seed_rand', 73)
*
*
* *** 0.) SET INPUT/OUTPUT PATHS ***
*
* Set path to data.
get_image_dir (HalconImages)
*
* Base directory containing the folders with the used images.
* There must be a 'good' or 'ok' subfolder in ImageDir.
* For training, only images in the 'good' or 'ok' folders are used.
* Images in subfolders named differently will be treated as containing anomalies.
* They can be used for qualitative evalu ation later.
*
* Note: You can immediately train an anomaly detection model on your own data as long
* as your images not showing any anomaly are located in a subfolder called 'good' or 'ok'.
ImageDir := 'C:/Workplace/DL Training/L113 FST HALCON/'
ImageSubDirs := ['Good','Bad']
* Test from Stemmer:
* ImageDir := HalconImages + '/bottles/'
* ImageSubDirs := ['good','broken_large']
* ImageSubDirs := ['Good']
*
* Folder where the ground truth anomaly regions are stored.
* This folder is not required for training.
* If there is no ground truth data available, set AnomalyDir to [].
AnomalyDir := []
*
* Folder where the preprocessed samples will be stored.
OutputDir := './CIM_pek_tests_domainreduce'
*
* Dataset specific preprocessing.
ExampleSpecificPreprocessing := false
*
* Set image size. It should be large enough if the defects are small.
* Please refer to the documentation of read_dl_model for possible restrictions
* depending on the network.
ImageWidth := 288
ImageHeight := 192
*
* Set the complexity of the model which roughly describes the capability of the model
* to handle complex application images. A higher value can improve the performance but
* increases the time needed to train the model.
Complexity := 75
*
*
* *** 1.) PREPARE ***
*
* Read and preprocess an anomaly detection dataset.
* For optimal performance it is beneficial to do a custom
* preprocessing step that defines input domains.
*
* Load and split the dataset.
create_dict (GenParamDataset)
* #read_dict ('C:/Workspace/HALCON/StarWheel/SegmentationSyringeStarWheel.hdict', [], [], DLDataset)
* #set_dict_tuple (DLDataset, ImageDir, ImageDir)
set_dict_tuple (GenParamDataset, 'image_sub_dirs', ImageSubDirs)
read_dl_dataset_anomaly (ImageDir, AnomalyDir, [], [], GenParamDataset, DLDataset)
*
* Note: make sure that every split contains at least one 'ok' image.
* For a small number of images in the dataset you might need to increase
* the validation fraction.
split_dl_dataset (DLDataset, 50, 20, [])
*
* Load the anomaly detection model and set parameters.
* For documentation see set_dl_model_param () and get_dl_model_param ().
read_dl_model ('initial_dl_anomaly_large.hdl', DLModelHandle)
set_dl_model_param (DLModelHandle, 'image_width', ImageWidth)
set_dl_model_param (DLModelHandle, 'image_height', ImageHeight)
set_dl_model_param (DLModelHandle, 'complexity', Complexity)
* In this example, the training of the anomaly detection model is done on the CPU.
* query_available_compute_devices (DeviceIdentifier)
query_available_dl_devices (['runtime','id'], ['gpu',0], DLDevice)
* query_available_dl_devices (['type','runtime','id'], ['gpu','gpu',0], DLDevice)
set_dl_model_param (DLModelHandle, 'device', DLDevice)
*
* Set preprocessing parameters and preprocess.
create_dict (PreprocessSettings)
set_dict_tuple (PreprocessSettings, 'overwrite_files', true)
create_dl_preprocess_param ('anomaly_detection', ImageWidth, ImageHeight, 3, [], [], 'constant_values', 'full_domain', [], [], [], [], DLPreprocessParam)
* Ændret ovenstående (ImageNumChannels) fra 3 til 1
preprocess_dl_dataset (DLDataset, OutputDir, DLPreprocessParam, PreprocessSettings, DLDatasetFileName)
*
* Run a specific preprocessing for this example.
get_dict_tuple (DLDataset, 'samples', DatasetSamples)
read_dl_samples (DLDataset, [0:|DatasetSamples| - 1], DLSampleBatch)
for SampleIndex := 0 to |DLSampleBatch| - 1 by 1
get_dict_object (SampleImage, DLSampleBatch[SampleIndex], 'image')
*
rectangle1_domain (SampleImage, SampleImageReduced, 0, 0, ImageHeight, ImageWidth)
*
* Update the sample.
set_dict_object (SampleImageReduced, DLSampleBatch[SampleIndex], 'image')
endfor
write_dl_samples (DLDataset, [0:|DatasetSamples| - 1], DLSampleBatch, [], [])
*
* Visually inspect ten randomly selected preprocessed DLSamples.
create_dict (WindowDict)
for Index := 0 to 10 by 1
SampleIndex := int(rand(1) * |DatasetSamples|)
read_dl_samples (DLDataset, SampleIndex, DLSample)
dev_display_dl_data (DLSample, [], DLDataset, 'anomaly_ground_truth', [], WindowDict)
dev_disp_text ('Press Run (F5) to continue', 'window', 'bottom', 'right', 'black', [], [])
*
get_dict_tuple (WindowDict, 'anomaly_ground_truth', WindowHandles)
dev_set_window (WindowHandles[0])
dev_disp_text ('Preprocessed image', 'window', 'top', 'left', 'black', [], [])
*
stop ()
endfor
dev_close_window_dict (WindowDict)
*
stop ()
*
*
* *** 2.) TRAIN ***
*
* Create training parameters.
*
* Control whether the training progress is displayed (true/false).
EnableDisplay := true
*
* Set a threshold for the training error and a maximum number of training epochs.
* If the training error falls below this threshold, the training is finished.
* Otherwise the training continues until the maximum number of epochs is reached.
ErrorThreshold := 0.0000001
MaxNumEpochs := 75
*
* Set the domain ratio which controls the fraction of each image used for training.
* The training result might be improved by setting a greater value, but this also
* increases the training time.
DomainRatio := 0.9
*
* Regularization noise can make the training more robust. In case the training fails,
* setting a higher value might help.
RegularizationNoise := 0.25
*
create_dict (TrainParamAnomaly)
set_dict_tuple (TrainParamAnomaly, 'regularization_noise', RegularizationNoise)
set_dict_tuple (TrainParamAnomaly, 'error_threshold', ErrorThreshold)
set_dict_tuple (TrainParamAnomaly, 'domain_ratio', DomainRatio)
create_dl_train_param (DLModelHandle, MaxNumEpochs, [], EnableDisplay, 73, 'anomaly', TrainParamAnomaly, TrainParam)
*
* The training and thus the call of train_dl_model_anomaly_dataset ()
* is done using the following procedure. This may take some time.
train_dl_model (DLDataset, DLModelHandle, TrainParam, 0, TrainResults, TrainInfos, EvaluationInfos)
dev_disp_text ('Press Run (F5) to continue', 'window', 'bottom', 'right', 'black', [], [])
* read_dl_model ('C:/Workspace/HALCON/StarWheel/model_final_kks.hdl', DLModelHandle)
stop ()
*
dev_close_window ()
*
*
* *** 3.) EVALUATE ***
*
* Calculate classification and segmentation thresholds and perform evaluation.
*
* Set the factor used to calculate the anomaly score. See the documentation of
* get_dl_model_param for details.
* This parameter can help to improve the anomaly score for a better classification
* between 'ok' and 'nok' images. For example, in case the defects are small, a larger
* value might be suitable.
StandardDeviationFactor := 15.0
set_dl_model_param (DLModelHandle, 'standard_deviation_factor', StandardDeviationFactor)
*
* Estimate threshold values. They are used to determine whether a pixel or image
* is regarded as anomalous. The procedure compute_dl_anomaly_thresholds returns
* possible suggestions based on the dataset used. Depending on the application, manual
* fine-tuning may be beneficial.
create_dict (GenParamThreshold)
set_dict_tuple (GenParamThreshold, 'enable_display', 'true')
compute_dl_anomaly_thresholds (DLModelHandle, DLDataset, GenParamThreshold, AnomalySegmentationThreshold, AnomalyClassificationThresholds)
dev_disp_text ('Press Run (F5) to continue', 'window', 'bottom', 'right', 'black', [], [])
stop ()
*
dev_close_window ()
*
* Set generic evaluation parameters and evaluate the model on the test split.
create_dict (GenParamEvaluation)
set_dict_tuple (GenParamEvaluation, 'measures', 'all')
set_dict_tuple (GenParamEvaluation, 'anomaly_classification_thresholds', AnomalyClassificationThresholds)
evaluate_dl_model (DLDataset, DLModelHandle, 'split', 'test', GenParamEvaluation, EvaluationResult, EvalParams)
*
* Set display parameters.
create_dict (GenParamDisplay)
*
* Visualize the histogram over the anomaly scores and with it the classification
* thresholds used for the evaluation.
set_dict_tuple (GenParamDisplay, 'display_mode', ['score_histogram','score_legend'])
create_dict (WindowDict)
dev_display_anomaly_detection_evaluation (EvaluationResult, EvalParams, GenParamDisplay, WindowDict)
get_dict_tuple (WindowDict, 'window_score_histogram', Tuple1)
get_dict_tuple(WindowDict, 'window_score_legend',Tuple2)
dump_window_image (Image1, Tuple1)
dump_window_image (Image2, Tuple2)
write_image (Image1, 'png', 16711680, 'C:/Workplace/DL Training/L113 FST HALCON/Result/WindowScoreHistogram.png')
write_image (Image2, 'png', 16711680, 'C:/Workplace/DL Training/L113 FST HALCON/Result/WindowScoreLegend.png')
dev_disp_text ('Press Run (F5) to continue', 'window', 'bottom', 'right', 'black', 'box', 'true')
stop ()
*
dev_close_window_dict (WindowDict)
*
* Visualize evaluation results such as precision, recall, and confusion matrix
* for a given classification threshold.
set_dict_tuple (GenParamDisplay, 'display_mode', ['pie_charts_precision','pie_charts_recall','absolute_confusion_matrix'])
* Select evaluation results for one threshold by its index in
* AnomalyClassificationThresholds for display. We use the last
* one of the computed AnomalyClassificationThresholds. Note that
* you have to use the AnomalyClassificationThreshold which suits
* your application best.
ClassificationThresholdIndex := |AnomalyClassificationThresholds| - 1
set_dict_tuple (GenParamDisplay, 'classification_threshold_index', ClassificationThresholdIndex)
create_dict (WindowDict)
dev_display_anomaly_detection_evaluation (EvaluationResult, EvalParams, GenParamDisplay, WindowDict)
get_dict_tuple (WindowDict, 'window_pie_charts_precision', Tuple1)
get_dict_tuple(WindowDict, 'window_pie_charts_recall',Tuple2)
get_dict_tuple(WindowDict, 'window_absolute_confusion_matrix',Tuple3)
dump_window_image (Image1, Tuple1)
dump_window_image (Image2, Tuple2)
dump_window_image (Image3, Tuple3)
write_image (Image1, 'png', 16711680, 'C:/Workplace/DL Training/L113 FST HALCON/Result/WindowPieChartsPrecision.png')
write_image (Image2, 'png', 16711680, 'C:/Workplace/DL Training/L113 FST HALCON/Result/WindowPieChartsRecall.png')
write_image (Image3, 'png', 16711680, 'C:/Workplace/DL Training/L113 FST HALCON/Result/WindowAbsoluteConfusionMatrix.png')
dev_disp_text ('Press Run (F5) to continue', 'window', 'bottom', 'right', 'black', [], [])
stop ()
*
dev_close_window_dict (WindowDict)
*
* Store the trained model.
write_dl_model (DLModelHandle, 'model_final_fst.hdl')
*
*
* *** 4.) INFER ***
* ImageSubDirs := ['Good']
read_dl_model ('C:/Workplace/DL Training/HALCON/model_final_fst.hdl', DLModelHandle)
* To demonstrate the inference steps, we apply the
* trained model to some randomly chosen example images.
list_image_files (ImageDir + '/' + ImageSubDirs, 'default', 'recursive', ImageFiles)
tuple_shuffle (ImageFiles, ImageFilesShuffled)
*
* Choose thresholds for inference.
InferenceClassificationThreshold := AnomalyClassificationThresholds[ClassificationThresholdIndex]
InferenceSegmentationThreshold := AnomalySegmentationThreshold
* #InferenceClassificationThreshold := InferenceSegmentationThreshold+0.03
* #InferenceSegmentationThreshold := InferenceClassificationThreshold
* #InferenceClassificationThreshold := 0.80
* #InferenceSegmentationThreshold := 0.30
*
* Create dictionary with dataset parameters used for display.
create_dict (DLDatasetInfo)
set_dict_tuple (DLDatasetInfo, 'class_names', ['ok','nok'])
set_dict_tuple (DLDatasetInfo, 'class_ids', [0,1])
*
create_dict (WindowDict)
for IndexInference := 0 to min2(|ImageFilesShuffled|,737) - 1 by 1
read_image (Image, ImageFilesShuffled[IndexInference])
gen_dl_samples_from_images (Image, DLSample)
preprocess_dl_samples (DLSample, DLPreprocessParam)
*
* Use the same dataset specific preprocessing as for training.
for SampleIndex := 0 to |DLSample| - 1 by 1
get_dict_object (SampleImage, DLSample[SampleIndex], 'image')
*
rectangle1_domain (SampleImage, SampleImageReduced, 0, 0, ImageHeight, ImageWidth)
*
* Update the sample.
set_dict_object (SampleImageReduced, DLSample[SampleIndex], 'image')
endfor
*
apply_dl_model (DLModelHandle, DLSample, [], DLResult)
*
* Apply thresholds to classify regions and the entire image.
threshold_dl_anomaly_results (InferenceSegmentationThreshold, InferenceClassificationThreshold, DLResult)
*
* Display the inference result.
dev_display_dl_data (DLSample, DLResult, DLDatasetInfo, ['anomaly_result','anomaly_image'], [], WindowDict)
dev_disp_text ('Press F5 (continue)', 'window', 'bottom', 'center', 'black', [], [])
stop ()
endfor
*
dev_close_window_dict (WindowDict)```

Related

How to minimise a multivariate cost function in Julia with Optim?

I am currently stuck trying to utilize the Optim package in Julia in an attempt to minimize a cost function. The cost function is the cost function for an L2 regularised logistic regression. It is constructed as follows;
using Optim
function regularised_cost(X, y, θ, λ)
m = length(y)
# Sigmoid predictions
h = sigmoid(X * θ)
# left side of the cost function
positive_class_cost = ((-y)' * log.(h))
# right side of the cost function
negative_class_cost = ((1 .- y)' * log.(1 .- h))
# lambda effect
lambda_regularization = (λ/(2*m) * sum(θ[2 : end] .^ 2))
# Current batch cost
𝐉 = (1/m) * (positive_class_cost - negative_class_cost) + lambda_regularization
# Gradients for all the theta members with regularization except the constant
∇𝐉 = (1/m) * (X') * (h-y) + ((1/m) * (λ * θ))
∇𝐉[1] = (1/m) * (X[:, 1])' * (h-y) # Exclude the constant
return (𝐉, ∇𝐉)
end
I would like to use LBFGS algorithm as a solver to find the best weights that minimize this function based on my training examples and labels which are defined as:
opt_train = [ones(size(X_train_scaled, 1)) X_train_scaled] # added intercept
initial_theta = zeros(size(opt_train, 2))
Having read the documentation, here's my current implementation which is currently not working:
cost, gradient! = regularised_cost(opt_train, y_train, initial_theta, 0.01)
res = optimize(regularised_cost,
gradient!,
initial_theta,
LBFGS(),
Optim.Options(g_tol = 1e-12,
iterations = 1000,
store_trace = true,
show_trace = true))
How do I pass my training examples and labels along with the gradients so that the solver (LBFGS) can find me the best weights for theta?

You need to close over your train data and make a loss function that only takes the parameters as inputs.
As per the docs on dealing with constant parameterised
It should be so.wthing like:
loss_and_grad(theta) = regularised_cost(opt_train, y_train, theta, 0.01)
loss(theta) = first(loss_and_grad(theta))
res = optimize(loss, initial_theta)
I will leave it to you to see how to hook the gradient in.
A reminder though: don't use non-const globals.
They are slow, in particular the way they are used in the loss_and_grad function I wrote will be slow.
So you should declare opt_train and y_train as const.
Or make a function that takes them and returns a loss function etc

YOLO: How to change the frequency of mAP calculation in darknet code

I am training YOLOv2 with -map option to print mean Average Precision. I need to change the frequency at which the map is calculated. At this time it is calculated every 300 iterations which is too frequent for me. I want this to be computed every (say 2000 iterations). Is there a way to change the switch ot code?
I do see the following code in detector.c file which needs to be changed. Any input?
int calc_map_for_each = 4 * train_images_num / (net.batch * net.subdivisions); // calculate mAP for each 4 Epochs
calc_map_for_each = fmax(calc_map_for_each, 100);
int next_map_calc = iter_map + calc_map_for_each;
next_map_calc = fmax(next_map_calc, net.burn_in);
next_map_calc = fmax(next_map_calc, 1000);
if (calc_map) {
printf("\n (next mAP calculation at %d iterations) ", next_map_calc);
if (mean_average_precision > 0) printf("\n Last accuracy mAP#0.5 = %2.2f %% ", mean_average_precision * 100);
}

I assume you are using Alexey's repo because of the -map option. In detector.c (https://github.com/AlexeyAB/darknet/blob/8c970498a296ed129ffef7d872ccc25d42d1afda/src/detector.c#L223) you may change the following code :
calc_map_for_each = fmax(calc_map_for_each, 100);
to
calc_map_for_each = fmax(calc_map_for_each, 1000);

RuntimeError: libpng signaled error while visualizing cnn layers

I am visualizing layers of cnn with keras. The visualization is on mnist test image.The model summary is here
The code for visualization is as follows:
layer_names = []
for layer in model.layers[:12]:
layer_names.append(layer.name) # Names of the layers, so you can have them as part of your plot
images_per_row = 16
for layer_name, layer_activation in zip(layer_names, activations): # Displays the feature maps
n_features = layer_activation.shape[-1] # Number of features in the feature map
size = layer_activation.shape[1] #The feature map has shape (1, size, size, n_features).
n_cols = n_features // images_per_row # Tiles the activation channels in this matrix
display_grid = np.zeros((size * n_cols, images_per_row * size))
for col in range(n_cols): # Tiles each filter into a big horizontal grid
for row in range(images_per_row):
channel_image = layer_activation[0,
:, :,
col * images_per_row + row]
channel_image -= channel_image.mean() # Post-processes the feature to make it visually palatable
channel_image /= channel_image.std()
channel_image *= 64
channel_image += 128
channel_image = np.clip(channel_image, 0, 255).astype('uint8')
display_grid[col * size : (col + 1) * size, # Displays the grid
row * size : (row + 1) * size] = channel_image
scale = 1. / size
plt.figure(figsize=(scale * display_grid.shape[1],
scale * display_grid.shape[0]))
plt.title(layer_name)
plt.grid(False)
plt.imshow(display_grid, aspect='auto', cmap='viridis')
This code visualize output of first two layers and show image with filters. But with the third layer it throws the error as follows:
RuntimeError: libpng signaled error
<Figure size 1152x0 with 1 Axes>
I have tried to uninstall and reinstall matplotlib but still it is not working.

It’s a logic error:
<Figure size 1152x0 with 1 Axes>
implies that scale * display_grid.shape[0] == 0 which can only happen if you set n_cols to zero in this line:
n_cols = n_features // images_per_row
caused by n_features being < images_per_row/2.
There should be a nicer error in future versions of matplotlib.

Tensorflow, How can I compute backward pass for a given forward function

I want to construct an L2-norm layer in Caffe style (well, I actually want to use Tensorflow in a pycaffe layer, since using CUDA to write .cu files in Caffe is an onerous task.)
Forward pass:
- input(x): n-D array
- output(y): n-D array that has the same shape of input
- operation:
y = x / sqrt(sum(x^2,axis=(0,1))) # channel wise L2 normalization
class L2NormLayer:
def __init__(self):
self.eps = 1e-12
self.sess = tf.Session()
def forward(self, in_x):
self.x = tf.constant(in_x)
self.xp2 = tf.pow(self.x, 2)
self.sum_xp2 = tf.reduce_sum(self.xp2, axis=(0, 1))
self.sqrt_sum_xp2 = tf.sqrt(self.sum_xp2 + self.eps)
self.hat = tf.div(self.x, self.sqrt_sum_xp2)
return self.sess.run(self.hat)
def backward(self, dl):
# 'dl' is loss calculated at upper layer (chain rule)
# how do I calculate this gradient automatically using Tensorflow
# hand-craft backward version
loss = tf.constant(dl)
d_x1 = tf.div(loss, self.sqrt_sum_xp2)
d_sqrt_sum_xp2 = tf.div(-tf.reduce_sum(self.x * dl, axis=(0, 1)), (self.eps + tf.pow(self.sqrt_sum_xp2, 2)))
d_sum_xp2 = tf.div(d_sqrt_sum_xp2, (self.eps + 2 * tf.sqrt(self.sum_xp2)))
d_xp2 = tf.ones_like(self.xp2) * d_sum_xp2
d_x2 = 2 * self.x * d_xp2
d_x = d_x1 + d_x2
return self.sess.run(d_x)
As commented in the code, how can I calcualte the gradient of the forward pass function by using Tensorflow automatically?

I think your best strategy would be to use existing caffe layers to achieve your goal.
First, use "Reduction" layer to compute the sq. L2 norm of x:
layer {
name: "norm_x_sq"
type: "Reduction"
bottom: "x"
top: "norm_x_sq"
reduction_param { operation: SUMSQ axis: 1 }
}
Use "Power" layer to take the square root of the norm and compute its reciprocal:
layer {
name: "norm_x-1"
type: "Power"
bottom: "norm_x_sq"
top: "norm_x-1"
power_param { power: -0.5 }
}
Once you have the denominator, you need to "Tile" it back to the same shape as x:
layer {
name: "denom"
type: "Tile"
bottom: "norm_x-1"
top: "denom"
tile_param { axis:1 tiles: N } # here you'll have to manually put the target dimension N
}
Finally, use "Eltwise" layer to normalize x:
layer {
name: "x_norm"
type: "Eltwise"
bottom: "x"
bottom: "denom"
top: "x_norm"
eltwise_param { operation: PROD }
}
Some additional notes:
1. Dividing by the norm might be numerically unstable if the norm is very little. You might want to consider adding a tiny constant to "norm_x_sq" before taking the reciprocal of the square root. You can do that using existing layers as well.
2. This example showed how to normalize according to axis=1 dimension. Depending how your vectors are arranged in the blob, you might be able to use "Scale" layer for the division instead of tile+eltwise.
3. You might also find this thread useful.

How does Tensorflow Batch Normalization work?

I'm using tensorflow batch normalization in my deep neural network successfully. I'm doing it the following way:
if apply_bn:
with tf.variable_scope('bn'):
beta = tf.Variable(tf.constant(0.0, shape=[out_size]), name='beta', trainable=True)
gamma = tf.Variable(tf.constant(1.0, shape=[out_size]), name='gamma', trainable=True)
batch_mean, batch_var = tf.nn.moments(z, [0], name='moments')
ema = tf.train.ExponentialMovingAverage(decay=0.5)
def mean_var_with_update():
ema_apply_op = ema.apply([batch_mean, batch_var])
with tf.control_dependencies([ema_apply_op]):
return tf.identity(batch_mean), tf.identity(batch_var)
mean, var = tf.cond(self.phase_train,
mean_var_with_update,
lambda: (ema.average(batch_mean), ema.average(batch_var)))
self.z_prebn.append(z)
z = tf.nn.batch_normalization(z, mean, var, beta, gamma, 1e-3)
self.z.append(z)
self.bn.append((mean, var, beta, gamma))
And it works fine both for training and testing phases.
However I encounter problems when I try to use the computed neural network parameters in my another project, where I need to compute all the matrix multiplications and stuff by myself. The problem is that I can't reproduce the behavior of the tf.nn.batch_normalization function:
feed_dict = {
self.tf_x: np.array([range(self.x_cnt)]) / 100,
self.keep_prob: 1,
self.phase_train: False
}
for i in range(len(self.z)):
# print 0 layer's 1 value of arrays
print(self.sess.run([
self.z_prebn[i][0][1], # before bn
self.bn[i][0][1], # mean
self.bn[i][1][1], # var
self.bn[i][2][1], # offset
self.bn[i][3][1], # scale
self.z[i][0][1], # after bn
], feed_dict=feed_dict))
# prints
# [-0.077417567, -0.089603029, 0.000436493, -0.016652612, 1.0055743, 0.30664611]
According to the formula on the page https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/nn/batch_normalization:
bn = scale * (x - mean) / (sqrt(var) + 1e-3) + offset
But as we can see,
1.0055743 * (-0.077417567 - -0.089603029)/(0.000436493^0.5 + 1e-3) + -0.016652612
= 0.543057
Which differs from the value 0.30664611, computed by Tensorflow itself.
So what am I doing wrong here and why I can't just calculate batch normalized value myself?
Thanks in advance!

The formula used is slightly different from:
bn = scale * (x - mean) / (sqrt(var) + 1e-3) + offset
It should be:
bn = scale * (x - mean) / (sqrt(var + 1e-3)) + offset
The variance_epsilon variable is supposed to scale with the variance, not with sigma, which is the square-root of variance.
After the correction, the formula yields the correct value:
1.0055743 * (-0.077417567 - -0.089603029)/((0.000436493 + 1e-3)**0.5) + -0.016652612
# 0.30664642276945747