Related
I want to use IMU (accelerometer and gyroscope) readings to compute the attitude via Neural Network. The input will be input_shape = (time steps, 6) and the output is in the quaternion form output_shape = (time steps,4).
Based on mathematical calculations the error between reference quaternion and the predicted quaternion is
y_pred[i,]=w0,x0,y0,z0
y_true[i,]=w1,x1,y1,z1
w = w0*w1 - x0*x1 - y0*y1 - z0*z1
x = w0*x1 + x0*w1 + y0*z1 - z0*y1
y = w0*y1 - x0*z1 + y0*w1 + z0*z1
z = w0*z1 + x0*y1 - y0*x1 + z0*w1
error_quaternion = [w, x, y, z]
To minimize the error, the scaler part of error quaternion (w) must be minimize. (please just ignore the math)
So to reach the best predication the w must be minimized (w is the shortest angle between the predicted and reference attitude) -
Ref = {Markley, F. Landis, and John L. Crassidis. Fundamentals of
spacecraft attitude determination and control. Vol. 1286. New York,
NY, USA:: Springer New York, 2014.}
I write this loss function
def LossQuat2(y_true, y_pred):
a, b = y_true.get_shape()
error = []
for i in range(a):
w0,x0,y0,z0 = tf.unstack(y_pred[i,])
w1,x1,y1,z1 = tf.unstack(y_true[i,])
x1 = -x1
y1 = -y1
z1 = -z1
w = w0*w1 - x0*x1 - y0*y1 - z0*z1
error.append(2*tf.math.acos(K.clip(tf.math.sqrt(w*w), -1., 1.)))
return tf.reduce_mean(error)
To validate it really calculate the error I try this code and the error calculated precisely
w0,x0,y0,z0 = y_pred[i,]
w1,x1,y1,z1 = y_true[i,]
x1 = -x1
y1 = -y1
z1 = -z1
w = w0*w1 - x0*x1 - y0*y1 - z0*z1
error = 2*math.acos(K.clip(np.sqrt(w*w), -1., 1.))
But after use this loss function to train the model, the output error is to much bigger than the MSE loss function. Also, it is too slow than MSE.
Why this loss function won't reduce the error correctly while mathematically it is correct?
How could I decrease the execution time of the loss function?
Is it true to use the for loop function? Is there any way to remove the for loop?
UPDATE:
Mathematics
Quaternion:
Quaternion is an attitude representation with 4 elements q=[w x y z]
w is the scalar part or real part
x y z are the vector part or imaginary part
Also, the quaternion could be written as:
q = [cos(theta/2) e*sin(theta/2)] , e is a unit vector (e=[i j k]
I intend to estimate the quaternion by the neural network
Quaternion Inverse
quaternion inverse or quaternion conjugate can be calculated by:
quaternion = [w x y z]
inverse(quaternion) = [w -x -y -z]
Quaternion Multiplication
To find the difference between the estimated attitude and the true(reference) attitude, estimated attitude(NN output) must be multiply by the quaternion reference using quaternion multiplication.
Quaternion multiplication:
q_m = q1 * inverse(q2)
or
q_m = q2 * inverse(q1)
both of them is the same.
If
q1=w0,x0,y0,z0
q2=w1,x1,y1,z1
Then q_m = [w x y z] and it could be calculated by:
w = w0*w1 - x0*x1 - y0*y1 - z0*z1
x = w0*x1 + x0*w1 + y0*z1 - z0*y1
y = w0*y1 - x0*z1 + y0*w1 + z0*z1
z = w0*z1 + x0*y1 - y0*x1 + z0*w1
The shortest angle between q1 and q2 is theta:
Theta = 2*acos(sqrt(w*w))
What I need is to write a loss function to minimize theta, if theta = 0, w will be equal to 1, so, the optimal q_m is:
q_m=[1 0 0 0]
Many thanks to david-harris #david-harris:
def loss(y_true, y_pred):
z = y_true * y_pred * tf.constant([[1., -1., -1., -1.]])
wtot = tf.reduce_sum(z, axis=1)
return tf.reduce_mean(2*tf.math.acos(tf.math.sqrt(wtot*wtot)))
It is much faster but it seems that but it reduce all value of the quaternion, so it does not work correctly.
**
So sorry for lots of math.
**
UPDATE 2
Based on David's suggested code, I wrote this:
def loss(y_true, y_pred):
z = y_true * (y_pred * tf.constant([1., -1., -1., -1.000000000]))
wtot = tf.reduce_sum(z,1)
return tf.reduce_mean(2*tf.math.acos(K.clip(tf.math.sqrt(wtot*wtot), -1.,1.)))
This code reduced the loss but MSE grows exponentially. I understand that this code wont optimized for MSE directly, but due to mathematics the MSE also must decreases.
After 10 epochs
loss: 0.0124 - mse: 227.4045
One of outputs based on the custom loss
Orange = Reference
Blue = Estimated by NN
One of outputs based on the MSE loss function
You should be able to vectorize (and speed up) the calculation using this approach. (I'm not sure I have all the signs right - don't understand why your lines 'x1 = -x1' are there. And I've temporarily remove the 'clip' part, you'll need to put that back if you want it)
def loss(y_true, y_pred):
z = y_true * y_pred * tf.constant([[1., -1., -1., -1.]])
wtot = tf.reduce_sum(z, axis=1)
return tf.reduce_mean(2*tf.math.acos(tf.math.sqrt(wtot*wtot)))
Can't see what the error in the maths is, sorry
I have a function f that is internally using some tf.while_loops and tf.gradients to compute the value y = f(x). Something like this
def f( x ):
...
def body( g, x ):
# Compute the gradient here
grad = tf.gradients( g, x )[0]
...
return ...
return tf.while_loop( cond, body, parallel_iterations=1 )
There are a few hundred lines of code. But I believe that those are the important points...
Now when I evaluate f(x), I get exactly the value I expect ..
y = known output of f(x)
with tf.Session() as sess:
fx = f(x)
print("Error = ", y - sess.run(fx, feed_dict)) # Prints 0
However, when I try to evaluate the gradient of f(x) with respect to x, that is,
grads = tf.gradients( fx, x )[0]
I get the error
AssertionError: gradients list should have been aggregated by now.
Here is the full trace:
File "C:/Dropbox/bob/tester.py", line 174, in <module>
grads = tf.gradients(y, x)[0]
File "C:\Anaconda36\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 649, in gradients
return [_GetGrad(grads, x) for x in xs]
File "C:\Anaconda36\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 649, in <listcomp>
return [_GetGrad(grads, x) for x in xs]
File "C:\Anaconda36\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 727, in _GetGrad
"gradients list should have been aggregated by now.")
AssertionError: gradients list should have been aggregated by now.
Could somebody please outline likely causes for this error? I have no idea where to even start looking for the issue...
Some observations:
Note that I have set the parallel iterations for the while loop to 1. This
should mean that there is no errors due to reading and writing from multiple threads.
If I discard the while loop, and just have f return body(), then the code runs:
# The following does not crash, but we removed the while_loop, so the output is incorrect
def f( x ):
...
def body( g, x ):
# Compute the gradient here
grad = tf.gradients( g, x )[0]
...
return ...
return body(...)
Obviously, the output is incorrect, but at least the gradients are computed.
I came across a similar issue. Some patterns I noted:
If the x used in tf.gradients was used in a manner that required dimension broadcasting in body, I got this error. If I changed it to one that didn't require broadcasting, tf.gradients returned [None]. I didn't test this extensively, so this pattern may not be consistent across all examples.
Both cases (returning [None] and raising this assertion error) can be resolved by differentiating tf.identity(y) rather than just y: grads = tf.gradients(tf.identity(y), xs) I have absolutely no idea why this works.
Let x and y be vectors of length N, and z is a function z = f(x,y). In Tensorflow v1.0.0, tf.hessians(z,x) and tf.hessians(z,y) both returns an N by N matrix, which is what I expected.
However, when I concatenate the x and y into a vector p of size 2*N using tf.concat, and run tf.hessian(z, p), it returns error "ValueError: None values not supported."
I understand this is because in the computation graph x,y ->z and x,y -> p, so there is no gradient between p and z. To circumvent the problem, I can create p first, slice it into x and y, but I will have to change a ton of my code. Is there a more elegant way?
related question: Slice of a variable returns gradient None
import tensorflow as tf
import numpy as np
N = 2
A = tf.Variable(np.random.rand(N,N).astype(np.float32))
B = tf.Variable(np.random.rand(N,N).astype(np.float32))
x = tf.Variable(tf.random_normal([N]) )
y = tf.Variable(tf.random_normal([N]) )
#reshape to N by 1
x_1 = tf.reshape(x,[N,1])
y_1 = tf.reshape(y,[N,1])
#concat x and y to form a vector with length of 2*N
p = tf.concat([x,y],axis = 0)
#define the function
z = 0.5*tf.matmul(tf.matmul(tf.transpose(x_1), A), x_1) + 0.5*tf.matmul(tf.matmul(tf.transpose(y_1), B), y_1) + 100
#works , hx and hy are both N by N matrix
hx = tf.hessians(z,x)
hy = tf.hessians(z,y)
#this gives error "ValueError: None values not supported."
#expecting a matrix of size 2*N by 2*N
hp = tf.hessians(z,p)
Compute the hessian by its definition.
gxy = tf.gradients(z, [x, y])
gp = tf.concat([gxy[0], gxy[1]], axis=0)
hp = []
for i in range(2*N):
hp.append(tf.gradients(gp[i], [x, y]))
Because tf.gradients computes the sum of (dy/dx), so when computing the second partial derivative, one should slice the vector into scalars and then compute the gradient. Tested on tf1.0 and python2.
I have a loss function I would like to try and minimize:
def lossfunction(X,b,lambs):
B = b.reshape(X.shape)
penalty = np.linalg.norm(B, axis = 1)**(0.5)
return np.linalg.norm(np.dot(X,B)-X) + lambs*penalty.sum()
Gradient descent, or similar methods, might be useful. I can't calculate the gradient of this function analytically, so I am wondering how I can numerically calculate the gradient for this loss function in order to implement a descent method.
Numpy has a gradient function, but it requires me to pass a scalar field at pre determined points.
You could try scipy.optimize.minimize
For your case a sample call would be:
import scipy.optimize.minimize
scipy.optimize.minimize(lossfunction, args=(b, lambs), method='Nelder-mead')
You could estimate the derivative numerically by a central difference:
def derivative(fun, X, b, lambs, h):
return (fun(X + 0.5*h,b,lambs) - fun(X - 0.5*h,b,lambs))/h
And use it like this:
# assign values to X, b, lambs
# set the value of h
h = 0.001
print derivative(lossfunction, X, b, lambs, h)
The code above is valid for dimX = 1, some modifications are needed to account for multidimensional vector X:
def gradient(fun, X, b, lambs, h):
res = []
for i in range (0,len(X)):
t1 = list(X)
t1[i] = t1[i] + 0.5*h
t2 = list(X)
t2[i] = t2[i] - 0.5*h
res = res + [(fun(t1,b,lambs) - fun(t2,b,lambs))/h]
return res
Forgive the naivity of the code, I barely know how to write some python :-)
Given a 2-dimensional tensor t, what's the fastest way to compute a tensor h where
h[i, :] = tf.histogram_fixed_width(t[i, :], vals, nbins)
I.e. where tf.histogram_fixed_width is called per row of the input tensor t?
It seems that tf.histogram_fixed_width is missing an axis parameter that works like, e.g., tf.reduce_sum's axis parameter.
tf.histogram_fixed_width works on the entire tensor indeed. You have to loop through the rows explicitly to compute the per-row histograms. Here is a complete working example using TensorFlow's tf.while_loop construct :
import tensorflow as tf
t = tf.random_uniform([2, 2])
i = 0
hist = tf.constant(0, shape=[0, 5], dtype=tf.int32)
def loop_body(i, hist):
h = tf.histogram_fixed_width(t[i, :], [0.0, 1.0], nbins=5)
return i+1, tf.concat_v2([hist, tf.expand_dims(h, 0)], axis=0)
i, hist = tf.while_loop(
lambda i, _: i < 2, loop_body, [i, hist],
shape_invariants=[tf.TensorShape([]), tf.TensorShape([None, 5])])
sess = tf.InteractiveSession()
print(hist.eval())
Inspired by keveman's answer and because the number of rows of t is fixed and rather small, I chose to use a combination of tf.gather to split rows and tf.pack to join rows. It looks simple and works, will see if it is efficient...
t_histo_rows = [
tf.histogram_fixed_width(
tf.gather(t, [row]),
vals, nbins)
for row in range(t_num_rows)]
t_histo = tf.pack(t_histo_rows, axis=0)
I would like to propose another implementation.
This implementation can also handle multi axes and unknown dimensions (batching).
def histogram(tensor, nbins=10, axis=None):
value_range = [tf.reduce_min(tensor), tf.reduce_max(tensor)]
if axis is None:
return tf.histogram_fixed_width(tensor, value_range, nbins=nbins)
else:
if not hasattr(axis, "__len__"):
axis = [axis]
other_axis = [x for x in range(0, len(tensor.shape)) if x not in axis]
swap = tf.transpose(tensor, [*other_axis, *axis])
flat = tf.reshape(swap, [-1, *np.take(tensor.shape.as_list(), axis)])
count = tf.map_fn(lambda x: tf.histogram_fixed_width(x, value_range, nbins=nbins), flat, dtype=(tf.int32))
return tf.reshape(count, [*np.take([-1 if a is None else a for a in tensor.shape.as_list()], other_axis), nbins])
The only slow part here is tf.map_fn but it is still faster than the other solutions mentioned.
If someone knows a even faster implementation please comment since this operation is still very expensive.
answers above is still slow running in GPU. Here i give an another option, which is faster(at least in my running envirment), but it is limited to 0~1 (you can normalize the value first). the train_equal_mask_nbin can be defined once in advance
def histogram_v3_nomask(tensor, nbins, row_num, col_num):
#init mask
equal_mask_list = []
for i in range(nbins):
equal_mask_list.append(tf.ones([row_num, col_num], dtype=tf.int32) * i)
#[nbins, row, col]
#[0, row, col] is tensor of shape [row, col] with all value 0
#[1, row, col] is tensor of shape [row, col] with all value 1
#....
train_equal_mask_nbin = tf.stack(equal_mask_list, axis=0)
#[inst, doc_len] float to int(equaly seg float in bins)
int_input = tf.cast(tensor * (nbins), dtype=tf.int32)
#input [row,col] -> copy N times, [nbins, row_num, col_num]
int_input_nbin_copy = tf.reshape(tf.tile(int_input, [nbins, 1]), [nbins, row_num, col_num])
#calculate histogram
histogram = tf.transpose(tf.count_nonzero(tf.equal(train_equal_mask_nbin, int_input_nbin_copy), axis=2))
return histogram
With the advent of tf.math.bincount, I believe the problem has become much simpler.
Something like this should work:
def hist_fixed_width(x,st,en,nbins):
x=(x-st)/(en-st)
x=tf.cast(x*nbins,dtype=tf.int32)
x=tf.clip_by_value(x,0,nbins-1)
return tf.math.bincount(x,minlength=nbins,axis=-1)