I want to create a neural network that takes a categorical tuple as input and passes its one-hot-encoded value to its layers.
For example, assuming that the tuple value limits were (2, 2, 3), I need a preprocessing layer that transforms the following three-dimensional list of values:
[
(1, 0, 0),
(0, 0, 1),
(1, 1, 2),
]
Into the following one-dimensional tensor:
[
0.0, 1.0, 0.0, 0.0, 0.0, 0.0,
1.0, 0.0, 0.0, 0.0, 0.0, 1.0,
]
Does such a function exist?
I assume that this custom layer operates on a batch having varied number of tuples per sample. For example, an input batch may be
[[(1, 0, 0), (0, 0, 1), (1, 1, 2)],
[(1, 0, 0), (1, 1, 2)]]
and the desired output tensors would be
[[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0],
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0]]
Since the samples can be of uneven sizes, the batch needs to be converted to tf.RaggedTensor (instead of normal Tensor) before being fed to the layer. However, the following solution works with both tf.Tensor and tf.RaggedTensor as input.
class FillOneLayer(tf.keras.layers.Layer):
def __init__(self, shape, *args, **kwargs):
super().__init__(*args, **kwargs)
self.shape = shape
def call(self, inputs, training=None):
num_samples = inputs.nrows() if isinstance(inputs, tf.RaggedTensor) else tf.shape(inputs)[0]
num_samples = tf.cast(num_samples, tf.int32)
ret = tf.TensorArray(tf.float32, size=num_samples, dynamic_size=False)
for i in range(num_samples):
sample = inputs[i]
sample = sample.to_tensor() if isinstance(sample, tf.RaggedTensor) else sample
updates_shape = tf.shape(sample)[:-1]
tmp = tf.zeros(self.shape)
tmp = tf.tensor_scatter_nd_update(tmp, sample, tf.ones(updates_shape))
ret = ret.write(i, tf.reshape(tmp, (-1,)))
return ret.stack()
Output for normal input tensor
>>> a = tf.constant([[(1, 0, 0), (0, 0, 1), (1, 1, 2)],
[(1, 0, 0), (0, 0, 1), (1, 0, 2)]])
>>> FillOneLayer((2,2,3))(a)
<tf.Tensor: shape=(2, 12), dtype=float32, numpy=
array([[0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1.],
[0., 1., 0., 0., 0., 0., 1., 0., 1., 0., 0., 0.]], dtype=float32)>
Output for ragged tensor
>>> a = tf.ragged.constant([[(1, 0, 0), (0, 0, 1), (1, 1, 2)],
[(1, 0, 0), (0, 0, 1)]])
>>> FillOneLayer((2,2,3))(a)
<tf.Tensor: shape=(2, 12), dtype=float32, numpy=
array([[0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1.],
[0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.]], dtype=float32)>
The solution also works when you decorate call() with tf.function, which is usually what happens when you call fit on a model whom this layer is a member of. In that case, to avoid graph retracing, you should ensure that all batches are of the same type, i.e., either all RaggedTensor or all Tensor.
Related
ffnn = Sequential([
Flatten(input_shape=X_train.shape[1:]),
Dense(512, activation='relu'),
Dropout(0.2),
Dense(512, activation='relu'),
Dropout(0.2),
Dense(10, activation='softmax')
])
ffnn_history = ffnn.fit(X_train,
y_train,
batch_size=batch_size,
epochs=epochs,
validation_split=0.2,
callbacks=[checkpointer, early_stopping],
verbose=1,
shuffle=True)
ffnn_accuracy = ffnn.evaluate(X_test, y_test, verbose=0)[1]
These codes are from https://github.com/stefan-jansen/machine-learning-for-trading/blob/main/18_convolutional_neural_nets/02_digit_classification_with_lenet5.ipynb.
I understand this network and how softmax function works. My question is, the output layer has 10 nodes. The output should be a vector of length 10 (the sum of the vector is 1). How does it matches the label y where y is an integer in the training and evaluating process (shouldn't it transform the output vector to the corresponding integer first)?
Does tensorflow automatically interpret the length-10 output vector to the corresponding integer or what?
In your case the labels are one-hot encoded by the loss function sparse_categorical_crossentropy():
>>> y_true = [1, 2]
>>> y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]
>>> tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred).numpy()
array([0.05129344, 2.3025851 ], dtype=float32)
The output softmax(x) can be interpreted as a probability distribution (Σ softmax(x) = 1.0). So e.g. argmax(softmax(x)) = id_maxprob is going to return you the index of the most probable class.
Hence, your target vector for your neural network is going to be 10-dimensional such that each integer [0, 1, .., 8, 9] corresponds to one node of the softmax-output.
With that being said, the target vector you're trying to predict is simply going to be one-hot encoded:
[1, 0, 0, 0, 0, 0, 0, 0, 0, 0] # == 0
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0] # == 1
..
[0, 0, 0, 0, 0, 0, 0, 0, 0, 1] # == 9
In other words: If you have a batch of images and feed it to your network, the output will be (n, num_classes) (here num_classes is 10) and it is you who is going to do the final interpretation of the output e.g. by using np.argmax in order to get your final predictions.
predictions = model(images)
predicted_ids = np.argmax(predictions, axis=1)
# Print each index == predicted integer
print(predicted_ids)
Also, note the following example:
>>> tf.one_hot([1, 2, 9], depth=10)
<tf.Tensor: shape=(3, 10), dtype=float32, numpy=
array([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)>
The matrix multiplication values vary when tensorflow is run in eager mode vs graph mode
The code flow is different for eager and non-eager executions within tensorflow. But the values must match ideally, which is not.
Eager execution:
import tensorflow as tf
from tensorflow.python.ops import gen_math_ops
import numpy as np
tf.enable_eager_execution()
dZ = np.array([[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, -0.9, 0.1, 0.1, 0.1]])
FC_W = np.array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])
import pdb
pdb.set_trace()
a = gen_math_ops.mat_mul(dZ, FC_W, False, True)
print(a)
Output of eager execution: [[-2.77555756e-17 -2.77555756e-17 -2.77555756e-17]
Graph execution:
import tensorflow as tf
from tensorflow.python.ops import gen_math_ops
import numpy as np
dZ = np.array([[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, -0.9, 0.1, 0.1, 0.1]])
FC_W = np.array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])
a = gen_math_ops.mat_mul(dZ, FC_W, False, True)
sess = tf.InteractiveSession()
print(str(sess.run(a)))
Output of graph execution: [[-5.55111512e-17 -5.55111512e-17 -5.55111512e-17]]
Isn't this too much difference in output, between the two modes, for a simple matrix multiplication?
(Although it is e-17)
The resulting differences are due to calculation accuracy and different ordering or grouping of operations. This leads to rounding effects.
I reproduced your findings with a C# program:
double[] a = new double[] { 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, -0.9, 0.1, 0.1, 0.1 };
double[] b = new double[] { 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0 };
double sum = 0;
for (int i = 0; i < a.Length; i++)
{
sum += a[i] * b[i];
}
Console.WriteLine($"{sum}");
sum = (a[0]*b[0] + a[1]*b[1])
+ (a[2]*b[2] + a[3]*b[3])
+ (a[4]*b[4] + a[5]*b[5])
+ (a[6]*b[6] + a[7]*b[7])
+ (a[8]*b[8] + a[9]*b[9]);
Console.WriteLine($"{sum}");
// output:
// -2.77555756156289E-17
// 5.55111512312578E-17
By the way:
Microsoft Excel365 delivers the proper zero as result without visible rounding.
I have a list of label names which I enuemrated and created a dictionary:
my_list = [b'airplane',
b'automobile',
b'bird',
b'cat',
b'deer',
b'dog',
b'frog',
b'horse',
b'ship',
b'truck']
label_dict =dict(enumerate(my_list))
{0: b'airplane',
1: b'automobile',
2: b'bird',
3: b'cat',
4: b'deer',
5: b'dog',
6: b'frog',
7: b'horse',
8: b'ship',
9: b'truck'}
Now I'm trying to cleaning map/apply the dict value to my target which is in an one-hot-encoded form.
y_test[0]
array([ 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.])
y_test[0].map(label_dict) should return:
'cat'
I was playing around with
(lambda key,value: value for y_test[0] == 1)
but couldn't come up with any concrete
Thank you.
Since we are working with one-hot encoded array, argmax could be used to get the index for one off 1 for each row. Thus, using the list as input -
[my_list[i] for i in y_test.argmax(1)]
Or with np.take to have array output -
np.take(my_list,y_test.argmax(1))
To work with dict and assuming sequential keys as 0,1,.., we could have -
np.take(label_dict.values(),y_test.argmax(1))
If the keys are not essentially in sequence but sorted -
np.take(label_dict.values(), np.searchsorted(label_dict.keys(),y_test.argmax(1)))
Sample run -
In [79]: my_list
Out[79]:
['airplane',
'automobile',
'bird',
'cat',
'deer',
'dog',
'frog',
'horse',
'ship',
'truck']
In [80]: y_test
Out[80]:
array([[ 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[ 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.]])
In [81]: [my_list[i] for i in y_test.argmax(1)]
Out[81]: ['cat', 'automobile', 'ship']
In [82]: np.take(my_list,y_test.argmax(1))
Out[82]:
array(['cat', 'automobile', 'ship'],
dtype='|S10')
we can use dot product to reverse one-hot encoding, if it really is ONE-hot.
Let's start with factorizing your list
f, u = pd.factorize(my_list)
now if you have an array you'd like to get back your strings with
a = np.array([0, 0, 0, 1, 0, 0, 0, 0, 0, 0])
Then use dot
a.dot(u)
'cat'
Now assume
y_test = np.array([
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 1, 0]
])
Then
y_test.dot(u)
array(['cat', 'automobile', 'ship'], dtype=object)
If it isn't one-hot but instead multi-hot, you could join with commas
y_test = np.array([
[0, 0, 0, 1, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 0, 1, 0, 0, 0, 0, 0, 1, 0]
])
[', '.join(u[y.astype(bool)]) for y in y_test]
['cat', 'automobile, truck', 'bird, ship']
I am creating a neural network in tensorflow and I have created the placeholders like this:
input_tensor = tf.placeholder(tf.float32, shape = (None,n_input), name = "input_tensor")
output_tensor = tf.placeholder(tf.float32, shape = (None,n_classes), name = "output_tensor")
During the training process, I was getting the following error:
Traceback (most recent call last):
File "try.py", line 150, in <module>
sess.run(optimizer, feed_dict={X: x_train[i: i + 1], Y: y_train[i: i + 1]})
TypeError: unhashable type: 'numpy.ndarray'
I identified that is because of the different datatypes of my x_train and y_train to the datatypes of the placeholders.
My x_train looks somewhat like this:
array([[array([[ 1., 0., 0.],
[ 0., 1., 0.]])],
[array([[ 0., 1., 0.],
[ 1., 0., 0.]])],
[array([[ 0., 0., 1.],
[ 0., 1., 0.]])]], dtype=object)
It was initially a dataframe like this:
0 [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]
1 [[0.0, 1.0, 0.0], [1.0, 0.0, 0.0]]
2 [[0.0, 0.0, 1.0], [0.0, 1.0, 0.0]]
I did x_train = train_x.values to get the numpy array
And y_train looks this:
array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])
x_train has dtype object and y_train has dtype float64.
What I want to know is that how I can change the datatypes of my training data so that it can work well with the tensorflow placeholders. Or please suggest if I am missing something.
It is little hard to guess what shape you want your data to be, but I am guessing one of the two combinations which you might be looking for. I will also try to simulate your data in Pandas dataframe.
df = pd.DataFrame([[[[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]],
[[[0.0, 1.0, 0.0], [1.0, 0.0, 0.0]]],
[[[0.0, 0.0, 1.0], [0.0, 1.0, 0.0]]]], columns = ['Mydata'])
print(df)
x = df.Mydata.values
print(x.shape)
print(x)
print(x.dtype)
Output:
Mydata
0 [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]
1 [[0.0, 1.0, 0.0], [1.0, 0.0, 0.0]]
2 [[0.0, 0.0, 1.0], [0.0, 1.0, 0.0]]
(3,)
[list([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]])
list([[0.0, 1.0, 0.0], [1.0, 0.0, 0.0]])
list([[0.0, 0.0, 1.0], [0.0, 1.0, 0.0]])]
object
Combination 1
y = [item for sub_list in x for item in sub_list]
y = np.array(y, dtype = np.float32)
print(y.dtype, y.shape)
print(y)
Output:
float32 (6, 3)
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 1. 0.]
[ 1. 0. 0.]
[ 0. 0. 1.]
[ 0. 1. 0.]]
Combination 2
y = [sub_list for sub_list in x]
y = np.array(y, dtype = np.float32)
print(y.dtype, y.shape)
print(y)
Output:
float32 (3, 2, 3)
[[[ 1. 0. 0.]
[ 0. 1. 0.]]
[[ 0. 1. 0.]
[ 1. 0. 0.]]
[[ 0. 0. 1.]
[ 0. 1. 0.]]]
Your x_train is a nested object containing arrays, so you have to unpack it and reshape it. Here's a general purpose hack:
def unpack(a, aggregate=[]):
for x in a:
if type(x) is float:
aggregate.append(x)
else:
unpack(x, aggregate=aggregate)
return np.array(aggregate)
x_train = unpack(x_train.values).reshape(x_train.shape[0],-1)
Once you've got a dense array (y_train is already dense), you can use a function like the following:
def cast(placeholder, array):
dtype = placeholder.dtype.as_numpy_dtype
return array.astype(dtype)
x_train, y_train = cast(X,x_train), cast(Y,y_train)
I used tf.nn.top_k()function from tensorflow to use the model's softmax probabilities to visualize the certainty of its predictions with 5 new images and with k=5. I have an output as follows which I am not sure how to exactly interpret. Could anyone explain the output please.
TopKV2(values=array([[ 1., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0.]], dtype=float32), indices=array([[13, 0, 1, 2, 3],
[13, 0, 1, 2, 3],
[13, 0, 1, 2, 3],
[26, 0, 1, 2, 3],
[13, 0, 1, 2, 3]], dtype=int32))
From the documentation, it returns two tensors: the first with the top K value and the second with the indices of these values in the original tensor.
So for your data what I see is that the original tensor is always one-hot (i.e. has a single 1.0 entry per row and is 0 everywhere else).