I'm running TF 2.7 (I need it in that version). I'm trying to use the Keras.losses.CosineSimilarity loss function to train a Keras model (Stellargraph GCN model). But everytime a strange error stops the training process. I can confirm that it is related to using CosineSimilarity.
The error:=
TypeError: Expected int64 passed to parameter 'y' of op 'Maximum', got 1e-12 of type 'float' instead. Error: Expected int64, but got 1e-12 of type 'float'.
Here is the code :=
from tensorflow.keras import layers, optimizers, losses, metrics, Model
from tensorflow.keras.metrics import Precision
from tensorflow.keras.callbacks import EarlyStopping
model = Model(inputs=x_inp, outputs=predictions)
similarity = losses.CosineSimilarity()
model.compile(
optimizer=optimizers.Adam(learning_rate = 0.001),
loss = similarity,
metrics= [Precision],
)
es_callback = EarlyStopping(monitor="val_F1", patience=50, restore_best_weights=True)
history = model.fit(
train_gen,
epochs=500,
validation_data=val_gen,
verbose=2,
shuffle=False,
callbacks=[es_callback],
)
I can confirm that all of my training & validation data are of the integer type.
It could be a confusing error message and you should actually cast your data to float32.
See here for an example of similar error.
In your case I think CosineSimilarity actually calculates the addition of the product of the norms (that is int32 here because your data are int32) and an epsilon (float32 seems to be equal to 1e-12) for denominator. It expects that epsilon to be a int32 to proceed the addition and raises this error. However in you case what you need is to change the data type to float32 instead.
Related
I would like to use TFP to write a neural network where the output are the probabilities of a categorical variable with 3 classes, and train it using the negative log-likelihood.
As I'm moving my first steps with TF and TFP, I started with a toy model where the input layer has only 1 unit receiving a null input, and the output layer has 3 units with softmax activation function. The idea is that the biases should learn (up to an additive constant) the log of the probabilities.
Here below is my code, true_p are the true parameters I use to generate the data and I would like to learn, while learned_p is what I get from the NN.
import numpy as np
import tensorflow as tf
from tensorflow import keras
from functions import nll
from tensorflow.keras.optimizers import SGD
import tensorflow.keras.layers as layers
import tensorflow_probability as tfp
tfd = tfp.distributions
# params
true_p = np.array([0.1, 0.7, 0.2])
n_train = 1000
# training data
x_train = np.array(np.zeros(n_train)).reshape((n_train,))
y_train = np.array(np.random.choice(len(true_p), size=n_train, p=true_p)).reshape((n_train,))
# model
input_layer = layers.Input(shape=(1,))
p_layer = layers.Dense(len(true_p), activation=tf.nn.softmax)(input_layer)
p_y = tfp.layers.DistributionLambda(tfd.Categorical)(p_layer)
model_p = keras.models.Model(inputs=input_layer, outputs=p_y)
model_p.compile(SGD(), loss=nll)
# training
hist_p = model_p.fit(x=x_train, y=y_train, batch_size=100, epochs=3000, verbose=0)
# check result
learned_p = np.round(model_p.layers[1].call(tf.constant([0], shape=(1, 1))).numpy(), 3)
learned_p
With this setup, I get the result:
>>> learned_p
array([[0.005, 0.989, 0.006]], dtype=float32)
I over-estimate the second category, and can't really distinguish between the first and the third one. What's worst, if I plot the probabilities at the end of each epoch, it looks like they are converging monotonically to the vector [0,1,0], which doesn't make sense (it seems to me the gradient should push in the opposite direction once I start to over-estimate).
I really can't figure out what's going on here, but have the feeling I'm doing something plain wrong. Any idea? Thank you for your help!
For the record, I also tried using other optimizers like Adam or Adagrad playing with the hyper-params, but with no luck.
I'm using Python 3.7.9, TensorFlow 2.3.1 and TensorFlow probability 0.11.1
I believe the default argument to Categorical is not the vector of probabilities, but the vector of logits (values you'd take softmax of to get probabilities). This is to help maintain precision in internal Categorical computations like log_prob. I think you can simply eliminate the softmax activation function and it should work. Please update if it doesn't!
EDIT: alternatively you can replace the tfd.Categorical with
lambda p: tfd.Categorical(probs=p)
but you'll lose the aforementioned precision gains. Just wanted to clarify that passing probs is an option, just not the default.
I get the error 2020-01-29 12:49:54.219205: W tensorflow/core/framework/op_kernel.cc:1632] OP_REQUIRES failed at cast_op.cc:123 : Unimplemented: Cast string to float is not supported 2020-01-29 12:49:54.219705: E tensorflow/core/common_runtime/executor.cc:
I have considered converting the strings to integers simply their place in the alffbhet will this work, or is there a better way of having string inputs in keras?
Here is my code
import tensorflow as tf
model = tf.keras.models.Sequential()
word = [['ja'],['ske'],['no'],['fork']]
language = [['dansk'],['dansk'],['engelsk'],['engelsk']]
tf.keras.layers.Flatten()
model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
model.add(tf.keras.layers.Dense(128, activation = tf.nn.relu))
model.add(tf.keras.layers.Dense(2, activation = tf.nn.softmax))
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy', #mean_absolute_percentage_error
metrics=['accuracy'])
model.fit(word,language, epochs = 100)
Unfortunately, you can't. The machine learning models don't understand words, they understand digits. It is mathematics basically, everything has to be transformed into digits for calculation.
You have to convert your model inputs into meaningful numerics. This can be done using techniques like one Hot encoding etc.
I am currently learning Deep learning and Keras. When I am executing this code I am getting weird error: "TypeError: Unable to build Dense layer with non-floating point dtype " and I can't figure out what is the problem. What am I missing? How to fix this weird error?
The error show at the model.fit(...
def create_nerual_network():
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu)) # Simple Dense Layer
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu)) # Simple Dense Layer
model.add(tf.keras.layers.Dense(2, activation=tf.nn.softmax)) # Output layer
return model
train_images, train_labels = load_dataset() #this function works fine
model = create_nerual_network()
model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy'])
model.fit(train_images, train_labels, epochs = 15, verbose=2)
train_loss, train_acc = model.evaluate(train_images, train_labels)
It is interesting that you do not specify your input shape anywhere before the model compilation but maybe newer versions of Keras can figure this out from provided input.
In which case I am quite certain that the problem is with train_images, look at what dtype is this array, it's probably int8 which is usual format for images as they use 8 bit integers for each color channel.
It is common practice to at least normalize your data before training and always convert it to float.
Try putting this before calling model.fit:
train_images = train_images / 256.
This will normalize your images into range [0, 1) and convert it to float array. It is possible that you have to convert to floats also your labels.
Trying to use non keras backend functions for custom loss calculation in keras models.
I am trying to make my keras cnn model use a custom loss function ( KAppa score). However since kappas is not defined in Keras backend , i need to used scikit-learn based kappa implementation. This sklearn function takes array of labels as the argument unlike keras backend functions which take tensors. The loss function call within keras mostly sends tensors Y_pred and Y_true. I did the implementation below using some quide i found online but I get errors .
import keras.backend as K
def cohen_kappa_score_func(y_true, y_pred):
sess = tf.Session()
with sess.as_default():
score = cohen_kappa_score(type(y_true.eval()),type(y_pred.eval()), weights='linear')#idea is to convert the tensor to array
sess.close()
return score
#use this later to compile the keras model with custom loss function as
model.compile(optimizer=optimizers.SGD(lr=0.001, momentum=0.9),
loss=cohen_kappa_score_func,
metrics=['categorical_crossentropy', 'mae','categorical_accuracy'])
This doesnt work and i get the following error
"InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'dense_15_target' with dtype float and shape [?,?]
[[node dense_15_target "
Please give me suggestios to solve this.
New to TensorFlow, so apologies for newbie question.
Following this tutorial but instead of using image data I am using numerical data.
Load the dataset:
train_dataset_url = "xxx.csv"
train_dataset_fp = tf.keras.utils.get_file(
fname=os.path.basename(train_dataset_url),
origin=train_dataset_url)
Make training dataset:
batch_size = 32
train_dataset = tf.contrib.data.make_csv_dataset(
train_dataset_fp,
batch_size,
column_names=column_names,
label_name=label_name,
num_epochs=1)
Train classified model using:
model = tf.keras.Sequential([
tf.keras.layers.Dense(10, activation=tf.nn.relu, input_shape=(1,)),
tf.keras.layers.Dense(10, activation=tf.nn.relu),
tf.keras.layers.Dense(4)
])
But when I "test" the model with the same inputs:
predictions = model(features)
I receive the error:
InvalidArgumentError: cannot compute MatMul as input #0(zero-based) was expected to be a float tensor but is a int32 tensor [Op:MatMul]
It's possible I have missed something fundamental. I feel like I need to specify a type somewhere.
The data which you feed in the model is a numpy array according to my assumption . The error states that the model requires a tensor with dtype=float32 or float64. You are providing a int32 numpy array. So, wherever you create a numpy array, just mention the dtype as float32.