I am running a keras model, LINK IS HERE. I have just changed the dataset for this model and when I run my model it throwing this error TypeError: tuple indices must be integers or slices, not str. As it's a image captioning model and the dataset is difficult for me to understand.
See the blow code and read also the location of the error.
`reduce_lr = keras.callbacks.ReduceLROnPlateau(
monitor="val_loss", factor=0.2, patience=3
)
# Create an early stopping callback.
early_stopping = tf.keras.callbacks.EarlyStopping(
monitor="val_loss", patience=5, restore_best_weights=True
)
history = dual_encoder.fit(
train_dataloader,
epochs=num_epochs,
#validation_data=val_dataloader,
#callbacks=[reduce_lr, early_stopping],
)
print("Training completed. Saving vision and text encoders...")
vision_encoder.save("vision_encoder")
text_encoder.save("text_encoder")
print("Models are saved.")
TypeError Traceback (most recent call last)
<ipython-input-31-745dd79762e6> in <module>()
15 history = dual_encoder.fit(
16 train_dataloader,
---> 17 epochs=num_epochs,
18 #validation_data=val_dataloader,
19 #callbacks=[reduce_lr, early_stopping],
11 frames
<ipython-input-26-0696c83bf387> in call(self, features, training)
16 with tf.device("/gpu:0"):
17 # Get the embeddings for the captions.
---> 18 caption_embeddings = text_encoder(features["caption"], training=training)
19 #caption_embeddings = text_encoder(train_inputs, training=training)
20 with tf.device("/gpu:1"):
TypeError: tuple indices must be integers or slices, not str'
The error is pointing to this location caption_embeddings = text_encoder(features["caption"], training=training)
Now I am confused, I don't know whether this error is due to the data which I am passing to my model like this history = dual_encoder.fit(train_dataloader) OR this error is related to caption_embeddings = text_encoder(features["caption"], training=training) and image_embeddings = vision_encoder(features["image"], training=training) which is defined in class DualEncoder.
Because I don't know what are these features["caption"] and features["image"] which is defined in Class DualEncoder as I have not changed these two with my new dataset if You check my CODE HERE IN THIS COLAB FILE.
The dataset (train_dataloader) seems to return a tuple of items: link. In particular, model input is a tuple (images, x_batch_input).
However, your code (in DualEncoder) seems to assume that it's a dict (with keys like "caption", "image", etc). I think that's the source of the mismatch.
Related
Background
My sentiment analysis research comes across a variety of datasets. Recently I've encountered one dataset that somehow I just cannot train successfully. I mostly work with open data in .CSV file format, hence Pandas and NumPy are heavily used.
During my research, one of the approaches is trying to integrate automated machine learning (AutoML), and the library I chose to use was Auto-Keras, mainly using its TextClassifier() wrapper function to achieve AutoML.
Main Problem
I've verified with official documentation, that the TextClassifier() takes data in the format of the NumPy array. However, when I load the data into Pandas DataFrame and used .to_numpy() on the columns that I need to train, the following error kept showing:
ValueError Traceback (most recent call last)
<ipython-input-13-1444bf2a605c> in <module>()
16 clf = ak.TextClassifier(overwrite=True, max_trials=2)
17
---> 18 clf.fit(x_train, y_train, epochs=3, callbacks=cbs)
19
20
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float).
Error-related code sectors
The sector where I drop the unneeded Pandas DataFrame columns using .drop(), and convert the needed columns to NumPy Array using the to_numpy() function that Pandas has provided.
df_src = pd.read_csv(get_data)
df_src = df_src.drop(columns=["Name", "Cast", "Plot", "Direction",
"Soundtrack", "Acting", "Cinematography"])
df_src = df_src.reset_index(drop=True)
X = df_src["Review"].to_numpy()
Y = df_src["Overall Sentiment"].to_numpy()
print(X, "\n")
print("\n", Y)
The main error code part, where I perform StratifedKFold() and at the same time, use TextClassifier() to train and test the model.
fold = 0
for train, test in skf.split(X, Y):
fold += 1
print(f"Fold #{fold}\n")
x_train = X[train]
y_train = Y[train]
x_test = X[test]
y_test = Y[test]
cbs = [tf.keras.callbacks.EarlyStopping(patience=3)]
clf = ak.TextClassifier(overwrite=True, max_trials=2)
# The line where it indicated the error.
clf.fit(x_train, y_train, epochs=3, callbacks=cbs)
pred = clf.predict(x_test) # result data type is in lists of `string`
ceval = clf.evaluate(x_test, y_test)
metrics_test = metrics.classification_report(y_test, np.array(list(pred), dtype=int))
print(metrics_test, "\n")
print(f"Fold #{fold} finished\n")
Supplementary
I am sharing the full code related to the error through Google Colab, which you can help me diagnose here.
Edit notes
I have tried the potential solution, such as:
x_train = np.asarray(x_train).astype(np.float32)
y_train = np.asarray(y_train).astype(np.float32)
or
x_train = tf.data.Dataset.from_tensor_slices((x_train,))
y_train = tf.data.Dataset.from_tensor_slices((y_train,))
However, the problem remains.
One of the strings is equal to nan. Just remove this entry and the corresponding label.
In TensorFlow 2.0, there's the class tf.keras.metrics.AUC. It can easily be added to the list of metrics of the compile method as follows.
# Example taken from the documentation
model.compile('sgd', loss='mse', metrics=[tf.keras.metrics.AUC()])
However, in my case, the output of my neural network is an NxM tensor, where N is the batch size and M is the number of separate outputs. I would like to compute the AUC metric for each of these M outputs separately (across all N instances of the batch). So, there should be M AUC metrics, each of them is computed with N observations. I tried to create a custom metric, but I am facing some issues. The following is my first attempt.
def get_custom_auc(output):
auc = tf.metrics.AUC()
#tf.function
def custom_auc(y_true, y_pred):
y_true = y_true[:, output]
y_pred = y_pred[:, output]
auc.update_state(y_true, y_pred)
return auc.result()
custom_auc.__name__ = "custom_auc_" + str(output)
return custom_auc
The need to rename custom_auc.__name__ is described in the following post: Is it possible to have a metric that returns an array (or tensor) rather than a number?. However, this implementation raises an error.
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [predictions must be >= 0] [Condition x >= y did not hold element-wise:] [x (strided_slice_1:0) = ] [3.14020467 3.06779885 2.86414027...] [y (Cast_1/x:0) = ] [0]
[[{{node metrics/custom_auc_2/StatefulPartitionedCall/assert_greater_equal/Assert/AssertGuard/else/_161/Assert}}]] [Op:__inference_keras_scratch_graph_5149]
I have also tried to create the AUC object inside the custom_auc, but this is not possible because I am using #tf.function, so I will get the error ValueError: tf.function-decorated function tried to create variables on non-first call.. Even if I remove the #tf.function (which I may need because I may use some if-else statements inside the implementation), I get another error
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable _AnonymousVar33 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar33/N10tensorflow3VarE does not exist.
[[node metrics/custom_auc_0/add/ReadVariableOp (defined at /train.py:173) ]] [Op:__inference_keras_scratch_graph_5174]
Note that, currently, I am adding these AUC metrics, one for each of the M outputs, as described in this answer. Furthermore, I cannot simply return the object auc, because apparently Keras expects the output of the custom metric to be a tensor and not an AUC object. So, if you do that, you get the following error.
TypeError: To be compatible with tf.contrib.eager.defun, Python functions must return zero or more Tensors; in compilation of .custom_auc at 0x1862e6680>, found return value of type , which is not a Tensor.
I've also tried to implement a custom metric class as follows.
class CustomAUC(tf.metrics.Metric):
def __init__(self, num_outputs, name="custom_auc", **kwargs):
super(CustomAUC, self).__init__(name=name, **kwargs)
assert num_outputs >= 1
self.num_outputs = num_outputs
self.aucs = [tf.metrics.AUC() for _ in range(self.num_outputs)]
def update_state(self, y_true, y_pred, sample_weight=None):
for output in range(self.num_outputs):
y_true1 = y_true[:, output]
y_pred1 = y_pred[:, output]
self.aucs[output].update_state(y_true1, y_pred1)
def result(self):
return [auc.result() for auc in self.aucs]
However, I am currently getting the error
ValueError: Shapes (200,) and () are incompatible
This error seems to be related to reset_states, so maybe I should also override this method. In fact, if I override reset_states with the following implementation
def reset_states(self):
for auc in self.aucs:
auc.reset_states()
I don't get this error anymore, but I get another error
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [predictions must be >= 0] [Condition x >= y did not hold element-wise:] [x (strided_slice_1:0) = ] [-1.38822043 1.24234951 -0.254447281...] [y (Cast_1/x:0) = ] [0]
[[{{node metrics/custom_auc/PartitionedFunctionCall/assert_greater_equal/Assert/AssertGuard/else/_98/Assert}}]] [Op:__inference_keras_scratch_graph_5248]
So, how do I implement this custom AUC metric, one for each of the M outputs of the network? Basically, I want to do something similar to the solution described in this answer, but with the AUC metric.
I have also opened the related issue on the TensorFlow's Github issue tracker.
I have a similar problem like yours. I have a model with 3 outputs and and i want to compute a custom metric (ConfusionMatricMetric) for the 3 outputs (that have different number of classes each). I used a solution in here https://keras.io/guides/customizing_what_happens_in_fit/ - Going lower level. My problem now is that I can't train the model because of
ValueError: tf.function-decorated function tried to create variables on non-first call.
then I used
tf.config.run_functions_eagerly(True)
and now the models train, very slow but it can be saved
P.S. I also used tf.keras.metrics.KLDivergence() instead of my custom metric and reproduced the same experiment with the same results as above - trained & saved (tf.saved_model.save)
I'm in the process of converting my pytorch models into tensorflow 2.0, so I'm still getting used to it. I have mostly gone off the API, I made a custom model, and defined it's call method with argument inputs:
class CustomModel(tf.keras.Model):
<... init...>
def call(self, inputs):
print("inputs: ", inputs)
self.sequential_convolution(inputs)
The sequential_convolution is a keras.Sequential of multiple convolution related layers. I can create the model object, compile it. It is variable length on both the output and input
model = CustomModel(inputs=tf.keras.Input(shape=(None, vdim)))
model.compile(optimizer=optimizer, loss=loss_func, metrics=[calc_accuracy])
for x, y in dataset:
print("x.shape: ", x.shape)
print("y.shape: ", y.shape)
model.fit(x, y, batch_size=1)
Where the shapes are x.shape: (244, 161) and y.shape: (40,). Both are Tensorflow tensors created from numpy arrays with tf.convert_to_tensor().
But when the model's call method prints the inputs, I get the following:
Tensor("input_1_1:0", shape=(None, 161), dtype=float32)
Which I should point out is not the Input defined on the model, this input is calculated from the actual input provided in the model.fit(), I manually changed the numbers to see what the causes were...
Which then ultimately leads to the stack trace:
x = self.sequential_conv(inputs)
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py:396 converted_call
return py_builtins.overload_of(f)(*args)
TypeError: 'NoneType' object is not callable
This error occurs in a function deemed internal use only, but not able to ascertain what the cause of my problem is.
As I can't find much information on the matter, I feel that it's most likely something simple I haven't done, but I'm not sure. Any help would be great...
Update: As rvinas pointed out, I had forgotten to add inputs_aux as second input in Model. Fixed now, and it works. So ConditionalRNN can readily be used to do what I want.
I'd like to treat time-series together with non-time-series characteristics in extended LSTM cells (a requirement also discussed here). ConditionalRNN (cond-rnn) for Tensorflow in Python seems to allow this.
Can it be used in Keras Functional API (without eager execution)?
That is, does anyone have a clue how to fix my failed approach below, or a different example where ConditionalRNN (or alternatives) are used to readily combine TS and non-TS data in LSTM-style cells or any equivalent?
I've seen the eager execution-bare tf example on Pilippe Remy's ConditionalRNN github page, but I did not manage to extend it to a readily fittable version in Keras Functional API.
My code looks as follows; it works if, instead of the ConditionalRNN, I use a standard LSTM cell (and adjust the model 'x' input correspondingly). With ConditionalRNN, I did not get it to execute; I receive either the must feed a value for placeholder tensor 'in_aux' error (cf. below), or instead some different type of input size complaints when I change the code, despite trying to be careful about data dimensions compatibility.
(Using Python 3.6, Tensorflow 2.1, cond-rnn 2.1, on Ubuntu 16.04)
import numpy as np
from tensorflow.keras.models import Model
from tensorflow.keras.layers import LSTM, Dense, Input
from cond_rnn import ConditionalRNN
inputs = Input(name='in',shape=(5,5)) # Each observation has 5 dimensions à 5 time-steps each
x = Dense(64)(inputs)
inputs_aux = Input(name='in_aux', shape=[5]) # For each of the 5 dimensions, a non-time-series observation too
x = ConditionalRNN(7, cell='LSTM')([x,inputs_aux]) # Updated Syntax for cond_rnn v2.1
# x = ConditionalRNN(7, cell='LSTM', cond=inputs_aux)(x) # Syntax for cond_rnn in some version before v2.1
predictions = Dense(1)(x)
model = Model(inputs=[inputs, inputs_aux], outputs=predictions) # With this fix, [inputs, inputs_aux], it now works, solving the issue
#model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='rmsprop', loss='mean_squared_error', metrics=['mse'])
data = np.random.standard_normal([100,5,5]) # Sample of 100 observations with 5 dimensions à 5 time-steps each
data_aux = np.random.standard_normal([100,5]) # Sample of 100 observations with 5 dimensions à only 1 non-time-series value each
labels = np.random.standard_normal(size=[100]) # For each of the 100 obs., a corresponding (single) outcome variable
model.fit([data,data_aux], labels)
The error I get is
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'in_aux' with dtype float and shape [?,5]
[[{{node in_aux}}]]
and the traceback is
Traceback (most recent call last):
File "/home/florian/temp_nonclear/playground/test/est1ls_bare.py", line 20, in <module>
model.fit({'in': data, 'in_aux': data_aux}, labels) #model.fit([data,data_aux], labels) # Also crashes when using model.fit({'in': data, 'in_aux': data_aux}, labels)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training.py", line 643, in fit
use_multiprocessing=use_multiprocessing)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 664, in fit
steps_name='steps_per_epoch')
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 383, in model_iteration
batch_outs = f(ins_batch)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/backend.py", line 3353, in __call__
run_metadata=self.run_metadata)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
run_metadata_ptr)
I noticed that you are not passing inputs_aux as input to your model. TF is complaining because this tensor is required to compute your output predictions and it is not being fed with any value. Defining your model as follows should solve the problem:
model = Model(inputs=[inputs, inputs_aux], outputs=predictions)
I'm pretty new to Tensorflow and have been running experiments with SSDs with the Tensorflow Object Detection API. I can successfully train a model, but by default, it only save the last n checkpoints. I'd like to instead save the last n checkpoints with the lowest loss (I'm assuming that's the best metric to use).
I found tf.estimator.BestExporter and it exports a saved_model.pb along with variables. However, I have yet to figure out how to load that saved model and run inference on it. After running models/research/object_detection/export_inference_graph.py on the checkpoiont, I can easily load a checkpoint and run inference on it using the object detection jupyter notebook: https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb
I've found documentation on loading saved models, and can load a graph like this:
with tf.Session(graph=tf.Graph()) as sess:
tags = [tag_constants.SERVING]
meta_graph = tf.saved_model.loader.load(sess, tags, PATH_TO_SAVED_MODEL)
detection_graph = tf.get_default_graph()
However, when I use that graph with the above jupyter notebook, I get errors:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-17-9e48f0d04df2> in <module>
7 image_np_expanded = np.expand_dims(image_np, axis=0)
8 # Actual detection.
----> 9 output_dict = run_inference_for_single_image(image_np, detection_graph)
10 # Visualization of the results of a detection.
11 vis_util.visualize_boxes_and_labels_on_image_array(
<ipython-input-16-0df86999596e> in run_inference_for_single_image(image, graph)
31 detection_masks_reframed, 0)
32
---> 33 image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')
34 # image_tensor = tf.get_default_graph().get_tensor_by_name('serialized_example')
35
~/anaconda3/envs/sb/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in get_tensor_by_name(self, name)
3664 raise TypeError("Tensor names are strings (or similar), not %s." %
3665 type(name).__name__)
-> 3666 return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
3667
3668 def _get_tensor_by_tf_output(self, tf_output):
~/anaconda3/envs/sb/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in as_graph_element(self, obj, allow_tensor, allow_operation)
3488
3489 with self._lock:
-> 3490 return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
3491
3492 def _as_graph_element_locked(self, obj, allow_tensor, allow_operation):
~/anaconda3/envs/sb/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _as_graph_element_locked(self, obj, allow_tensor, allow_operation)
3530 raise KeyError("The name %s refers to a Tensor which does not "
3531 "exist. The operation, %s, does not exist in the "
-> 3532 "graph." % (repr(name), repr(op_name)))
3533 try:
3534 return op.outputs[out_n]
KeyError: "The name 'image_tensor:0' refers to a Tensor which does not exist. The operation, 'image_tensor', does not exist in the graph."
Is there a better way to load the saved model or convert it to an inference graph?
Thanks!
Tensorflow detection API supports different input formats during exporting as discribed in documentation of file export_inference_graph.py:
image_tensor: Accepts a uint8 4-D tensor of shape [None, None, None, 3]
encoded_image_string_tensor: Accepts a 1-D string tensor of shape [None]
containing encoded PNG or JPEG images. Image resolutions are expected to be
the same if more than 1 image is provided.
tf_example: Accepts a 1-D string tensor of shape [None] containing
serialized TFExample protos. Image resolutions are expected to be the same
if more than 1 image is provided.
So you should check that you use image_tensor input_type. The chosen input node will be named as "inputs" in exported model. So I suppose that replacing image_tensor:0 with inputs (or maybe inputs:0) will solve your problem.
Also I would like to recommend a useful tool to run exported models with several lines of code: tf.contrib.predictor.from_saved_model. Here is example of how to use it:
import tensorflow as tf
import cv2
img = cv2.imread("test.jpg")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img_rgb = np.expand_dims(img, 0)
predict_fn = tf.contrib.predictor.from_saved_model("./saved_model")
output_data = predict_fn({"inputs": img_rgb})
print(output_data) # detector output dictionary