I've been trying to make an LSTM model using this code provided in a Deeplearning.AI tutorial on Coursera.
model = tf.keras.Sequential([
tf.keras.layers.Embedding(10000, 64),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])
However, I get this error:
"Cannot convert a symbolic Tensor (bidirectional_4/forward_lstm_4/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported"
If I drop the bidirectional layers, the code runs fine.
I know that there are several other questions dealing with the issue of converting tensors to numpy arrays. However I can't find one that addresses my issue because:
a) They all deal with doing so outside of a model. My issue is that the model is failing to even be instantiated because one layer seems to have trouble talking to another and I haven't found a solution that deals with that and
b) This is the exact same code that runs just fine inside a Colab notebook (with the same TF version as on my desktop) but fails on my desktop.
Thanks,
It turns out that this is a problem with numpy versions 2.0 and above. When I downgraded my numpy version to 1.19.5, it worked just fine.
Related
I am new to TensorFlow and I am wanting to use tensorflow.config.legacy_seq2se, specifically embedding_rnn_seq2seq() and I can't figure out how to use it (or if there is an equivalent method) for TensorFlow 2.
I know that in TensorFlow 2, TensorFlow removed contrib and according to this document
tf.contrib.legacy_seq2seq has been deleted and replaced with tf.seq2seq in TensorFlow 2, but I can't find embedding_rnn_seq2seq() in the tf.seq2seq documentation I have seen.
The reason I want to use it is I am trying to implement something similar to what is done with embedding_rnn_seq2seq() in this article. So is there an equivalent in tensorflow 2, or is there a different way to achieve the same goal?
According to https://docs.w3cub.com/tensorflow~python/tf/contrib/legacy_seq2seq/embedding_rnn_seq2seq , contrib.legacy_rnn_seq2seq createsan embedding of an argument that you pass, encoder_inputs (the shape is num_encoder_symbols x input_size). It then runs an RNN to encode the embedded encoder_inputs to convert it into a state vector. Then it embeds another argument you pass decoder_inputs (the shape is num_decoder_symbols x input_size). Next it runs an RNN decoder initialized with with the last encoder state, on the embedded decoder_inputs.
Contrib was a community maintained part of Tensorflow, and seq2seq was part of it. In Tensorflow 2 it was removed.
You could just use a Tensorflow_addons which contains community made add ons including seq2seq I believe.
You can import Tensorflow add ons via
import tensorflow_addons
Or you could just use a Tensorflow version that still has Seq2Seq (I believe 1.1 is the latest).
There are also things like bi-directional recurrent neural networks and dynamic RNNs (they are basically a new version of seq2seq) that may work.
I am trying to use a custom optimiser to train a NN in Keras. The original algorithm has been developed to train CNNs and is based on using different adaptive learning rates for every weight of the network. The algorithm is called WAME (weight-wise adaptive learning rates with moving average estimator).
The optimiser has been developed by a former University of London student and can be found in this GitHub repo (lines 54 to 153).
As you can see in the code, it is built as a subclass of the optimizer_v2 superclass available in TensorFlow. This class is called WAMEprop.
What I am trying to do is simply:
pasting the code that defines the class into my Google Colab notebook
using the new optimizer as follows:
#building the model
wame_model = models.Sequential()
wame_model.add(layers.Dense(32, activation='relu', input_shape=(11,)))
wame_model.add(layers.Dense(32, activation='relu'))
wame_model.add(layers.Dense(1))
#compiling the model using WAMEprop as optimizer
wame_model.compile(optimizer=WAMEprop(name='wame'), loss='mse', metrics=['mae'])
#fitting the model
history = wame_model.fit(train_features, train_targets,
validation_data=(test_features, test_targets),
epochs=50, batch_size=1, verbose=1)
Now, I get the following error:
ValueError Traceback (most recent call last)
<ipython-input-55-a13c54fc61f2> in <module>()
5 wame_model.add(layers.Dense(32, activation='relu'))
6 wame_model.add(layers.Dense(1))
----> 7 wame_model.compile(optimizer=WAMEprop(name='wame'), loss='mse', metrics=['mae'])
8
9 history = wame_model.fit(white_train_features_df_s.to_numpy(), white_train_targets.to_numpy(),
5 frames
/usr/local/lib/python3.7/dist-packages/keras/optimizers.py in get(identifier)
131 else:
132 raise ValueError(
--> 133 'Could not interpret optimizer identifier: {}'.format(identifier))
ValueError: Could not interpret optimizer identifier: <__main__.WAMEprop object at 0x7fd4bc1d01d0>
Since I am new to Keras and I, unfortunately, don't know Tensorflow, I am not sure what exactly it is not able to find.
Did I get some import wrong?
Did I use the keras compile() method in the wrong way?
Also, if I don't pass the parameter name='wame' to the WAMEprop() call, I get an error message saying that the positional argument 'name' is required. Strangely enough, there is no parameter 'name' in the class constructor. Does this depend on the interaction with the superclass?
Thank you very much a lot in advance for any help you could offer!
Cheers!
UPDATE:
the error message refers to a method get() that takes as input identifier in the optimizers.py file that must have been installed with TensorFlow. Now, this function is expecting to get a string (I guess fr the readily available optimizers), a configuration dictionary, an optimizer_v2.OptimizerV2 object or a tf.compat.v1.train.Optimizer object.
I think the object I am passing as an optimizer is no one of these.
If I run:
my_optimizer = WAMEprop(name='wame')
print(type(my_optimizer))
I get <class '__main__.WAMEprop'>.
So, I suspect the object I am dealing with is something different from what Keras is expecting.
UPDATE2: it runs on my laptop, I have tensorflow installed within an Anaconda environment. Now, I am convinced there is some installation or import problem in Google Colab
Problem solved. I needed to uninstall tensorflow 2.6.0 from Google Colab and install tensorflow 2.0.0 instead.
What are symbolic tensors in TensorFlow and Keras? How are they different than other tensors? Why do they even exist? Where do they come up in TensorFlow and Keras? How should we deal with them or what problems can we face when dealing with them?
In the past, I had faced certain issues related to symbolic tensors, such as the _SymbolicException, but the documentation does not describe this concept. There's also another post where this question is also asked, but, in this post, I am focusing on this specific question, so that answers can be later used as a reference.
According to blog.tensorflow.org, a symbolic tensor differs from other tensors in that they do not specifically hold values.
Let's consider a simple example.
>>> a = tf.Variable(5, name="a")
>>> b = tf.Variable(7, name="b")
>>> c = (b**2 - a**3)**5
>>> print(c)
The output is as follows:
tf.Tensor(1759441920, shape=(), dtype=int32)
For the above, the values are specifically defined in tf.Variable format, and the output is in Tensor format. However, the tensor must contain a value in order to be considered as such.
Symbolic tensors are different in that no explicit values are required to define the tensor, and this has implications in terms of building neural networks with TensorFlow 2.0, which now uses Keras as the default API.
Here is an example of a Sequential neural network that is used to build a classification model for predicting hotel cancellation incidences (full Jupyter Notebook here if interested):
from tensorflow.keras import models
from tensorflow.keras import layers
model = models.Sequential()
model.add(layers.Dense(8, activation='relu', input_shape=(4,)))
model.add(layers.Dense(1, activation='sigmoid'))
This is a symbolically defined model, as no values are explicitly being defined in the network. Rather, a framework is created for the input variables to be read by the network, and then generate predictions.
In this regard, Keras has become quite popular given that it allows for building of graphs using symbolic tensors, while at the same time maintaining an imperative layout.
I was trying to build a LSTM neural net with Keras to predict tags for words in a set of sentences.
The implementation is all pretty straight forward, but the surprising thing was that
given the exactly same and otherwise correctly implemented code and
using Tensorflow 1.4.0 with Keras running on Tensorflow Backend,
on some people's computers, it returned tensors with wrong dimensions, while for others it worked perfectly.
The problem occured in the following context:
First, we turned the list of training sentences (sentences as a list of word indeces) into a 2-D matrix using the pad_sequences method from Keras (https://keras.io/preprocessing/sequence/):
def do_padding(sequences, length, padding_value):
return pad_sequences(sequences, maxlen=length, padding='post',
truncating='post', value=padding_value)
train_sents_padded = do_padding(train_sents, MAX_LENGTH,
word_to_id[PAD_TOKEN])
Next, we used our do_padding method on the corresponding training labels to turn them into a padded matrix. At the same time, we used the Keras to_categorical method (https://keras.io/utils/#to_categorical) to add a one-hot encoded vector to the created label matrix (one one-hot vector for each cell in the matrix, that means for word in each training sentence):
train_labels_padded = to_categorical(do_padding(train_labels, MAX_LENGTH,
label_to_id["O"]), NUM_LABELS)
We expected the resulting shape to be 3-D: (len(train_labels), MAX_LENGTH, NUM_LABELS). Yet, we found that the resulting shape was 2-D and basically looked like this: ((len(train_labels) x MAX_LENGTH), NUM_LABELS), meaning the numbers on the two expected dimensions len(train_labels) and MAX_LENGTH were multiplied and flattened into one dimension.
Interestingly, this problem as said before only occured for about 50% of the people, using Tensorflow 1.4.0 and Keras running on Tensorflow Backend.
We managed to solve the problem by reshaping the label matrix manually:
train_labels_padded = np.reshape(train_labels_padded, (len(train_labels),
MAX_LENGTH, NUM_LABELS))
I was just wondering if any of you have experienced a similar problem and have figured out the reason why this happens.
I'm struggling running Tensorflow (v1.1) code multiple times in Jupyter Notebook.
For example, I execute this simple code snippet that creates an encoding layer for a seq2seq model:
# Construct encoder layer (LSTM)
encoder_cell = tf.contrib.rnn.LSTMCell(encoder_hidden_units)
encoder_outputs, encoder_final_state = tf.nn.dynamic_rnn(
encoder_cell, encoder_inputs_embedded,
dtype=tf.float32, time_major=False
)
First time is totally fine, my encoder is created.
However, if I rerun it (no matter the changes I've applied), I get this error:
Attempt to have a second RNNCell use the weights of a variable scope that already has weights
It's very annoying as it forces me to restart the kernel every time I want to change a layer.
Can someone explain me why this happens and how I can fix this ?
Thanks!
You are trying to build the exact same graph twice and therefore TensorFlow complains because the variables already exist in the default graph.
What you could do is to call tf.reset_default_graph() before trying to call the method a second time to ensure you create a new graph when required.
Just in case, I would also suggest using an interactive session as described here in the Start TensorFlow InteractiveSession section:
import tensorflow as tf
sess = tf.InteractiveSession()