I am trying to build an LSTM network using an Estimator. My data looks like
X = [[1,2,3], [2,3,4], ... , [98,99,100]]
y = [2, 3, ... , 99]
I am using an Estimator:
regressor = learn.Estimator(model_fn=lstm_model,
params=model_params,
)
where the lstm_model function is
def lstm_model(features, targets, mode, params):
def lstm_cells(layers):
if isinstance(layers[0], dict):
return [tf.nn.rnn_cell.BasicLSTMCell(layer['steps'],state_is_tuple=True) for layer in layers]
return [tf.nn.rnn_cell.BasicLSTMCell(steps, state_is_tuple=True) for steps in layers]
stacked_lstm = tf.nn.rnn_cell.MultiRNNCell(lstm_cells(params['rnn_layers']), state_is_tuple=True)
output, layers = tf.nn.rnn(stacked_lstm, [features], dtype=tf.float32)
return learn.models.linear_regression(output, targets)
and params are
model_params = {
'steps': 1000,
'learning_rate': 0.03,
'batch_size': 24,
'time_steps': 3,
'rnn_layers': [{'steps': 3}],
'dense_layers': [10, 10]
}
and then I do the fitting
regressor.fit(X, y)
The issue I am facing is
output, layers = tf.nn.rnn(stacked_lstm, [features], dtype=tf.float32)
requires a sequence but I am not sure how to split my features to into list of tensors. The shape of features inside the lstm_model function is (?, 3)
I have two questions, how do I do the training in batches? and how do I split 'features' so
output, layers = tf.nn.rnn(stacked_lstm, [features], dtype=tf.float32)
doesn't throw and error. The error I am getting is
raise TypeError("%s that don't all match." % prefix)
TypeError: Tensors in list passed to 'values' of 'Concat' Op have types [float64, float32] that don't all match.
I am using tensorflow 0.12
I had to set the shape for features to be
(batch_size, time_step, 1) or (None, time_step, 1) and then unstack the features to go in the rnn. Unstacking the features in the "time_step" so you have a list of tensors with the size of time steps and the shape for each tensor should be (None, 1) or (batch_size, 1)
Related
I have a Keras model that takes an input layer with shape (n, 288, 1), of which 288 is the number of features. I am using a TensorFlow dataset tf.data.experimental.make_batched_features_dataset and my input layer will be (n, 1, 1) which means it gives one feature to the model at a time. How can I make an input tensor with the shape of (n, 288, 1)? I mean how can I use all my features in one tensor?
Here is my code for the model:
def _gzip_reader_fn(filenames):
"""Small utility returning a record reader that can read gzip'ed files."""
return tf.data.TFRecordDataset(filenames, compression_type='GZIP')
def _input_fn(file_pattern, tf_transform_output, batch_size):
"""Generates features and label for tuning/training.
Args:
file_pattern: input tfrecord file pattern.
tf_transform_output: A TFTransformOutput.
batch_size: representing the number of consecutive elements of returned
dataset to combine in a single batch
Returns:
A dataset that contains (features, indices) tuple where features is a
dictionary of Tensors, and indices is a single Tensor of label indices.
"""
transformed_feature_spec = (
tf_transform_output.transformed_feature_spec().copy())
dataset = tf.data.experimental.make_batched_features_dataset(
file_pattern=file_pattern,
batch_size=batch_size,
features=transformed_feature_spec,
reader=_gzip_reader_fn,
label_key=features.transformed_name(features.LABEL_KEY))
return dataset
def _build_keras_model(nb_classes=2, input_shape, learning_rate):
# Keras needs the feature definitions at compile time.
input_shape = (288,1)
input_layer = keras.layers.Input(input_shape)
padding = 'valid'
if input_shape[0] < 60:
padding = 'same'
conv1 = keras.layers.Conv1D(filters=6, kernel_size=7, padding=padding, activation='sigmoid')(input_layer)
conv1 = keras.layers.AveragePooling1D(pool_size=3)(conv1)
conv2 = keras.layers.Conv1D(filters=12, kernel_size=7, padding=padding, activation='sigmoid')(conv1)
conv2 = keras.layers.AveragePooling1D(pool_size=3)(conv2)
flatten_layer = keras.layers.Flatten()(conv2)
output_layer = keras.layers.Dense(units=nb_classes, activation='sigmoid')(flatten_layer)
model = keras.models.Model(inputs=input_layer, outputs=output_layer)
optimizer = keras.optimizers.Adam(lr=learning_rate)
# Compile Keras model
model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['accuracy'])
model.summary(print_fn=logging.info)
return model
This is the error:
tensorflow:Model was constructed with shape (None, 288, 1) for input Tensor("input_1:0", shape=(None, 288, 1), dtype=float32), but it was called on an input with incompatible shape (128, 1, 1).
I tried Convolution Neural Network with Tensorflow.
However, Shape causes an error.
First is part of the main function.
while True:
with mss.mss() as sct:
Game_Scr = np.array(sct.grab(Game_Scr_pos))[:,:,:3]
cv2.imshow('Game_Src', Game_Scr)
cv2.waitKey(0)
Game_Scr = cv2.resize(Game_Scr, dsize=(960, 540), interpolation=cv2.INTER_AREA)
print(Game_Scr.shape)
print(Convolution(Game_Scr))
Second is my called function.
def Convolution(img):
kernel = tf.Variable(tf.truncated_normal(shape=[4], stddev=0.1))
sess = tf.Session()
with tf.Session() as sess:
img = img.astype('float32')
Bias1 = tf.Variable(tf.truncated_normal(shape=[4],stddev=0.1))
conv2d = tf.nn.conv2d(img, kernel, strides=[1, 1, 1, 1], padding='SAME')# + Bias1
conv2d = sess.run(conv2d)
return conv2d
ValueError: Shape must be rank 4 but is rank 3 for 'Conv2D' (op: 'Conv2D') with input shapes: [540,960,3], [4].
I tried changing the shape many times, but I get the same error.
Try replacing
img = img.astype('float32')
with
img = tf.expand_dims(img.astype('float32'), 0)
The dimention of tf.nn.conv2d input shoul be 4, (batch_size, image_hight, image_with, image_channels). You where missing the batch_size, tf.expand_dims just add that dimention (with a batch_size of 1 since you only have one image).
As per the official documentation here, input tensor should be of shape [batch, in_height, in_width, in_channels] and a filter / kernel tensor should be of shape [filter_height, filter_width, in_channels, out_channels].
Try by changing your Convolution function to something like this:
def Convolution(img):
kernel = tf.Variable(tf.truncated_normal(shape=[200, 200, 3, 3], stddev=0.1))
sess = tf.Session()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
img = img.astype('float32')
conv2d = tf.nn.conv2d(np.expand_dims(img, 0), kernel, strides=[1, 1, 1, 1], padding='SAME')# + Bias1
conv2d = sess.run(conv2d)
return conv2d
Following the example at https://www.tensorflow.org/guide/datasets#preprocessing_data_with_datasetmap, I want to create a tf.Dataset which takes in paths to images, and maps these to image tensors.
My first attempt was the following, which is very similar to the example in the above link:
def input_parser(image_path):
image_data_string = tf.read_file(image_path)
image_decoded = tf.image.decode_png(image_data_string, channels=3)
image_float = tf.image.convert_image_dtype(image_decoded, dtype=tf.float32)
return image_float
def train_model():
image_paths = ['test_image1.png', .test_image2.png', 'test_image3.png']
dataset = tf.data.Dataset.from_tensor_slices(image_paths)
dataset = dataset.map(map_func=input_parser)
iterator = dataset.make_initializable_iterator()
input_images = iterator.get_next()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
sess.run(iterator.initializer)
for i in range(3):
x = sess.run(input_images)
print(x.shape)
This seemed to work ok, and printed out:
(64, 64, 3)
(64, 64, 3)
(64, 64, 3)
Which are indeed the dimensions of my images.
So then I tried to actually feed this data into a network to train, and modified the code accordingly:
def input_parser(image_path):
image_data_string = tf.read_file(image_path)
image_decoded = tf.image.decode_png(image_data_string, channels=3)
image_float = tf.image.convert_image_dtype(image_decoded, dtype=tf.float32)
return image_float
def train_model():
image_paths = ['test_image1.png', .test_image2.png', 'test_image3.png']
dataset = tf.data.Dataset.from_tensor_slices(image_paths)
dataset = dataset.map(map_func=input_parser)
iterator = dataset.make_initializable_iterator()
input_images = iterator.get_next()
x = tf.layers.conv2d(inputs=input_images, filters=50, kernel_size=[5, 5], name='layer1')
x = tf.layers.flatten(x, name='layer2')
prediction = tf.layers.dense(inputs=x, units=4, name='layer3')
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
sess.run(iterator.initializer)
for i in range(3):
p = sess.run(prediction)
print(p)
This then gave me the following error message:
ValueError: Input 0 of layer layer1 is incompatible with the layer: expected ndim=4, found ndim=3. Full shape received: [None, None, 3]
I have two questions about this:
1) Why is my network receiving an input of shape [None, None, 3], when as we have seen, the data read by the iterator is of shape [64, 64, 3].
2) Why isn't the shape of the input actually [1, 64, 64, 3], i.e. with 4 dimensions? I thought that the first dimension would be 1 because this is the batch size (I am not batching the data, so effectively this is a batch size of 1).
Thanks!
The shape is None in the spatial dimensions because in principle you could be loading images of any size. There is no guarantee that they will be 64x64 so Tensorflow uses None shapes to allow for inputs of any size. Since you know that the images will always be the same size, you can use a Tensor's set_shape method to give this information. Just include a line image_float.set_shape((64, 64, 3)) in your parse function. Note that this seems to modify the tensor in place. There is even an example using images here.
You are not batching the data, so no batch axis is added at all. The elements of the dataset are simply images of shape (64, 64, 3) and these elements are returned one by one by the iterator. If you want batches of size 1 you should use dataset = dataset.batch(1). Now the elements of the dataset are image "batches" of shape (1, 64, 64, 3). Of course you could also use any other method to add an axis in front, such as tf.expand_dims.
I have a Tensorflow layer with 2 nodes. These are the output nodes of another 2 larger hidden layers. Now I want to add 2 new nodes to this layer, so I end up with 4 nodes in total, and do some last computation. The added nodes are implemented as Placeholders so far, and have a dynamic shape depending on the batch size. Here is a sketch of the net:
Now I want to concatenate Nodes 3 and 4 to the nodes 1 and 2 of the previously computed layer. I know there is tf.concat for this, but I don't understand how to do this correctly.
How do I add Placeholders of the same batchsize as the original net input to a specific layer?
EDIT:
When I use tf.concat over axis=1, I end up with the following problem:
z = tf.placeholder(tf.float32, shape=[None, 2])
Weight_matrix = weight_variable([4, 2])
bias = bias_variable([4, 2])
concat = tf.concat((dnn_out, z), 1)
h_fc3 = tf.nn.relu(tf.matmul(concat, Weight_matrix) + bias)
Adding the bias to the tf.matmul result throws an InvalidArgumentError: Incompatible shapes: [20,2] vs. [4,2].
Since your data is batched, probably over the first dimension, you need to concatenate over the second (axis=1):
import tensorflow as tf
import numpy as np
dnn_output = tf.placeholder(tf.float32, (None, 2)) # replace with your DNN(input) result
additional_nodes = tf.placeholder(tf.float32, (None, 2))
concat = tf.concat((dnn_output, additional_nodes), axis=1)
print(concat)
# > Tensor("concat:0", shape=(?, 4), dtype=float32)
dense_output = tf.layers.dense(concat, units=2)
print(dense_output)
# > Tensor("dense/BiasAdd:0", shape=(?, 2), dtype=float32)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print(sess.run(dense_output, feed_dict={dnn_output: np.ones((5, 2)),
additional_nodes: np.zeros((5, 2))}))
I am building a rnn and I use tf.nn.dynamic_rnn to yield the output and state.
The code is as follows (tf version 1.3):
import tensorflow as tf
def lstm_cell():
return tf.contrib.rnn.DropoutWrapper(tf.contrib.rnn.BasicLSTMCell(128), output_keep_prob=0.7)
cell= tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(3)])
initial_state= cell.zero_state(1, tf.float32)
layer = tf.placeholder(tf.float32, [1,1,36])
outputs, state=tf.nn.dynamic_rnn(cell=cell, inputs=layer, initial_state=initial_state)
Since the input tensor is always of batch size =1, the initial_state and state also have a batch size 1.
layer is an input of batch_size=1 as well, and for each cell there are 36 nodes(size of the embedded sequence). Each layer has lstm_size 128.
The problem comes when I loop the rnn cell.
rnn_outputs_sequence=outputs
for i in range(1, num_pics, 1):
outputs, state=tf.nn.dynamic_rnn(cell=cell, inputs=outputs, initial_state=state)
rnn_outputs_sequence=tf.concat((rnn_outputs_sequence, outputs),axis=1)
rnn_outputs_sequence is expected to have shape [1, num_pics, 36].However, this triggers an error:
Trying to share variable rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel, but specified shape (256, 512) and found shape (164, 512).
I cannot figure out this shape [164, 512].
Can anyone help me with this out?
Thanks.
import tensorflow as tf
def lstm_cell():
return tf.contrib.rnn.DropoutWrapper(tf.contrib.rnn.BasicLSTMCell(128), output_keep_prob=0.7)
cell= tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(2)])
initial_state= cell.zero_state(1, tf.float32)
layer = tf.placeholder(tf.float32, [1,1,36])
outputs, state=tf.nn.dynamic_rnn(cell=cell, inputs=layer, initial_state=initial_state)
outputs = tf.reshape(outputs, shape=[1, -1])
outputs = tf.layers.dense(outputs, 36,\
kernel_initializer=tf.contrib.layers.xavier_initializer(uniform=False))
outputs = tf.reshape(outputs, shape=[1, 1, -1])
rnn_outputs_sequence=outputs
print(outputs)
for i in range(1, 16, 1):
outputs, state=tf.nn.dynamic_rnn(cell=cell, inputs=outputs, initial_state=state)
outputs = tf.reshape(outputs, shape=[1, -1])
outputs = tf.layers.dense(outputs, 36,\
kernel_initializer=tf.contrib.layers.xavier_initializer(uniform=False))
outputs = tf.reshape(outputs, shape=[1, 1, -1])
print(outputs)
rnn_outputs_sequence=tf.concat((rnn_outputs_sequence, outputs),axis=1)
print(rnn_outputs_sequence)