I am building a 1d convolutional neural net for my own data (spectra) and am having an issue with tf.reshape. First I load in the data with pandas, and convert these to numpy arrays, composed of 708 training example spectra, each of length 2151,
import pandas as pd
import numpy as np
data = pd.read_csv('test.csv',header=None)
yTrue = data.ix[:,0].as_matrix()
data = data - data.mean()
data = data.ix[:,1:].as_matrix()
where I subtract the mean value in each column. So data is of dimensions 708 x 2151 here. I then create a network that starts with,
sess = tf.InteractiveSession()
## define inputs
x_ = tf.placeholder(tf.float32, shape=[None, 2151])
x_ = tf.reshape(x_, [-1,1,2151,1])
y_ = tf.placeholder(tf.float32, shape=[None])
which are inputs for my 1d convolutional neural net (with kernels with a width of 10, and 32 feature maps),
W_conv1 = weight_variable([1, 10, 1, 32])
b_conv1 = bias_variable([32])
h_conv1 = tf.nn.relu(conv2d(x_, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)
I then build the rest of the network and then try to run ADAM on it,
cost_function = tf.reduce_mean(tf.pow(y_out - y_, 2))/(2 * samples_number) #L2 loss
train_step = tf.train.AdamOptimizer(1e-4).minimize(cost_function)
correct_prediction = tf.equal(tf.argmax(y_out,1), tf.argmax(y_,1))
sess.run(tf.initialize_all_variables())
for i in range(20000):
print(i)
sess.run(train_step, feed_dict={x_: data, y_: yTrue})
However I get the following error:
ValueError: Cannot feed value of shape (708, 2151) for Tensor u'Reshape_26:0',
which has shape '(?, 1, 2151, 1)'
I have looked at these answers: TensorFlow/TFLearn: ValueError: Cannot feed value of shape (64,) for Tensor u'target/Y:0', which has shape '(?, 10)'; Tensorflow error using my own data which suggest that I need to be doing some reshaping before I pass my data to the network. However, I am not sure what this should be? Particularly since the following works on the first row of the data,
t = tf.constant(data[0])
tf.reshape(t,[1,1,2151,1])
Does anyone have any ideas here?
Best,
Ben
The issue is that feed_dict can replace any Tensor, and since you've changed x_ to reference the reshape op, that's the thing that it's trying to replace. It should work if you just use different Python variables to reference the placeholder and the reshape op:
x_placeholder_ = tf.placeholder(tf.float32, shape=[None, 2151])
x_ = tf.reshape(x_placeholder_, [-1,1,2151,1])
Then when feeding, use x_placeholder_:
sess.run(train_step, feed_dict={x_placeholder_: data, y_: yTrue})
Related
I am building a rnn and I use tf.nn.dynamic_rnn to yield the output and state.
The code is as follows (tf version 1.3):
import tensorflow as tf
def lstm_cell():
return tf.contrib.rnn.DropoutWrapper(tf.contrib.rnn.BasicLSTMCell(128), output_keep_prob=0.7)
cell= tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(3)])
initial_state= cell.zero_state(1, tf.float32)
layer = tf.placeholder(tf.float32, [1,1,36])
outputs, state=tf.nn.dynamic_rnn(cell=cell, inputs=layer, initial_state=initial_state)
Since the input tensor is always of batch size =1, the initial_state and state also have a batch size 1.
layer is an input of batch_size=1 as well, and for each cell there are 36 nodes(size of the embedded sequence). Each layer has lstm_size 128.
The problem comes when I loop the rnn cell.
rnn_outputs_sequence=outputs
for i in range(1, num_pics, 1):
outputs, state=tf.nn.dynamic_rnn(cell=cell, inputs=outputs, initial_state=state)
rnn_outputs_sequence=tf.concat((rnn_outputs_sequence, outputs),axis=1)
rnn_outputs_sequence is expected to have shape [1, num_pics, 36].However, this triggers an error:
Trying to share variable rnn/multi_rnn_cell/cell_0/basic_lstm_cell/kernel, but specified shape (256, 512) and found shape (164, 512).
I cannot figure out this shape [164, 512].
Can anyone help me with this out?
Thanks.
import tensorflow as tf
def lstm_cell():
return tf.contrib.rnn.DropoutWrapper(tf.contrib.rnn.BasicLSTMCell(128), output_keep_prob=0.7)
cell= tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(2)])
initial_state= cell.zero_state(1, tf.float32)
layer = tf.placeholder(tf.float32, [1,1,36])
outputs, state=tf.nn.dynamic_rnn(cell=cell, inputs=layer, initial_state=initial_state)
outputs = tf.reshape(outputs, shape=[1, -1])
outputs = tf.layers.dense(outputs, 36,\
kernel_initializer=tf.contrib.layers.xavier_initializer(uniform=False))
outputs = tf.reshape(outputs, shape=[1, 1, -1])
rnn_outputs_sequence=outputs
print(outputs)
for i in range(1, 16, 1):
outputs, state=tf.nn.dynamic_rnn(cell=cell, inputs=outputs, initial_state=state)
outputs = tf.reshape(outputs, shape=[1, -1])
outputs = tf.layers.dense(outputs, 36,\
kernel_initializer=tf.contrib.layers.xavier_initializer(uniform=False))
outputs = tf.reshape(outputs, shape=[1, 1, -1])
print(outputs)
rnn_outputs_sequence=tf.concat((rnn_outputs_sequence, outputs),axis=1)
print(rnn_outputs_sequence)
I have a problem with linear regression and 3d matrices.
They are all floating point numbers, with labels.
I got started from this code but I changed the matrix:
https://aqibsaeed.github.io/2016-07-07-TensorflowLR/
With 2 dimensions, it is working well but, with 3, I can not get it running.
this is the shape
(387, 7, 10) shape train_x
(387, 1) shape train_x
(43, 7, 10) test_x.shape
(43, 1) test_y.shape
n_dim = f.shape[1]
train_x, test_x, train_y, test_y = train_test_split(f,l,test_size=0.1, shuffle =False)
print(train_x.shape)
print(train_y.shape)
print(test_x.shape)
print(test_y.shape)
learning_rate = 0.01
training_epochs = 1000
cost_history = np.empty(shape=[1],dtype=float)
X = tf.placeholder(tf.float32,[None,n_dim])
Y = tf.placeholder(tf.float32,[None,1])
W = tf.Variable(tf.ones([n_dim,1]))
#init = tf.initialize_all_variables()
init = tf.global_variables_initializer()
y_ = tf.matmul(X, W)
cost = tf.reduce_mean(tf.square(y_ - Y))
training_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
sess = tf.Session()
sess.run(init)
for epoch in range(training_epochs):
sess.run(training_step,feed_dict={X:train_x,Y:train_y})
cost_history = np.append(cost_history,sess.run(cost,feed_dict={X: train_x,Y: train_y}))
plt.plot(range(len(cost_history)),cost_history)
plt.axis([0,training_epochs,0,np.max(cost_history)])
plt.show()
pred_y = sess.run(y_, feed_dict={X: test_x})
mse = tf.reduce_mean(tf.square(pred_y - test_y))
print("MSE: %.4f" % sess.run(mse))
fig, ax = plt.subplots()
ax.scatter(test_y, pred_y)
ax.plot([test_y.min(), test_y.max()], [test_y.min(), test_y.max()], 'k--', lw=3)
ax.set_xlabel('Measured')
ax.set_ylabel('Predicted')
plt.show()
</ blink>
this is the mistake
\session.py", line 1100, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (387, 7, 10) for Tensor 'Placeholder_12:0', which has shape '(?, 7)'
Your error message shows exact reason why it is raised.
The dimension between placeholder and train_x doesn't fit.
train_x has a (387, 7, 10) shape. In usual convention, you have 387 datapoint which has (7, 10) dimension.
But, X (placeholder, the bucket you will put train_x in) has a [None, n_dim] (I guess n_dim is 7) shape.
Using [None, ~] in the first element is only accepted as the number of datapoints, not dimension of your data.
So you need to change [None, n_dim] to [None, 7, 10] in this case.
edited)
In this case, X is not exacty 3D data. just a bunch of 2D data. For direct weight multiplication of 2D data, you need convolution step. That is CNN. But you only have very small dimension data matrix, you just need to reshape (7,10) matrix shape data to (7*10) vector shape data.
Using tf.reshape function.tf.reshape(X, shape=[387, 7*10]) will be works, and also change your W to right dimension to multiply. like, tf.Variable(tf.ones([7*10,1])).
these are my first tensorflow steps, and I would like if somebody else had the same issues as me and if there is a way around it.
I am coding the mnist tutorial and my current code-snippet is:
#placeholder for input
x = tf.placeholder(tf.float32,[None,784]) # None means a dimension can be of any length
#Weights for the model: 784 pixel maps to ten results
W = tf.Variable(tf.zeros([784,10]))
#bias
b = tf.Variable( tf.zeros([10]))
#implementing the model
y = tf.matmul(x,W) + b
#implementing cross-entropy
y_ = tf.placeholder(tf.float32,[None,10])
#cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
cross_entropy = tf.reduce_mean(
tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
sess=tf.InteractiveSession()
tf.global_variables_initializer().run()
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
for _ in range(1000):
batch_xs, batch_xy64 = mnist.train.next_batch(100)
batch_xy = batch_xy64.astype(np.float32)
sess.run(train_step , feed_dict={x:batch_xs,y:batch_xy})
correct_prediction = tf.equal(tf.argmax(y,1),tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
print (sess.run(accuracy,feed_dict={x:mnist.test.images, y_:mnist.test.labels}))
First I tried cross_entropy from the MNIST description and the the one in the provided source-code, which made no difference.
Note that I explicitly try to cast the batch_xy, as it is returned as a float 64.
This also seems to be the problem, as in the session.run float32 tensors and variables seem to be expected.
As far as I saw debugging the code, the labes in the mnist are returned as float64 - perhaps that explains my error:
...
File "/home/braunalx/python-workspace/LearnTensorFlow/firstSteps/MNIST_Start.py", line 40, in mnist_run
y_ = tf.placeholder(tf.float32,[None,10])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1548, in placeholder
return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)
...
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder_1' with dtype float and shape [?,10]
[[Node: Placeholder_1 = Placeholder[dtype=DT_FLOAT, shape=[?,10], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Is there any issue with the provided mnist data?
The error says that you didn't feed a value for a placeholder that is needed. Replace y with y_ on this line sess.run(train_step , feed_dict={x:batch_xs,y:batch_xy}).
I am looking methods to embed variable length sequences with float values to fixed size vectors. The input formats as following:
[f1,f2,f3,f4]->[f1,f2,f3,f4]->[f1,f2,f3,f4]-> ... -> [f1,f2,f3,f4]
[f1,f2,f3,f4]->[f1,f2,f3,f4]->[f1,f2,f3,f4]->[f1,f2,f3,f4]-> ... -> [f1,f2,f3,f4]
...
[f1,f2,f3,f4]-> ... -> ->[f1,f2,f3,f4]
Each line is a variable length sequnece, with max length 60. Each unit in one sequece is a tuple of 4 float values. I have already paded zeros to fill all sequences to the same length.
The following architecture seems solve my problem if I use the output as the same as input, I need the thought vector in the center as the embedding for the sequences.
In tensorflow, I have found tow candidate methods tf.contrib.legacy_seq2seq.basic_rnn_seq2seq and tf.contrib.legacy_seq2seq.embedding_rnn_seq2seq.
However, these tow methos seems to be used to solve NLP problem, and the input must be discrete value for words.
So, is there another functions to solve my problems?
All you need is only an RNN, not the seq2seq model, since seq2seq goes with an additional decoder which is unecessary in your case.
An example code:
import numpy as np
import tensorflow as tf
from tensorflow.contrib import rnn
input_size = 4
max_length = 60
hidden_size=64
output_size = 4
x = tf.placeholder(tf.float32, shape=[None, max_length, input_size], name='x')
seqlen = tf.placeholder(tf.int64, shape=[None], name='seqlen')
lstm_cell = rnn.BasicLSTMCell(hidden_size, forget_bias=1.0)
outputs, states = tf.nn.dynamic_rnn(cell=lstm_cell, inputs=x, sequence_length=seqlen, dtype=tf.float32)
encoded_states = states[-1]
W = tf.get_variable(
name='W',
shape=[hidden_size, output_size],
dtype=tf.float32,
initializer=tf.random_normal_initializer())
b = tf.get_variable(
name='b',
shape=[output_size],
dtype=tf.float32,
initializer=tf.random_normal_initializer())
z = tf.matmul(encoded_states, W) + b
results = tf.sigmoid(z)
###########################
## cost computing and training components goes here
# e.g.
# targets = tf.placeholder(tf.float32, shape=[None, input_size], name='targets')
# cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=targets, logits=z))
# optimizer = tf.train.AdamOptimizer(learning_rate=0.1).minimize(cost)
###############################
init = tf.global_variables_initializer()
batch_size = 4
data_in = np.zeros((batch_size, max_length, input_size), dtype='float32')
data_in[0, :4, :] = np.random.rand(4, input_size)
data_in[1, :6, :] = np.random.rand(6, input_size)
data_in[2, :20, :] = np.random.rand(20, input_size)
data_in[3, :, :] = np.random.rand(60, input_size)
data_len = np.asarray([4, 6, 20, 60], dtype='int64')
with tf.Session() as sess:
sess.run(init)
#########################
# training process goes here
#########################
res = sess.run(results,
feed_dict={
x: data_in,
seqlen: data_len})
print(res)
To encode sequence to a fixed length vector you typically use recurrent neural networks (RNNs) or convolutional neural networks (CNNs).
If you use a recurrent neural network you can use the output at the last time step (last element in your sequence). This corresponds to the thought vector in your question. Have a look at tf.dynamic_rnn. dynamic_rnn requires you to specify to type of RNN cell you want to use. tf.contrib.rnn.LSTMCell and tf.contrib.rnn.GRUCell are most common.
If you want to use CNNs you need to use 1 dimensional convolutions. To build CNNs you need tf.layers.conv1d and tf.layers.max_pooling1d
I have found a solution to my problem, using the following architecture,
,
The LSTMs layer below encode the series x1,x2,...,xn. The last output, the green one, is duplicated to the same count as the input for the decoding LSTM layers above. The tensorflow code is as following
series_input = tf.placeholder(tf.float32, [None, conf.max_series, conf.series_feature_num])
print("Encode input Shape", series_input.get_shape())
# encoding layer
encode_cell = tf.contrib.rnn.MultiRNNCell(
[tf.contrib.rnn.BasicLSTMCell(conf.rnn_hidden_num, reuse=False) for _ in range(conf.rnn_layer_num)]
)
encode_output, _ = tf.nn.dynamic_rnn(encode_cell, series_input, dtype=tf.float32, scope='encode')
print("Encode output Shape", encode_output.get_shape())
# last output
encode_output = tf.transpose(encode_output, [1, 0, 2])
last = tf.gather(encode_output, int(encode_output.get_shape()[0]) - 1)
# duplite the last output of the encoding layer
decoder_input = tf.stack([last for _ in range(conf.max_series)], axis=1)
print("Decoder input shape", decoder_input.get_shape())
# decoding layer
decode_cell = tf.contrib.rnn.MultiRNNCell(
[tf.contrib.rnn.BasicLSTMCell(conf.series_feature_num, reuse=False) for _ in range(conf.rnn_layer_num)]
)
decode_output, _ = tf.nn.dynamic_rnn(decode_cell, decoder_input, dtype=tf.float32, scope='decode')
print("Decode output", decode_output.get_shape())
# Loss Function
loss = tf.losses.mean_squared_error(labels=series_input, predictions=decode_output)
print("Loss", loss)
what I have is the following, which I believe is a network with one hidden LSTM layer:
# Parameters
learning rate = 0.001
training_iters = 100000
batch_size = 128
display_step = 10
# Network Parameters
n_input = 13
n_steps = 10
n_hidden = 512
n_classes = 13
# tf Graph input
x = tf.placeholder("float", [None, n_steps, n_input])
y = tf.placeholder("float", [None, n_classes])
# Define weights
weights = {
'out' : tf.Variable(tf.random_normal([n_hidden, n_classes]))
}
biases = {
'out' : tf.Variable(tf.random_normal([n_classes]))
}
However, I am trying to build an LSTM network using TensorFlow to predict power consumption. I have been looking around to find a good example, but I could not find any model with 2 hidden LSTM layers. Here's the model that I would like to build:
1 input layer,
1 output layer,
2 hidden LSTM layers(with 512 neurons in each),
time step(sequence length): 10
Could anyone guide me to build this using TensorFlow? ( from defining weights, building input shape, training, predicting, use of optimizer or cost function, etc), any help would be much appreciated.
Thank you so much in advance!
Here is how I do it in a translation model with GRU cells. You can just replace the GRU with an LSTM. It is really easy just use tf.nn.rnn_cell.MultiRNNCell with a list of the multiple cells it should wrap. In the code bellow I am manually unrolling it but you can pass it to tf.nn.dynamic_rnn or tf.nn.rnn as well.
y = input_tensor
with tf.variable_scope('encoder') as scope:
rnn_cell = rnn.MultiRNNCell([rnn.GRUCell(1024) for _ in range(3)])
state = tf.zeros((BATCH_SIZE, rnn_cell.state_size))
output = [None] * TIME_STEPS
for t in reversed(range(TIME_STEPS)):
y_t = tf.reshape(y[:, t, :], (BATCH_SIZE, -1))
output[t], state = rnn_cell(y_t, state)
scope.reuse_variables()
y = tf.pack(output, 1)
First you need some placeholders to put your training data (one batch)
x_input = tf.placeholder(tf.float32, [batch_size, truncated_series_length, 1])
y_output = tf.placeholder(tf.float32, [batch_size, truncated_series_length, 1])
A LSTM need a state, which consists of two components, the hidden state and the cell state, very good guide here: https://arxiv.org/pdf/1506.00019.pdf. For every layer in the LSTM you have one cell state and one hidden state.
The problem is that Tensorflow stores this in a LSTMStateTuple which you can not send into placeholder. So you need to store it in a Tensor, and then unpack it into a tuple:
state_placeholder = tf.placeholder(tf.float32, [num_layers, 2, batch_size, state_size])
l = tf.unpack(state_placeholder, axis=0)
rnn_tuple_state = tuple(
[tf.nn.rnn_cell.LSTMStateTuple(l[idx][0], l[idx][1])
for idx in range(num_layers)]
)
Then you can use the built-in Tensorflow API to create the stacked LSTM layer.
cell = tf.nn.rnn_cell.LSTMCell(state_size, state_is_tuple=True)
cell = tf.nn.rnn_cell.MultiRNNCell([cell]*num_layers, state_is_tuple=True)
outputs, state = tf.nn.dynamic_rnn(cell, x_input, initial_state=rnn_tuple_state)
From here you continue with the outputs to calculate logits and then a loss with respect to the y_inputs.
Then you run each batch with the sess.run-command, with truncated backpropagation (good explanation here http://r2rt.com/styles-of-truncated-backpropagation.html)
init_state = np.zeros((num_layers, 2, batch_size, state_size))
...current_state... = sess.run([...state...], feed_dict={x_input:batch_in, state_placeholder:current_state ...})
current_state = np.array(current_state)
You will have to convert the state to a numpy array before feeding it again.
Perhaps it is better to use a librarly like Tflearn or Keras instead?