tensorflow MNIST fully_connected_feed.py fails: range() takes at least 2 arguments (1 given) - tensorflow

I'm having trouble running the example in one of the tensor flow tutorials. The tutorial says to run I just need to type python fully_connected_feed.py. When I do this it gets through fetching the input data, but then fails, like so:
Extracting data/train-images-idx3-ubyte.gz
Extracting data/train-labels-idx1-ubyte.gz
Extracting data/t10k-images-idx3-ubyte.gz
Extracting data/t10k-labels-idx1-ubyte.gz
Traceback (most recent call last):
File "fully_connected_feed.py", line 225, in <module>
tf.app.run()
File "/Users/me/anaconda/lib/python2.7/site-packages/tensorflow/python/platform/default/_app.py", line 11, in run
sys.exit(main(sys.argv))
File "fully_connected_feed.py", line 221, in main
run_training()
File "fully_connected_feed.py", line 141, in run_training
loss = mnist.loss(logits, labels_placeholder)
File "/Users/me/tftmp/mnist.py", line 96, in loss
indices = tf.expand_dims(tf.range(batch_size), 1)
TypeError: range() takes at least 2 arguments (1 given)
I think this error is caused because there is some problem with session setup and/or tensor evaluation. This is the function in mnist.py causing the problem:
def loss(logits, labels):
"""Calculates the loss from the logits and the labels.
Args:
logits: Logits tensor, float - [batch_size, NUM_CLASSES].
labels: Labels tensor, int32 - [batch_size].
Returns:
loss: Loss tensor of type float.
"""
# Convert from sparse integer labels in the range [0, NUM_CLASSSES)
# to 1-hot dense float vectors (that is we will have batch_size vectors,
# each with NUM_CLASSES values, all of which are 0.0 except there will
# be a 1.0 in the entry corresponding to the label).
batch_size = tf.size(labels)
labels = tf.expand_dims(labels, 1)
indices = tf.expand_dims(tf.range(batch_size), 1)
concated = tf.concat(1, [indices, labels])
onehot_labels = tf.sparse_to_dense(
concated, tf.pack([batch_size, NUM_CLASSES]), 1.0, 0.0)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, onehot_labels,
name='xentropy')
loss = tf.reduce_mean(cross_entropy, name='xentropy_mean')
return loss
If I put all the code in the loss function inside a with tf.Session(): block, it gets past this error. However, I get other errors later about uninitialised variables, so I'm guessing something major is going wrong with session setup or initialisation, or something. Being new to tensor flow I'm a little at a loss. Any ideas?
[NB: I havent edited the code at all, just downloaded from the tensorflow tutorials and tried to run as instructed, with python fully_connected_feed.py]

This issue arises because in the latest version of the TensorFlow source on GitHub, tf.range() has been updated to be more permissive with its arguments (previously it required two arguments; now it has the same semantics as Python's range() built-in function), and the fully_connected_feed.py example has been updated to exploit this.
However, if you try to run this version against the binary distribution of TensorFlow, you will get this error because the change to tf.range() has not been incorporated into the binary package.
The easiest solution is to download the old version of mnist.py. Alternatively, you could build from source to use the latest version of the tutorial.

you can right result fix mnist code like this :
indices = tf.expand_dims(tf.range(0,batch_size),1)

TypeError: range() takes at least 2 arguments (1 given)
That's the error.
Looking at the tensorflow docs for range, we can see that range has a function signature of start, limit, delta=1, name='range'. This means that at least two arguments are required for function invocation. Your example only shows one argument provided.
An example can be found in the docs:
# 'start' is 3
# 'limit' is 18
# 'delta' is 3
tf.range(start, limit, delta) ==> [3, 6, 9, 12, 15]

Related

Graph disconnected: cannot obtain value for tensor Tensor() at layer "input_1"

The code for this problem is quite complex because I'm trying to implement fractalNet but changing the convolution base block to just a dense layer. I'm trying to separately build two fractalNets (one after the other so I don't think they should be interfering). One for the policy and one for the value function.
There are also a number of issues I have seen so far that may or may not be related. One is that I can't import numpy as np and use np which is why I've been forced to use numpy(). The other is that my code seems to trying to be working on tensors tf.Tensor[stuff] as well as Tensor[stuff] in different sections at the same time. The build_model function below outputs Tensor[stuff] from the Input call whereas the neural network builder code uses tf.Tensor[stuff]. I tried but to no avail to stick to type.
Here is the complete error that keeps killing the code:
/home/ryan/.local/lib/python3.6/site-packages/keras/engine/network.py:190: UserWarning: Model inputs must come from `keras.layers.Input` (thus holding past layer metadata), they cannot be the output of a previous non-Input layer. Here, a tensor specified as input to your model was not an Input tensor, it was generated by layer activation_1.
Note that input tensors are instantiated via `tensor = keras.layers.Input(shape)`.
The tensor that caused the issue was: activation_1/Relu:0
str(x.name))
Traceback (most recent call last):
File "train.py", line 355, in <module>
main(**vars(args))
File "train.py", line 302, in main
val_func = NNValueFunction(bl,c,layersizes,dropout,deepest,obs_dim) # Initialize the value function
File "/home/ryan/trpo_fractalNN/trpo/value.py", line 37, in __init__
self.model = self._build_model()
File "/home/ryan/trpo_fractalNN/trpo/value.py", line 56, in _build_model
model = Model(inputs=obs_input, outputs=outputs)
File "/home/ryan/.local/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/ryan/.local/lib/python3.6/site-packages/keras/engine/network.py", line 94, in __init__
self._init_graph_network(*args, **kwargs)
File "/home/ryan/.local/lib/python3.6/site-packages/keras/engine/network.py", line 241, in _init_graph_network
self.inputs, self.outputs)
File "/home/ryan/.local/lib/python3.6/site-packages/keras/engine/network.py", line 1511, in _map_graph_network
str(layers_with_complete_input))
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(None, 29), dtype=float32) at layer "input_1". The following previous layers were accessed without issue: []
So here is the part of the code that I'm suspicious of at the moment because of the fact that somehow it is breaking at the very beginning on the value function's neural net.
def _build_model(self):
""" Construct TensorFlow graph, including loss function, init op and train op """
# hid1 layer size is 10x obs_dim, hid3 size is 10, and hid2 is geometric mean
# hid3_units = 5 # 5 chosen empirically on 'Hopper-v1'
# hid2_units = int(np.sqrt(hid1_units * hid3_units))
# heuristic to set learning rate based on NN size (tuned on 'Hopper-v1')
obs = keras.layers.Input(shape=(self.obs_dim,))
# I'm not sure why it won't work with np??????????????????????????????????????????????????????????????????????????????????
obs_input = Dense(int(self.layersizes[0][0].numpy()))(obs) # Initial fully-connected layer that brings obs number up to a len that will work with fractal architecture
obs_input = Activation('relu')(obs_input)
self.lr = 1e-2 / np.sqrt(self.layersizes[2][0]) # 1e-2 empirically determined
print('Value Params -- lr: {:.3g}'
.format(self.lr))
outputs = fractal_net(self,bl=self.bl,c=self.c,layersizes=self.layersizes,
drop_path=0.15,dropout=self.dropout,
deepest=self.deepest)(obs_input)
model = Model(inputs=obs_input, outputs=outputs)
optimizer = Adam(self.lr)
model.compile(optimizer=optimizer, loss='mse')
return model
I found out the issue. The problem was that since I was trying to combine multiple files, I had a 'Dense' call to bring the obs_len to the desired size and then took that and plugged it into the fractalNet code. However, I didn't realize that this would break things. I solved the issue by removing the initial Dense call and placing it inside the fractalNet code itself.
So moral of the story, don't try to break up different parts of the NN layers into separate files. Just as a side comment, In the current fractalNN code, it calls fractal_net and then a Dense layer afterwards and apparently this still works. But I think it breaks things to try to reverse this order. I hope this helps someone else.

Combine Time-series with time-invariant data in RNN/LSTM using Keras Functional API

Update: As rvinas pointed out, I had forgotten to add inputs_aux as second input in Model. Fixed now, and it works. So ConditionalRNN can readily be used to do what I want.
I'd like to treat time-series together with non-time-series characteristics in extended LSTM cells (a requirement also discussed here). ConditionalRNN (cond-rnn) for Tensorflow in Python seems to allow this.
Can it be used in Keras Functional API (without eager execution)?
That is, does anyone have a clue how to fix my failed approach below, or a different example where ConditionalRNN (or alternatives) are used to readily combine TS and non-TS data in LSTM-style cells or any equivalent?
I've seen the eager execution-bare tf example on Pilippe Remy's ConditionalRNN github page, but I did not manage to extend it to a readily fittable version in Keras Functional API.
My code looks as follows; it works if, instead of the ConditionalRNN, I use a standard LSTM cell (and adjust the model 'x' input correspondingly). With ConditionalRNN, I did not get it to execute; I receive either the must feed a value for placeholder tensor 'in_aux' error (cf. below), or instead some different type of input size complaints when I change the code, despite trying to be careful about data dimensions compatibility.
(Using Python 3.6, Tensorflow 2.1, cond-rnn 2.1, on Ubuntu 16.04)
import numpy as np
from tensorflow.keras.models import Model
from tensorflow.keras.layers import LSTM, Dense, Input
from cond_rnn import ConditionalRNN
inputs = Input(name='in',shape=(5,5)) # Each observation has 5 dimensions à 5 time-steps each
x = Dense(64)(inputs)
inputs_aux = Input(name='in_aux', shape=[5]) # For each of the 5 dimensions, a non-time-series observation too
x = ConditionalRNN(7, cell='LSTM')([x,inputs_aux]) # Updated Syntax for cond_rnn v2.1
# x = ConditionalRNN(7, cell='LSTM', cond=inputs_aux)(x) # Syntax for cond_rnn in some version before v2.1
predictions = Dense(1)(x)
model = Model(inputs=[inputs, inputs_aux], outputs=predictions) # With this fix, [inputs, inputs_aux], it now works, solving the issue
#model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='rmsprop', loss='mean_squared_error', metrics=['mse'])
data = np.random.standard_normal([100,5,5]) # Sample of 100 observations with 5 dimensions à 5 time-steps each
data_aux = np.random.standard_normal([100,5]) # Sample of 100 observations with 5 dimensions à only 1 non-time-series value each
labels = np.random.standard_normal(size=[100]) # For each of the 100 obs., a corresponding (single) outcome variable
model.fit([data,data_aux], labels)
The error I get is
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'in_aux' with dtype float and shape [?,5]
[[{{node in_aux}}]]
and the traceback is
Traceback (most recent call last):
File "/home/florian/temp_nonclear/playground/test/est1ls_bare.py", line 20, in <module>
model.fit({'in': data, 'in_aux': data_aux}, labels) #model.fit([data,data_aux], labels) # Also crashes when using model.fit({'in': data, 'in_aux': data_aux}, labels)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training.py", line 643, in fit
use_multiprocessing=use_multiprocessing)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 664, in fit
steps_name='steps_per_epoch')
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 383, in model_iteration
batch_outs = f(ins_batch)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/backend.py", line 3353, in __call__
run_metadata=self.run_metadata)
File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
run_metadata_ptr)
I noticed that you are not passing inputs_aux as input to your model. TF is complaining because this tensor is required to compute your output predictions and it is not being fed with any value. Defining your model as follows should solve the problem:
model = Model(inputs=[inputs, inputs_aux], outputs=predictions)

ValueError from tensorflow estimator RNNClassifier with gcloud ml-engine job

I am working on the task.py file for submitting a gcloud MLEngine job. Previously I was using tensorflow.estimator.DNNClassifier successfully to submit jobs with my data (which consists solely of 8 columns of sequential numerical data for cryptocurrency prices & volume; no categorical).
I have now switched to the tensorflow contrib estimator RNNClassifier. This is my current code for the relevant portion:
def get_feature_columns():
return [
tf.feature_column.numeric_column(feature, shape=(1,))
for feature in column_names[:len(column_names)-1]
]
def build_estimator(config, learning_rate, num_units):
return tf.contrib.estimator.RNNClassifier(
sequence_feature_columns=get_feature_columns(),
num_units=num_units,
cell_type='lstm',
rnn_cell_fn=None,
optimizer=tf.train.AdamOptimizer(learning_rate=learning_rate),
config=config)
estimator = build_estimator(
config=run_config,
learning_rate=args.learning_rate,
num_units=[32, 16])
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
However, I'm getting the following ValueError:
ValueError: All feature_columns must be of type _SequenceDenseColumn. You can wrap a sequence_categorical_column with an embedding_column or indicator_column. Given (type <class 'tensorflow.python.feature_column.feature_column_v2.NumericColumn'>): NumericColumn(key='LTCUSD_close', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None)
I don't understand this, as the data is not categorical.
As #Ben7 pointed out sequence_feature_columns accepts columns like sequence_numeric_column. However, according to the documentation, RNNClassifier sequence_feature_columns expects SparseTensors and sequence_numeric_column is a dense tensor. This seems to be contradictory.
Here is a workaround I used to solve this issue (I took the to_sparse_tensor function from this answer):
def to_sparse_tensor(dense):
# sequence_numeric_column default is float32
zero = tf.constant(0.0, dtype=tf.dtypes.float32)
where = tf.not_equal(dense, zero)
indices = tf.where(where)
values = tf.gather_nd(dense, indices)
return tf.SparseTensor(indices, values, tf.shape(dense, out_type=tf.dtypes.int64))
def get_feature_columns():
return [
tf.feature_column.sequence_numeric_column(feature, shape=(1,), normalizer_fn=to_sparse_tensor)
for feature in column_names[:len(column_names)-1]
]
you got this error because you use a numeric feature column whereas this kind of estimator can only accept sequence feature columns as you can see it on the init function.
So, instead of using numeric column you have to use sequence_numeric_column.

TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

I am using a modified predict.py for testing a pruned SqueezeNet Model
[phung#archlinux SqueezeNet-Pruning]$ python predict.py --image 3_100.jpg --model model_prunned --num_class 2
prediction in progress
Traceback (most recent call last):
File “predict.py”, line 66, in
prediction = predict_image(imagepath)
File “predict.py”, line 52, in predict_image
index = output.data.numpy().argmax()
TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
[phung#archlinux SqueezeNet-Pruning]$
I understand that numpy does not support GPU yet.
How shall I modify the code to get away from this error without invoking tensor copy data operation, tensor.cpu() ?
Change
index = output.data.numpy().argmax()
to
index = output.cpu().data.numpy().argmax()
This means data is first moved to cpu and then converted to numpy array.
I found out that I can just use
output.argmax()
You can use torch.max function like follows:
value, index = torch.max(output,1)
I am facing the same error for the following lines of code
with torch.no_grad():
for idx, (inputs, targets) in enumerate(ploader):
inputs, targets = inputs.to(device), targets.to(device)
outputs = net(inputs)
scores, predicted = outputs.max(1)
# save top1 confidence score
outputs = F.normalize(outputs, dim=1)
probs = F.softmax(outputs, dim=1)
top1_scores.append(probs[0][predicted.item()])
progress_bar(idx, len(ploader))
idx = np.argsort(top1_scores) # Error for this line

TensorFlow in_top_k evaluation input argumants

I am following the tutorial in this link and trying to change the evaluation method for the model (at the bottom). I would like to get a top-5 evaluation and I'm trying to use to following code:
topFiver=tf.nn.in_top_k(y, y_, 5, name=None)
However, this yields the following error:
File "AlexNet.py", line 111, in <module>
topFiver = tf.nn.in_top_k(pred, y, 5, name=None)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 346, in in_top_k
targets=targets, k=k, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 486, in apply_op
_Attr(op_def, input_arg.type_attr))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 59, in _SatisfiesTypeConstraint
", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: DataType float32 for attr 'T' not in list of allowed values: int32, int64
As far as I can tell, the problem is that tf.nn.in_top_k() only works for tf.int32 or tf.int64 data, but my data is in tf.float32 format. Is there any workaround for this?
The targets argument to tf.nn.in_top_k(predictions, targets, k) must be a vector of class IDs (i.e. indices of columns in the predictions matrix). This means that it only works for single-class classification problems.
If your problem is a single-class problem, then I assume that your y_ tensor is a one-hot encoding of the true labels for your examples (for example because you also pass them to an op like tf.nn.softmax_cross_entropy_with_logits(). In that case, you have two options:
If the labels were originally stored as integer labels, pass them directly to tf.nn.in_top_k() without converting them to one-hot. (Also, consider using tf.nn.sparse_softmax_cross_entropy_with_logits() as your loss function, because it may be more efficient.)
If the labels were originally stored in the one-hot format, you can convert them to integers using tf.argmax():
labels = tf.argmax(y_, 1)
topFiver = tf.nn.in_top_k(y, labels, 5)