Change number of classes of MNIST Tensorflow - tensorflow

Hi, Im trying to adapt the beginners tutorial of Tensorflow with MNIST and softmax. In the tutorial you have 10 clases (for digits 0-9).
Now, with a different dataset (EMNIST) I have 62 classes for digits and letters.
What I have in the model of the orginal example is:
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.matmul(x, W) + b`
Where 784 stands for the total pixels of a 28x28 image and 10 is the number of classes. What I want is:
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 62]))
b = tf.Variable(tf.zeros([62]))
y = tf.matmul(x, W) + b`
For 62 classes.
But when I reach this part of the code, where the next batch is called for execution:
for _ in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
I get this error...
Traceback (most recent call last):
File "calligraphy.py", line 77, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "C:\Users\Willy Barales\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "calligraphy.py", line 64, in main
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
File "C:\Users\Willy Barales\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 789, in run
run_metadata_ptr)
File "C:\Users\Willy Barales\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 975, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (100, 10) for Tensor 'Placeholder_1:0', which has shape '(?, 62)'
Any ideas on how to change the dataset for this example?
Do I have to change something in the mnist.py file where .next_batch() is implemented?
As far as I know, EMNIST has the exact same format as MNIST.
Thanks in advance.
Info on the new dataset:
http://biometrics.nist.gov/cs_links/EMNIST/Readme.txt

All I have to do was to edit in the mnist.py file the part where the one hot vectors were created from labels, since those are the ones corresponding to batch_ys, thanks to the enlightment of Neijla.
def extract_labels(f, one_hot=False, num_classes=62)
Besides of course, changing the number of the classes in the model as I stated first in my question.

Related

Trouble with TensorFlow and MNIST recognition

Beforehand, I thank you for analyzing my post and helping out. I've recently gotten interested in ML with Tensorflow,
but I've encountered a problem with my code. I'm reading a book called Learning TensorFlow, and I've written out the whole thing
from the first example. They are analyzing MNIST images, and I've also added my own comments with my perspective on how things work
in the code. When I run the code, however, I get an error. Here's my code, and the error.
#Import tensorflow under the name of ts
import tensorflow as tf
#Import MNIST tutorial data from tensorflow
from tensorflow.examples.tutorials.mnist import input_data
#Declare constants
#Data path
DATA_DIR = 'C:/tmp/data'
#Number of steps
NUM_STEPS = 1000
#Number of examples per step
MINIBATCH_SIZE = 100
#When we read the data-set it saves it locally under our data path, or under c:/tmp/data
data = input_data.read_data_sets(DATA_DIR, one_hot = True)
#Our placeholder X is the image. Placeholders are supplied when running the computation graph
x = tf.placeholder(tf.float32, [None, 784])
#Create a variable representing the weights. Variables are manipulated by the computation graph
W = tf.Variable(tf.zeros([784, 10]))
y_true = tf.placeholder(tf.float32, [None, 784])
y_pred = tf.matmul(x, W)
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
logits=y_pred, labels=y_true))
gd_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_mask = tf.equal(tf.argmax(y_pred, 1), tf.argmax(y_true, 1))
accuracy = tf.reduce_mean(tf.cast(correct_mask, tf.float32))
with tf.Session() as sess:
#Initialize global variables
sess.run(tf.global_variables_initializer())
for _ in range(NUM_STEPS):
batch_xs, batch_ys = data.train.next_batch(MINIBATCH_SIZE)
sess.run(gd_step, feed_dict={x: batch_xs, y_true: batch_ys})
ans = sess.run(accuracy, feed_dict={x: data.test.images,
y_true: data.test.labels})
print("Accuracy: {:.4}%".format(ans*100))
Now here's the error.
runfile('C:/Users/user/.spyder-py3/temp.py', wdir='C:/Users/user/.spyder-py3')
Extracting C:/tmp/data\train-images-idx3-ubyte.gz
Extracting C:/tmp/data\train-labels-idx1-ubyte.gz
Extracting C:/tmp/data\t10k-images-idx3-ubyte.gz
Extracting C:/tmp/data\t10k-labels-idx1-ubyte.gz
Traceback (most recent call last):
File "<ipython-input-11-bf503334b166>", line 1, in <module>
runfile('C:/Users/user/.spyder-py3/temp.py', wdir='C:/Users/CwWJc/.spyder-py3')
File "C:\Users\user\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)
File "C:\Users\user\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/user/.spyder-py3/temp.py", line 38, in <module>
sess.run(gd_step, feed_dict={x: batch_xs, y_true: batch_ys})
File "C:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
run_metadata_ptr)
File "C:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1149, in _run
str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (100, 10) for Tensor
'Placeholder_15:0', which has shape '(?, 784)'
Any help is greatly appreciated. Sorry if I'm making a stupid mistake. I find that I often do, though. Thanks in advance! Also, sorry for garbage formatting. :)
Hahaha! I got y_true mixed up. Sorry for the hassle everyone.

Confusion Matrix with Tensorflow

I am using finetune AlexNet architecture written by #kratzert on my own dataset which, works properly (I got the code from here: https://github.com/kratzert/finetune_alexnet_with_tensorflow) and I want to figure out how to build confusion matrix from his code. I have tried to use tf.confusion_matrix(labels, predictions, num_classes) to build confusion matrix but I can't. I am confused what should be the values for labels and predictions, I mean, I know what should be but each time I feed these value got an error. Can anyone help me on this or have a look at the code (above link) and guide me?
I added these two lines in finetune.py exactly after calculating accuracy to make the labels and the predictions as the number of the class.
with tf.name_scope("accuracy"):
correct_pred = tf.equal(tf.argmax(score, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))
**true_class = tf.argmax(y, 1)
predicted_class = tf.argmax(score, 1)**
and I have added tf.confusion_matrix() inside my session at the very bottom before saving checkpoint of the model
for _ in range(val_batches_per_epoch):
img_batch, label_batch = sess.run(next_batch)
acc, cost = sess.run([accuracy, loss], feed_dict={x: img_batch,
y: label_batch,
keep_prob: 1.})
test_acc += acc
test_count += 1
test_acc /= test_count
print("{} Validation Accuracy = {:.4f} -- Validation Loss = {:.4f}".format(datetime.now(),test_acc, cost))
print("{} Saving checkpoint of model...".format(datetime.now()))
**print(sess.run(tf.confusion_matrix(true_class, predicted_class, num_classes)))**
# save checkpoint of the model
checkpoint_name = os.path.join(checkpoint_path,
'model_epoch'+str(epoch+1)+'.ckpt')
save_path = saver.save(sess, checkpoint_name)
print("{} Model checkpoint saved at {}".format(datetime.now(),
checkpoint_name))
I have tried other places as well but each time I will get an error:
Caused by op 'Placeholder_1', defined at:
File "/home/armin/Desktop/Alexnet_DataPipeline/finetune.py", line 85, in <module>
y = tf.placeholder(tf.float32, [batch_size, num_classes])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_ops.py", line 1777, in placeholder
return gen_array_ops.placeholder(dtype=dtype, shape=shape, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 4521, in placeholder
"Placeholder", dtype=dtype, shape=shape, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3290, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1654, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder_1' with dtype float and shape [128,3]
any help will be appreciated, Thanks.
It's a fairly long piece of code you're referring to, and you did not specify where you put your confusion matrix line.
Just by experience, the most frequent problem with confusion matrices is that tf.confusion_matrix() requires both the labels and the predictions as the number of the class, not as one-hot vectors. In other words, the label and the prediction should be in the form of the number 5 instead of [ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0 ].
In the code you refer to, y is in the one-hot format. The output of the network, score is a vector, giving the probability of each class. That is also not the required format. You need to do something like
true_class = tf.argmax( y, 1 )
predicted_class = tf.argmax( score, 1 )
and use those with the confusion matrix like
tf.confusion_matrix( true_class, predicted_class, num_classes )
(Basically, if you take a look at line 123 of finetune.py, that has both of those elements for determining accuracy, but they are not saved in separate tensors.)
If you want to keep a running total of confusion matrices of all batches, you just have to add them up - since each cell of the matrix counts the number of examples falling into that category, an element-wise addition creates the confusion matrix for the whole set:
cm_running_total = None
cm_nupmy_array = sess.run(tf.confusion_matrix(true_class, predicted_class, num_classes), feed_dict={x: img_batch, y: label_batch, keep_prob: 1.} )
if cm_running_total is None:
cm_running_total = cm_numpy_array
else:
cm_running_total += cm_numpy_array

Tensorflow won't matmul inputs and weights. "Dimensions must be equal"

I've been working on a simple tensor flow neural network. My input placeholder is
x = tf.placeholder(tf.float32, shape=[None, 52000, 3]).
My weight matrix is initialized to all zeros as
W = tf.Variable(tf.zeros([52000, 10])).
I tried different combinations with and without the 3 for color channels, but I guess I'm just not understanding the dimensionality because I got the error:
Traceback (most recent call last): File
"C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py",
line 686, in _call_cpp_shape_fn_impl
input_tensors_as_shapes, status) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py",
line 473, in exit
c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape
must be rank 2 but is rank 3 for 'MatMul' (op: 'MatMul') with input
shapes: [?,52000,3], [52000,10].
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "rating.py", line 65, in
y = tf.matmul(x, W) + b # "fake" outputs to train/test File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\math_ops.py",
line 1891, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name) File
"C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\gen_math_ops.py",
line 2436, in _mat_mul
name=name) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\op_def_library.py",
line 787, in _apply_op_helper
op_def=op_def) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py",
line 2958, in create_op
set_shapes_for_outputs(ret) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py",
line 2209, in set_shapes_for_outputs
shapes = shape_func(op) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py",
line 2159, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py",
line 627, in call_cpp_shape_fn
require_shape_fn) File "C:\Users\Everybody\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\common_shapes.py",
line 691, in _call_cpp_shape_fn_impl
raise ValueError(err.message) ValueError: Shape must be rank 2 but is rank 3 for 'MatMul' (op: 'MatMul') with input shapes: [?,52000,3],
[52000,10].
At first, I thought my next_batch() function was the culprit because I had to make my own due to the fact that I uploaded my images "manually" using scipy.misc.imread(), whose definition reads:
q = 0
def next_batch(batch_size):
x = images[q:q + batch_size]
y = one_hots[q:q + batch_size]
q = (q + batch_size) % len(images)
return x, y
However, after looking through, I don't see what's wrong with this, so I imagine that I'm just confused about dimensionality. It is supposed to be a "flattened" 200x260 color image. It just occurred to me now that maybe I have to flatten the color channels as well? I will place my full code below if curious. I'm a bit new to Tensorflow, so thanks, all. (Yes, it is not a CNN yet, I decided to start simple just to make sure I'm importing my dataset right. And, I know it is tiny, I'm starting my dataset small too.)
############# IMPORT DEPENDENCIES ####################################
import tensorflow as tf
sess = tf.InteractiveSession() #start session
import scipy.misc
import numpy as np
######################################################################
#SET UP DATA #########################################################
images = []
one_hots = []
########### IMAGES ##################################################
#put all the images in a list
for i in range(60):
images.append(scipy.misc.imread('./shoes/%s.jpg' % str(i+1)))
print("One image appended...\n")
#normalize them, "divide" by 255
for image in images:
print("One image normalized...\n")
for i in range(260):
for j in range(200):
for c in range(3):
image[i][j][c]/=255
for image in images:
tf.reshape(image, [52000, 3])
########################################################################
################# ONE-HOT VECTORS ######################################
f = open('rateVectors.txt')
lines = f.readlines()
for i in range(0, 600, 10):
fillerlist = []
for j in range(10):
fillerlist.append(float(lines[i+j][:-1]))
one_hots.append(fillerlist)
print("One one-hot vector added...\n")
########################################################################3
#set placeholders and such for input, output, weights, biases
x = tf.placeholder(tf.float32, shape=[None, 52000, 3])
y_ = tf.placeholder(tf.float32, shape=[None, 10])
W = tf.Variable(tf.zeros([52000, 10])) # These are our weights and biases
b = tf.Variable(tf.zeros([10])) # initialized as zeroes.
#########################################################################
sess.run(tf.global_variables_initializer()) #initialize variables in the session
y = tf.matmul(x, W) + b # "fake" outputs to train/test
##################### DEFINING OUR MODEL ####################################
#our loss function
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y, y_))
#defining our training as gradient descent
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
###################### TRAINING #############################################
#################### OUR CUSTOM BATCH FUNCTION ##############################
q = 0
def next_batch(batch_size):
x = images[q:q + batch_size]
y = one_hots[q:q + batch_size]
q = (q + batch_size) % len(images)
return x, y
#train
for i in range(6):
batch = next_batch(10)
train_step.run(feed_dict={x: batch[0], y_: batch[1]})
print("Batch Number: " + i + "\n")
print("Done training...\n")
################ RESULTS #################################################
#calculating accuracy
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
#print accuracy
print(accuracy.eval(feed_dict={x: images, y_: one_hots}))
Your placeholder should have the dimension [None, 200, 260, 3] where None is the batch size, 200, 260 is the image size, and 3 is the channels.
Your weights should be [filter_height, filter_width, num_channels, num_filters]
Your bias should be [num_filters]
And the dimensions for the labels should be [None, num_classes] where None is the batch size, and num_classes is the number of classes that your images have.
These are just to make sure that math works.
I took these codes from here

Applying gradients with feed_dict for gradients

I want to do some non-tensorflow processing on the computed gradients, before applying them on the variables.
My plan was to run the gradient ops that I get from the compute_gradients function , do my processing (in python without tensorflow), and then run the apply operation I get from the apply_gradients function and feed the processed gradients in the feed_dict. Unfortunately, this doesn't work in my scenario.
I managed to narrow it down to some issue with tf.nn.embedding_lookup (same happens with tf.gather), and the error can be reproduced as follows (using tf1.4):
import tensorflow as tf
x = tf.placeholder(dtype=tf.float32, shape=[])
z = tf.placeholder(dtype=tf.int32, shape=[])
emb_mat = tf.get_variable('w', [100, 5], initializer=tf.truncated_normal_initializer(stddev=0.1))
emb = tf.nn.embedding_lookup(emb_mat, z)
loss = x - tf.reduce_sum(emb) # Just some silly loss
opt = tf.train.GradientDescentOptimizer(0.1)
grads_and_vars = opt.compute_gradients(loss, tf.trainable_variables())
train_op = opt.apply_gradients(grads_and_vars)
grads = [g for g,v in grads_and_vars]
tsess = tf.Session()
tsess.run(tf.global_variables_initializer())
gradsres = tsess.run(grads, {x: 1.0, z: 1})
tsess.run(train_op, {g:r for g,r in zip(grads, gradsres)})
which results in the error
Traceback (most recent call last):
File "/home/cruvadom/.p2/pool/plugins/org.python.pydev_6.0.0.201709191431/pysrc/_pydevd_bundle/pydevd_exec.py", line 3, in Exec
exec exp in global_vars, local_vars
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1098, in _run
raise ValueError('Tensor %s may not be fed.' % subfeed_t)
ValueError: Tensor Tensor("gradients/Gather_1_grad/ToInt32:0", shape=(2,), dtype=int32, device=/device:GPU:0) may not be fed.
It seems there is some additional tensor I need to feed to the graph for the computation. What is the right way to do why I want to do?
Thanks!
If you run the training operation, it will automatically calculate the gradients. You can retrieve the gradients from the session:
tsess = tf.Session()
tsess.run(tf.global_variables_initializer())
_, grads_and_vars, loss = tsess.run([train_op ,grads, loss], {x: 1.0, z: 1})
assert not np.isnan(loss), 'Something wrong! loss is nan...'
#Get the gradients
for g, v in grads_and_vars:
if g is not None:
grad_hist_summary = tf.summary.histogram("{}/grad_histogram".format(v.name), g)
sparsity_summary = tf.summary.scalar("{}/grad/sparsity".format(v.name), tf.nn.zero_fraction(g))

TensorFlow shape error in feed_dict

I'm trying to adapt existing logreg example to my data and are getting the following error:
Epoch: 0001 cost=
Traceback (most recent call last):
File "tflin.py", line 64, in <module>
print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(sess.run(cost, feed_dict={X: train_X, Y:train_Y})), \
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 315, in run
return self._run(None, fetches, feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 506, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (60000, 6) for Tensor u'Placeholder:0', which has shape '(6,)'
Source code can be found here: https://github.com/ilautar/tensorflow-test/blob/master/tflin.py
I'm sure it is obvious, any pointers?
Thank you,
Igor
The error occurs because you are trying to feed a 60000 x 6 matrix into a tf.placeholder() that is defined to be a vector of length 6. This happens when you try to feed the whole train_X matrix (as opposed to feeding a single row, which succeeds).
The best way to make this work is to do the following:
Define your placeholders (and model) in terms of batch inputs, which can have varying shape:
# tf Graph Input
X = tf.placeholder(tf.float32, [None, n_input])
Y = tf.placeholder(tf.float32, [None])
When feeding in a single example, extend it to be a 1 x 6 matrix using numpy.newaxis:
# Fit all training data
for epoch in range(training_epochs):
for (x, y) in zip(train_X, train_Y):
sess.run(optimizer, feed_dict={X: x[numpy.newaxis, ...],
Y: y[numpy.newaxis, ...]})