So apparently the Yann LeCun's website is down so the following lines for reading mnist with tensorflow don't seem to be working :
FROM tensorflow.examples.tutorials.mnist IMPORT input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot = true)
Any ideas how can i read the mnist without using these above lines?
You can access the website here: https://web.archive.org/web/20160117040036/http://yann.lecun.com/exdb/mnist/ - download the data, and read it in from a local copy...
Edit
Here is a code example for reading a local mnist dataset
You can download the dataset individually and put it in the directory you created.Then the code can run normally.
http://yann.lecun.com/exdb/mnist/
Ok guys i fixed my problem,my mistake was that i was extracting the files(to the \MNIST_data directory) from the .gz files where i should have left them as they were unextracted instead.Thanks for the help.
Related
Summary
My question is composed by:
A context in which I present my project, my working environment and my workflow
The detailed problem
The concerned parts of my code
The solutions I tried to solve my problem
The question reminder
Context
I've written a Python Keras implementation of a downgraded version of the original Super-Resolution GAN. Now I want to test it using Google Firebase Machine Learning Kit, by hosting it in the Google servers. That's why I have to convert my Keras program to a TensorFlow Lite one.
Environment and workflow (with the problem)
I'm training my program on Google Colab working environment: there, I've installed TF 2.0.0-beta1 (this choice is motivated by this uncorrect answer: https://datascience.stackexchange.com/a/57408/78409).
Workflow (and problem):
I write locally my Python Keras program, keeping in mind that it will run on TF 2. So I use TF 2 imports, for example: from tensorflow.keras.optimizers import Adam and also from tensorflow.keras.layers import Conv2D, BatchNormalization
I send my code to my Drive
I run without any problem my Google Colab Notebook: TF 2 is used.
I get the output model in my Drive, and I download it.
I try to convert this model to the TFLite format by executing the following CLI: tflite_convert --output_file=srgan.tflite --keras_model_file=srgan.h5: here the problem appears.
The problem
Instead of outputing the TF Lite converted model from the TF (Keras) model, the previous CLI outputs this error:
ValueError: Unknown loss function:build_vgg19_loss_network
The function build_vgg19_loss_network is a custom loss function that I've implemented and that must be used by the GAN.
Parts of code that rise this problem
Presenting the custom loss function
The custom loss function is implemented like that:
def build_vgg19_loss_network(ground_truth_image, predicted_image):
loss_model = Vgg19Loss.define_loss_model(high_resolution_shape)
return mean(square(loss_model(ground_truth_image) - loss_model(predicted_image)))
Compiling the generator network with my custom loss function
generator_model.compile(optimizer=the_optimizer, loss=build_vgg19_loss_network)
What I've tried to do in order to solve the problem
As I read it on StackOverflow (link at the beginning of this question), TF 2 was thought to be sufficient to output a Keras model which would be correctly processed by my tflite_convert CLI. But it's not, obviously.
As I read it on GitHub, I tried to manually set my custom loss function among Keras' loss functions, by adding these lines: import tensorflow.keras.losses
tensorflow.keras.losses.build_vgg19_loss_network = build_vgg19_loss_network. It didn't work.
I read on GitHub I could use custom objects with load_model Keras function: but I only want to use compile Keras function. Not load_model.
My final question
I want to do only minor changes to my code, since it works fine. So I don't want, for example, to replace compile with load_model. With this constraint, could you help me, please, to make my CLI tflite_convert works with my custom loss function?
Since you are claiming that TFLite conversion is failing due to a custom loss function, you can save the model file without keep the optimizer details. To do that, set include_optimizer parameter to False as shown below:
model.save('model.h5', include_optimizer=False)
Now, if all the layers inside your model are convertible, they should get converted into TFLite file.
Edit:
You can then convert the h5 file like this:
import tensorflow as tf
model = tf.keras.models.load_model('model.h5') # srgan.h5 for you
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
Usual practice to overcome the unsupported operators in TFLite conversion is documented here.
I had the same error. I recommend changing the loss to "mse" since you already have a well-trained model and you don't need to train with the .tflite file.
I'm not familiar with using python as a ML tool and wanted to train the MNIST data set. I have downloaded the MNIST library using
pip install python-mnist but do not know what my next step should be. What would an import statement look like? Should I also import TensorFlow and or Keras to train the data?
I know the MNIST dataset is available in TensorFlow and Keras, however, importing via pip a necessary solution for my use case. Both TensorFlow and Keras have tutorials using the MNIST data set, but I was wondering if it is possible to use the data set without using their pre-downloaded library.
The import statement should look like this
from mnist import MNIST
mndata = MNIST('./dir_with_mnist_data_files')
images, labels = mndata.load_training()
then you can work directly with the arrays of raw images and labels.
I've beeen trying out the Tensorflow 2 alpha and I have been trying to freeze and export a model to a .pb graphdef file.
In Tensorflow 1 I could do something like this:
# Freeze the graph.
frozen_graph_def = tf.graph_util.convert_variables_to_constants(
sess,
sess.graph_def,
output_node_names)
# Save the frozen graph to .pb file.
with open('model.pb', 'wb') as f:
f.write(frozen_graph_def.SerializeToString())
However this doesn't seem possible anymore as convert_variables_to_constants is removed and use of sessions is discouraged.
I looked and found there is the freeze graph util
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py that works with SavedModel exports.
Is there some way to do it within Python still or I am meant to switch and use this tool now?
I have also faced this same problem while migrating from tensorflow1.x to tensoflow2.0 beta.
This problem can be solved by 2 methods:
1st is to go to the tensflow2.0 docs search for the methods you have used and change the syntax for each line &
To use google's tf_ugrade_v2 script
tf_upgrade_v2 --infile your_tf1_script_file --outfile converted_tf2_file
You try above command to change your tensorflow1.x script to tensorflow2.0, it will solve all your problem.
Also, you can rename the method (Manual step by refering documentation)
Rename 'tf.graph_util.convert_variables_to_constants' to 'tf.compat.v1.graph_util.convert_variables_to_constants'
The measure problem is that in tensorflow2.0 is that many syntax and function has changed try referring the tensoflow2.0 docs or use the google's tf_upgrade_v2 script
Not sure if you've seen this Tensorflow 2.0 issue, but this response seems to be a work-around:
https://github.com/tensorflow/tensorflow/issues/29253#issuecomment-530782763
Note: this hasn't worked for my nlp model but maybe it will work for you. The suggested work-around is to use model.save_weights('weights.h5') while in TF 2.0 environment. Then create new environment with TF 1.14 and do all following steps in TF 1.14 env. Build your model model = create_model() and use model.load_weights('weights.h5') to load weights back into your model. Then save entire model with model.save('final_model.h5'). If you manage to have success with the above steps, then follow the rest of the steps in the link to use freeze_graph.
one can use:
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST/", one_hot=True)
But I don't want to download dataset everytime. What is the best way to download the dataset ONLY in case if it is not in MNIST/?
I tried: if not os.path.isdir("MNIST/"): but this way the mnist is not initialised. I want to use this input_data but only its reading ability without the download.
If you don't want to download data-set everytime, you can initialize the first parameter with absolute path, just like this:
input_data.read_data_sets("/your/absolute/path/", one_hot=True)
The function will not download the file if the file is detected. You can debug the function, and you will know. Also, you could modify the code in your way, using the part which you really need.
For running without downloading MNIST data, do following (it worked for me)
input_data.read_data_set('--absolute-path-of-MNIST-DATASET--')
I'm new in machine learning and I am following tensorflow's tutorial to create some simple Neural Networks which learn the MNIST data.
i wanna run a code that do the recognition hand writing digits using the MNIST data but i don't know how to run it ... should i dowload the data on my machine and extracted and put it on a file and then set the path on the code or did tensorflow contain the data ...but when i do import input_data i get
No module named 'input_data' also when i do
from tensorflow.examples.tutorials.mnist import input_data ==> No module named 'tensorflow.examples'
ps:when i do import tensorflow as tf i get no erreur so it's fine with tensorflow i think
could u help me plz for example i wanna run the code below what should i do
https://github.com/hwalsuklee/tensorflow-mnist-cnn
If you cannot import tensorflow.examples I'm guessing something went wrong with the installation. Try reinstalling tensorflow with the latest version.
You don't need to download the data on your own, tensorflow will put it in the path you provide. But first, try these steps:
I'm currently using tf 1.2.0 and I'm not getting that error.
If you want to know which version you have installed:
import tensorflow as tf
print(tf.__version__)
After everything is installed try:
from tensorflow.examples.tutorials.mnist import input_data
input_data.read_data_sets("./data/", one_hot=True)
That should copy the data to a "data" folder inside your working folder (the "data" folder will be created and all the files will be available there).
If the above lines of code run with no errors, you should be able to run the example.