I downloaded a copy of mnist data from a random website because the official website was down.
Which directory can I put the mnist file for tensorflow's input_data method to pick it up
Maybe this Link help you.
Another way to use MNIST dataset: some Frameworks like Tensorflow and Keras has this dataset. So you can use this:
# Import MNIST
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
# Load data
X_train = mnist.train.images
Y_train = mnist.train.labels
X_test = mnist.test.images
Y_test = mnist.test.labels
The sample example in here:
In Keras also its same and you can import and use. Both Frameworks download it first and then just need to import.
Related
I tried to import some examples from mnist in tensorflow version 2.
But the examples are all about tensorflow version 1 and I modifed my tensorflow using
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
here, I tried to use
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
but ModuleNotFoundError: No module named 'tensorflow.examples' appeared in my VSCode terminal, so I tried with
from keras import Sequential
from tensorflow import keras
import os
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
In this way, data loading was successful but the example code,
batch_xs, batch_ys = mnist.next_batch(100)
did not work.
To get the tutorial folder in my tensorflow, I also tried to download from github, but there was no tutorial folder according to ModuleNotFoundError: No module named 'tensorflow.examples'
However, there was no folder named tensorflow_core in my C:\Users\User\anaconda3\envs\tensorflow\Lib\site-packages
help me
I have followed this TensorFlow tutorial to classify images using transfer learning approach. Using almost 16,000 manually classified images (with about 40/60 split of 1/0) added on top of the pre-trained MobileNet V2 model, my model achieved 96% accuracy on the hold out test set. I then saved the resulting model.
Next, I would like to use this trained model to classify new images. To do so, I have adapted one of the portions of the tutorial's code (in the end where it says #Retrieve a batch of images from the test set) in the way described below. The code works, however, it only processes one batch of 32 images and that's it (there are hundreds of images in the source folder). What am I missing here? Please advise.
# Import libraries
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import preprocessing
from tensorflow.keras.preprocessing import image_dataset_from_directory
import matplotlib.pyplot as plt
import numpy as np
import os
# Load saved model
model = tf.keras.models.load_model('/model')
# Re-compile model
base_learning_rate = 0.0001
model.compile(optimizer=tf.keras.optimizers.Adam(lr=base_learning_rate),
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=['accuracy'])
# Define paths
PATH = 'Data/'
new_dir = os.path.join(PATH, 'New_images') # New_images must contain at least one class (sub-folder)
IMG_SIZE = (640, 640)
BATCH_SIZE = 32
new_dataset = image_dataset_from_directory(new_dir, shuffle=True, batch_size=BATCH_SIZE, image_size=IMG_SIZE)
# Retrieve a batch of images from the test set
image_batch, label_batch = new_dataset.as_numpy_iterator().next()
predictions = model.predict_on_batch(image_batch).flatten()
# Apply a sigmoid since our model returns logits
predictions = tf.nn.sigmoid(predictions)
predictions = tf.where(predictions < 0.5, 0, 1)
print('Predictions:\n', predictions.numpy())
len(new_dataset) # equals 25, i.e., there are 25 batches
Replace this code:
# Retrieve a batch of images from the test set
image_batch, label_batch = new_dataset.as_numpy_iterator().next()
predictions = model.predict_on_batch(image_batch).flatten()
with this one:
predictions = model.predict(new_dataset,batch_size=BATCH_SIZE).flatten()
tf.data.Dataset objects can be directly passed to the method predict(). Reference
I am trying to solve an exercise from a machine learning book, where a classifier should be trained on the cifar10 dataset using tensorflow and keras. I have attached a code example. The code is running in a Jupyter notebook inside PyCharm.
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import time
import os
import tensorflow as tf
from tensorflow import keras
tf.random.set_seed(42)
np.random.seed(42)
def build_model():
# Build a model as instructed
model = keras.Sequential()
model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))
for i in range(20):
model.add(keras.layers.Dense(100,
activation=keras.activations.elu,
kernel_initializer=tf.keras.initializers.HeNormal))
model.add(keras.layers.Dense(10, activation=keras.activations.softmax))
return model
model = build_model()
# Load the CIFAR10 image dataset
cifar10 = keras.datasets.cifar10.load_data()
X_train = cifar10[0][0] / 255.
X_test = cifar10[1][0] / 255.
y_train = cifar10[0][1]
y_test = cifar10[1][1]
print(X_train.max())
from sklearn.model_selection import train_test_split
# Split the data
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.1, shuffle= True)
model = build_model()
root_logdir = os.path.join(os.curdir, "my_logs_11-8")
def get_run_logdir():
run_id = time.strftime("run_%Y_%m_%d_%H_%M_%S")
return os.path.join(root_logdir, run_id)
run_logdir = get_run_logdir()
tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)
model.compile(optimizer=keras.optimizers.Nadam(lr=5e-5), loss="sparse_categorical_crossentropy")
history = model.fit(X_train, y_train, epochs=100,
validation_data=(X_valid, y_valid),
callbacks=[tensorboard_cb],
batch_size=32)
I am already running into problems here:
I do not get past a few epochs (10-20), because with each epoch, the python process takes up more and more memory. My machine has 16 GB, of which 12 GB are usually free. When memory is full, my IDE (PyCharm) crashes and the python process is killed. Is that due to a memory leak? What can I do to fix it?
For each epoch, keras measures an estimated time of how long that epoch took. However, the time measured is much smaller than walltime (24s/epoch vs. ~60s/epoch). The book I am following seems to reach much faster training on a much weaker machine. How can this be?
If I want to implement a classifier using the sklearn library. Is there a way to save the model or convert the file into a saved tensorflow file in order to convert it to tensorflow lite later?
If you replicate the architecture in TensorFlow, which will be pretty easy given that scikit-learn models are usually rather simple, you can explicitly assign the parameters from the learned scikit-learn models to TensorFlow layers.
Here is an example with logistic regression turned into a single dense layer:
import tensorflow as tf
import numpy as np
from sklearn.linear_model import LogisticRegression
# some random data to train and test on
x = np.random.normal(size=(60, 21))
y = np.random.uniform(size=(60,)) > 0.5
# fit the sklearn model on the data
sklearn_model = LogisticRegression().fit(x, y)
# create a TF model with the same architecture
tf_model = tf.keras.models.Sequential()
tf_model.add(tf.keras.Input(shape=(21,)))
tf_model.add(tf.keras.layers.Dense(1))
# assign the parameters from sklearn to the TF model
tf_model.layers[0].weights[0].assign(sklearn_model.coef_.transpose())
tf_model.layers[0].bias.assign(sklearn_model.intercept_)
# verify the models do the same prediction
assert np.all((tf_model(x) > 0)[:, 0].numpy() == sklearn_model.predict(x))
It is not always easy to replicate a scikit model in tensorflow. For instance scitik has a lot of on the fly imputation libraries which will be a bit tricky to implement in tensorflow
I have a code in Keras (or its TF version). I want to have a CUDA code which is equivalence to it. Is there a way to get it?
I know that from Keras I can look at the basic graph topology using the following code:
# LSTM for sequence classification in the IMDB dataset
import numpy
from keras.datasets import imdb
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers.embeddings import Embedding
from keras import backend as K
from keras.preprocessing import sequence
# fix random seed for reproducibility
numpy.random.seed(7)
# load the dataset but only keep the top n words, zero the rest
top_words = 5000
max_review_length = 500
# create the model
embedding_vecor_length = 32
model = Sequential()
model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
g = K.get_session().graph
# GIVES THE GRAPH TOPOLOGY!:
graph_def = g.as_graph_def()
Is there a way to have the .cc file that represent this code?
Thanks!
There is no functionality in TensorFlow to generate C++ CUDA source code from a graph, but the XLA framework supports ahead-of-time compilation, which generates efficient bytecode from your TensorFlow graph, which you can then execute on your CUDA-capable GPU.