I have trying to import resnet34 and resent50 into a Kaggle Kernal with no success. When I run:
from keras.applications.resnet50 import ResNet50 as resnet50
resnet = ResNet50(weights='imagenet')
learn = ConvLearner.pretrained(resnet, data, precompute=True)
It starts downloading : 'Downloading: "https://download.pytorch.org/models/resnet34-333f7ec4.pth" to /tmp/.torch/models/resnet34-333f7ec4.pth'
but says "URL fetch failure on https://github.com/fchollet/deep-learning-models/releases/download/v0.2/resnet50_weights_tf_dim_ordering_tf_kernels.h5: None -- [Errno -2] Name or service not known"
and with resnet34:
arch=resnet34
learn = ConvLearner.pretrained(arch, data, precompute=True)
I get: URLError: <urlopen error [Errno -2] Name or service not known>
Others seem to be importing using these same line.
You're getting this error because Kaggle kernels don't (currently) have internet access, so you're not able to fetch things via URL.
You can add these models to your kernel by adding the relevent datasets (linked below) and then reading them in as you would any other file from the file path "../input/[name_of_dataset]/[name_of_file]". (You should replace [name_of_dataset] and [name_of_file] with the actual names of your dataset and desired file, of course. :)
ResNet50 dataset
ResNet34 dataset
Hope that helps!
Related
I have used Recommenders https://github.com/microsoft/recommenders library to train an NCF recommendation model. Currently I'm getting issues in deployment through Amazon TensorflowModel library
Model is saved using the following code
def save(self, dir_name):
"""Save model parameters in `dir_name`
Args:
dir_name (str): directory name, which should be a folder name instead of file name
we will create a new directory if not existing.
"""
# save trained model
if not os.path.exists(dir_name):
os.makedirs(dir_name)
saver = tf.compat.v1.train.Saver()
saver.save(self.sess, os.path.join(dir_name, MODEL_CHECKPOINT))
Files exported in the process are 'checkpoint', 'model.ckpt.data-00000-of-00001', 'model.ckpt.index', 'model.ckpt.meta'
They follow the structure of
- model.tar.gz
- 00000000
- checkpoint
- model.ckpt.data-00000-of-00001
- model.ckpt.index
- model.ckpt.meta
I have tried various deployment processes, however they all give the same error. Here's the latest one that I implemented following this example https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-script-mode/pytorch_bert/code/inference_code.py
from sagemaker.tensorflow.model import TensorFlowModel
model = TensorFlowModel(
entry_point="tf_inference.py",
model_data=zipped_model_path,
role=role,
model_version='1',
framework_version="2.7"
)
predictor = model.deploy(
initial_instance_count=1, instance_type="ml.g4dn.2xlarge", endpoint_name='endpoint-name3'
)
All Solutions end with the same error over and over again
Traceback (most recent call last):
File "/sagemaker/serve.py", line 502, in <module>
ServiceManager().start()
File "/sagemaker/serve.py", line 482, in start
self._create_tfs_config()
File "/sagemaker/serve.py", line 153, in _create_tfs_config
raise ValueError("no SavedModel bundles found!")
These 2 links helped me resolve the issue
https://github.com/aws/sagemaker-python-sdk/issues/599
https://www.tensorflow.org/guide/migrate/saved_model#1_save_the_graph_as_a_savedmodel_with_savedmodelbuilder
Sagemaker has weird directory structure that you need to strictly follow. The first one shares the starting directories and 2nd one shares the process of saving the model for TF1 and TF2
I am trying to create several LSTM models for time series prediction (e.g. Vanilla, Stacked, Bidirectional). After creating a model I want to save it using tf.keras.models.save_model
This works fine for the LSTM architectures I describe above, but when trying to save a ConvLSTM model I get the following error: ValueError: Object dictionary contained a non-trackable object: (None, None) (for key states)
I am using Keras with backend TensorFlow (2.X) on a Colab notebook. I've created a notebook where the problem can be reproduced.
Any help would be appreciated!
Edit: the model should be saved in the Tensorflow SavedModel format (save_format='tf')
There are two ways you can save your model.
model.save('model.h5')
Your method but you're missing model name and extension.
Go to gdrive directory using cd.
% cd /content/gdrive
Save with the file name and extension.
# save model to drive
tf.keras.models.save_model(
model = model,
filepath = 'model2.h5',
overwrite=True,
include_optimizer=True,
save_format=None,
signatures=None
)
Update: I can confirm this issue has been fixed in Tensorflow 2.3.0 https://github.com/tensorflow/tensorflow/issues/40081
I am using some implementation for creating a face recognition which uses this file:
"facenet.load_model("20170512-110547/20170512-110547.pb")"
What is the use of this file? I am not sure how it works.
console log :
Model filename: 20170512-110547/20170512-110547.pb
distance = 0.72212267
Github link of the actual owner of the code
https://github.com/arunmandal53/facematch
pb stands for protobuf. In TensorFlow, the protbuf file contains the graph definition as well as the weights of the model. Thus, a pb file is all you need to be able to run a given trained model.
Given a pb file, you can load it as follow.
def load_pb(path_to_pb):
with tf.gfile.GFile(path_to_pb, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
with tf.Graph().as_default() as graph:
tf.import_graph_def(graph_def, name='')
return graph
Once you have loaded the graph, you can basically do anything. For instance, you can retrieve tensors of interest with
input = graph.get_tensor_by_name('input:0')
output = graph.get_tensor_by_name('output:0')
and use regular TensorFlow routine like:
sess.run(output, feed_dict={input: some_data})
Explanation
The .pb format is the protocol buffer (protobuf) format, and in Tensorflow, this format is used to hold models. Protobufs are a general way to store data by Google that is much nicer to transport, as it compacts the data more efficiently and enforces a structure to the data. When used in TensorFlow, it's called a SavedModel protocol buffer, which is the default format when saving Keras/ Tensorflow 2.0 models. More information about this format can be found here and here.
For example, the following code (specifically, m.save), will create a folder called my_new_model, and save in it, the saved_model.pb, an assets/ folder, and a variables/ folder.
# first download a SavedModel from TFHub.dev, a website with models
m = tf.keras.Sequential([
hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_130_224/classification/4")
])
m.build([None, 224, 224, 3]) # Batch input shape.
m.save("my_new_model") # defaults to save as SavedModel in tensorflow 2
In some places, you may also see .h5 models, which was the default format for TF 1.X. source
Extra information: In TensorFlow Lite, the library for running models on mobile and IoT devices, instead of protocol buffers, flatbuffers are used. This is what the TensorFlow Lite Converter converts into (.tflite format). This is another Google format which is also very efficient: it allows access to any part of the message without deserialization (unlike json, xml). For devices with less memory (RAM), it makes more sense to load what you need from the model file, instead of loading the entire thing into memory to deserialize it.
Loading SavedModels in TensorFlow 2
I noticed BiBi's answer to show loading models was popular, and there is a shorter way to do this in TF2:
import tensorflow as tf
model_path = "/path/to/directory/inception_v1_224_quant_20181026"
model = tf.saved_model.load(model_path)
Note,
the directory (i.e. inception_v1_224_quant_20181026) has to have a saved_model.pb or saved_model.pbtxt, otherwise the code will crash. You cannot specify the .pb path, specify the directory.
you might get TypeError: 'AutoTrackable' object is not callable for older models, fix here.
If you load a TF1 model, I found that I don't get any errors, but the loaded file doesn't behave as expected. (e.g. it doesn't have any functions on it, like predict)
I was given a TensorFlow checkpoint and also an exported model, but to serve a model using Google ML Cloud, I need a saved_model.pbtxt file. It seems that I need to load the checkpoint and use SavedModelBuilder but SavedModelBuilder wants a dictionary of the names of the input and output nodes.
My question is, given the checkpoint or the exported model (below), how can I find the names of the nodes needed to generate the pbtxt file I need to serve the model via Google's ML Cloud service?
checkpoint
export.data-00000-of-00001
export.index
export.meta
options.json
The export.meta should be a MetaGraphDef proto. So you should be able to parse the proto to get the graph. You can then search through the nodes to find the node of interest.
Something like:
import argparse
from tensorflow.core.protobuf import meta_graph_pb2
import logging
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description='Argument parser.')
parser.add_argument('--path',
required=True,
help='The path to the metadata graph file.')
args = parser.parse_args()
with open(args.path, 'r') as hf:
graph = meta_graph_pb2.MetaGraphDef.FromString(hf.read())
print "graph: \n{0}".format(graph)
I think you should also be able to point TensorBoard at the directory containing that file and TensorBoard will render the graph and use that to identify the names of the input/output nodes.
So apparently the Yann LeCun's website is down so the following lines for reading mnist with tensorflow don't seem to be working :
FROM tensorflow.examples.tutorials.mnist IMPORT input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot = true)
Any ideas how can i read the mnist without using these above lines?
You can access the website here: https://web.archive.org/web/20160117040036/http://yann.lecun.com/exdb/mnist/ - download the data, and read it in from a local copy...
Edit
Here is a code example for reading a local mnist dataset
You can download the dataset individually and put it in the directory you created.Then the code can run normally.
http://yann.lecun.com/exdb/mnist/
Ok guys i fixed my problem,my mistake was that i was extracting the files(to the \MNIST_data directory) from the .gz files where i should have left them as they were unextracted instead.Thanks for the help.