I am trying to transform my Keras model in the Google cloud console into a TPU model. Unfortunatelly I am getting an error as shown below. My minimal example is the following:
import keras
from keras.models import Sequential
from keras.layers import Dense, Activation
import tensorflow as tf
import os
model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Dense(32))
model.add(Activation('relu'))
model.compile(optimizer='rmsprop', loss='mse')
tpu_model = tf.contrib.tpu.keras_to_tpu_model(
model,
strategy=tf.contrib.tpu.TPUDistributionStrategy(
tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)))
My output is:
Using TensorFlow backend.
Traceback (most recent call last):
File "cloud_python4.py", line 11, in <module>
tpu_model = tf.contrib.tpu.keras_to_tpu_model(AttributeError: module 'tensorflow.contrib.tpu' has no attribute 'keras_to_tpu_model'
The keras_to_tpu_model method seems experimental as indicated on the tensorflow website. Has it recently been removed? If so, how can I proceed to make use of TPUs to estimate my Keras model? If the keras_to_tpu_model method would be still available, why can I not invoke it?
I am assuming you defined you TPU_WORKER as below
import os
TPU_WORKER = ‘grpc://’ + os.environ[‘COLAB_TPU_ADDR’]
Instead of converting your model to TPU, build a distribution strategy. This is the method by which the batch will be distributed to the eight TPUs and how the loss from each will be calculated.
resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)
tf.contrib.distribute.initialize_tpu_system(resolver)
strategy = tf.contrib.distribute.TPUStrategy(resolver)
With the strategy build and compile your model. This should work quite nicely for regression.
with strategy.scope():
model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Dense(32))
model.add(Activation('relu'))
model.compile(optimizer='rmsprop', loss='mse')
Import keras from tensorflow.
This is because tf.contrib.tpu.keras_to_tpu_model( )' requires a tensorflow version Model, not the keras version.
For example, use from tensorflow.keras.layers import Dense, Activation instead. And so on.
Related
I implemented a local version of an RNN and a Colab TPU version of an RNN(code-below). When I execute the Colab TPU version(code-below), the training speed is very slow like my local version running on my laptop's CPU.
Does Colab TPU support RNN networks?
Am I missing something here?
import tensorflow as tf
import os
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, SimpleRNN
resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
print("All devices: ", tf.config.list_logical_devices('TPU'))
strategy = tf.distribute.TPUStrategy(resolver)
with strategy.scope():
model = Sequential()
model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
model.add(Dense(16, activation="relu"))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='rmsprop')
model.fit(X,y, epochs=50, batch_size=16, verbose=0)
ctrl-f on this page for RNN. It seems like it should work if you can make the RNN static enough.
In general, dynamic operations don't work well with TPUs since it needs to recompile the model graph for each new training example.
I want to change the follow pytorch network (v1.2) to tensorflow. I am confusing between tf.nn.conv2d and tf.keras.layers.Conv2D what should I choose?
import torch.nn as nn
nn.Sequential(nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, dilation=dilation, bias=True),
nn.BatchNorm2d(out_planes),
nn.ReLU(inplace=True))
tf.nn.conv2d is functional api and tf.keras.layers.Conv2D is layer-class api. You should use the latter one. It's quite as similar as the relationship between torch.nn.functional.conv2d and torch.nn.Conv2D.
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Conv2D, ReLU, BatchNormalization
model = Sequential()
model.add(Conv2D(filters=10, kernel_size=3, strides=1))
model.add(BatchNormalization())
model.add(ReLU())
When using SavedModels in Tensorflow 2.0, is it possible to access activations from intermediate layers? For example, with one of the models here: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md, I can run, for example,
model = tf.saved_model.load('faster_rcnn_inception_v2_coco_2018_01_28/saved_model').signatures['serving_default']
outputs = model(input_tensor)
to get output predictions and bounding boxes. I would like to be able to access layers other than the outputs, but there doesn't seem to be any documentation for Tensorflow 2.0 on how to do this. The downloaded models also include checkpoint files, but there doesn't seem to be very good documentation for how to load those with Tensorflow 2.0 either...
If you are generating saved models using TensorFlow 2.0, it is possible to extract individual layers. But the model which you are referring to has been saved in TensorFlow 1.x. With TF 1.x saved models, you cannot individually extract layers.
Here is an example on how you can extract layers from a saved model in TensorFlow 2.0
import tensorflow as tf
import numpy as np
model = tf.keras.Sequential([
tf.keras.layers.Dense(64, activation='relu', input_shape=(100,)),
tf.keras.layers.Dense(10, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])
# Compile and fit the model
model.save('save_model', save_format='tf')
Then load the model as shown.
model = tf.keras.models.load_model('save_model')
layer1 = model.get_layer(index=1)
Is it possible to convert a keras model (h5 file of network architecture and weights) into a tensorflow model? Or is there an equivalent function to model.save of keras in tensorflow?
Yes, it is possible, because Keras, since it uses Tensorflow as backend, also builds computational graph. You just need to get this graph from your Keras model.
"Keras only uses one graph and one session. You can access the session
via: K.get_session(). The graph associated with it would then be:
K.get_session().graph."
(from fchollet: https://github.com/keras-team/keras/issues/3223#issuecomment-232745857)
Or you can save this graph in checkpoint format (https://www.tensorflow.org/api_docs/python/tf/train/Saver):
import tensorflow as tf
from keras import backend as K
saver = tf.train.Saver()
sess = K.get_session()
retval = saver.save(sess, ckpt_model_name)
By the way, since tensorflow 13 you can use keras right from it:
from tensorflow.python.keras import models, layers
I have a code in Keras (or its TF version). I want to have a CUDA code which is equivalence to it. Is there a way to get it?
I know that from Keras I can look at the basic graph topology using the following code:
# LSTM for sequence classification in the IMDB dataset
import numpy
from keras.datasets import imdb
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers.embeddings import Embedding
from keras import backend as K
from keras.preprocessing import sequence
# fix random seed for reproducibility
numpy.random.seed(7)
# load the dataset but only keep the top n words, zero the rest
top_words = 5000
max_review_length = 500
# create the model
embedding_vecor_length = 32
model = Sequential()
model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
g = K.get_session().graph
# GIVES THE GRAPH TOPOLOGY!:
graph_def = g.as_graph_def()
Is there a way to have the .cc file that represent this code?
Thanks!
There is no functionality in TensorFlow to generate C++ CUDA source code from a graph, but the XLA framework supports ahead-of-time compilation, which generates efficient bytecode from your TensorFlow graph, which you can then execute on your CUDA-capable GPU.