I want to change the follow pytorch network (v1.2) to tensorflow. I am confusing between tf.nn.conv2d and tf.keras.layers.Conv2D what should I choose?
import torch.nn as nn
nn.Sequential(nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, dilation=dilation, bias=True),
nn.BatchNorm2d(out_planes),
nn.ReLU(inplace=True))
tf.nn.conv2d is functional api and tf.keras.layers.Conv2D is layer-class api. You should use the latter one. It's quite as similar as the relationship between torch.nn.functional.conv2d and torch.nn.Conv2D.
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Conv2D, ReLU, BatchNormalization
model = Sequential()
model.add(Conv2D(filters=10, kernel_size=3, strides=1))
model.add(BatchNormalization())
model.add(ReLU())
Related
I have an One-hot-encoded sparse matrix which can't be transformed into a normal matrix due to its size.
I would like to reduce the dimensions using an autoencoder. Currently I am trying to use Tensorflow and its Keras library for that.
The Tensorflow docs state that sparse tensors exist and that they can be used in Keras (see https://www.tensorflow.org/guide/sparse_tensor).
The Problem is that all autoencoders I've found in the internet do not seem to work with sparse tensors.
I have prepared a small code example which stops after the first training epoch with the error message: "Failed to convert elements of SparseTensor to Tensor. Consider casting elements to a supported type.".
My Questions would be:
Do you have an idea to improve the Code or ideally do you have an example which I can look up?
If not: Do you have other ideas on how to do what I would like to do (e.g. another library, other method, etc.)?
Code Example:
#necessary imports
import tensorflow as tf
from keras.models import Model, Sequential
from keras.layers import Input, Dense, ActivityRegularization
from tensorflow.keras import backend as K
from tensorflow.keras import regularizers
#example one-hot-encoded matrix with 10 records with each one out of 4 distinct categories
sparse_tensor = tf.sparse.SparseTensor(indices=[[0,3], [1,3], [2,0], [3,1], [4,0], [5,2], [6,2], [7,1], [8,3], [9,1]],
values=[1 for i in range(10)],
dense_shape=[10, 4])
encoder = Sequential([
Input(shape=(4,), sparse=True),
Dense(1, activation = 'relu'),
ActivityRegularization(l1=1e-3)
])
decoder = Sequential([
Dense(4, activation = 'sigmoid', input_shape = (1, )),
])
autoencoder = Sequential([encoder, decoder])
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(x=sparse_tensor, y=sparse_tensor, epochs=5, batch_size=5, shuffle=True)
I implemented a local version of an RNN and a Colab TPU version of an RNN(code-below). When I execute the Colab TPU version(code-below), the training speed is very slow like my local version running on my laptop's CPU.
Does Colab TPU support RNN networks?
Am I missing something here?
import tensorflow as tf
import os
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, SimpleRNN
resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
print("All devices: ", tf.config.list_logical_devices('TPU'))
strategy = tf.distribute.TPUStrategy(resolver)
with strategy.scope():
model = Sequential()
model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
model.add(Dense(16, activation="relu"))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='rmsprop')
model.fit(X,y, epochs=50, batch_size=16, verbose=0)
ctrl-f on this page for RNN. It seems like it should work if you can make the RNN static enough.
In general, dynamic operations don't work well with TPUs since it needs to recompile the model graph for each new training example.
I am trying to transform my Keras model in the Google cloud console into a TPU model. Unfortunatelly I am getting an error as shown below. My minimal example is the following:
import keras
from keras.models import Sequential
from keras.layers import Dense, Activation
import tensorflow as tf
import os
model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Dense(32))
model.add(Activation('relu'))
model.compile(optimizer='rmsprop', loss='mse')
tpu_model = tf.contrib.tpu.keras_to_tpu_model(
model,
strategy=tf.contrib.tpu.TPUDistributionStrategy(
tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)))
My output is:
Using TensorFlow backend.
Traceback (most recent call last):
File "cloud_python4.py", line 11, in <module>
tpu_model = tf.contrib.tpu.keras_to_tpu_model(AttributeError: module 'tensorflow.contrib.tpu' has no attribute 'keras_to_tpu_model'
The keras_to_tpu_model method seems experimental as indicated on the tensorflow website. Has it recently been removed? If so, how can I proceed to make use of TPUs to estimate my Keras model? If the keras_to_tpu_model method would be still available, why can I not invoke it?
I am assuming you defined you TPU_WORKER as below
import os
TPU_WORKER = ‘grpc://’ + os.environ[‘COLAB_TPU_ADDR’]
Instead of converting your model to TPU, build a distribution strategy. This is the method by which the batch will be distributed to the eight TPUs and how the loss from each will be calculated.
resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)
tf.contrib.distribute.initialize_tpu_system(resolver)
strategy = tf.contrib.distribute.TPUStrategy(resolver)
With the strategy build and compile your model. This should work quite nicely for regression.
with strategy.scope():
model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Dense(32))
model.add(Activation('relu'))
model.compile(optimizer='rmsprop', loss='mse')
Import keras from tensorflow.
This is because tf.contrib.tpu.keras_to_tpu_model( )' requires a tensorflow version Model, not the keras version.
For example, use from tensorflow.keras.layers import Dense, Activation instead. And so on.
I am trying to run LeNet on Keras with double precision and it fails with an error: TypeError: Input 'filter' of 'Conv2D' Op has type float64 that does not match type float32 of argument 'input'.. The code I am using is as follows:
import numpy as np
from sklearn.utils import shuffle
import keras
from keras.models import Sequential
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, Dropout,Flatten
from keras import backend as K
from keras.models import Model
from keras.utils import np_utils
import time
import tensorflow as tf
K.set_floatx('float64') # Note: the code works if we comment this line, i.e., with single precision
from mlxtend.data import mnist_data
X, y = mnist_data()
X = X.astype(np.float64)
X, y = shuffle(X, y)
keras_model = Sequential()
keras_model.add(Conv2D(32, kernel_size=(5, 5), activation='relu', input_shape=(28,28,1), padding='same'))
keras_model.add(MaxPooling2D(pool_size=(2, 2)))
keras_model.add(Conv2D(64, (5, 5), activation='relu', padding='same'))
keras_model.add(MaxPooling2D(pool_size=(2, 2)))
keras_model.add(Flatten())
keras_model.add(Dense(512, activation='relu'))
keras_model.add(Dropout(0.5))
keras_model.add(Dense(10, activation='softmax'))
keras_model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.SGD(lr=0.01, momentum=0.95, decay=5e-4, nesterov=True))
keras_model.fit(X.reshape((-1, 28,28, 1)), np_utils.to_categorical(y, 10), epochs=1, batch_size=64)
Any suggestions is much appreciated :)
You have a gaming NVIDIA GPU.
You can use only float32 or int32, that's all.
It's the default of TensorFlow.
This default is introduced by Tensorflow due to limitations of CUDA capable GPUs of Nvidia. Best explanation I found here. So the premium Tesla GPUs work well on float16 and float64 as well, but the gaming GPUs work only on float32 and perform very bad for float16 or float64.
I think we are all looking at OpenCL that is supported by AMD GPUs that are more pricey. Unfortunately OpenCL is not supported by TensorFlow as of now.
Suggestion#1:
You are stuck with float32. Forget changing it while having that hardware.
Suggestion#2:
Once you get a GPU good with float16 change to that. Machine learning doesn't require high precision above that.
I have a code in Keras (or its TF version). I want to have a CUDA code which is equivalence to it. Is there a way to get it?
I know that from Keras I can look at the basic graph topology using the following code:
# LSTM for sequence classification in the IMDB dataset
import numpy
from keras.datasets import imdb
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers.embeddings import Embedding
from keras import backend as K
from keras.preprocessing import sequence
# fix random seed for reproducibility
numpy.random.seed(7)
# load the dataset but only keep the top n words, zero the rest
top_words = 5000
max_review_length = 500
# create the model
embedding_vecor_length = 32
model = Sequential()
model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
g = K.get_session().graph
# GIVES THE GRAPH TOPOLOGY!:
graph_def = g.as_graph_def()
Is there a way to have the .cc file that represent this code?
Thanks!
There is no functionality in TensorFlow to generate C++ CUDA source code from a graph, but the XLA framework supports ahead-of-time compilation, which generates efficient bytecode from your TensorFlow graph, which you can then execute on your CUDA-capable GPU.