I am learning Keras and Thensorflow for deep learning and I have a question.
With this imports:
import tensorflow as tf
from keras import backend as K
Is there a difference between these two calls:
K.foo
and
tf.foo
?
In which conditions are they equivalent ?
Yes, there may be a difference.
Keras is built on top of a backend. This backend may be Tensorflow, Theano or CNTK.
So, a function from keras will call a function from the backend, something like this:
#at keras.backend
def foo(args**):
#there may be some preprocessing or inversion in dimensions
return tf.foo(args_that_may_be_different**)
It's impossible to have an answer for all functions. Some are indeed exactly the same, some may have a difference.
You can search the backend codes, speficially the tensorflow backend and see how keras handles each function.
Related
I would like to benchmark some TensorFlow operations (for example between them or against PyTorch). However most of the time I will write something like:
import numpy as np
import tensorflow as tf
tf_device = '/GPU:0'
a = np.random.normal(scale=100, size=shape).astype(np.int64)
b = np.array(7).astype(np.int64)
with tf.device(tf_device):
a_tf = tf.constant(a)
b_tf = tf.constant(b)
%timeit tf.math.floormod(a_tf, b_tf)
The problem with this approach is that it does the computation in eager-mode (I think in particular that it has to perform GPU to CPU placement). Eventually, I want to use those ops in a tf.keras model and therefore would like to evaluate their performance in graph mode.
What is the preferred way to do it?
My google searches have led to nothing and I don't know how to use sessions like in tf 1.x.
What you are looking for is tf.function. Check this tutorial and this docs.
As the tutorial says, in TensorFlow 2, eager execution is turned on by default. The user interface is intuitive and flexible (running one-off operations is much easier and faster), but this can come at the expense of performance and deployability. To get performant and portable models, use tf.function to make graphs out of your programs.
Check this code:
import numpy as np
import tensorflow as tf
import timeit
tf_device = '/GPU:0'
shape = [100000]
a = np.random.normal(scale=100, size=shape).astype(np.int64)
b = np.array(7).astype(np.int64)
#tf.function
def experiment(a_tf, b_tf):
tf.math.floormod(a_tf, b_tf)
with tf.device(tf_device):
a_tf = tf.constant(a)
b_tf = tf.constant(b)
# warm up
experiment(a_tf, b_tf)
print("In graph mode:", timeit.timeit(lambda: experiment(a_tf, b_tf), number=10))
print("In eager mode:", timeit.timeit(lambda: tf.math.floormod(a_tf, b_tf), number=10))
I'm not familiar with using python as a ML tool and wanted to train the MNIST data set. I have downloaded the MNIST library using
pip install python-mnist but do not know what my next step should be. What would an import statement look like? Should I also import TensorFlow and or Keras to train the data?
I know the MNIST dataset is available in TensorFlow and Keras, however, importing via pip a necessary solution for my use case. Both TensorFlow and Keras have tutorials using the MNIST data set, but I was wondering if it is possible to use the data set without using their pre-downloaded library.
The import statement should look like this
from mnist import MNIST
mndata = MNIST('./dir_with_mnist_data_files')
images, labels = mndata.load_training()
then you can work directly with the arrays of raw images and labels.
When I build a model using Lambda layer to calculate the sum along an axis, model.evaluate() gives different result in comparison to manually calculating from model.predict(). I have checked the output data types and shapes and it did not help. The problem only occurs when I use tensorflow as Keras backend. It works fine when I use CNTK as backend.
Here is a minimal code that can reproduce this behavior.
from keras.layers import Input,Lambda
import keras.backend as K
from keras.models import Model
import numpy as np
inp=Input((2,))
out=Lambda(lambda x:K.sum(x,axis=-1),output_shape=(1,))(inp)
model=Model(input=inp,output=out)
model.compile(loss="mse",optimizer="sgd")
xs=np.random.random((3,2))
ys=np.sum(xs,axis=1)
print(np.mean((model.predict(xs)-ys)**2)) # This is zero
print(model.evaluate(xs,ys)) # This is not zero
I establish a network with no parameters and should simply calculate the sum for each x, and ys are constructed so that it should be the same with the output from the model. model.predict() gives the same results as ys, but model.evaluate() gives something non-zero when I use tensorflow as backend.
Any idea about why this should happen? Thanks!
I have some working Python3 sources gotten from the internet where initial Keras imports are direct like this:
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
...
In TensorFlow documentation instead I see the following indirect form:
import tensorflow as tf
from tensorflow.keras import layers
...
To me they seem to mean respectively, that Keras can be used without knowing that TensorFlow is behind, and, that Keras is provided (again?) as a part of TensorFlow. (I kind of expect that also Keras similarly provides references to TensorFlow in the former case)
What is the difference? Does it depend on how Keras and TensorFlow are installed, or rather on the way they are used? Is it a potential source of confusion that I have to get rid of? In other words, should I fix my installation, and how? Or should I just accept that there are two, and manage their respective usages to live with them safely?
Background: my installation is under Ubuntu Linux, with Python3.5.2, where pip3 list shows the following packages:
Keras (2.2.4)
Keras-Applications (1.0.6)
Keras-Preprocessing (1.0.5)
tensorboard (1.12.0)
tensorflow (1.12.0)
BTW, I have checked that they are really different:
import keras as keras
import tensorflow.keras as tf_keras
print( keras is tf_keras )
---> False
print( [keras.__version__ , tf_keras.__version__] )
---> ['2.2.4', '2.1.6-tf']
print( [len(dir(keras)) , len(dir(tf_keras)) ] )
---> [32, 30]
print( [ len(dir(keras.models)) , len(dir(tf_keras.models)) ] )
---> [27, 17]
print( [ len(dir(keras.layers)) , len(dir(tf_keras.layers)) ] )
---> [167, 117]
and indeed it seems that I have two different Keras and that the former is higher versioned and richer.
Related readings, useful but not enough to solve the "does it need a fix?" question:
Import statments when using Tensorflow contrib keras
what's the difference between "import keras" and "import tensorflow.keras"
Difference between Keras and tf.keras: should old Keras code be changed?
Why keras does not allow to add a convolutional layer in this way?
Thanks!
Rather than posting my own answer, I'll point you to a very exhaustive answer, much better than I would be able to write: it is here (thanks to Adrian Rosebrock).
Disclaimer: I have no link with Adrian or his activity. I have greatly appreciated his explanation though.
There is no fix needed. They're two different packages and you just manage their respective usages.
Official word as of September 2021:
User should always use from tensorflow import keras which will give them the public API.
import keras will directly access the keras PIP package, which is not 100% same as the public API namespace. It will probably give you keras.Model/layers.* etc, but not all the APIs. Under the hood, we populate all the APIs under keras/api/* and tensorflow __init__ files will pick them up from there.
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999/10
Is it possible to define a graph in native TensorFlow and then convert this graph to a Keras model?
My intention is simply combining (for me) the best of the two worlds.
I really like the Keras model API for prototyping and new experiments, i.e. using the awesome multi_gpu_model(model, gpus=4) for training with multiple GPUs, saving/loading weights or whole models with oneliners, all the convenience functions like .fit(), .predict(), and others.
However, I prefer to define my model in native TensorFlow. Context managers in TF are awesome and, in my opinion, it is much easier to implement stuff like GANs with them:
with tf.variable_scope("Generator"):
# define some layers
with tf.variable_scope("Discriminator"):
# define some layers
# model losses
G_train_op = ...AdamOptimizer(...)
.minimize(gloss,
var_list=tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
scope="Generator")
D_train_op = ...AdamOptimizer(...)
.minimize(dloss,
var_list=tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
scope="Discriminator")
Another bonus is structuring the graph this way. In TensorBoard debugging complicated native Keras models are hell since they are not structured at all. With heavy use of variable scopes in native TF you can "disentangle" the graph and look at a very structured version of a complicated model for debugging.
By utilizing this I can directly setup custom loss function and do not have to freeze anything in every training iteration since TF will only update the weights in the correct scope, which is (at least in my opinion) far easier than the Keras solution to loop over all the existing layers and set .trainable = False.
TL;DR:
Long story short: I like the direct access to everything in TF, but most of the time a simple Keras model is sufficient for training, inference, ... later on. The model API is much easier and more convenient in Keras.
Hence, I would prefer to set up a graph in native TF and convert it to Keras for training, evaluation, and so on. Is there any way to do this?
I don't think it is possible to create a generic automated converter for any TF graph, that will come up with a meaningful set of layers, with proper namings etc. Just because graphs are more flexible than a sequence of Keras layers.
However, you can wrap your model with the Lambda layer. Build your model inside a function, wrap it with Lambda and you have it in Keras:
def model_fn(x):
layer_1 = tf.layers.dense(x, 100)
layer_2 = tf.layers.dense(layer_1, 100)
out_layer = tf.layers.dense(layer_2, num_classes)
return out_layer
model.add(Lambda(model_fn))
That is what sometimes happens when you use multi_gpu_model: You come up with three layers: Input, model, and Output.
Keras Apologetics
However, integration between TensorFlow and Keras can be much more tighter and meaningful. See this tutorial for use cases.
For instance, variable scopes can be used pretty much like in TensorFlow:
x = tf.placeholder(tf.float32, shape=(None, 20, 64))
with tf.name_scope('block1'):
y = LSTM(32, name='mylstm')(x)
The same for manual device placement:
with tf.device('/gpu:0'):
x = tf.placeholder(tf.float32, shape=(None, 20, 64))
y = LSTM(32)(x) # all ops / variables in the LSTM layer will live on GPU:0
Custom losses are discussed here: Keras: clean implementation for multiple outputs and custom loss functions?
This is how my model defined in Keras looks in Tensorboard:
So, Keras is indeed only a simplified frontend to TensorFlow so you can mix them quite flexibly. I would recommend you to inspect source code of Keras model zoo for clever solutions and patterns that allows you to build complex models using clean API of Keras.
You can insert TensorFlow code directly into your Keras model or training pipeline! Since mid-2017, Keras has fully adopted and integrated into TensorFlow. This article goes into more detail.
This means that your TensorFlow model is already a Keras model and vice versa. You can develop in Keras and switch to TensorFlow whenever you need to. TensorFlow code will work with Keras APIs, including Keras APIs for training, inference and saving your model.