I'm very new to Machine Learning.
I have found this example at Github: Code
I implemented the q_learning_keras function and chaned the Keras model to a tf.keras model.
With the Keras model 1000 Episodes take about 30 minutes.
With the tf.keras model 1000 Episodes take about 110 minutes.
Also the tf.keras variant need about 5-7% less cpu power.
I'm using Tensorflow version 1.13.1 and Keras version 2.2.4.
I'm runing the code on a virtual machine with 4 Cores # 2.6 GHz.
Keras model:
model = Sequential()
model.add(InputLayer(batch_input_shape=(1, 5)))
model.add(Dense(10, activation='sigmoid'))
model.add(Dense(2, activation='linear'))
model.compile(loss='mse', optimizer='adam', metrics=['mae'])
tf.keras model:
model = tf.keras.models.Sequential([
tf.keras.layers.InputLayer(batch_input_shape=(1, 5)),
tf.keras.layers.Dense(10, activation='sigmoid'),
tf.keras.layers.Dense(2, activation='linear')
])
The rest of the code is in both excatly the same, ase in the Github example linked above.
In the most over post I have found exactly the opposite problem, that Keras was slower then tf.keras. So I also hoped to get an increase in speed or at least to stay the same.
So I think, I have maybe make a mistake but I don't find it.
Edit:
Done some more resarch, with Tensorboard to see how the model look like.
The models look very similar at the most things. But the tf.keras model have some overhead. I don't know if this is normal for tf.keras model or if it comes from my way I build the model.
tf.keras model
keras model
Related
I'm adding data augmentation to my tensorflow model like so:
data_augmentation = keras.Sequential([
layers.experimental.preprocessing.RandomRotation(factor=0.4, fill_mode="wrap"),
layers.experimental.preprocessing.RandomTranslation(height_factor=0.2, width_factor=0.2, fill_mode="wrap"),
layers.experimental.preprocessing.RandomFlip("horizontal"),
layers.experimental.preprocessing.RandomContrast(factor=0.2),
layers.experimental.preprocessing.RandomHeight(factor=0.2),
layers.experimental.preprocessing.RandomWidth(factor=0.2)
])
input_shape = (299, 299, 3)
inceptionV3_base = tf.keras.applications.InceptionV3(
input_shape=input_shape,
include_top=False,
weights='imagenet'
)
tf_model = tf.keras.Sequential([
tf.keras.layers.InputLayer(input_shape=input_shape),
data_augmentation,
inceptionV3_base,
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(1024, activation='relu'),
tf.keras.layers.Dense(len(classes), activation='softmax')
])
Adding the data_augmentation layer to model slows down training by 13x. Am I using the keras preprocessing layers correctly?
I had the same issue - was missing ptxas (nvidia-cuda-toolkit package for me).
There are two ways of adding data augmentation:
1- Inside the model, just like the way you did.
2- Outside the model, and before training, using tf.data.Dataset.map()
Maybe trying option2 could make your model training faster. Try it!
More details here: https://keras.io/guides/preprocessing_layers/
There are many other ways of optimizing the training, such as the way you feed data into your model. Are you using tf.data.Dataset methods like cache() and prefetch(). More details can be found here: https://www.tensorflow.org/tutorials/load_data/images#configure_the_dataset_for_performance
I have 4 classes, each with 1350 images. The validation set has 20% of the total images (it is generated automatically). The training model uses MobilenetV2 network:
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE, include_top=False, weights='imagenet')
The model is created:
model = tf.keras.Sequential([
base_model,
tf.keras.layers.Conv2D(32, 3, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(4, activation='softmax', kernel_regularizer=regularizers.l2(0.001))
])
The model is trained through 20 epochs and then fine tunning is done in 15 epochs. The result is as follows:
Image of the model trained before fine tunning
Image of the model trained after 15 epochs and fine tunning
A bit difficult to tell without the numeric values of validation loss but I would say the results before fine tuning are slightly over fitting and for after fine tuning less over fitting. A couple of additional things you could do. One is try using an adjustable learning rate using the callback tf.keras.callbacks.ReduceLROnPlateau. Set it up to monitor validation loss. Documentation is here. I set factor=.5 and patience=1. Second replace the Flatten layer with tf.keras.layers.GlobalMaxPool2D and see if it improves the validation loss.
I have keras pretrained model(model.h5). And I want to prune that model with tensorflow Magnitude-based weight pruning with Keras. One curious things is that my pretrained model is built with original keras model > I mean that is not from tensorflow.keras. Inside tensorflow Magnitude-based weight pruning with Keras example, they show how to do with tensorflow.keras model. I want to ask is that can I use their tool to prune my original keras pretrained model?
inside their weight pruning toolkit ,there is two way. one is pruned the model layer by layer while training and second is pruned the whole model. I tried the second way to prune the whole pretrained model. below is my code.
inside their weight pruning toolkit ,there is two way. one is pruned the model layer by layer while training and second is pruned the whole model. I tried the second way to prune the whole pretrained model. below is my code.
For my original pretrained model, I load the weight from model.h5 and can call model.summary() after I apply prune_low_magnitude() none of the method from model cannot call including model.summary() method. And show the error like AttributeError: 'NoneType' object has no attribute 'summary'
model = get_training_model(weight_decay)
model.load_weights('model/keras/model.h5')
model.summary()
epochs = 1
end_step = np.ceil(1.0 * 100 / 2).astype(np.int32) * epochs
print(end_step)
new_pruning_params = {
'pruning_schedule': tfm.sparsity.keras.PolynomialDecay(initial_sparsity=0.1,
final_sparsity=0.90,
begin_step=40,
end_step=end_step,
frequency=30)
}
new_pruned_model = tfm.sparsity.keras.prune_low_magnitude(model, **new_pruning_params)
print(new_pruned_model.summary())
Inside their weight pruning toolkit
enter link description here ,there is two way. one is pruned the model layer by layer while training and second is pruned the whole model. I tried the second way to prune the whole pretrained model. below is my code.
For my original pretrained model, I load the weight from model.h5 and can call model.summary() after I apply prune_low_magnitude() none of the method from model cannot call including model.summary() method. And show the error like
AttributeError: 'NoneType' object has no attribute 'summary'
I hope this answer still helps, but I recently had the same issue that prune_low_magnitude() returns an object of type 'None'. Also new_pruned_model.compile() would not work.
The model I had been using was a pretrained model that could be imported from tensorflow.python.keras.applications.
For me this worked:
(0) Import the libraries:
from tensorflow_model_optimization.python.core.api.sparsity import keras as sparsity
from tensorflow.python.keras.applications.<network_type> import <network_type>
(1) Define the pretrained model architecture
# define model architecture
loaded_model = <model_type>()
loaded_model.summary()
(2) Compile the model architecture and load the pretrained weights
# compile model
opt = SGD(lr=learn_rate, momentum=momentum)
loaded_model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
loaded_model.load_weights('weight_file.h5')
(3) set pruning parameters and assign pruning schedule
# set pruning parameters
pruning_params = {
'pruning_schedule': sparsity.PolynomialDecay(...)
}
# assign pruning schedule
model_pruned = sparsity.prune_low_magnitude(loaded_model, **pruning_params)
(4) compile model and show summary
# compile model
model_pruned.compile(
loss=tf.keras.losses.categorical_crossentropy,
optimizer='SGD',
metrics=['accuracy'])
model_pruned.summary()
It was important to import the libraries specifically from tensorflow.python.keras and use this keras model from the TensorFlow library.
Also, it was important to use the TensorFlow Beta Release (pip install tensorflow==2.0.0b1), otherwise still an object with type 'None' would be returned by prune_low_magnitude.
I am using PyCharm 2019.1.3 (x64) as IDE. Here is the link that led me to this solution: https://github.com/tensorflow/model-optimization/issues/12#issuecomment-526338458
How can a Tensorflow feature_column be used in conjunction with a Keras model?
E.g. for a Tensorflow estimator, we can use an embedding column from Tensorflow Hub:
embedded_text_feature_column = hub.text_embedding_column(
key="sentence",
module_spec="https://tfhub.dev/google/nnlm-en-dim128/1")
estimator = tf.estimator.DNNClassifier(
hidden_units=[100],
feature_columns=[embedded_text_feature_column],
n_classes=2,
optimizer=tf.train.AdamOptimizer(learning_rate=0.001))
However, I would like to use the TF Hub text_embedding_column as input to a Keras model. E.g.
net = tf.keras.layers.Input(...) # use embedding column here
net = tf.keras.layers.Flatten()
net = Dense(100, activation='relu')(net)
net = Dense(2)(net)
Is this possible?
The answer seems to be that you don't use feature columns. Keras comes with its own set of preprocessing functions for images and text, so you can use those.
So basically the tf.feature_columns are reserved for the high level API. Then the tf.keras.preprocessing() functions are used with tf.keras models.
Here is a link to the section on preprocessing data in the keras documentation.
https://keras.io/preprocessing/text/
Here is another Stackoverflow post that has an example of this approach.
Add Tensorflow pre-processing to existing Keras model (for use in Tensorflow Serving)
The keras functional api is a viable way to do this, but if you want to use feature_columns this tutorial shows you how:
https://www.tensorflow.org/beta/tutorials/keras/feature_columns
Basically it's this DenseFeatures layer that does the job:
feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
model = tf.keras.Sequential([
feature_layer,
layers.Dense(128, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(1, activation='sigmoid')
])
I built a simple LSTM model using keras and trained as follows:
model = Sequential()
model.add(LSTM(activation='tanh',input_dim=6,output_dim=50,return_sequences=False))
model.add(Dense(output_dim=1,activation = 'sigmoid'))
model.compile(loss='binary_crossentropy', optimizer =optimizers.Adam(lr = 0.01),metrics=['accuracy'])
model.fit(X_train,y_train,batch_size=256,nb_epoch=1000,validation_data = (X_test,y_test))
model.save('model_params.h5')
The model almost converged. Therefore I want to fine tune the model by resuming training using smaller learning rate (i.e., 0.001). How could I achieve this?
New answer
If your optimizer has an lr property, and this property is a tensor, you can change it with:
keras.backend.set_value(model.optimizer.lr, new_value)
Old answer, with some side effects
You just need to compile the model again:
model.compile(loss='binary_crossentropy',
optimizer= optimizers.Adam(lr=0.001),...)
But usually, Adam is a very good optimizer and doesn't need those changes. It's normal for it to find alone its ways.
It's very normal for the training to diverge when you compile with a new optimizer. It takes a few epochs until the optimizer adjusts itself.