I am using the DeepMind Sonnet Library, https://sonnet.readthedocs.io/en/latest/
After defining a simple MLP, I run it with a mock datapoint in order to set the input shape.
import tensorflow as tf
import sonnet as snt
mlp = snt.nets.MLP([32, 32, 1])
mlp(np.array([[1.0, 1.0, 1.0]])) # MLP has structure 3 -> 32 -> 32 -> 1
I am now looking for a way to set my predefined weights for this network. Something like mlp.set_weights() for instance. Is there a way to do that?
Related
I have a very basic code that tries to create a single-layered Dense neural net and predicts the output for a deterministic input. The code is as follows:
import tensorflow as tf
from tensorflow.keras import layers
model = tf.keras.models.Sequential()
model.add(layers.Dense(units = 10))
import numpy as np
inp = np.ones((1,10))
model.predict(inp)
But the output that I am getting isn't being deterministic. I think it is related to initializing the weights and biases. So, how do I fix this without writing the initializing function from scratch?
Set global seed before initializing model tf.random.set_seed(42)
You can also set seed for specific parts of model, e.g. kernel_initializer in Dense layer, but with this approach, you may miss initializers that will still be nondeterministic. In your case setting it globally will be the best solution.
I regularly use scikit-learn pipelines to streamline model processing, and I'm wondering the easiest way to do something similar with Keras in Tensorflow 2.0.
What I'd like to do is deploy a Keras model as an API endpoint, and then submit a piece of text in a numpy array to it and have it tokenized, padded and predicted. But I don't know the shortest path to do this.
Here's some sample code:
from tensorflow import keras
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Embedding, Dense, Flatten
import numpy as np
sample_words = [
'The sky is blue',
'The sky delivers us many gifts',
'Wise men appreciate gifts for what they are, not what they are not',
'Wherever you go, there you are',
'Don\'t pass judgment onto others, or you will quickly be judged yourself'
]
y = np.array([1, 0, 1, 1, 0])
tokenizer = Tokenizer(num_words=10)
tokenizer.fit_on_texts(sample_words)
train_sequences = tokenizer.texts_to_sequences(sample_words)
train_sequences = pad_sequences(train_sequences, maxlen=7)
mod = Sequential([
Embedding(10, 2, input_length=7),
Flatten(),
Dense(3, activation='relu'),
Dense(1, activation='sigmoid')
])
mod.compile(optimizer='adam', loss='binary_crossentropy')
mod.fit(train_sequences, y)
The idea is that if I have a web form and someone submits a form with the words 'The sky is pretty today', I can wrap it in a numpy array, send it to the endpoint (which will be setup on Google Cloud), and have it padded, tokenized, and predicted.
In scikit learn it would be as simple as: pipe = make_pipeline(tokenizer, mod), and then go from there.
I have a feeling there are some solutions that include td.Datasets, but I was hoping keras had something in it that was more user friendly.
Keras is easy in a way that there is no need to explicitly build any pipelines.
The Keras model is using Tensorflow backend to create a computation graph which could be loosely said as similar to scikit-learn's pipeline.
Thus your mod is in itself equivalent to a pipeline having the operations: Embedding -> Flatten -> Dense -> Dense. The mod.compile() method is generating the tensorflow computation graph.
Then everything comes together in model.fit() method where you plug in your inputs to your model (i.e. pipeline) and then the method trains on your data.
In order to have the tokenization be a part of your model, the TextVectorization layer can be used.
This layer has basic options for managing text in a Keras model. It transforms a batch of strings (one sample = one string) into either a list of token indices (one sample = 1D tensor of integer token indices) or a dense representation (one sample = 1D tensor of float values representing data about the sample's tokens)
Code snapshot:
vectorize_layer = TextVectorization(
max_tokens=max_features,
output_mode='int',
output_sequence_length=max_len
)
model.add(vectorize_layer)
input_data = [["foo qux bar"], ["qux baz"]]
model.predict(input_data)
>>>
array([[2, 1, 4, 0],
[1, 3, 0, 0]])
If I want to implement a classifier using the sklearn library. Is there a way to save the model or convert the file into a saved tensorflow file in order to convert it to tensorflow lite later?
If you replicate the architecture in TensorFlow, which will be pretty easy given that scikit-learn models are usually rather simple, you can explicitly assign the parameters from the learned scikit-learn models to TensorFlow layers.
Here is an example with logistic regression turned into a single dense layer:
import tensorflow as tf
import numpy as np
from sklearn.linear_model import LogisticRegression
# some random data to train and test on
x = np.random.normal(size=(60, 21))
y = np.random.uniform(size=(60,)) > 0.5
# fit the sklearn model on the data
sklearn_model = LogisticRegression().fit(x, y)
# create a TF model with the same architecture
tf_model = tf.keras.models.Sequential()
tf_model.add(tf.keras.Input(shape=(21,)))
tf_model.add(tf.keras.layers.Dense(1))
# assign the parameters from sklearn to the TF model
tf_model.layers[0].weights[0].assign(sklearn_model.coef_.transpose())
tf_model.layers[0].bias.assign(sklearn_model.intercept_)
# verify the models do the same prediction
assert np.all((tf_model(x) > 0)[:, 0].numpy() == sklearn_model.predict(x))
It is not always easy to replicate a scikit model in tensorflow. For instance scitik has a lot of on the fly imputation libraries which will be a bit tricky to implement in tensorflow
I am trying to implement following network Fots for Text detection using the new tensorflow 2. The authors use the resnet as the backbone of their network, so my first thought was to use the tensoflow hub resnet for loading a pretrained network. But the problem is that i can not find a way to print the summary of the module, which is loaded from tfhub?
Is there any way to see the layers of the loaded modules from tf-hub?
Thanks
Update
Unfortunately is the resnet not available for tf2-hub, so i deceided to use the builtin keras implementation of resent, at least till there is a hub impl. of it.
Here is how i get the intermediate layers of resnet using tf2.keras.applications:
import numpy as np
import tensorflow as tf
from tensorflow import keras
layers_out = ["activation_9", "activation_21", "activation_39", "activation_48"]
imgs = np.random.randn(2, 640, 640, 3).astype(np.float32)
model = keras.applications.resnet50.ResNet50(input_shape=(640, 640, 3), include_top=False)
intermid_outputs= [model.get_layer(layer_name).output for layer_name in layers_out]
shared_conds = keras.Model(inputs=model.input, outputs=intermid_outputs)
Y = conv_shared(imgs)
shapes = [y.shape for y in Y]
print(shapes)
You can do something like this to examine the intermediate outputs:
resnet = hub.Module("https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/3")
outputs = resnet(np.random.rand(1,224,224,3), signature="image_feature_vector", as_dict=True)
for intermediate_output in outputs.keys():
print(intermediate_output)
Then, if you want to link an intermediate layer of the hub module to the rest of your graph, you can do:
resnet = hub.Module("https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/3")
features = resnet(images, signature="image_feature_vector", as_dict=True)["resnet_v2_50/block4"]
flatten = tf.reshape(features, (-1, features.shape[3]))
Assuming that we want to extract the features from the last block of the ResNet.
Assuming that you wanted to get the intermidiate outputs, and updating #gorjan answer for newer versions of resnet in tfhub, you could try something like this.
First load the model using the hub.KerasLayer with the argument return_endpoints=True (this is not available for all models and is not documented afaik, results may vary):
hub_model_layer = hub.KerasLayer("https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/5",
trainable=True,
arguments=dict(return_endpoints=True))
Then you can compile a model like this:
input = tf.keras.layers.Input((244, 244, 3))
dict_output = hub_model_layer(input)
C2 = dict_output['resnet_v2_50/block1/unit_1/bottleneck_v2/shortcut']
C3 = dict_output['resnet_v2_50/block2/unit_1/bottleneck_v2/shortcut']
C4 = dict_output['resnet_v2_50/block3/unit_1/bottleneck_v2/shortcut']
C5 = dict_output['resnet_v2_50/block4/unit_1/bottleneck_v2/shortcut']
model = tf.keras.Model(inputs=input, outputs=[C2, C3, C4, C5])
The variable dict_output is a dictionary with all the avilable endpoints, you can print it to search for the outputs that you want to use. They are not in order and I was not abe to find a way to recover the graph, but you can guess them by the layer names.
I have been working with Keras and really liked the model.summary()
It gives a good overview of the size of the different layers and especially an overview of the number of parameters the model has.
Is there a similar function in Tensorflow? I could find nothing on Stackoverflow or the Tensorflow API documentation.
Looks like you can use Slim
Example:
import numpy as np
from tensorflow.python.layers import base
import tensorflow as tf
import tensorflow.contrib.slim as slim
x = np.zeros((1,4,4,3))
x_tf = tf.convert_to_tensor(x, np.float32)
z_tf = tf.layers.conv2d(x_tf, filters=32, kernel_size=(3,3))
def model_summary():
model_vars = tf.trainable_variables()
slim.model_analyzer.analyze_vars(model_vars, print_info=True)
model_summary()
Output:
---------
Variables: name (type shape) [size]
---------
conv2d/kernel:0 (float32_ref 3x3x3x32) [864, bytes: 3456]
conv2d/bias:0 (float32_ref 32) [32, bytes: 128]
Total size of variables: 896
Total bytes of variables: 3584
Also here is an example of custom function to print model summary:
https://github.com/NVlabs/stylegan/blob/f3a044621e2ab802d40940c16cc86042ae87e100/dnnlib/tflib/network.py#L507
If you already have .pb tensorflow model you can use: inspect_pb.py to print model info or use tensorflow summarize_graph tool with --print_structure flag, also it's nice that it can detect input and output names.
I haven't seen anything like model.summary() for the tensorflow... However, I don't think you need it. There is a TensorBoard, where you can easily check the architecture of the NN.
https://www.tensorflow.org/get_started/graph_viz
You can use keras with the tensorflow backend to get the best features of either keras or tensorflow.