Accessing to Weights and Layers in Tensorflow Hub - tensorflow

When I try to get the model from tensorflow-hub resporitory.
I can see it as a Saved Model format, but I cant get access to model architecture as well as weights store for each layer.
import tensorflow_hub as hub
model = hub.load("https://tfhub.dev/tensorflow/centernet/hourglass_512x512/1")
)
Is there any formal way to work with it?
All the attribute I can get through model.__dict__ is not clear for a specific layer in the original model.
{'_self_setattr_tracking': True,
'_self_unconditional_checkpoint_dependencies': [TrackableReference(name='_model', ref=<tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject object at 0x7fe4e4914710>),
TrackableReference(name='signatures', ref=_SignatureMap({'serving_default': <ConcreteFunction signature_wrapper(input_tensor) at 0x7FE4E601F210>})),
TrackableReference(name='_self_saveable_object_factories', ref=DictWrapper({}))],
'_self_unconditional_dependency_names': {'_model': <tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject at 0x7fe4e4914710>,
'signatures': _SignatureMap({'serving_default': <ConcreteFunction signature_wrapper(input_tensor) at 0x7FE4E601F210>}),
'_self_saveable_object_factories': {}},
'_self_unconditional_deferred_dependencies': {},
'_self_update_uid': 176794,
'_self_name_based_restores': set(),
'_self_saveable_object_factories': {},
'_model': <tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject at 0x7fe4e4914710>,
'signatures': _SignatureMap({'serving_default': <ConcreteFunction signature_wrapper(input_tensor) at 0x7FE4E601F210>}),
'__call__': <tensorflow.python.saved_model.function_deserialization.RestoredFunction at 0x7fe315a28950>,
'graph_debug_info': ,
'tensorflow_version': '2.4.0',
'tensorflow_git_version': 'unknown'}
I have also tried with model.signatures['serving_default'].__dict__, the Tensor represents for each layer is not visible
[<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>,
<tf.Tensor: shape=(), dtype=resource, numpy=<unprintable>>],

With the CLI tool saved_model_cli provided by the package tensorflow-serving-api it's possible to inspect a saved model. In the first step I downloaded and cached the model:
from os import environ
import tensorflow_hub as hub
environ['TFHUB_CACHE_DIR'] = '/Users/you/.cache/tfhub_modules'
hub.load("https://tfhub.dev/tensorflow/centernet/hourglass_512x512/1")
Then I inspected the signatures and layers:
saved_model_cli show --dir /Users/you/.cache/tfhub_modules/3085eb2fbe2ad0b69801d50844c97b7a7a5ecade --all
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['__saved_model_init_op']:
The given SavedModel SignatureDef contains the following input(s):
The given SavedModel SignatureDef contains the following output(s):
outputs['__saved_model_init_op'] tensor_info:
dtype: DT_INVALID
shape: unknown_rank
name: NoOp
Method name is:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_tensor'] tensor_info:
dtype: DT_UINT8
shape: (1, -1, -1, 3)
name: serving_default_input_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 4)
name: StatefulPartitionedCall:0
outputs['detection_classes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:1
outputs['detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:2
outputs['num_detections'] tensor_info:
dtype: DT_FLOAT
shape: (1)
name: StatefulPartitionedCall:3
Method name is: tensorflow/serving/predict
After that I used the debugger to understand how a saved model works internally and found the member fields variables and trainable_variables in model.signatures['serving_default'] which stores the data (weights, ...) of the model. Here you see the output of model.signatures['serving_default'].variables:

Short summary for the answer. We can access to the variables of a layer by model.signatures['serving_default'].variables

Related

Are those Keras and PyTorch snippets equivalent?

I am wondering if I succeeded in translating the following definition in PyTorch to Keras?
In PyTorch, the following multi-layer perceptron was defined:
from torch import nn
hidden = 128
def mlp(size_in, size_out, act=nn.ReLU):
return nn.Sequential(
nn.Linear(size_in, hidden),
act(),
nn.Linear(hidden, hidden),
act(),
nn.Linear(hidden, hidden),
act(),
nn.Linear(hidden, size_out),
)
My translation is
from tensorflow import keras
from keras import layers
hidden = 128
def mlp(size_in, size_out, act=keras.layers.ReLU):
return keras.Sequential(
[
layers.Dense(hidden, activation=None, name="layer1", input_shape=(size_in, 1)),
act(),
layers.Dense(hidden, activation=None, name="layer2", input_shape=(hidden, 1)),
act(),
layers.Dense(hidden, activation=None, name="layer3", input_shape=(hidden, 1)),
act(),
layers.Dense(size_out, activation=None, name="layer4", input_shape=(hidden, 1))
])
I am particularly confused about the input/output arguments, because that seems to be where tensorflow and PyTorch differ.
From the documentation:
When a popular kwarg input_shape is passed, then keras will create an
input layer to insert before the current layer. This can be treated
equivalent to explicitly defining an InputLayer.
So, did I get it right?
In Keras, you can provide an input_shape for the first layer or alternatively use the tf.keras.layers.Input layer. If you do not provide either of these details, the model gets built the first time you call fit, eval, or predict, or the first time you call the model on some input data. So the input shape will actually be inferred if you do not provide it. See the docs for more details. PyTorch generally infers the input shape at runtime.
def keras_mlp(size_in, size_out, act=layers.ReLU):
return keras.Sequential([layers.Input(shape=(size_in,)),
layers.Dense(hidden, name='layer1'),
act(),
layers.Dense(hidden, name='layer2'),
act(),
layers.Dense(hidden, name='layer3'),
act(),
layers.Dense(size_out, name='layer4')])
def pytorch_mlp(size_in, size_out, act=nn.ReLU):
return nn.Sequential(nn.Linear(size_in, hidden),
act(),
nn.Linear(hidden, hidden),
act(),
nn.Linear(hidden, hidden),
act(),
nn.Linear(hidden, size_out))
You can compare their summary.
For Keras:
>>> keras_mlp(10, 5).summary()
Model: "sequential_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
layer1 (Dense) (None, 128) 1408
re_lu_6 (ReLU) (None, 128) 0
layer2 (Dense) (None, 128) 16512
re_lu_7 (ReLU) (None, 128) 0
layer3 (Dense) (None, 128) 16512
re_lu_8 (ReLU) (None, 128) 0
layer4 (Dense) (None, 5) 645
=================================================================
Total params: 35,077
Trainable params: 35,077
Non-trainable params: 0
_________________________________________________________________
For PyTorch:
>>> summary(pytorch_mlp(10, 5), (1,10))
============================================================================
Layer (type:depth-idx) Output Shape Param #
============================================================================
Sequential [1, 5] --
├─Linear: 1-1 [1, 128] 1,408
├─ReLU: 1-2 [1, 128] --
├─Linear: 1-3 [1, 128] 16,512
├─ReLU: 1-4 [1, 128] --
├─Linear: 1-5 [1, 128] 16,512
├─ReLU: 1-6 [1, 128] --
├─Linear: 1-7 [1, 5] 645
============================================================================
Total params: 35,077
Trainable params: 35,077
Non-trainable params: 0
Total mult-adds (M): 0.04
============================================================================
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.14
Estimated Total Size (MB): 0.14
============================================================================

use tfds.load download the datasets error

when i want to download the datasets by tfds.load(),just like that
ratings = tfds.load('movielens/100k-ratings', split="train")
the error is:
Downloading and preparing dataset Unknown size (download: Unknown size, generated: Unknown size, total: Unknown size) to C:\Users\samsung\tensorflow_datasets\movielens\100k-ratings\0.1.0...
Dl Completed...: 0%
0/1 [00:21<?, ? url/s]
Dl Size...:
0/0 [00:21<?, ? MiB/s]
Extraction completed...:
0/0 [00:21<?, ? file/s]
HTTPConnectionPool(host='files.grouplens.org', port=80): Max retries exceeded with url: /datasets/movielens/ml-100k.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001E81008F910>: Failed to establish a new connection: [WinError 10060]
by the way, I use the company computer.
could someone help me.help!!!
Do you have the same dataset at the download floder
It cannot use the shuffle Fn but you can export and updates
ds = tfds.load('movielens/100k-ratings', split='train', shuffle_files=True)
👉👉👉 ds = ds.shuffle(1024).batch(64).prefetch(tf.data.experimental.AUTOTUNE)
assert isinstance(ds, tf.data.Dataset)
for example in ds.take(1):
print(example)
# {'bucketized_user_age': <tf.Tensor: shape=(), dtype=float32, numpy=45.0>, 'movie_genres': <tf.Tensor: shape=(1,), dtype=int64, numpy=array([7], dtype=int64)>,
# 'movie_id': <tf.Tensor: shape=(), dtype=string, numpy=b'357'>, 'movie_title': <tf.Tensor: shape=(), dtype=string, numpy=b"One Flew Over the Cuckoo's Nest (1975)">,
# 'raw_user_age': <tf.Tensor: shape=(), dtype=float32, numpy=46.0>, 'timestamp': <tf.Tensor: shape=(), dtype=int64, numpy=879024327>,
# 'user_gender': <tf.Tensor: shape=(), dtype=bool, numpy=True>, 'user_id': <tf.Tensor: shape=(), dtype=string, numpy=b'138'>,
# 'user_occupation_label': <tf.Tensor: shape=(), dtype=int64, numpy=4>, 'user_occupation_text': <tf.Tensor: shape=(), dtype=string, numpy=b'doctor'>,
# 'user_rating': <tf.Tensor: shape=(), dtype=float32, numpy=4.0>, 'user_zip_code': <tf.Tensor: shape=(), dtype=string, numpy=b'53211'>}

module 'tensorflow_core._api.v2.data' has no attribute 'Iterator'

Can't figure out what to use instead of Iterator
I tried tf.compat.v1.data.Iterator instead but got another error - AttributeError: 'PrefetchDataset' object has no attribute 'output_types'
code:
train_ds = prepare_for_train(labeled_ds)
val_ds = tf.data.Dataset.from_tensor_slices(test_data)
#create a iterator with shape and type
iter = tf.data.Iterator.from_structure(train_ds.output_types, train_ds.output_shapes)
"""iter= tf.compat.v1.data.Iterator.from_structure(train_ds.output_types, train_ds.output_shapes)"""
print(iter)
*AttributeError: module 'tensorflow_core._api.v2.data' has no attribute 'Iterator'*
My TF version 2.2.0-dev20200212
Thank you!
I was able to reproduce your error. Here is how you can fix it in Tensorflow Version 2.x.
You need to define iter as below -
iter = tf.compat.v1.data.Iterator.from_structure(tf.compat.v1.data.get_output_types(train_dataset),
tf.compat.v1.data.get_output_shapes(train_dataset))
Below is an example -
Code -
%tensorflow_version 2.x
import tensorflow as tf
print(tf.__version__)
import numpy as np
# Reinitializable iterator to switch between Datasets
EPOCHS = 10
# making fake data using numpy
train_data = (np.random.sample((100,2)), np.random.sample((100,1)))
# create two datasets, one for training and one for test
train_dataset = tf.data.Dataset.from_tensor_slices(train_data)
# create a iterator of the correct shape and type
iter = tf.compat.v1.data.Iterator.from_structure(tf.compat.v1.data.get_output_types(train_dataset),
tf.compat.v1.data.get_output_shapes(train_dataset))
# create the initialisation operations
train_init_op = iter.make_initializer(train_dataset)
features, labels = iter.get_next()
for _ in range(EPOCHS):
print([features, labels])
Output -
2.1.0
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/iterator_ops.py:347: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/iterator_ops.py:348: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/iterator_ops.py:350: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
[<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.35431711, 0.07564416])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.38728039])>]
Hope this answers your question. Happy Learning.
tf.compat.v1.disable_eager_execution()
train_ds = prepare_for_train(labeled_ds)
val_ds = tf.data.Dataset.from_tensor_slices(test_data)
#create a iterator with shape and type
iter = tf.data.Iterator.from_structure(train_ds.output_types, train_ds.output_shapes)
"""iter= tf.compat.v1.data.Iterator.from_structure(train_ds.output_types, train_ds.output_shapes)"""
print(iter)
Using this should solve the issue
*AttributeError: module 'tensorflow_core._api.v2.data' has no attribute 'Iterator'*

The initial state or constants of an RNN layer cannot be specified with a mix of Keras tensors and non-Keras tensors

As we know the decoder takes the encoder hidden states as the initial state ...
encoder_output , state_h, state_c = LSTM(cellsize, return_state=True)(embedded_encoder_input)
encoder_states = [state_h, state_c]
decoder_lstm = LSTM(cellsize, return_state=True, return_sequences=True)
decoder_outputs, state_dh, state_dc = decoder_lstm(embedded_decoder_inputs, initial_state=encoder_states)
Assume I want to replace the initial state of the decoder to be encoder_output and features from me from other resources
encoder_states = [encoder_output , my_state]
But I face the following error:
ValueError: The initial state or constants of an RNN layer cannot be
specified with a mix of Keras tensors and non-Keras tensors (a "Keras
tensor" is a tensor that was returned by a Keras layer, or by Input)
Although I print state_h & stat_c & encoder_output & my_state, all have the same type and shape, example:
state_h: Tensor("lstm_57/while/Exit_2:0", shape=(?, 128), dtype=float32)
my_state: Tensor("Reshape_17:0", shape=(?, 128), dtype=float32)
What am I understanding that it will not accept inputs not produced from the previous layer, and as Keras tensor?
Update
After convert tensor to Keras tensor, The new error:
ValueError: Input tensors to a Model must come from
keras.layers.Input. Received: Tensor("Reshape_18:0", shape=(?, 128),
dtype=float32) (missing previous layer metadata).
I guess you mixed tensorflow tensor and keras tensor. Although the results of state_h and my_state are tensor, they are actually different. You can use K.is_keras_tensor() to distinguish them. An example:
import tensorflow as tf
import keras.backend as K
from keras.layers import LSTM,Input,Lambda
my_state = Input(shape=(128,))
print('keras input layer type:')
print(my_state)
print(K.is_keras_tensor(my_state))
my_state = tf.placeholder(shape=(None,128),dtype=tf.float32)
print('\ntensorflow tensor type:')
print(my_state)
print(K.is_keras_tensor(my_state))
# you may need it
my_state = Lambda(lambda x:x)(my_state)
print('\nconvert tensorflow to keras tensor:')
print(my_state)
print(K.is_keras_tensor(my_state))
# print
keras input layer type:
Tensor("input_3:0", shape=(?, 128), dtype=float32)
True
tensorflow tensor type:
Tensor("Placeholder:0", shape=(?, 128), dtype=float32)
False
convert tensorflow to keras tensor:
Tensor("lambda_1/Identity:0", shape=(?, 128), dtype=float32)
True

Calculate gradients of intermediate nodes in tensorflow eager execution

I use tensorflow eager execution to do the following calculation:
y = x^2
z = y + 2.
My goal is to calculate dz/dx and dz/dy (the gradients of z over y and z)
dx, dy = GradientTape.gradient(z, [x, y]).
However, only dy is calculated and dx is None. Namely, only the gradients of tensors that directly rely on z can be calculated.
[None, <tf.Tensor: id=11, shape=(), dtype=float32, numpy=1.0>]
[None, <tf.Tensor: id=11, shape=(), dtype=float32, numpy=1.0>]
[None, <tf.Tensor: id=11, shape=(), dtype=float32, numpy=1.0>]
[None, <tf.Tensor: id=11, shape=(), dtype=float32, numpy=1.0>]
[None, <tf.Tensor: id=11, shape=(), dtype=float32, numpy=1.0>]
The following is the full code.
from __future__ import absolute_import, division, print_function
import tensorflow as tf
tf.enable_eager_execution()
tfe = tf.contrib.eager
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "6"
import warnings
warnings.filterwarnings('ignore')
train_steps = 5
for i in range(train_steps):
x = tf.contrib.eager.Variable(0.)
with tf.GradientTape() as tape:
y = tf.square(x)
z = y + 2
print(tape.gradient(z, [x,y]))
Any solution?