for learning i tried to to use tensorflow probability to fit a simple 1 dimensional function (x,y) but i have realized i do something wrong when i call a model inside another model and i really need to understand what.
here is my model
mlp1 = tf.keras.Sequential(
[tf.keras.layers.Input(shape=(1), dtype=tf.float32),
Dense(20, activation="relu", name="layer1"),
Dense(20, activation="relu", name="layer2"),
Dense(1, activation="softplus",name="layer3"),
]
)
class subq0(tf.keras.Model):
def __init__(self):
super().__init__()
self.mlp1=mlp1()
def call(self, inputs):
sigma= self.mlp1(inputs)
return sigma
polynom = subq0()
optimizer = tf.optimizers.SGD(learning_rate=0.0001,momentum=0.9)
polynom.compile(optimizer=optimizer, loss= "mse" )
polynom.build(input_shape=(1,))
polynom.summary()
polynomial.fit(X_train, y_train , epochs= 100, verbose=0, batch_size=64 ,validation_data=(X_val,y_val) )
the error i get is
ValueError: You cannot build your model by calling build if your
layers do not support float type inputs. Instead, in order to
instantiate and build your model, call your model on real tensor data
(of the correct dtype).
The actual error from call is:
'tensorflow.python.framework.ops.EagerTensor' object is not callable.
Related
I am trying to create a custom layer in tensorflow to output the running maximum of its inputs. The layer has a memory variable and comparison function. I wrote the following
class ComputeMax(tf.keras.layers.Layer):
def __init__(self):
super(ComputeMax, self).__init__()
def build(self, input_shape):
self.maxval = tf.Variable(initial_value=tf.zeros((input_shape)),
trainable=False)
def call(self, inputs):
self.maxval.assign(tf.maximum(inputs, self.maxval))
return self.maxval
my_sum = ComputeMax()
x = tf.ones((1,2))
y = my_sum(x)
print(y.numpy()) # [1, 1]
y = my_sum(x)
print(y.numpy()) # [1, 1]
It works as above. When I try it in a test model:
model = Sequential()
model.add(tf.keras.Input(shape=(2)))
model.add(Dense(1, activation='relu'))
model.add(ComputeMax())
model.compile(optimizer='adam', loss='mse')
I get the error on compile:
ValueError: Cannot convert a partially known TensorShape to a Tensor: (None, 1)
What am I missing?
Actually, the layer needs to know the input neurons from the previous layer, which is the last value in input_shape. You are using input_shape as it is which is actually batch shape, leading to a layer of the shape of batch.
This implementation might help.
class ComputeMax(tf.keras.layers.Layer):
def __init__(self):
super(ComputeMax, self).__init__()
def build(self, input_shape):
self.maxval = tf.Variable(initial_value=tf.zeros((input_shape[-1])),
trainable=False)
def call(self, inputs):
self.maxval.assign(tf.maximum(inputs, self.maxval))
return self.maxval
But probably it won't give you answers with numpy 1d array.
I created a simple cnn to detect custom digits and I am trying to visualize the activations of my layers. When I run the following code layer_outputs = [layer.output for layer in model.layers[:9]] I get the error Layer conv2d has no inbound nodes
When I searched online, it said to define input shape of first layer, but I've already done that and I'm not sure why that is happening. Below is my model.
class myModel(Model):
def __init__(self):
super().__init__()
self.conv1 = Conv2D(filters=32, kernel_size=(3,3), activation='relu', padding='same',
input_shape=(image_height, image_width, num_channels))
self.maxPool1 = MaxPool2D(pool_size=(2,2))
self.conv2 = Conv2D(filters=64, kernel_size=(3,3), activation='relu', padding='same')
self.maxPool2 = MaxPool2D(pool_size=(2,2))
self.conv3 = Conv2D(filters=64, kernel_size=(3,3), activation='relu', padding='same')
self.maxPool3 = MaxPool2D(pool_size=(2,2))
self.flatten = Flatten()
self.d1 = Dense(128, activation='relu')
self.d2 = Dense(10, activation='softmax')
def call(self, x):
x = self.conv1(x)
x = self.maxPool1(x)
x = self.conv2(x)
x = self.maxPool2(x)
x = self.conv3(x)
x = self.maxPool3(x)
x = self.flatten(x)
x = self.d1(x)
x = self.d2(x)
return x
Based on your stated goal and what you've posted, I believe the problem here is slightly (and very understandably) misunderstanding the way the TensorFlow APIs work. The model object and its constituent parts only store state for the model, not the evaluation of it, for example the hyperparameters you've set and the parameters the model learns when its fed training data. Even if you worked to fix the problem with what you're trying, the .output of the layer objects as part of the model wouldn't return the activations you want to visualize. It instead returns the part of the TensorFlow graph that represents that part of the computation.
For what you want to do, you'll need to manipulate an object that's the result of calling the .predict function on the model that you've set up and trained. Or you could drop down to below the Keras abstractions and manipulate the tensors directly.
If I gave this more thought, there's probably a reasonably elegant way to get this by only evaluating your graph (i.e., calling .predict) once, but the most obvious naïve way is simply to instantiate several new models (or several subclasses of your model) with each of the layers of interest as the terminal output, which should get you what you want.
For example, you could do something like this for each of the layers whose outputs you're interested in:
my_test_image = # get an image
input = Input(shape=(None, 256, 256, 3)) # input size will need to be set according to the relevant model
outputs_of_interest = Model(input, my_model.layers[-2].output)
outputs_of_interest.predict(my_test_image) # <=== this has the output you want
I'm using tf.data.experimental.make_csv_dataset function to create the input to a Keras sequential model. My first layer is a DenseFeature that receives a list of tf.feature_column (indicator, bucketized, numeric etc). The following layers are Dense using relu activation. When I run the fit function I get the error: "Cannot create a tensor proto whose content is larger than 2GB.". What do I need to change to make this model train?
The below is the main part of the code:
train_input = tf.data.experimental.make_csv_dataset(["df_train.csv"], batch_size=64, label_name="loss_rate", num_epochs=1)
eval_input = tf.data.experimental.make_csv_dataset(["df_val.csv"], batch_size=64, label_name="loss_rate", shuffle=False, num_epochs=1)
#all_features is generated by a function (it has 87 tf.feature_column objects)
feature_layer = layers.DenseFeatures(all_features)
def deep_sequential_model():
model = tf.keras.Sequential([
feature_layer,
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(1, activation='sigmoid')
])
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(loss='mse',
optimizer=optimizer,
metrics=['mae', 'mse'])
return model
model = deep_sequential_model()
model.fit(train_input,
validation_data=eval_input,
epochs=10)
I'm getting the error:
/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py in __init__(self, node_def, g, inputs, output_types, control_inputs, input_types, original_op, op_def)
1696 "Cannot create a tensor proto whose content is larger than 2GB.")
1697 if not _VALID_OP_NAME_REGEX.match(node_def.name):
-> 1698 raise ValueError("'%s' is not a valid node name" % node_def.name)
1699 c_op = None
1700 elif type(node_def).__name__ == "SwigPyObject":
ValueError: '_5' is not a valid node name```
I've just found the problem. The csv that I was loading had an index column that was creating a Tensor without any name and I believe this was causing issues. I just removed the index from the csv and it worked.
I'm trying to reduce the GPU memory cost of DenseNet. I found someone use tf.contrib.layers.recompute_grad to complete this work in tensorflow. I have a Keras model and I want to use tf.contrib.layers.recompute_grad in Keras. I tried to use a Keras custom layer to wrap it up.
class Back_Recompute(Layer):
def __init__(self, filters, kernel_size, w_decay, **kwargs):
super(Back_Recompute, self).__init__(**kwargs)
self.n_filters = filters
self.we_decay = w_decay
self.ks = kernel_size
def call(self, ip):
global brcount
with tf.variable_scope('BR_{}'.format(brcount), use_resource=True):
def _x(inner_ip):
x = Conv2D(self.n_filters, self.ks, kernel_initializer='he_normal', padding='same', use_bias=False,
kernel_regularizer=l2(self.we_decay))(inner_ip)
return x
brcount = brcount + 1
_x = tf.contrib.layers.recompute_grad(_x)
return _x(ip)
def compute_output_shape(self, input_shape):
return (input_shape[0], input_shape[1], input_shape[2], self.n_filters)
But weird problem is that my train code was stuck in fit_generator and cannot execute without error message. Then I thought that was probably because of Keras layer, so I have changed Keras layer Conv2D to tf.layers.conv2d. When I run the train code, I got a error message: AttributeError: 'Activation' object has no attribute 'outbound_nodes'. I don't know how to use tf.contrib.layers.recompute_grad in Keras. Does someone can help me?
When I have defined a model like this:
def create_basic_model_terse(input, out_dims):
with default_options(activation=relu):
model = Sequential([
LayerStack(3, lambda i: [
Convolution((5,5), [32,32,64][i], init=glorot_uniform(), pad=True),
MaxPooling((3,3), strides=(2,2))
]),
Dense(64, init=glorot_uniform()),
Dense(out_dims, init=glorot_uniform(), activation=None)
])
return model(input)
How can I get some kind of information about each layer in the network like output shape / dimensions?
You can look at CNTK 202 tutorials. There are other tutorials such as CNTK 105 that also shows how to get different attributes of models.
For a model
def create_model():
with default_options(initial_state=0.1):
return Sequential([
Embedding(emb_dim),
Recurrence(LSTM(hidden_dim), go_backwards=False),
Dense(num_labels)
])
model = create_model()
print(len(model.layers))
print(model.layers[0].E.shape)
print(model.layers[2].b.value)