Correct way to iterate over Keras ragged tensor - tensorflow

I have an input Tensorflow ragged tensor structured like this [batch num_images width height channels] and I need to iterate over the dimension num_images to extract some features relevant for downstream applications.
Example code is the following:
from tensorflow.keras.applications.efficientnet import EfficientNetB7
from tensorflow.keras.layers import Input
import tensorflow as tf
eff_net = EfficientNetB7(weights='imagenet', include_top=False)
input_claim = Input(shape=(None, 600, 600, 3), name='input_1', ragged=True)
eff_out = tf.map_fn(fn=eff_net,
elems=input_claim, fn_output_signature=tf.float32)
The first Input dimension is set to None as it can differ across data points, and for this reason the input receives instances of tf.RaggedTensor.
This code breaks with a TypeError in this way TypeError: Could not build a TypeSpec for KerasTensor(type_spec=RaggedTensorSpec(TensorShape([None, None, 600, 600, 3]), tf.float32, 1, tf.int64), name='input_1', description="created by layer 'input_1'") of unsupported type <class 'keras.engine.keras_tensor.RaggedKerasTensor'>.
I suspect there is a better way to perform this type of preprocessing though
Update: num_images is needed because (although not described here) I am doing some following reduce operation on this dimension

You can use tf.ragged.map_flat_values to achieve the same
Create a model like:
def eff_net(x): #dummy eff_net for testing that returns [batch, dim]
return tf.random.normal(shape=tf.shape(x)[:2])
input_claim = keras.Input(shape=(None, 600, 600, 3), name='input_1', ragged=True)
class RaggedMapLayer(layers.Layer):
def call(self, x):
return tf.ragged.map_flat_values(eff_net, x)
outputs = RaggedMapLayer()(input_claim)
model = keras.Model(inputs=input_claim, outputs=outputs)
testing,
inputs = tf.RaggedTensor.from_row_splits( tf.random.normal(shape=(10, 600, 600, 3)), row_splits=[0, 2, 5,10])
#shape [3, None, 600, 600, 3]
model(inputs).shape
#[3, None, 600]

Related

Why does the same Tensorflow model work with a list of arrays but doesn't work with tf.data.Dataset unbatched?

I have the following simple set up:
import tensorflow as tf
def make_mymodel():
class MyModel(tf.keras.models.Model):
def __init__(self):
super(MyModel, self).__init__()
self.model = tf.keras.Sequential([
tf.keras.layers.Input(shape=(1, 2)),
tf.keras.layers.Dense(1)
])
def call(self, x):
return self.model(x)
mymodel = MyModel()
return mymodel
model = make_mymodel()
X = [[[1, 1]],
[[2, 2]],
[[10, 10]],
[[20, 20]],
[[50, 50]]]
y = [1, 2, 10, 20, 50]
# ds_n_X = tf.data.Dataset.from_tensor_slices(X)
# ds_n_Y = tf.data.Dataset.from_tensor_slices(y)
# ds = tf.data.Dataset.zip((ds_n_X, ds_n_Y))
#
# for input, label in ds:
# print(input.numpy(), label.numpy())
loss_fn = tf.keras.losses.BinaryCrossentropy(from_logits=False)
model.build((1, 2))
model.compile(optimizer='adam',
loss=loss_fn)
model.summary()
model.fit(X, y, epochs=10)
print(model.predict([
[[25, 25]]
]))
This works fine (although I get strange predictions), but when I uncomment the ds lines and change model.fit(X, y, epochs=10) to model.fit(ds, epochs=10), I get the following error:
Traceback (most recent call last):
File "example_dataset.py", line 51, in <module>
model.fit(ds, epochs=10)
...
ValueError: slice index 0 of dimension 0 out of bounds. for '{{node strided_slice}} = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](Shape, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)' with input shapes: [0], [1], [1], [1] and with computed input tensors: input[1] = <0>, input[2] = <1>, input[3] = <1>.
The error gets solved when I run model.fit(ds.batch(2), epochs=10) (I added a batch instruction to the dataset).
I expect to be able to use a list of arrays and tf.data.Dataset interchangeably but, for some reason, I need to add a batch dimension to the dataset in order to use tf.data.Dataset. Is this expected behavior or am I conceptually missing something?
Because the model expects input as (batch_dim, input_dim). So, for your data, each input to the model should be like (None, 1, 2).
Let's explore the dimensions of your data by array and by dataset. While you define your input as array the shape is:
>>> print(np.array(X).shape)
(5, 1, 2)
It is compatible with what the model expects. But when you define a dataset using your array the shape is:
>>> for input, label in ds.take(1):
print(input.numpy().shape)
(1, 2)
And this is incompatible with what model expects, and if we batch the data:
>>> ds = ds.batch(1)
>>> for input, label in ds.take(1):
print(input.numpy().shape)
(1, 1, 2)
Then, it will be fine to pass dataset to the model.fit().

Input 0 is incompatible with layer model_1: expected shape=(None, 244, 720, 3), found shape=(None, 720, 3)

I wanted to test my model by uploading an image but I got this error. And I think I got the error somewhere in these lines, I'm just not sure how to fix.
IMAGE_SIZE = [244,720]
inception = InceptionV3(input_shape=IMAGE_SIZE + [3], weights='imagenet',include_top=False)
Also here's the code of uploading my test image
picture = image.load_img('/content/DSC_0365.JPG', target_size=(244,720))
img = img_to_array(picture)
prediction = model.predict(img)
print (prediction)
I'm still a newbie in Machine learning so my knowledge right now is not yet that deep.
This is mostly because you didn't prepare your input (its dimension) for your inception model. Here is one possible solution.
Model
from tensorflow.keras.applications import *
IMAGE_SIZE = [244,720]
inception = InceptionV3(input_shape=IMAGE_SIZE + [3],
weights='imagenet', include_top=False)
# check it's input shape
inception.input_shape
(None, 244, 720, 3)
Inference
Let's test a sample by passing it to the model.
from PIL import Image
a = Image.open('/content/1.png').convert('RGB')
display(a)
Check its basic properties.
a.mode, a.size, a.format
('RGB', (297, 308), None)
So, its shape already in (297 x 308 x 3). But to able to pass it to the model, we need an extra axis which is the batch axis. To do that, we can do
import tensorflow as tf
import numpy as np
a = tf.expand_dims(np.array(a), axis=0)
a.shape
TensorShape([1, 308, 297, 3])
Much better. Now, we may want to normalize our data and resize it according to the model input shape. To do that, we can do:
a = tf.divide(a, 255)
a = tf.image.resize(a, [244,720])
a.shape
TensorShape([1, 244, 720, 3])
And lastly, pass it to the model.
inception(a).shape
TensorShape([1, 6, 21, 2048])
# or, preserve the prediction to later analysis
y_pred = inception(a)
Updated
If you're using the [tf.keras] image processing function which loads the image into PIL format, then we can do simply:
image = tf.keras.preprocessing.image.load_img('/content/1.png',
target_size=(244,720))
input_arr = tf.keras.preprocessing.image.img_to_array(image)
input_arr = np.array([input_arr]) # Convert single image to a batch.
inception(input_arr).shape
TensorShape([1, 6, 21, 2048])

Tensorflow avoid shape information with crop

again I have some issue with Tensorflow. I am using a FCN model and need to apply a random crop due to memory usage.
tf.random_crop(combined, size=[512, 512, 4])
unfortunately now the new size "sticks" to the tensor and I can not get rid of it.
The issue caused by this is, that the resulting model only accepts input of size 512x512, which cannot be worked around in a nice way, as far as I know.
Is there any solution to either remove the shape information caused by random_crop or to easily adapt the size afterwards after obtaining a trained model?
Thank you in advance.
I don't know if it will completely suit your use-case, but the size parameter of tf.random_crop() can be a tensor, so you can for instance use a placeholder as shown in the example below.
import tensorflow as tf
import numpy as np
image = tf.placeholder(tf.float64, [None, None, 4])
cropped_size = tf.placeholder(tf.int32, [2])
cropped_image = tf.random_crop(image, size=[cropped_size[0], cropped_size[1], 4])
print(cropped_image.get_shape().as_list())
# [None, None, 4]
with tf.Session() as sess:
res = sess.run(cropped_image,
feed_dict={image: np.random.rand(900, 600, 4), cropped_size: [512, 512]})
print(res.shape)
# (512, 512, 4)
EDIT:
There may be different solutions to have the value of cropped_size assigned without using a feed_dict, depending how the crop dimensions are stored ; e.g. using TF file readers (the values would stay unknown till read).
Another simple hack otherwise: take advantage of tf.placeholder_with_default(default_val, shape) (doc), providing default_val with the crop dimensions acquired anyhow. As tf.placeholder_with_default() value isn't actually assigned until runtime (in case you you want to feed this placeholder with a different value), your dimensions would stay None in the graph:
import tensorflow as tf
image = tf.random_uniform((900, 600, 4)) # image tensor, acquired anyhow e.g. from tf.data
cropped_size_for_this_run = [512, 512] # crop dimensions, acquired anyhow
cropped_size = tf.placeholder_with_default(cropped_size_for_this_run, shape=[2])
cropped_image = tf.random_crop(image, size=[cropped_size[0], cropped_size[1], 4])
print(cropped_image.get_shape().as_list())
# [None, None, 4]
with tf.Session() as sess:
# You can leave cropped_size with its default value assigned at runtime:
res = sess.run(cropped_image)
print(res.shape)
# (512, 512, 4)
# ... or you can specify a new one if you wish so:
res = sess.run(cropped_image, feed_dict={cropped_size: [256, 256]})
print(res.shape)
# (256, 256, 4)
# ... It would switch back to the default value if you don't feed one:
res = sess.run(cropped_image)
print(res.shape)
# (512, 512, 4)

Tensorflow: TypeError with numpy_input_fn

I am coding a Convolutional Neural Network to classify images in TensorFlow but there is a problem:
When I try to feed my NumPy array of flattened images (3 channels with RGB values from 0 to 255) to a tf.estimator.inputs.numpy_input_fn I get the following error:
TypeError: Failed to convert object of type <class 'dict'> to Tensor.
Contents: {'x': <tf.Tensor 'random_shuffle_queue_DequeueMany:1' shape=(8,
196608) dtype=uint8>}. Consider casting elements to a supported type.
My numpy_imput_fn looks like this:
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={'x': train_x},
y=train_y,
batch_size=8,
num_epochs=None,
shuffle=True)
In the documentation for the function it is said that x should be a dict of NumPy array:
x: dict of numpy array object.
Nevermind, for those having the same problem I fixed it. In my model function i had:
input_layer = tf.reshape(features, [-1, 256, 256, 1])
Which raised the type error. To fix it you have to access the 'x' key in the features dictionary:
input_layer = tf.reshape(features['x'], [-1, 256, 256, 1])

Using Estimator for building an LSTM network

I am trying to build an LSTM network using an Estimator. My data looks like
X = [[1,2,3], [2,3,4], ... , [98,99,100]]
y = [2, 3, ... , 99]
I am using an Estimator:
regressor = learn.Estimator(model_fn=lstm_model,
params=model_params,
)
where the lstm_model function is
def lstm_model(features, targets, mode, params):
def lstm_cells(layers):
if isinstance(layers[0], dict):
return [tf.nn.rnn_cell.BasicLSTMCell(layer['steps'],state_is_tuple=True) for layer in layers]
return [tf.nn.rnn_cell.BasicLSTMCell(steps, state_is_tuple=True) for steps in layers]
stacked_lstm = tf.nn.rnn_cell.MultiRNNCell(lstm_cells(params['rnn_layers']), state_is_tuple=True)
output, layers = tf.nn.rnn(stacked_lstm, [features], dtype=tf.float32)
return learn.models.linear_regression(output, targets)
and params are
model_params = {
'steps': 1000,
'learning_rate': 0.03,
'batch_size': 24,
'time_steps': 3,
'rnn_layers': [{'steps': 3}],
'dense_layers': [10, 10]
}
and then I do the fitting
regressor.fit(X, y)
The issue I am facing is
output, layers = tf.nn.rnn(stacked_lstm, [features], dtype=tf.float32)
requires a sequence but I am not sure how to split my features to into list of tensors. The shape of features inside the lstm_model function is (?, 3)
I have two questions, how do I do the training in batches? and how do I split 'features' so
output, layers = tf.nn.rnn(stacked_lstm, [features], dtype=tf.float32)
doesn't throw and error. The error I am getting is
raise TypeError("%s that don't all match." % prefix)
TypeError: Tensors in list passed to 'values' of 'Concat' Op have types [float64, float32] that don't all match.
I am using tensorflow 0.12
I had to set the shape for features to be
(batch_size, time_step, 1) or (None, time_step, 1) and then unstack the features to go in the rnn. Unstacking the features in the "time_step" so you have a list of tensors with the size of time steps and the shape for each tensor should be (None, 1) or (batch_size, 1)