Are there any pros to having a convolution layer using a filter the same size as the input data (i.e. the filter can only fit over the input one way)?
A filter the same size as the input data will collapse the output dimensions to
1 x 1 x n_filters, which could be useful towards the end of a network that has a low dimensional output like a single number for example.
One place this is used is in sliding window object detection, where redundant computation is saved by making only one forward pass to compute the output on all windows.
However, it is more typical to add one or more dense layers that give the desired output dimension instead of fully collapsing your data with convolution layers.
Related
My 1-layer CNN neuron network's input data_set has 10 channels. If I set filter channel equal to 16, then there will be 10*16=160 filters.
I want to use same 16 filter channel's weights for each input channel. means only use 16 filters for my input data_set. means the 10 input channels share same convolution filter weights.
Dose any one know how to do this in tensorflow? thanks a lot.
You could use the lower level tf.nn.conv1d with a filters arg constructed by tiling the same single-channel filters.
f0 = tf.get_variable('filters', shape=(kernel_width, 1, filters_out), initializer=...)
f_tiled = tf.tile(f0, (1, filters_in, 1))
output = tf.nn.conv1d(input, f_tiled, ...)
However, you would get the same effect (and it would be much more efficient and less error prone) to simply add all your input channels together to form a single-channel input then use the higher-level layers API.
conv_input = tf.reduce_sum(input, axis=-1, keepdis=True))
output = tf.layers.conv1d(conv_input, filters=...)
Note unless all your channels are almost equivalent, this is probably a bad idea. If you want to reduce the number of free parameters, consider multiple convolutions - a 1x1 to reduce the number of filters, other convolutions with wide kernels and non-linearities, then a 1x1 convolution to get back to a large number of filters. The reduce_sum in the above implementation is effectively a 1x1 convolution with fixed weights of tf.ones, and unless your dataset is tiny you'll almost certainly get a better result from learning the weights followed by some non-linearity.
I have two huge grids (input and output) representing some spatial data of the same area. I want to be able to generate the output pixel-by-pixel by feeding a neural network a small part of the input grid, around the pixel of interest.
The naive way of training and evaluating on the CNN would be to extract sections separately, and giving those to the fit() function. But if the sub-grid the CNN operates on is e.g. a 256×256 area of the input, then I would copy each data point 65536 (!!!) times per epoch.
So is there any way to have karas just use subsections of a bigger data structure as training?
To me, this sounds a bit like training RNN's on sequencial sections of a data series, instead of copying each section separately.
The performance consideration is mainly in the case of evaluating the model. I want to use this model to generate output grid of a huge geographical area (denmark) with a resolution of 12,5 cm
It seems to me that you are looking for a fully convolutional network (FCN).
By using only layers that scale in size with their inputs (banishing the use of dense layers specifically), an FCN is able to produce an output with a spatial range that grows proportionally with that of the input — typically, the ouput has the same resolution as the input, as in your case.
If your inputs are very large, you can still train an FCN on subimages. Then for inference, you can
run the network on your entire image: indeed, sometimes the inputs are too big to be batched together during training, but can be feed alone for inference.
or split your input into subimages and tile the results back. In that case, I would probably use overlapping tiles to avoid potential border effects.
You can probably go well with a Sequence generator.
You will still have to create slices for each batch, but taking slices isn't slow at all compared with the CNN operations.
And by using a keras.utils.Sequence, the generation of the batches is parallel with the model's execution, so no penalty:
class GridGenerator(keras.utils.Sequence):
def __init__(self, originalGrid_maybeFileName, outputGrid, subGridSize):
self.originalGrid = originalGrid_maybeFileName
self.outputGrid = outputGrid
self.subgridSize = subgridSize
def __len__(self):
#naive implementation, if grids are squares and the sizes are multiples of each other
self.divs = self.originalGrid.shape[:,:,1] // self.subgridSize
return self.divs * self.divs
def __getitem__(self,i):
row, column = divmod(i, self.divs)
#using channels_last
x= self.originalGrid[:,row:row+self.subgridSize, column:column+self.subgridSize]
y= self.outputGrid[:,row:row+self.subgridSize, column:column+self.subgridSize]
return x,y
If the full grid doesn't fit your PC's memory, then you should find ways of loading parts of the grid at a time. (Use the generator to load these parts)
Create the generator and train with fit_generator:
generator = GridGenerator(xGrid, yGrid, subSize)
#you can create additional generators to take a part of that as training and another part as validation
model.fit_generator(generator, len(generator), ...., workers = 4)
The workers argument determines how many batches will be loaded in parallel before sent to the model.
I'm working on a problem using Keras that has been presenting me with issues:
My X data is all of shape (num_samples, 8192, 8), but my Y data is of shape (num_samples, 4), where 4 is a one-hot encoded vector.
Both X and Y data will be run through LSTM layers, but the layers are rejecting the Y data because it doesn't match the shape of the X data.
Is padding the Y data with 0s so that it matches the dimensions of the X data unreasonable? What kind of effects would that have? Is there a better solution?
Edited for clarification:
As requested, here is more information:
My Y data represents the expected output of passing the X data through my model. This is my first time working with LSTMs, so I don't have an architecture in mind, but I'd like to use an architecture that works well with classifying long (8192-length) sequences of words into one of several categories. Additionally, the dataset that I have is of an immense size when fed through an LSTM, so I'm currently using batch-training.
Technologies being used:
Keras (Tensorflow Backend)
TL;DR Is padding one tensor with zeroes in all dimensions to match another tensor's shape a bad idea? What could be a better approach?
First of all, let's make sure your representation is actually what you think it is; the input to an LSTM (or any recurrent layer, for that matter) must be of dimensionality: (timesteps, shape), i.e. if you have 1000 training samples, each consisting of 100 timesteps, with each timestep having 10 values, your input shape will be (100,10,). Therefore I assume from your question that each input sample in your X set has 8192 steps and 8 values per step. Great; a single LSTM layer can iterate over these and produce 4-dimensional representations with absolutely no problem, just like so:
myLongInput = Input(shape=(8192,8,))
myRecurrentFunction = LSTM(4)
myShortOutput = myRecurrentFunction(myLongInput)
myShortOutput.shape
TensorShape([Dimension(None), Dimension(4)])
I assume your problem stems from trying to apply yet another LSTM on top of the first one; the next LSTM expects a tensor that has a time dimension, but your output has none. If that is the case, you'll need to let your first LSTM also output the intermediate representations at each time step, like so:
myNewRecurrentFunction=LSTM(4, return_sequences=True)
myLongOutput = myNewRecurrentFunction(myLongInput)
myLongOutput.shape
TensorShape([Dimension(None), Dimension(None), Dimension(4)])
As you can see the new output is now a 3rd order tensor, with the second dimension now being the (yet unassigned) timesteps. You can repeat this process until your final output, where you usually don't need the intermediate representations but rather only the last one. (Sidenote: make sure to set the activation of your last layer to a softmax if your output is in one-hot format)
On to your original question, zero-padding has very little negative impact on your network. The network will strain itself a bit in the beginning trying to figure out the concept of the additional values you have just thrown at it, but will very soon be able to learn they're meaningless. This comes at a cost of a larger parameter space (therefore more time and memory complexity), but doesn't really affect predictive power most of the time.
I hope that was helpful.
I am trying to model a classifier that contain Multi Dimensional Feature as input. Can any one knew of a dataset that contain multi dimensional Features?
Lets say for example: In mnist data we have pixel location as feature & feature value is a Single Dimensional grey scale value that varies from (0 - 255), But if we consider a colour image then in that case a single grey scale value is not sufficient, in this case also we will take the pixel location as feature but feature value will be of 3 Dimension( R(0-255) as one dimension, G(0-255) as second dimension and B(0-255) as third dimension) So in this case how can one solve using FeedForward Neural network?
SMALL SUGGESTIONS ALSO ACCEPTED.
The same way.
If you plug the pixels into your network directly just reshape the tensor to have H*W*3 length.
If you use convolutions note the the last parameter is the number of input/output dimensions. Just make sure the first convolution uses 3 as input.
I'm following udacity MNIST tutorial and MNIST data is originally 28*28 matrix. However right before feeding that data, they flatten the data into 1d array with 784 columns (784 = 28 * 28).
For example,
original training set shape was (200000, 28, 28).
200000 rows (data). Each data is 28*28 matrix
They converted this into the training set whose shape is (200000, 784)
Can someone explain why they flatten the data out before feeding to tensorflow?
Because when you're adding a fully connected layer, you always want your data to be a (1 or) 2 dimensional matrix, where each row is the vector representing your data. That way, the fully connected layer is just a matrix multiplication between your input (of size (batch_size, n_features)) and the weights (of shape (n_features, n_outputs)) (plus the bias and the activation function), and you get an output of shape (batch_size, n_outputs). Plus, you really don't need the original shape information in a fully connected layer, so it's OK to lose it.
It would be more complicated and less efficient to get the same result without reshaping first, that's why we always do it before a fully connected layer. For a convolutional layer, on the opposite, you'll want to keep the data in original format (width, height).
That is a convention with fully connected layers. Fully connected layers connect every node in the previous layer with every node in the successive layer so locality is not an issue for this type of layer.
Additionally by defining the layer like this we can efficiently calculate the next step by calculating the formula: f(Wx + b) = y. This would not be as easily possible with multidimensional input and reshaping the input is low cost and easy to accomplish.