How do I perform sequence labeling in caffe using LSTM - sequence

Question

I have looked at the LRCN example (http://tutorial.caffe.berkeleyvision.org/caffe-cvpr15-sequences.pdf) which use LSTM for classification. For video classification a majority voting is done. Why is that? I would assume one waits until the end of a sequence?
In my toy, example binary counting, I inputted the labels in two different ways.
First I labelled every timestep with the sequence label. Secondly I labelled every timestep with an ignore_label but the last one. For simplicity I used a sequence length of 50 and a batchsize of 50 as well.
Both approaches lead to a network, where when I deploy it, I receive the same output for every timestep.
Edit:
The toy example works, if instead of classifying a whole sequence, one predicts the next number. Thus for each number a label exists. This is no solution for a real-world sequence classification task. Using the post by Kaparthy (http://karpathy.github.io/2015/05/21/rnn-effectiveness/) I have created following network:
name: "BasicLstm"
layer {
name: "data"
type: "HDF5Data"
top: "data"
top: "cont"
top: "label"
include {
phase: TRAIN
}
hdf5_data_param {
source: "./path_to_txt.txt"
batch_size: 2000
}
}
layer {
name: "lstm1"
type: "LSTM"
bottom: "data"
bottom: "cont"
top: "lstm1"
recurrent_param {
num_output: 5
weight_filler {
type: "uniform"
min: -0.08
max: 0.08
}
bias_filler {
type: "constant"
value: 0
}
}
}
layer {
name: "lstm2"
type: "LSTM"
bottom: "lstm1"
bottom: "cont"
top: "lstm2"
recurrent_param {
num_output: 4
weight_filler {
type: "uniform"
min: -0.08
max: 0.08
}
bias_filler {
type: "constant"
value: 0
}
}
}
layer {
name: "predict"
type: "InnerProduct"
bottom: "lstm2"
top: "predict"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 2
decay_mult: 0
}
inner_product_param {
num_output: 39
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
axis: 2
}
}
layer {
name: "softmax_loss"
type: "SoftmaxWithLoss"
bottom: "predict"
bottom: "label"
top: "loss"
loss_weight: 20
softmax_param {
axis: 2
}
loss_param {
ignore_label: -1
}
}
The important parts of the solver: I already have played a little bit with lr_policy: INV but in the end I tried it with fixed
net: "Basic.prototxt"
test_initialization: false
base_lr: 0.001
momentum: 0.9
lr_policy: "fixed"
display: 50
max_iter: 1000000
solver_mode: GPU
No sequence ranges over 2000.
I have put 10 sequences side by side.
I embedded my data in a one hot vector of size 132.
My data HDF5 file has these dimensions: XX*10*132*1
My data has a label at the end of each sequence. Every other label is -1 and will be ignored during backpropagation.
To increase efficiency I packed multiple short sequences together (they are below 2000 timesteps).
For classification I have used the python interface. When I classify a sequence following arises:
net.blobs['data'].reshape(726, 1, 132, 1)
net.blobs['cont'].reshape(726, 1)
net.blobs['data'].data[...] = data
net.blobs['cont'].data[...] = cont
output = net.forward()
output_prob = output['prob']
for i in range(726):
plt.plot(output_prob[i][0])
In the image one can see that for every timestep the same probabilities have been calculated.