Weighted sampling sequences to train a neural network in CNTK - cntk

Question

I am training a LSTM on weighted sequences with CNTK. I started from the following example of language understanding: https://github.com/Microsoft/CNTK/blob/master/Tutorials/CNTK_202_Language_Understanding.ipynb
To train the network, they produce a CNTK Text Format (CTF) file like :
19 |S0 178:1 |# BOS |S1 14:1 |# flight |S2 128:1 |# O
19 |S0 770:1 |# show |S2 128:1 |# O
19 |S0 429:1 |# flights |S2 128:1 |# O
19 |S0 444:1 |# from |S2 128:1 |# O
I have a weight associated to each sequence. Therefore, I generated the following output:
19 |weight 10 |S0 178:1 |# BOS |S1 14:1 |# flight |S2 128:1 |# O
19 |weight 10 |S0 770:1 |# show |S2 128:1 |# O
19 |weight 10 |S0 429:1 |# flights |S2 128:1 |# O
19 |weight 10 |S0 444:1 |# from |S2 128:1 |# O
I want to take the weight into account when training the network. One possible way to take it into account is modifying the loss function as follow: I multiply the cross entropy by the weight of an instance.
def create_criterion_function(model):
labels = Placeholder(name='labels')
weight = Placeholder(name='weight')
ce = weight * cross_entropy_with_softmax(model, labels)
errs = classification_error (model, labels)
return combine ([ce, errs]) # (features, labels) -> (loss, metric)
However, when I have many sequences, the network seem to not learn. I have been told that this is a case of catastrophic forgetting:
Catastrophic forgetting (also: catastrophic interference) is a term, often used in connectionist literature, to describe a common problem with many traditional artificial neural network models. It refers to the catastrophic loss of previously learned responses, whenever an attempt is made to train the network with a single new (additional) response.
Another solution might be that I could sample the instances in the minibatch according to the weight: sequences with higher weight should appear more often in the minibatches. Is there a way to do it in CNTK?

Answer 1

I think your approach is correct. However, note that scaling up the objective by 10 will scale up your gradient by 10 for these samples. Before looking into catastrophic forgetting, I would first try reducing your learning rate by 10, as to bring back the gradient steps into the same range as before.