CNTK LSTM input shape - cntk

Question

We are building a LSTM network using the C# API for CNTK but finds it very difficult based on the current level of the CNTK documentation to settle on the proper shape / dimensions of the inputs.
We have a time series with a value (one number) at each time t and we want to use a sequence of the previous 744 values of the time series to make a prediction using LSTM. furthermore, is we want to make a minibatch with 25 sequences how the shape of the CNTK.InputVariable should look like:
[0] 744
[1] 1
[2] 25
or
[0] 1
[1] 744
[2] 25
… and then, if we instead of one value at each time t have two values, how will the CNTK.InputVariable shape then look like?

Answer 1

If you use recurrent networks (LSTM, GRU), then you need to know what static and dynamic axes are.
The static axis is used to describe the input data forms (in the first case it is a vector of rank 1 and size 1: new int {1}).
The dynamic axis is used to specify the sequence (variable length in your case 744) of your input data (in your case new int {1}). To indicate that the dynamic axis should be used for sequences, specify this in the input parameter dynamicAxes: new[] { Axis.DefaultBatchAxis() }
var inputDimension = 1; //for two values is 2 etc.
var inputShape = new { inputDimension };
var input = Variable.InputVariable(inputShape, DataType.Double, "input", new[] { Axis.DefaultBatchAxis() });
And be sure to prepare the minibatches correctly (example of creating one minibatch):
var device = DeviceDescriptor.CPUDevice;
var inputDimension = 1;
var outputDimension = 1;
var minibatchSize = 25;
var oneMinibatchFeaturesData = new List<List<double[]>>(minibatchSize)
{
new List<double[]> //first sequence
{
new double[] { 23 },//t=1. Array.Length = inputDimension
new double[] { 25 },//t=2
//...
new double[] { 65 },//t=744
},
new List<double[]> //second seqeunce
{
new double[] { 76 }, //t=1
new double[] { 236 },//t=2
//...
new double[] { 87 }, //t=744
},
//...
new List<double[]> //twenty fifth sequence
{
new double[] { 9 }, //t=1
new double[] { 2 },//t=2
//...
new double[] { 90 }, //t=744
},
};
var oneMinibatchLabelsData = new List<double[]>(minibatchSize)
{
new double[] { 1 },//label of first sequence. Array.Length = outputDimension
new double[] { 5 },//label of second sequence
//...
new double[] { 3 }//label of twenty fifth sequence
};
var features = Value.CreateBatchOfSequences(new[] { inputDimension }, oneMinibatchFeaturesData.Select(sequence => sequence.SelectMany(value => value)), device);
var labels = Value.CreateBatch(new[] { outputDimension }, oneMinibatchLabelsData.SelectMany(value => value), device);
The length of the sequence can be arbitrary. One minibatch may contain sequences of different lengths.
LSTM is difficult to train on sequences of this length. If the length of your sequence is always 744, then you should probably use a simple FNN with input dimension 744.

Answer 2

In depth study of the CNTK Readers is what I recommend!
#Stanislav Grigorev is correct.
The Input Dimensions is entirely dependant on your Dataset. For example, the ATIS in the example here, looks like this:
The code example can be found here.
Data is read in with the reader:
IList<StreamConfiguration> streamConfigurations = new StreamConfiguration[]
{
new StreamConfiguration(featuresName, inputDim, true, "S0"),
// new StreamConfiguration(featuresName, inputDim, true, "S1"), // Not used in the old example.
new StreamConfiguration(labelsName, numOutputClasses, false, "S2")
};
and the reading with the TextFormatMinibatchSource:
var minibatchSource = MinibatchSource.TextFormatMinibatchSource(
Path.Combine(DataFolder, "Train.ctf"),
streamConfigurations,
MinibatchSource.InfinitelyRepeat,
true);
var featureStreamInfo = minibatchSource.StreamInfo(featuresName);
var labelStreamInfo = minibatchSource.StreamInfo(labelsName);
then the line, in the while loop:
var minibatchData = minibatchSource.GetNextMinibatch(minibatchSize, device);
Reads each minibatch. This is all obvious to anyone reading the code, but to illustrate the way the data is read in, I have provided this example.
The Dataset Parameters are given in the code example:
const int inputDim = 2000;
const int numOutputClasses = 5;
It is important these numbers are correct!
I have started a website: http://www.cntking.com/ to try to get the ball rolling on C# and CNTK, I think its a very underestimated Language C# for Machine Learning.