How does you create an LSTM layer in CNTK, from C++? - cntk

Question

CNTK generally provides a great C++ API, but I'm struggling to find out how to build LSTM or GRU layers from the C++ API. The only function I can find is OptimizedRNNStack. That function seems quite self-explanatory, except for the weights variable. So far I haven't managed to figure out how to initialize that weights variable. Looking at CNTK.core.bs, it seems to initialize the weights with:
ParameterTensor {0:0, initFilterRank=0, initOutputRank=-1, init=init,
initValueScale=initValueScale}`
but I can't figure out how to translate that into C++. For context - I'm trying to use CTC to build an OCR pipeline. Building everything in C++ is great, because I can use all the native data synthesis tools, and the entire pipeline can be trained and tested end-to-end. However, if I must build the model in Brainscript, I guess that's fine too.

Answer 1

The C++ API does not have a Layers library equivalent. I have struggled to do that as the static-typing nature of C++ makes it hard to support all those options. Let me share a private piece of C++ code that creates a GRU simimilar to the Layers library (without all the options).
Sorry this is not directly copy-pastable; please try to change the return value to Function, and change the lambda signature by creating two PlaceholderVariables, dh, and x. That funny let is short for const auto.
static BinaryModel GRU(size_t outputDim, const DeviceDescriptor& device)
{
let activation = [](const Variable& x) { return Tanh(x); };
auto W = Parameter({ outputDim * 3, NDShape::InferredDimension }, DataType::Float, GlorotUniformInitializer(), device, L"W");
auto R = Parameter({ outputDim * 2, outputDim }, DataType::Float, GlorotUniformInitializer(), device, L"R");
auto R1 = Parameter({ outputDim , outputDim }, DataType::Float, GlorotUniformInitializer(), device, L"R1");
auto b = Parameter({ outputDim * 3 }, 0.0f, device, L"b");
let stackAxis = vector<Axis>{ Axis(0) };
let stackedDim = (int)outputDim;
let one = Constant::Scalar(1.0f, device); // for "1 -"...
// e.g. https://en.wikipedia.org/wiki/Gated_recurrent_unit
return BinaryModel({ W, R, R1, b }, [=](const Variable& dh, const Variable& x)
{
let& dhs = dh;
// projected contribution from input(s), hidden, and bias
let projx3 = b + Times(W, x);
let projh2 = Times(R, dh);
let zt_proj = Slice(projx3, stackAxis, 0 * stackedDim, 1 * stackedDim) + Slice(projh2, stackAxis, 0 * stackedDim, 1 * stackedDim);
let rt_proj = Slice(projx3, stackAxis, 1 * stackedDim, 2 * stackedDim) + Slice(projh2, stackAxis, 1 * stackedDim, 2 * stackedDim);
let ct_proj = Slice(projx3, stackAxis, 2 * stackedDim, 3 * stackedDim);
let zt = Sigmoid(zt_proj)->Output(); // fun update gate z(t)
let rt = Sigmoid(rt_proj); // reset gate r(t)
let rs = dhs * rt; // "cell" c
let ct = activation(ct_proj + Times(R1, rs));
let ht = (one - zt) * ct + zt * dhs; // hidden state ht / output
return ht;
});
}