Loss tensor being pruned out of graph in PopART - ipu

Question

I’ve written a very simple PopART program using the C++ interface, but every time I try to compile it to run on an IPU device I get the following error:
terminate called after throwing an instance of ‘popart::error’
what(): Could not find loss tensor ‘L1:0’ in main graph tensors
I’m defining the loss in my program like so:
auto loss = builder->aiGraphcoreOpset1().l1loss({outputs[0]}, 0.1f, popart::ReductionType::Sum, “l1LossVal”);
Is there something wrong with my loss definition that’s resulting in it being pruned out of the graph? I’ve followed the same structure as one of the Graphcore examples here.

Answer 1

This error usually happens when the model protobuf you pass to the TrainingSession or InferenceSession objects doesn’t contain the loss tensor. A common reason for this is when you call builder->getModelProto() before you add the loss tensor to the graph. To ensure your loss tensor is part of the protobuf your calls should be in the following order:
...
auto loss = builder->aiGraphcoreOpset1().l1loss(...);
auto proto = builder->getModelProto();
auto session = popart::TrainingSession::createFromOnnxModel(...);
...
The key point is that the getModelProto() call should be the last call from the builder interface before setting up the PopART session.