Is it necessary to make the tensorflow serving system a lib jar package? - tensorflow-serving

Question

First,I have to approval that the structure of the serving is very good. But in some scenario for example 'picture object detection' ,when a picture comes ,the picture need to be processed by many models, if we loop send the image to the remote server ,and wait the returns, it would be get a large delay, and Image transfer is very resource intensive。
Because our company use java to provide external RPC services,so
I packaged the tensorflow serving into a '.so' lib ,
then I provide a java api ,Java calls the native method of the c lib package
so users can call serving locally as if they were calling serving remotely. and At the same time, saving the time of remote transmission。 below is my java structure:
java
and the code in java is very simple:
public class TensorflowServerPredictorImpl {
static {
try {
NativeLibLoader.initLoad();
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
private final long handle;//server c impl handle
public native long init(byte[] options);
private native byte[] predict(long handle,byte[] request);
public TensorflowServerPredictorImpl(ServerOptions.ServerConfig config){
handle = init(config.toByteArray());
}
public Predict.PredictResponse Predict(Predict.PredictRequest request)throws Exception{
byte[] requestByteArray = request.toByteArray();
byte[] responseByteArray = predict(this.handle,requestByteArray);
Predict.PredictResponse response = Predict.PredictResponse.parseFrom(responseByteArray);
return response;
}
}
and the use of lib is just like:
public class Test{
public static void main(String[] args)throws Exception{
URL url = Test.class.getResource("/");
String path = url.getPath()+"model_config_file.cfg";
ServerOptions.ServerConfig.Builder builder = ServerOptions.ServerConfig.newBuilder();
builder.setModelConfigFile(path);
ServerOptions.ServerConfig config = builder.build();
TensorflowServerPredictorImpl predictor = new TensorflowServerPredictorImpl(config);
Predict.PredictRequest request =buildRequest(1);
Predict.PredictResponse response = predictor.Predict(request);
}
the predictor Support for multithreading。
How do other people solve such problems? Does it make sense for me to do this?

Answer 1

I had a similar problem to solve and installed a tensorflow serving application in a docker container. The requests for classification (in my case time series) are sent to the serving via grpc (protobuf) which is a binary format. This worked well so far. Although making the protobuf interface work in java was quite a steep learning phase.
But now I have also to serve several models in different technologies (tensoflow, python sklearn, ...) and some models cannot be served by tensorflow serving. So I have to setup a broker application in python which receives the requests from the java application and sends them out to many models. A new nice feature of the broker is that it can now do a voting over the results of the different models and send back only the voting result to java.