I would like to serve about ~600 models with Tensorflow Serving.
I am trying to find a solution to eventually reduce the number of models:
My models have the same architecture, only the weights changes.
Is it possible to load only one model and changing the weights?
Would it be possible to aggregate all those models together and effectively, the first layer of the model would be an ID and the input features for that model?
Has anyone tried having couple of hundreds models running on one machine? I have find this cortex solution, but wanted to avoid using another tech.
https://towardsdatascience.com/how-to-deploy-1-000-models-on-one-cpu-with-tensorflow-serving-ec4297bff54b
If the models have the same architecture but different weight, you can try merging all those model into a "super model". However I would need to know more about the task to see if that's possible.
To serve 600 models, you would need a very powerful machine and lot of memory (depending on how big your models are and how much you use them in parallel).
You can either run TFServe yourself, or use a provider such as Inferrd.com/Google/AWS.
Related
I am new to the Transformers concept and I am going through some tutorials and writing my own code to understand the Squad 2.0 dataset Question Answering using the transformer models. In the hugging face website, I came across 2 different links
https://huggingface.co/models
https://huggingface.co/transformers/pretrained_models.html
I want to know the difference between these 2 websites. Does one link have just a pre-trained model and the other have a pre-trained and fine-tuned model?
Now if I want to use, let's say an Albert Model For Question Answering and train with my Squad 2.0 training dataset on that and evaluate the model, to which of the link should I further?
I would formulate it like this:
The second link basically describes "community-accepted models", i.e., models that serve as the basis for the implemented Huggingface classes, like BERT, RoBERTa, etc., and some related models that have a high aceptance or have been peer-reviewed.
This list has bin around much longer, whereas the list in the first link only recently got introduced directly on the Huggingface website, where the community can basically upload arbitrary checkpoints that are simply considered "compatible" with the library. Oftentimes, these are additional models trained by practitioners or other volunteers, and have a task-specific fine-tuning. Note that al models from /pretrained_models.html are also included in the /models interface as well.
If you have a very narrow usecase, you might as well check and see if there was already some model that has been fine-tuned on your specific task. In the worst case, you'll simply end up with the base model anyways.
Folks, I am writing an application which will produce recommendations based on ML model call. The application will have different models, some of them should be called in sequence. A data scientist should be able, to upload a model in the system. This means that the application should have logic to store models metadata as well as address of a model server. A model server will be instantiated dynamically on a model upload event.
I would like to use a cluster of TensorFlow Serving here, however I am stacked with a question of architecture.
Is there a way to have something like service registry for TensorFlow servers? What is the best way to build such a cluster of servers with different models?
I need some clarification on what you're trying to do. Is the feature vector for all the models the same? If not then it will be quite a bit harder to do this. Trained models are encapsulated in the SavedModel format. It sounds like you're trying to train an ensemble, but some of the models are frozen? You could certainly write a custom component to make an inference request as part of the input to Trainer, if that's what you need.
UPDATE 1
From your comment below it sounds like what you might be looking for is a service mesh, such as Istio for example. That would help manage the connections between services running inside containers, and the connections between users and services. In this case tf.Serving instances running your models are the services, but the basic request-response pattern is the same. Does that help?
I'm trying to run several models of neural networks in tensorflow in parallel, each model is independent of the rest. It is necessary to create a session for each of the executions I launch with tensorflow or I could reuse the same session for each of the models ?. Thank you
A session is linked to a specific Tensorflow Graph instance. If you want to have one session for all, you need to put all your models in the same graph. This may cause you naming problems for tensors and is IMO generally a bad idea (you should keep things that are not related to each other separate).
Having everything in the same graph also raises your model's resources requirements (you always load everything even if you run only a sub-graph), which is another reason to split things in independent graphs. With independent graphs, you'll have to use multiple sessions.
Let us take a practical situation a researcher often finds him/herself into when using TensorFlow :
Multiple GPUs are available for training and I'd like to use them for speedup.
Subsequently I'd like to give the trained model to a colleague or collaborator with a different (maybe 1 !!) number of GPUs.
It is important that the code need not be modified for use when shared with multiple collaborators.
However, TensorFlow documentation/examples are not very clear/explained for such a scenario.
Basic questions are :
How do I write a code which involves training a model with multiple GPUs and where the model can be easily restored from checkpoints ?
How do I deal with the situation where my collaborators have different number of GPU resources ? More precisely, what best practices should I follow to ensure that the code and model I share with them is easily usable by them ?
Are there some examples or best practices other TensorFlow users (facing the same situation !!) can share ?
NOTE : I am not looking for a readymade solution. My prime purpose is to understand a TensorFlow feature which is not very well documented.
The tensorflow text summarization model as described here https://github.com/tensorflow/models/tree/master/textsum requires a multi GPU architecture in order to train. My repeated attempts at training the model has resulted in memory exceptions, machine crashing for various reasons. Is the trained summarisation model available so can make use of the summarization model without the need for training? The summarization model is trained using the not free Gigaword dataset, if the trained model is not available from Google is this a factor in reason why ?
So as far as I can tell, no one has put the trained model out there that is referenced. I too was originally running into memory issues on my macbook pro and eventually ended up using my gaming laptop which had a much better GPU.
The other option of course is to take advantage of AWS and use something like their g2.2xlarge instance. They also have their P2 instances as well, but I have not checked that out yet.
With regards to the Gigaword dataset, it simply comes down to licensing. It is not a free license from LDC and often many of the academics working on this have the dataset provided to them via their Universities or companies. I have not had luck finding it, however LDC did get back to me and advised that they do have other article datasets that have a pricetag of around $300 which is much more reasonable for those of use just trying to learn TF. That said, if you didn't want to buy anything, you can always write your own page scraper and format the data for the textsum model. https://github.com/tensorflow/models/pull/379/files
Hope this helps some. Good luck!