I'm serving a model using TensorFlow serving. After attacking to the system to serve 10 requests per second, the status of my server is:
It shows that all CPUs are busy while my GPU is idle. I found that about 50% of my requests takes longer than 30 seconds.
Why TensorFlow Serving doesn't leverage my GPU?
Related
I'm working remotely and from time to time I need to use the GPU for model training. I connect to the company network using ssh. Is there a way to see if someone is currently using the GPU for training?
For the past week, I have been failing to connect to a GPU, even though I have no active sessions whatsoever.
The message that keeps getting popped is the following:
Cannot connect to GPU backend
You cannot currently connect to a GPU due to usage limits in Colab. Learn more
As a Colab Pro subscriber you have higher usage limits than non-subscribers, but availability is not unlimited. To get the most out of Colab Pro, avoid using GPUs when they are not necessary for your work.
Note that I have a Colab Pro account.
If you excessively use GPUs you will go over the Colab Pro quota of 24h. Then, you will be restricted from usage for at least 12h.
Colab Pro is better and more flexible than the free version, but it still has its limitations.
I am running some edge tpu model on edge tpu devices. Is there anyway I can read how much memory from TPU is consuming during the run?
I've been enjoying the free colab TPUs and I am looking to upgrade to the GCP ones, but I am a little concerned about the time limits for TPU colabs, I heard colab only allows a certain number of hours for each user.
So I am wondering if I could just use a CPU or GPU instance, and connect to the TPU on my GCP.
I have a Tensorflow model which is working perfectly fine on my laptop (Tf 1.8 on OS HighSierra). However, I wanted to scale my operations up and use Amazon's Virtual Machine to run predictions faster. What is the best way to use my saved model and classify images in jpeg format which are stored locally? Thank you!
you have two options:
1) Start a virtual machine on AWS (known as an Amazon EC2 instance). You can pick from many different instance types, including GPU instances. You'll have full administrative access on this machine, meaning that you can copy you TF model to it and predict just like you would on your own machine.
More details on getting started with EC2 here: https://aws.amazon.com/ec2/getting-started/
I would also recommend using the Deep Learning Amazon Machine Image, which bundles all the popular ML/DL tools as well as the NVIDIA environment for GPU training/prediction : https://aws.amazon.com/machine-learning/amis/
2) If you don't want to manage virtual machines, I'd recommend looking at Amazon SageMaker. You'll be able to import your TF model and to deploy it on fully-managed infrastructure for prediction.
Here's a sample notebook showing you how to bring your own TF model to SageMaker: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/tensorflow_iris_byom/tensorflow_BYOM_iris.ipynb
Hope this helps.