I am trying to run a code that requires tpu name and zone in config.
Using a TPU runtime, how can I find the TPU name and zone in Google Colab?
Google Colab doesn't expose TPU name or its zone.
However you can get the TPU IP using the following code snippet:
tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
print('Running on TPU ', tpu.cluster_spec().as_dict())
Related
I am trying to train chatbot model using tensorflow and seq to seq architecture using sagemaker also I have completed coding in spyder but when
I am trying to access cornel movie corpus dataset from s3 bucket into sagemaker it says no such file or directory even granting access to s3 bucket
if you're in a notebook: aws s3 cp s3://path_to_the_file /home/ec2-user/SageMaker will copy data from s3 to your SageMaker directory in the notebook (if you have the IAM permissions to do so)
if you're in the docker container of a SageMaker training job: you need to pass the s3 path to the SDK training call: estimator.fit({'mydata':'s3://path_to_the_file'}) and in the docker your tensorflow code must read from this path: opt/ml/input/data/mydata
The output of export_inference_graph.py is
- model.ckpt.data-00000-of-00001
- model.ckpt.info
- model.ckpt.meta
- frozen_inference_graph.pb
+ saved_model (a directory)
while the output of export_tflite_ssd_graph.py
- tflite_graph.pbtxt
- tflite_graph.pb
What is difference in both the frozen files?
I assume you are trying to use your object detection model on mobile devices. For which you need to convert your model to tflite version.
But, you cannot convert models like fasterRCNN to tflite. You need to go for SSD models to be used for mobile devices.
Another way to use model like fasterRCNN in your deployment is,
Use AWS EC2 tensorflow AMI, deploy your model on cloud and have it routed to your website domain or mobile device. When server gets an image through http form that user fills, model will process it on your cloud server and send it back to your required terminal.
I have tried using PyDrive to authenticate and get access of Google Drive.
I followed every step in here, https://pythonhosted.org/PyDrive/quickstart.html.
After downloading and renaming “client_secrets.json”, where should I put or use this file in Jupyter notebook environment, in order to access google-drive in google-colab for local runtime?
I am getting error, InvalidConfigError: Invalid client secrets file while saving PyDrive credentials.
In order to use Google Colab in 'local runtime'.
You have to install Jupyter and related packages, that I have been able to do, as mentioned here https://research.google.com/colaboratory/local-runtimes.html.
But how to access 'google-drive' now?
Disclaimer: I am very new to Neural Network and Tensorflow.
I am trying to create a QA application where user asks a question and the application gives the answer. Most of the traditional methods I tried did not work or is not accurate enough or requires manual intervention. I was researching about unsupervised QA application, that is when I came across BERT.
BERT as google claims is state of the art neural network model and achieved highest score in leader board for Squad 2.0. I wish to use this model for my application and test it's performance.
I have created a Windows 2012 Datacenter edition Virtual Machine in Compute Engine. I have created Cloud TPU using ctpu.
I have the BERT large uncased model in Cloud Storage.
How do I train the BERT large uncased model with SQUAD 2.0?
Please feel free to correct me if I am wrong, I have the understanding that Cloud TPU is just a device like CPU or GPU. However if you read this, they are explaining like Cloud TPU is a virtual machine ("On Cloud TPU you can run with BERT-Large as...").
Where do I run run_squad.py as mentioned in here?
python run_squad.py \
--vocab_file=$BERT_LARGE_DIR/vocab.txt \
--bert_config_file=$BERT_LARGE_DIR/bert_config.json \
--init_checkpoint=$BERT_LARGE_DIR/bert_model.ckpt \
--do_train=True \
--train_file=$SQUAD_DIR/train-v2.0.json \
--do_predict=True \
--predict_file=$SQUAD_DIR/dev-v2.0.json \
--train_batch_size=24 \
--learning_rate=3e-5 \
--num_train_epochs=2.0 \
--max_seq_length=384 \
--doc_stride=128 \
--output_dir=gs://some_bucket/squad_large/ \
--use_tpu=True \
--tpu_name=$TPU_NAME \
--version_2_with_negative=True
How to access the the storage bucket files from Virtual Machine for this argument vocab_file?
Is the external IP address the value for $TPU_NAME environment variable?
So TPUs currently only read from GCS. The model that you've downloaded should be uploaded to another GCS bucket of your own creation. That's how the TPU will access vocab_file and other files.
I have developed a model in Keras that works perfectly when reading data stored locally. However, I now want to take advantage of Google Cloud Platform's GPUs for training the model. I have set up the GPU on GCP and am working in a Jupyter notebook. I have moved my images to Google Cloud Storage.
My question is:
How can I access these images (specifically the directories - training, validation, test) directly from Cloud Storage using the Keras' flow_from_directory method of the ImageDataGenerator class?
here's my directory structure in Google Cloud Storage (GCS):
mybucketname/
class_1/
img001.jpg
img002.jpg
...
class_2/
img001.jpg
img002.jpg
...
class_3/
img001.jpg
img002.jpg
...
While I haven't yet figured out a way to read the image data directly from GCS, in the meantime I can copy the files directly from Cloud Storage to the VM via import os, sys os.system('gsutil cp -r gs://mybucketname/ .')