I see tensorflow support AWS s3 file system (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/platform/s3) but I am unable to use the S3 path with tensorboard.
I tried latest nightly 0.4.0rc3 but no luck. I built locally also and made sure Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: set to YES but still i don't see tensorboard --logdir=s3://bucket/path working at all.
Am I missing something here?
If you start a tensorboard by using AWS s3 file, you should do as follows:
(1) add ENV VARIBLES
export AWS_ACCESS_KEY_ID=******
export AWS_SECRET_ACCESS_KEY=*******
export S3_ENDPOINT=******
export S3_VERIFY_SSL=0
export S3_USE_HTTPS=0
(2) upgrade tensorflow to newest version by using pip:
pip install tensorflow==1.4.1
(3) you don't need to upgrade tensorboard, because it is revolved in the previous step
Then you can start you tensorboard by using you code
tensorboard --logdir=s3://bucket/path
Related
I installed Uiniversal Sentence Encoder (Tensorflow 2) in 2 virtual environment with Ananconda. One is on Mac, anther is on Ubuntu.
All worked with following:
module_url = "https://tfhub.dev/google/universal-sentence-encoder/4"
model = hub.load(module_url)
Installed with:
conda create -n my-tf2-env python=3.6 tensorflow
conda init bash
conda activate my-tf2-env
conda install -c conda-forge tensorflow-hub
But, for unknown reason after 3 weeks, Mac does not work with following error which fails at:
model = hub.load(module_url)
Error: SavedModel file does not exist at: /var/folders/99/8rwn_9hx3jj9x3qz6yf0j2f00000gp/T/tfhub_modules/063d866c06683311b44b4992fd46003be952409c/{saved_model.pbtxt|saved_model.pb}
On Mac, I recreated new env with same procedure but has same error.
On Ubuntu, all works well.
I want to know how to fix Mac. Thank you for help.
What I attempted on Mac is that I tried to download "https://tfhub.dev/google/universal-sentence-encoder/4" to local drive and load it from local drive in future, not from web url. This process was not finished and not successful yet. I don't remember if there is anything downloaded to Mac with this attempt, that might corrupted Tensorflow-hub on login user account of my Mac.
This error usually occurs when the saved_model.pb is not present in the path specified in the module_url.
For example, if we consider the Folder structure as shown in the screenshot below,
The code,
import tensorflow_hub as hub
module_url = "https://tfhub.dev/google/universal-sentence-encoder/4"
model = hub.load(module_url)
and
import tensorflow_hub as hub
module_url = "/home/mothukuru/Downloads/Hub"
model = hub.load(module_url)
work successfully.
But if saved_model.pb is not present in that Folder as shown below,
Executing the code,
import tensorflow_hub as hub
module_url = "/home/mothukuru/Downloads/Hub"
model = hub.load(module_url)
results in the below error,
OSError: SavedModel file does not exist at: /home/mothukuru/Downloads/Hub/{saved_model.pbtxt|saved_model.pb}
In your specific case, executing the code while the Download of the Model was in progress might have resulted in the error.
As stated in the comment, deleting the Downloaded File can fix the problem.
Please let me know if this answer has not resolved your issue and I will be happy to modify it accordingly.
TF Published some additional guidelines on caching models apparently in response to questions about this issue.
In my case, I was running this locally on Mac via a jupyter notebook.
I was not sure how to "Delete the download file" as suggest in the other answer, but I found this resolved my issue:
https://www.tensorflow.org/hub/caching#reading_from_remote_storage
Reading from remote storage
Users can instruct the tensorflow_hub
library to directly read models from remote storage (GCS) instead of
downloading the models locally with
os.environ["TFHUB_MODEL_LOAD_FORMAT"] = "UNCOMPRESSED"
or by setting the command-line flag --tfhub_model_load_format to UNCOMPRESSED. This way, no caching directory is needed, which is especially helpful in environments that provide little disk space but a fast internet connection.
I ran that command in my notebook, and then the error was immediately resolved.
Note: I assume this is slower, especially if you do not have a fast internet connection, since what you are doing is telling the program to not locally cache (store) a copy and to just download it on demand.
Stylegan2 uses network pickle files to store ML models. I transfer trained one model, which I am able to open up on cloud servers. I have been generating images from this model fine with the following setup:
Google Colab: Python 3.6.9, CUDA 10.1, tensorflow-gpu 1.15, CuDNN 7.6.5
However, I cannot open the network pickle file on my local machine, even though I've been trying to replicate that cloud setup the best I can. (I have the right GPU hardware/drivers/etc.)
Local (Windows 10) Python 3.6.9, CUDA 10.1, tensorflow-gpu 1.15, CuDNN 7.6.5
Requires a library 'dnnlib' to be in the PYTHONPATH and for a tf.Session() to be initialized
I get the an assertion error about the pickle.
**Assertion error**: `assert state["version"] in [2,3]`
I find this error very odd because the network pickle works on the cloud--so it was saved properly. Additionally, my local set up can open up other network pickles(ie. ones downloaded from the internet through GET requests), making me think that I have properly set up my PYTHONPATH and initialized a tf.Session. These are prerequisites listed in the Stylegan repo:
"You can import the networks in your own Python code using pickle.load(). For this to work, you need to include the dnnlib source directory in PYTHONPATH and create a default TensorFlow session by calling dnnlib.tflib.init_tf()"
I'm not sure why I cannot open up this pickle in one environment, but can in another. Does anyone have any suggestions as to where I might start looking?
Actually, I figured it out by printing out what version was throwing the error. The version printed was '4'. I realized that this matched the pickle (HIGHEST_PROTOCOL) and that what I needed was the newest pull of the Stylegan2 repository, which included pickle format_version 4 in their allowed versions.
I am trying to follow this tutorial:
https://medium.com/#natu.neeraj/training-a-keras-model-on-google-cloud-ml-cb831341c196
to upload and train a Keras model on Google Cloud Platform, but I can't get it to work.
Right now I have downloaded the package from GitHub, and I have created a cloud environment with AI-Platform and a bucket for storage.
I am uploading the files (with the suggested folder structure) to my Cloud Storage bucket (basically to the root of my storage), and then trying the following command in the cloud terminal:
gcloud ai-platform jobs submit training JOB1
--module-name=trainer.cnn_with_keras
--package-path=./trainer
--job-dir=gs://mykerasstorage
--region=europe-north1
--config=gs://mykerasstorage/trainer/cloudml-gpu.yaml
But I get errors, first the cloudml-gpu.yaml file can't be found, it says "no such folder or file", and trying to just remove it, I get errors because it says the --init--.py file is missing, but it isn't, even if it is empty (which it was when I downloaded from the tutorial GitHub). I am Guessing I haven't uploaded it the right way.
Any suggestions of how I should do this? There is really no info on this in the tutorial itself.
I have read in another guide that it is possible to let gcloud package and upload the job directly, but I am not sure how to do this or where to write the commands, in my terminal with gcloud command? Or in the Cloud Shell in the browser? And how do I define the path where my python files are located?
Should mention that I am working with Mac, and pretty new to using Keras and Python.
I was able to follow the tutorial you mentioned successfully, with some modifications along the way.
I will mention all the steps although you made it halfway as you mentioned.
First of all create a Cloud Storage Bucket for the job:
gsutil mb -l europe-north1 gs://keras-cloud-tutorial
To answer your question on where you should write these commands, depends on where you want to store the files that you will download from GitHub. In the tutorial you posted, the writer is using his own computer to run the commands and that's why he initializes the gcloud command with gcloud init. However, you can submit the job from the Cloud Shell too, if you download the needed files there.
The only files we need from the repository are the trainer folder and the setup.py file. So, if we put them in a folder named keras-cloud-tutorial we will have this file structure:
keras-cloud-tutorial/
├── setup.py
└── trainer
├── __init__.py
├── cloudml-gpu.yaml
└── cnn_with_keras.py
Now, a possible reason for the ImportError: No module named eager error is that you might have changed the runtimeVersion inside the cloudml-gpu.yaml file. As we can read here, eager was introduced in Tensorflow 1.5. If you have specified an earlier version, it is expected to experience this error. So the structure of cloudml-gpu.yaml should be like this:
trainingInput:
scaleTier: CUSTOM
# standard_gpu provides 1 GPU. Change to complex_model_m_gpu for 4 GPUs
masterType: standard_gpu
runtimeVersion: "1.5"
Note: "standard_gpu" is a legacy machine type.
Also, the setup.py file should look like this:
from setuptools import setup, find_packages
setup(name='trainer',
version='0.1',
packages=find_packages(),
description='Example on how to run keras on gcloud ml-engine',
author='Username',
author_email='user#gmail.com',
install_requires=[
'keras==2.1.5',
'h5py'
],
zip_safe=False)
Attention: As you can see, I have specified that I want version 2.1.5 of keras. This is because if I don't do that, the latest version is used which has compatibility issues with versions of Tensorflow earlier than 2.0.
If everything is set, you can submit the job by running the following command inside the folder keras-cloud-tutorial:
gcloud ai-platform jobs submit training test_job --module-name=trainer.cnn_with_keras --package-path=./trainer --job-dir=gs://keras-cloud-tutorial --region=europe-west1 --config=trainer/cloudml-gpu.yaml
Note: I used gcloud ai-platform instead of gcloud ml-engine command although both will work. At some point in the future though, gcloud ml-engine will be deprecated.
Attention: Be careful when choosing the region in which the job will be submitted. Some regions do not support GPUs and will throw an error if chosen. For example, if in my command I set the region parameter to europe-north1 instead of europe-west1, I will receive the following error:
ERROR: (gcloud.ai-platform.jobs.submit.training) RESOURCE_EXHAUSTED:
Quota failure for project . The request for 1 K80
accelerators exceeds the allowed maximum of 0 K80, 0 P100, 0 P4, 0 T4,
0 TPU_V2, 0 TPU_V3, 0 V100. To read more about Cloud ML Engine quota,
see https://cloud.google.com/ml-engine/quotas.
- '#type': type.googleapis.com/google.rpc.QuotaFailure violations:
- description: The request for 1 K80 accelerators exceeds the allowed maximum of
0 K80, 0 P100, 0 P4, 0 T4, 0 TPU_V2, 0 TPU_V3, 0 V100.
subject:
You can read more about the features of each region here and here.
EDIT:
After the completion of the training job, there should be 3 folders in the bucket that you specified: logs/, model/ and packages/. The model is saved on the model/ folder a an .h5 file. Have in mind that if you set a specific folder for the destination you should include the '/' at the end. For example, you should set gs://my-bucket/output/ instead of gs://mybucket/output. If you do the latter you will end up with folders output, outputlogs and outputmodel. Inside output there should be packages. The job page link should direct to output folder so make sure to check the rest of the bucket too!
In addition, in the AI-Platform job page you should be able to see information regarding CPU, GPU and Network utilization:
Also, I would like to clarify something as I saw that you posted some related questions as an answer:
Your local environment, either it is your personal Mac or the Cloud Shell has nothing to do with the actual training job. You don't need to install any specific package or framework locally. You just need to have the Google Cloud SDK installed (in Cloud Shell is of course already installed) to run the appropriate gcloud and gsutil commands. You can read more on how exactly training jobs on the AI-Platform work here.
I hope that you will find my answer helpful.
I got it to work halfway now by not uploading the files but just running the upload commands from cloud at my local terminal... however there was an error during it running ending in "job failed"
Seems it was trying to import something from the TensorFlow backend called "from tensorflow.python.eager import context" but there was an ImportError: No module named eager
I have tried "pip install tf-nightly" which was suggested at another place, but it says I don't have permission or I am loosing the connection to cloud shell(exactly when I try to run the command).
I have also tried making a virtual environment locally to match that on gcloud (with Conda), and have made an environment with Conda with Python=3.5, Tensorflow=1.14.0 and Keras=2.2.5, which should be supported for gcloud.
The python program works fine in this environment locally, but I still get the (ImportError: No module named eager) when trying to run the job on gcloud.
I am putting the flag --python-version 3.5 when submitting the job, but when I write the command "Python -V" in the google cloud shell, it says Python=2.7. Could this be the issue? I have not fins a way to update the python version with the cloud shell prompt, but it says google cloud should support python 3.5. If this is anyway the issue, any suggestions on how to upgrade python version on google cloud?
It is also possible to manually there a new job in the google cloud web interface, doing this, I get a different error message: ERROR: Could not find a version that satisfies the requirement cnn_with_keras.py (from versions: none) and No matching distribution found for cnn_with_keras.py. Where cnn_with_keras.py is my python code from the tutorial, which runs fine locally.
Really don't know what to do next. Any suggestions or tips would be very helpful!
The issue with the GPU is solved now, it was something so simple as, my google cloud account had GPU settings disabled and needed to be upgraded.
I don't think I'm asking this question right but I have jupyter notebook that launches a Tensorflow training job with a python training script I wrote.
That training script requires certain modules. Seems my sagemaker training job is failing because some of the modules don't exist.
How can I ensure that my training job script has all the modules it needs?
Edit
An example of one of these modules is keras.
The odd thing is, I can import keras in the jupyter notebook, but when that import statement is in my training script then I get the No module named keras error
If you want to install multiple packages, one way is to upgrade to Sagemaker Python SDK v2. With this, you can create a requirements.txt in the same directory as your notebook, and run the training. Sagemaker will automatically take care of the installation.
If you want to stay on v1 SDK, you can add the following snippet to your entry_point script.
import subprocess
import sys
def install(package):
subprocess.check_call([sys.executable, "-q", "-m", "pip", "install", package])
install('keras')
The module script runs within a docker container which obviously does not have the dependency installed. Jupyter notebook on the other hand has keras pre-installed.
Easy way to do this is to have a requirements.txt file with all the requirements and then pass that on when creating your model.
env = {
'SAGEMAKER_REQUIREMENTS': 'requirements.txt', # path relative to `source_dir` below.
}
sagemaker_model = TensorFlowModel(model_data = 's3://mybucket/modelTarFile,
role = role,
entry_point = 'entry.py',
code_location = 's3://mybucket/runtime-code/',
source_dir = 'src',
env = env,
name = 'model_name',
sagemaker_session = sagemaker_session,
)
You can upload your requirements.txt file to s3 bucket which can be
accessible by sagemaker and download the file to your working
directory of the container using boto3. Install the libraries from
requirements.txt the entry file.
import os
import boto3
s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', '/opt/ml/code/requirements.txt')
os.command('pip install -r /opt/ml/code/requirements.txt')
The other way you can do it is by building your own container using
bring your own algorithm option provided by aws.
Ref-links:
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb
The EstimatorBase class (and TensorFlow class) accept the parameter dependencies which you can use as follows to pass your requirements.txt:
estimator = TensorFlow(
dependencies=['requirements.txt'], # copies this file
)
e.g.
estimator = TensorFlow(
entry_point='src/train.py',
dependencies=['requirements.txt'], # copies this file
)
or
estimator = TensorFlow(
source_dir='src', # this copies the entire src folder
entry_point='train.py', # when using source_dir has to be directly under that dir
dependencies=['requirements.txt'], # copies this file
)
This copies the requirements.txt file into your sourcedir.tar.gz along with the training code.
This may only work on newer image versions. I read that in older versions you may need to put the requirements.txt file in the same folder as your training code.
If this doesn't work, you can use pip download to download your dependencies defined in requirements.txt locally, then use the dependencies parameter to specify the folder to which you downloaded your dependencies.
Another option is in your entry_point .py file you can add
import os
if __name__ == "__main__":
os.system('pip install mymodule')
import mymodule
# rest of code goes here
This worked for me for simple modules such as pyparsing, but I think with keras you better just use a Tensorflow container that has keras preinstalled, as mentioned above.
The environment on your notebook instance is exclusive from the environment of your training job on SageMaker, unless it is local mode.
If you're using a custom docker image, then most likely your docker image doesn't have Keras installed.
If you are using the SageMaker predefined TensorFlow container, which is most likely invoked through the following code:
https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/estimator.py#L170
TensorFlow(entry_point='training_code.py',
blah,
blah
)
Then you will need to install your dependencies within that container. There are currently two modes for training for TensorFlow on SageMaker, "framework" and "script" mode.
If training through "framework" mode, which is only available with 1.12 and below, then you will be limited to using a keras_model_fn defined here:
https://github.com/aws/sagemaker-python-sdk/tree/v1.12.0/src/sagemaker/tensorflow#preparing-the-tensorflow-training-script
Installing your dependencies would be done by passing in a requirements.txt.
On "script mode", which is introduced with TensorFlow 1.11 and above:
https://github.com/aws/sagemaker-python-sdk/tree/master/src/sagemaker/tensorflow#training-with-tensorflow
Requirements.txt is not supported for "script" mode and instead it is recommended to install your dependencies within your user script, which would be your Python file that contains all of your Keras code.
Please let me know if there is anything I can clarify.
For examples:
https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-python-sdk/tensorflow_script_mode_quickstart
https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-python-sdk/tensorflow_iris_dnn_classifier_using_estimators
I want to use TensorBoard to visualize results stored on an S3 server, without downloading them to my machine. Ideally, this would work:
$ tensorboard --logdir s3://mybucket/summary
Assuming the tfevents files are stored under summary. However this does not work and returns UnimplementedError: File system scheme s3 not implemented.
Is there some workaround to enable TensorBoard to access the data on the server?
The S3 File system plugin for tensorflow was released in Version 1.4 in early October. You'll need to make sure your tensorflow-tensorboard version is at least pip install tensorflow-tensorboard==0.4.0-rc1
Then you can start the server:
tensorboard --logdir=s3://root-bucket/jobs/4/train