In a Google Collab notebook once I execute the commands:
%load_ext tensorboard
%tensorboard --logdir <data_directory>
I am unable to save the model in the cloud, not only automatically but also manually.
Someone has an idea on why this can be happening?
If its helpful for someone else; executing in an independent module at the beginning the command:
%load_ext tensorboard
And at the end of the program in a separated module
%tensorboard --logdir <folder_name>
Worked for me and allows to save my work despite the TensorBoard module being loaded in the system
Related
I was trying to run a command to test local predict in my computer. However, the command failed every time with this error.
ERROR: gcloud crashed (PermissionError): [WinError 5] Access is denied
This is the command:
gcloud ai-platform local predict --model-dir model_final --json-instances image_b64.json --framework tensorflow
I am positive 101% positive that I have followed everything in the doc by Google.
First, the command required a model file to be saved in TensorFlow SavedModel format, which, since I use Keras, I can just do model.save("model_final").
If you have used Keras for training, use tf.keras.Model.save to export a SavedModel
So I did, at it only output a single file, so I can only assume it's the file to be placed in the --model-dir parameter. I admit doing model.save("model_final") created a file, not a dir, which is a bit weird but the document for Keras just said use that so there is no way I could be wrong.
And also:
If you export your SavedModel using tf.keras.Model.save, then you do not need to specify a serving input function.
If you export a SavedModel from tf.keras or from a TensorFlow estimator, the exported graph is ready for serving by default.
The "image_b64.json" file follows this format:
{"image_bytes":{"b64": base64_jpeg_data )}}
So after 3 hours and having followed everything required by Google, and somehow the gloud still throws me that error. And, yes, of course I have run the command line under Administrator Mode. I also tried it in two of my computers, and I got the same error. I am using Windows, Tensorflow 1.15.
Can anyone point out what is the problem with my implementation, or Google Doc/Keras is just lack lustering. Thank you.
I’m using a python notebook in gGloud datalab to re-train a neural network.
From the notebook I call retrain.py with
!python -m retrain --bottleneck_dir=../tf_files/bottlenecks --how_many_training_steps=500 --model_dir=../tf_files/models/ --summaries_dir=../tf_files/training_summaries/'mobilenet_1.0_224' --output_graph=../tf_files/retrained_graph.pb --output_labels=../tf_files/retrained_labels.txt --architecture='mobilenet_1.0_224' --image_dir=../tf_files/flower_photos
Within retrain.py I import the Tensorboard with
from google.datalab.ml import TensorBoard as tb
followed by the main function that does a bunch of things for the training process followed by:
inti=tf.global_variables_initializer()
sess.run(init)
tb.start('./tmp/retrain_logs’)
Executing retrain.py, a neural network will be trained, and TensorBoard will be activated (as stated in the output of my notebook copied below)
TensorBoard 1.8.0 at http://3439c553be9b:59199 (Press CTRL+C to quit)
{'text/html':TensorBoard was started successfully with pid 7707. Click here to access it.}
I tried to see the TensorBoard by:
clicking on the link provided (http://3439c553be9b:59199). A site on my web-brower opens but is empty.
I used gCloud Shell to connect with 'datalab connect --port=59199 .' This brings me to my files on gCloud, but not to a TensorBoard.
Can someone tell me how to access the TensorBoard please?
Thank you,
Julia
Just clicking on the link should work.
Could you check to see if you have a firewall rule that is preventing this?
I think the first string "TensorBoard 1.8.0 at http://3439c553be9b:59199 (Press CTRL+C to quit) " was output by the new tensorboard version. The direct link won't work.
The second string "TensorBoard was started successfully with pid 7707. Click here to access it.", the word "here" should be backed up by a hyperlink. See code https://github.com/googledatalab/pydatalab/blob/master/google/datalab/ml/_tensorboard.py#L73. That link should work. Do you get that link?
Recently I wrote a my custom operator (and its gradient) in python following this post
Tensorflow: Custom operation used in two networks simultaneously produces nan
Tensorflow runs with no error and the prediction gives expected accuracy. However, when I want to visualize this graph with tensorboard. I find that I cannot open the subgraph to see its structures. But it's gradient subgraph can be opened and seen. Does anyone has some idea about this problem?
Fig.1: subgraph fc1 cannot be opened but gradient/fc1 can be opened.
I can open the fc1 metanode on TensorBoard 0.1.8.
What version of TensorBoard you are using? You can find the version via running
python -c 'from tensorboard import version; print(version.VERSION)'
After that, could you try upgrading tensorboard via
pip install --upgrade tensorflow-tensorboard
and let me know if the issue persists?
Well I figure eight hours is enough time trying to fix this on my own, so I'll just ask folks:
I am running tensorflow-gpu 1.1.0 just fine in my virtual environment named 'tensorflow' outside of jupyterhub and Jupyter notebook. That is, I can import tensorflow and run scripts with it using the gpu.
When I'm inside my tensorflow virtualenv and using jupyterhub, Jupyter can not seem to 'see' tensorflow. I get the following error:
ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory
1) This is a common seen error message indicating tensorflow install problems, yet my paths and environment variables seem fine. After all, I can use tensorflow-gpu just fine outside of Jupyter.
2) Typing 'which jupyter' shows ~/anaconda3/envs/hub/bin/jupyter, so I believe that I am referencing jupyter inside my virtualenv.
3) Pip freeze shows that I have jupyterhub and tensorflow-gpu. I even did a pip3 freeze and it shows both packages as well.
Any ideas? Can tensorflow-gpu be run from Jupyter notebooks?
I got the solution from here:
[https://github.com/jupyter/notebook/issues/1290][1]
Basically, something was 'wrong' with jupyter in that it could not read my LD_LIBRARY_PATH variable. I did put everything correctly inside .bashrc so I don't know why.
Switch to the command line (terminal). Switch into your virtual environment if you have one.
type in: jupyter notebook --generate-config
It will tell you the directory in which your jupyter configuration file is stored. If you want to list it again type: jupyter --config-dir
Mine jupyter_notebook_config.py file is located here: /home/me/.jupyter/jupyter_notebook_config.py
At the very top of this file, jupyter_notebook_config.py, add in the following code:
import os
c = get_config()
os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda-8.0/lib64:usr/local/cuda-8.0/lib64/libcudart.so.8.0'
c.Spawner.env.update('LD_LIBRARY_PATH')
Then restart jupyterhub or jupyter notebook (type in at the command line: jupyter notebook
Tensorflow gpu should work.
The same thing applies even if you are running jupyterhub. Make the changes in jupyter, not jupyterhub. (Each user of jupyterhub will have their own jupyter process, so make the changes not at the 'hub' level, but the jupyter notebook level.
I'm using Tensorflow on Windows but when I try to launch Tensorboard opening http://localhost:6006 the browser shows a blank page
I have added the codeline
writer = tf.train.SummaryWriter('mypath/my_graph', sess.graph)
to my Tensorflow model and launched tensorboard with
tensorboard --logdir="mypath/my_graph"
Here the console output:
Following mrry suggestion I have updated to 0.12.0rc1 and now the Tensorboard page is shown but unfortunately I cannot see any graph, and is missing also the left panel to upload a graph file manually that I can see in some screenshots of the official guide.
Tried also to use
writer = tf.summary.FileWriter('mypath/my_graph', sess.graph)
following the deprecation hint
EDIT
I have found the problem. If I launch tensorboard --logdir="mypath/my_graph" TensorBoard is unable to load the path and looks always for the graph files in the default user path C:\Users\andrew\mygraph\ if I run console as user or C:\Windows\System32 if I run console as administrator. This is a bug and should be fixed.
The 0.12.0rc0 (Release Candidate 0) release of TensorFlow on Windows contains a broken version of TensorBoard. We recently made a new release (0.12.0rc1, Release Candidate 1) that contains a fix for TensorBoard on Windows. You can upgrade by following the instructions for installing the latest release on Windows, or simply typing pip install --upgrade tensorflow at the command prompt.
In ubuntu we can use:
tensorboard --logdir=/home/user/graph/
In Windows we have to change the command prompt to the directory in which the graph file is placed and then use:
tensorboard --logdir=\home\user\graph\