Hello,
I am trying to launch Tensor board on Tensorflow/Keras on Azure Databricks,
but it will go to '502 Bad Gateway' after clicking 'View TensorBoard'.
Please advise how to fix this.
Thanks.
Looking for active tensorboard process...
Active tensorboard processkilled...
Starting tensorboard process...
Tensorboard process started.
TensorBoard log directory set to: /dbfs/ml/lenet/train/1570698969.0092866. View TensorBoard
Related
In a Google Collab notebook once I execute the commands:
%load_ext tensorboard
%tensorboard --logdir <data_directory>
I am unable to save the model in the cloud, not only automatically but also manually.
Someone has an idea on why this can be happening?
If its helpful for someone else; executing in an independent module at the beginning the command:
%load_ext tensorboard
And at the end of the program in a separated module
%tensorboard --logdir <folder_name>
Worked for me and allows to save my work despite the TensorBoard module being loaded in the system
I use a google instance to train my model.
I want to check the performance via tensorboard
I ran : tensorboard --logdir ./data/log and got : TensorBoard 2.1.1 at http://localhost:6006/ (Press CTRL+C to quit)
But trying to access http://localhost:6006/ in chorme, it doesnt work ( This site can’t be reached )
NOTE : I'm running this in google server instance and want to access it on my local chorme.
I'm trying to add TensorBoard functionality to this SageMaker example: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/keras_bring_your_own/hpo_bring_your_own_keras_container.ipynb
The issue is that SageMaker's Estimator.fit() does not seem to support Keras models compiled with callbacks.
Now from this git issue post it was described that what I need to do for TensorBoard functionality is,
"You need your code inside the container to save checkpoints to S3,
and you need to periodically sync your local Tensorboard log directory
with your S3 checkpoints."
So to sum it all up, to enable TensorBoard in SageMaker with this custom Keras docker image, it looks like I need a way of periodically uploading a file to an S3 bucket during training without using callbacks. Is this possible to do? I was considering trying to shove this code into a custom loss function, but I'm not sure if this would be the way to go about it. Any help is greatly appreciated!
I’m using a python notebook in gGloud datalab to re-train a neural network.
From the notebook I call retrain.py with
!python -m retrain --bottleneck_dir=../tf_files/bottlenecks --how_many_training_steps=500 --model_dir=../tf_files/models/ --summaries_dir=../tf_files/training_summaries/'mobilenet_1.0_224' --output_graph=../tf_files/retrained_graph.pb --output_labels=../tf_files/retrained_labels.txt --architecture='mobilenet_1.0_224' --image_dir=../tf_files/flower_photos
Within retrain.py I import the Tensorboard with
from google.datalab.ml import TensorBoard as tb
followed by the main function that does a bunch of things for the training process followed by:
inti=tf.global_variables_initializer()
sess.run(init)
tb.start('./tmp/retrain_logs’)
Executing retrain.py, a neural network will be trained, and TensorBoard will be activated (as stated in the output of my notebook copied below)
TensorBoard 1.8.0 at http://3439c553be9b:59199 (Press CTRL+C to quit)
{'text/html':TensorBoard was started successfully with pid 7707. Click here to access it.}
I tried to see the TensorBoard by:
clicking on the link provided (http://3439c553be9b:59199). A site on my web-brower opens but is empty.
I used gCloud Shell to connect with 'datalab connect --port=59199 .' This brings me to my files on gCloud, but not to a TensorBoard.
Can someone tell me how to access the TensorBoard please?
Thank you,
Julia
Just clicking on the link should work.
Could you check to see if you have a firewall rule that is preventing this?
I think the first string "TensorBoard 1.8.0 at http://3439c553be9b:59199 (Press CTRL+C to quit) " was output by the new tensorboard version. The direct link won't work.
The second string "TensorBoard was started successfully with pid 7707. Click here to access it.", the word "here" should be backed up by a hyperlink. See code https://github.com/googledatalab/pydatalab/blob/master/google/datalab/ml/_tensorboard.py#L73. That link should work. Do you get that link?
tensorboard --logdir=-logs
Starting TensorBoard b'54' at http://ASUS-PC:6006
(Press CTRL+C to quit)
I try to use tensorboard, connecting http://ASUS-PC:6006 or http://localhost:6006, but I can't see anything. How can I fix it? I use pycharm. Sorry I don't have enough reputation in it.