I am running the latest version of tensorflow (1.7.0) in Windows. It added a new feature for debugging in tensorboard. However, I am not able to get it to connect. I was previously using the LocalCLIDebugWrapperSession debugger. However, I wanted to play with the new features in TensorBoardDebugWrapperSession.
When I run tensorboard using the command "tensorboard --logdir=dir --debugger_port 6064" it tells me "Creating InteractiveDebuggerPlugin at port 6064".
However, when I run my model in tensorflow (after swapping out the LocalCLIDebugWrapperSession for the TensorBoardDebugWrapperSession) it never seems to connect to the interactive debugger. I don't get an error in the output for either tensorboard or tensorflow. Nothing happens and on the tensorboard debugger page it just shows: "Debugger is waiting for Session.run() connections..."
It seems that tensorflow does not have support for grpc connection on windows. A workaround is to use windows subsystem. Note that there is no direct access to system GPU from the subsystem. So it cannot be use with tensorflow-gpu version
start_epoch: grpc:// debug URL scheme is not implemented on Windows yet.
https://github.com/tensorflow/tensorflow/issues/17933
Related
I am using the pre-built deep learning VM instances offered by google cloud, with an Nvidia tesla K80 GPU attached. I choose to have Tensorflow 2.5 and CUDA 11.0 automatically installed. When I start the instance, everything works great - I can run:
Import tensorflow as tf
tf.config.list_physical_devices()
And my function returns the CPU, accelerated CPU, and the GPU. Similarly, if I run tf.test.is_gpu_available(), the function returns True.
However, if I log out, stop the instance, and then restart the instance, running the same exact code only sees the CPU and tf.test.is_gpu_available() results in False. I get an error that looks like the driver initialization is failing:
E tensorflow/stream_executor/cuda/cuda_driver.cc:355] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
Running nvidia-smi shows that the computer still sees the GPU, but my tensorflow can’t see it.
Does anyone know what could be causing this? I don’t want to have to reinstall everything when I’m restarting the instance.
Some people (sadly not me) are able to resolve this by setting the following at the beginning of their script/main:
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
I had to reinstall CUDA drivers and from then on it worked even after restarting the instance. You can configure your system settings on NVIDIAs website and it will provide you the commands you need to follow to install cuda. It also asks you if you want to uninstall the previous cuda version (yes!).This is luckily also very fast.
I fixed the same issue with the commands below, taken from https://issuetracker.google.com/issues/191612865?pli=1
gsutil cp gs://dl-platform-public-nvidia/b191551132/restart_patch.sh /tmp/restart_patch.sh
chmod +x /tmp/restart_patch.sh
sudo /tmp/restart_patch.sh
sudo service jupyter restart
Option-1:
Upgrade a Notebooks instance's environment. Refer the link to upgrade.
Notebooks instances that can be upgraded are dual-disk, with one boot disk and one data disk. The upgrade process upgrades the boot disk to a new image while preserving your data on the data disk.
Option-2:
Connect to the notebook VM via SSH and run the commands link.
After execution of the commands, the cuda version will update to 11.3 and the nvidia driver version to 465.19.01.
Restart the notebook VM.
Note: Issue has been solved in gpu images. New notebooks will be created with image version M74. About new image version is not yet updated in google-public-issue-tracker but you can find the new image version M74 in console.
I am having trouble getting Tensorboard Profiler to run on my windows machine. I am using the tf.profiler progrtammatic mode as described here:
tf.profiler.experimental.start('logdir')
# Train the model here
tf.profiler.experimental.stop()
I then ran TB from the terminal as such:
tensorboard --logdir=<path> --host localhost --port 6006
However when I open the profiler tab I see this:
I suspect this is because I am training on Windows and the instructions are made for Linux. For example, insuring CUPTI exists on path (see here). Has anyone got profiler to work on Windows? If so, where there any nuances to the installation beyond simply pip install tensorboard_plugin_profile?
Doing a course in Machine Learning and can't get Tensorboard to work. I have saved runs from running a DQN and I write:
tensorboard -logdir runs
With the folliwng result:
2019-12-28 18:32:04.265065: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
TensorBoard 1.7.0 at http://david-linux:6006 (Press CTRL+C to quit)
So I click the link and get:
No dashboards are active for the current data set.
Probable causes:
You haven’t written any data to your event files
TensorBoard can’t find your event files.
I also get this result after having the code running for a while:
"W1228 18:34:34.186506 Thread-2 application.py:272] path /[[_dataImageSrc]] not found, sending 404
W1228 18:34:34.205581 Thread-2 application.py:272] path /[[_imageURL]] not found, sending 404"
Running this on Linux using Anaconda Python version 3.6 because that is what the course book uses. Have no idea what the above errors means, quite new to coding in general and reinforment learning in particular.
It could be caused if the browser isn't updated. You could also try installing the latest version of Tensorboard:
pip uninstall tensorflow-tensorboard
pip install tensorboard
Also try using different browsers.
Can you just try going to http://localhost:6006 instead? It looks like your hostname is not one that actually resolves in DNS.
I am trying to run tensorflow on a remote machine's GPU through Jupyter notebook. However, if I print the available devices using tf, I only get CPUs. I have never used a GPU before and am relatively new at using conda / jupyter notebook remotely as well, so I am not sure how to set up using the GPU in jupyter notebook.
I am using an environment set up by someone else who already executed the same code on the same GPU, but they did it via python script, not in a jupyter notebook.
this is the only code in the other person's file that had to do with the GPU
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
set_session(tf.Session(config=config))
I think the problem was that I had tensorflow in my environment instead of tensorflow-gpu. But now I get this message "cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version" and I don't know how to update the driver through terminal
How is your environment set up? Specifically, what is your remote environment, and what is your local environment? Sounds like your CUDA drivers are out of date, but it could be more than just that. If you are just getting started, I would recommend finding an environment that requires little to no configuration work on your part, so you can get started more easily/quickly.
For example, you can run GPUs on the cloud, and connect to them via local terminal. You also have your "local" frontend be Colab by connecting it to a local runtime. (This video explains that particular setup, but there's lots of other options)
You may also want to try running nvidia-smi on the remote machine to see if the GPUs are visible.
Here is another solution, that describes how to set up a GPU-Jupyterlab instance with Docker.
To update your drivers via terminal, run:
ubuntu-drivers devices
sudo ubuntu-drivers autoinstall
sudo reboot
Are your CUDA paths set appropriately? Like that?
export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Is it possible to run Tensorboard on a machine without CUDA support?
I'm working at a computation center (via ssh) which has two major clusters:
CPU-Cluster which is a general workhorse without CUDA support (no dedicated GPU)
GPU-Cluster with dedicated GPUs e.g. for running neural networks with tensorflow-gpu.
The access to the GPU-cluster is limited to Training etc. such that I can't afford to run Tensorboard on a machine with CUDA-support. Instead, I'd like to run Tensorboard on the CPU-Cluster.
With the TF bundled Tensorboard I get import errors due to missing CUDA support.
It seems reasonable that the official Tensorboard should have a mode for running with CPU-only. Is this true?
I've also found an inofficial standalone Tensorboard version (github.com/dmlc/tensorboard), does this work without CUDA-support?
Solved my problem: just install tensorflow instead of tensorflow-gpu.
Didn't work for me for a while due to my virtual environment (conda), which didn't properly remove tensorflow-gpu.
Tensorboard is not limited by whether a machine has GPU or not.
And as far as I know, what Tensorboard do is parsing events pb files and display them on web. There is not computing, so it doesn't need GPU.