This question already has an answer here:
ipython notebook on linux VM running matplotlib interactive with nbagg
(1 answer)
Closed 6 years ago.
I use an ipython kernel on a remote machine via:
user#remote_machine$ ipython kernel
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-24970.json
and then through manual ssh tunneling (see here) connect a qtconsole on my local machine to it:
user#local_machine$ for port in $(cat kernel-24970.json | grep '_port' | grep -o '[0-9]\+'); do ssh remote_machine -Y -f -N -L $port:127.0.0.1:$port; done
user#local_machine$ ipython qtconsole --existing kernel-24970.json
This works fine. However, to visualize my data while debugging, i want to use matplotlib.pyplot. Although I have enabled X11 forwarding on my ssh tunnel (through -Y), when I try plotting something, I get the following error:
TclError: no display name and no $DISPLAY environment variable
as if X11 forwarding does not have any effect.
Furthermore, once when I had access to the remote machine, I started the remote kernel with:
user#remote_machine$ ipython qtconsole
and repeated the same process from my local machine. This time, I wasn't getting any errors. But the figures were being plotted on the remote machine instead of my local machine.
So, does anyone know if it's possible to connect to a remote ipython kernel, and display plots locally? (please note that inline mode works, and shows the plots in the local qtconsole, but that's not useful for me as I frequently need to zoom in).
A simpler and more robust approach is to run ipython remotely as you did, and instead of trying to plot the figures remotely, instead save them remotely. At the same time mount the remote directory using sftp, and open it in your local file browser.
Make sure to refresh your directory view in case the images that were saved remotely are not visible (otherwise it can take some time for this to happen). One simple way for refreshing the remote directory's view is noted here.
Related
I'm using VScode with Windows Subsystem for Linux (WSL) and working through an SSH connection.
Unfortunately, the Remote - SSH extension by Microsoft doesn't enable X11 forwarding (#267), and thus an extension such as Remote X11 has been created.
Sadly, this extension doesn't work for me as I'm working through a jump host/gateway, and therefore have to manually change the DISPLAY variable in the VScode terminal to the DISPLAY variable found in another terminal that is used for displaying stuff.
I guess it is possible to get the DISPLAY variable by running tr '\0' '\n' < /proc/<pid>/environ | grep DISPLAY(Environment variables of a running process on Unix?), with the <pid> being run from the local terminal, and now it's just a problem of automating the search for this process.
So i guess the stuff I'm asking for, is a way to:
Get the pid from a process on a local terminal
Get the DISPLAY variable from this process
Set the DISPLAY variable in every newly opened terminal in VScode (perhaps running step 2 in the workspace-settings.json)
PS. I'm using the VcXsrv X Server
I am running a tensorflow experiment on a remote machine continuously writing to the same events.out.tfevents.xxx file. I would expect tensorboard to refresh automatically every minute or so displaying the new logs. This does work when using sshfs to mount the remote machine on my laptop and using the mounted directory to run tensorboard on.
However, when using rsync to copy the files over and run tensorboard on the local files, the tensorboard never refreshes, I have to restart it in order to get the updates.
This is my rsync command:
rsync -aP --del -e ssh server_name:folder_on_server local_folder --exclude='*checkpoints*' --exclude='*.json' --exclude='*.DS_Store'
Any help would be greatly appreciated!
It's a known issue with the Tensorboard, see this issue on github.
Here's an quote from the issue (emphasis is mine) :
It looks like when the tensorboard reads an event file from local directory - it will not notice that the event file was deleted and recreated (which is quite valid case when you are using [...] rsync to sync the data)
One workaround is to use --inplace as an option in your rsync command.
I am curious to find out why packer is failing to get ssh access on an ESXi server. The build works just fine for vmware_fusion locally.
As JSON does not seem to display nicely directly here on SF - a link to a gist with the builder configuration: https://gist.github.com/geoHeil/5acf06cb0f3afadfa347d437c2695a7c
When running
packer build -var-file variables.json -only=vmwarevmwareRemote template.json
the kickstart file is loaded, configured and installed. However, in the case of ESXi as the builder the build seems to be stuck on waiting for SSH to become available.
I noticed in the logs that:
/var/log/auth.log
2017-02-08T17:33:20Z sshd[94210]: User 'root' running command 'esxcli --formatter csv network vm list\n'
2017-02-08T17:33:25Z sshd[94210]: User 'root' running command 'esxcli --formatter csv network vm list\n'
displays a lot of the same commands.
Executing this command manually shows
esxcli --formatter csv network vm list
Name,Networks,NumPorts,WorldID,
ubunu-test,"VM Network,",1,87986,
someOther,"VM Network,",1,84833,
What could be wrong here?
edit
packer version is latest 0.12.2, esxi 6.5
edit2
when applying the suggestion of setting a network the same problem persists. But now I see 2 commands in the logs
[root#vm-bd-dev:/var/log] tail -f auth.log
2017-02-09T09:05:56Z sshd[111376]: User 'root' running command 'esxcli --formatter csv network vm list\n'
2017-02-09T09:05:56Z sshd[111376]: User 'root' running command 'esxcli --formatter csv network vm port list -w 111433\n'
The second (new) one has the following output:
ActiveFilters,DVPortID,IPAddress,MACAddress,PortID,Portgroup,TeamUplink,UplinkPortID,vSwitch,
,,0.0.0.0,00:0c:29:47:d5:3d,33554450,VM Network,vmnic2,33554437,vSwitch0,
You probably need some more vmx_data settings for the network, something like:
"vmx_data": {
"ethernet0.networkName": "VM Network",
"ethernet0.present": "true",
"ethernet0.virtualDev": "vmxnet3",
"ethernet0.startConnected": "true",
"ethernet0.addressType": "generated"
}
Switching the network interface to something not hard coded like
network --bootproto=dhcp --ipv6=auto --activate
solved the problem for me.
Apparently different interfaces (no eth0) were available on ESXi.
The Question
I'm trying to enable X11 forwarding through the PyCharm SSH Terminal which can be executed via
"Tools -> Start SSH session..."
Unfortunately, It seems there is no way of specifying the flags like I would do in my shell for enabling the X11 Forwarding:
ssh -X user#remotehost
Do you know some clever way of achieving this?
Current dirty solution
The only dirty hack I found is to open an external ssh connection with X11 forwarding and than manually update the environment variable DISPLAY.
For example I can run on my external ssh session:
vincenzo#remotehost:$ echo $DISPLAY
localhost:10.0
And than set on my PyCharm terminal:
export DISPLAY=localhost:10.0
or update the DISPLAY variable in the Run/Debug Configuration, if I want to run the program from the GUI.
However, I really don't like this solution of using an external ssh terminal and manually update the DISPLAY variable and I'm sure there's a better way of achieving this!
Any help would be much appreciated.
P.s. Making an alias like:
alias ssh='ssh -X'
in my .bashrc doesn't force PyCharm to enable X11 forwarding.
So I was able to patch up jsch and test this out and it worked great.
Using X11 forwarding
You will need to do the following to use X11 forwarding in PyCharm:
- Install an X Server if you don't already have one. On Windows this might be the VcXsrv project, on Mac OS X the XQuartz project.
- Download or compile the jsch package. See instructions for compilation below.
- Backup jsch-0.1.54.jar in your pycharm's lib folder and replace it with the patched version. Start Pycharm with a remote environment and make sure to remove any instances of the DISPLAY environment variable you might have set in the run/debug configuration.
Compilation
Here is what you need to do on a Mac OS or Linux system with Maven installed.
wget http://sourceforge.net/projects/jsch/files/jsch/0.1.54/jsch-0.1.54.zip/download
unzip download
cd jsch-0.1.54
sed -e 's|x11_forwarding=false|x11_forwarding=true|g' -e 's|xforwading=false|xforwading=true|g' -i src/main/java/com/jcraft/jsch/*.java
sed -e 's|<version>0.1.53</version>|<version>0.1.54</version>|g' -i pom.xml
mvn clean package
This will create jsch-0.1.54.jar in target folder.
Update 2020:
I found a very easy solution. It may be due to the updated PyCharm version (2020.1).
Ensure that X11Forwarding is enabled on server: In /etc/ssh/sshd_config set
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost no
On client (MacOS for me): In ~/.ssh/config set
ForwardX11 yes
In PyCharm deselect Include system environment variables. This resolves the issue that the DISPLAY variable gets set to the system variable.
EDIT: As seen in the below image it works. For example I used the PyTorch implementation of DeepLab and visualize sample images from PASCAL VOC:
X11 forwarding was implemented in 2021.1 for all IntelliJ-based IDEs. If it still doesn't work, please consider creating a new issue at youtrack.jetbrains.com.
By the way, the piece of advice about patching jsch won't work for any IDE newer than 2019.1.
In parallel, open MobaXTerm and connect while X11 forwarding checkbox is enabled. Now PyCharm will forward the display through MobaXTerm X11 server.
This until PyCharm add this 'simple' feature.
Also, set DISPLAY environment variable in PyCharm run configuration like this:
DISPLAY=localhost:10.0
(the right hand side should be obtained with the command echo $DISPLAY in the server side)
Update 2022: for PyCharm newer than 2022.1: Plotting in SciView works by only setting ForwardX11 yes in .ssh/config (my laptop OS is ubuntu 22.04). I did not set any other parameters either on the server or local side.
I am developing a project in PyCharm using a Docker interpreter, but I am running into issues when doing most "interactive" things. e.g.,
import matplotlib.pyplot as plt
plt.plot([1, 2, 3], [4, 5, 6])
gives
RuntimeError: Invalid DISPLAY variable
I can circumvent this using
import matplotlib
matplotlib.use('agg')
which gets rid of the error, but no plot is produced when I do plt.show(). I also get the same error as in the thread [pycharm remote python console]: "cannot connect to X server" error with import pandas when trying to debug after importing Pandas, but I cannot SSH into my docker container, so the solution proposed there doesn't work. I have seen the solution of passing "-e DISPLAY=$DISPLAY" into the "docker run" command, but I don't believe PyCharm has any functionality for specifying command-line parameters like this with a Docker interpreter. Is there any way to set up some kind of permanent, generic X11 forwarding (if that is indeed the root cause) so that the plots will be appropriately passed to the DISPLAY on my local machine? More generally, has anyone used matplotlib with a Docker interpreter in PyCharm successfully?
Here's the solution I came up with. I hope this helps others. The steps are as follows:
Install and run Socat
socat TCP-LISTEN:6000,reuseaddr,fork UNIX-CLIENT:\"$DISPLAY\"
Install and run XQuartz (probably already installed)
Edit the PyCharm run/debug configuration for your project, setting the appropriate address for the DISPLAY variable (in my case 192.168.0.6:0)
Running/debugging the project results in a new quartz popup displaying the plotted graph, without any need to save to an image, etc.
Run xhost + on the host and add these options to the docker run: -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix