I have a dataframe say dfdata in a jupyter server notebook running on a remote machine).
I want to access the dataframe in the remote machine memory to my local machine, say to paste the dfdata to Excel.
Normally (when the notebook server is running locally), I do dfdata.to_clipboard() to copy the dataframe to clipboard and now able to paste it to Excel.
However, since the dfdata is now on the remote machine, the dfdata.to_clipboard() does not have the copy of the dataframe in clipboard.
How to make this work, i.e. copy paste dataframe from remote machine to locally running Excel, Textfile, etc.? Any alternative methods, if to_clipboard() by design will not work across remote server due to any security restrictions / limitations.
As an alternative you can use copydf - github and pypi.
This should be as simple as pip install copydf.
Then in your jupyter session:
from copydf import copyDF
copyDf( df )
I wrote a Jupyter nbextension to do this (installation instructions in the README).
It overwrites Panda's pandas.io.clipboard.copy & pandas.io.clipboard.clipboard_set (which is an embedded copy of pyperclip) to send messages to the Jupyter frontend via the Comm mechanism. I had to add a simple bootstrap-based UI that pops up when the client-side JS receives a message, as browsers won't let you push data onto the system clipboard without explicit user interaction (security!).
If you have another terminal session connected to the same remote server, you can try getting the clipboard system to output to the terminal.
I tried xclip -o in another ssh session (after calling df.to_clipboard() in the notebook) and it printed the contents of my DataFrame to the terminal. I was able to copy that text and paste it to Google Sheets successfully, split correctly into columns.
It may depend on what's installed on the server. There is apparently another clipboard system called xsel, but xclip worked for me on Ubuntu Server 16.04.
Related
I need to run a Jupyter notebook 24/7 on a virtual environment. I've never done this before, and as a finance student I barely heard of this.
I have the putty connection, then the "something like cmd" opens, and HERE I am... (Really don't know where I am). There is also a virtual environment created (not by me), which has some capacity (like ram, storage, cpu, and stuff).
How do I open the jupyter notebook on the virtual environment?
Is there a way to do this or should I just delete all of this and run task scheduler?
PuTTY allows you to SSH to a server.
From this server's terminal session, you would need to run jupyter server (or jupyter lab). This would start the Jupyter Server/Lab web server on its default port. Read the documentation on running (and securing) a public server.
Using the same hostname/IP that you used in PuTTY, copy it, and open http://<that address>:<jupyter port> in a browser window, and you should be presented with a Jupyter login page where you'd need to copy a token that is generated when Jupyter is started (assuming you've not configured any other authentication methods).
To get Python code to run constantly, it would be recommended to just use cron with a regular Python script, and not use a notebook file. Otherwise, you'd probably want to look at another CLI tool called papermill, along with cron
I have a Raspberry Zero connected to a SIM7600G-H 4G HAT with a camera module connected. I want to use it as a webcam, who makes a picture in a defined cycle and send's it via scp to a web server who display it on a homepage.The created shell script is started via a CRONJob every 2 hours.
The whole setup works very well if I have a good, powerful SIM connection. However, as soon as I operate the setup at the desired location, a strange behavior appears.
At the location where I run the webcam I only have a relatively poor 3G connection, if I run the scp command from a connected laptop it works fine. I can therefore assume that the problem has nothing to do with the SIM module.
The Raspian shows two peculiar behaviors.
Even though i created a key and gave it to the webserver, every now and then it wants me to enter the password when i run the scp command.This does not happen when I connect directly to the webserver via ssh.
The first few images the raspian loads without problems using scp command on the webserver, but then suddenly it does not work anymore.
I send two pictures each. I replace one with an existing one on the web server. This is the image that is displayed on the homepage and another one I put in an archive folder named after the timestamp. It looks like this:
scp foo.jpg <username>#webserver:dir/to/folder/default.jpg
FILENAME=`date +"%Y-%m-%d_%H-%M-%S"`
scp foo.jpg <username>#webserver:dir/to/archive_folder/${FILENAME}.jpg
Because of the password issue I downloaded an additional service called sshpass and added in addition to the scp command the following command:
sshpass -p <password>
However, it seems like the issue is not related to sshpass since it also happens if I try it only with the scp command and enter the password by my self.
At the end for the "new file" which goes into the archive folder, the raspian creates the filename at the web server, but he does not transmit the data of the file. At the end, the file remains empty.
The file which should be replaced "default.jpg" is not touched at all.
I tried to find out what happens via the debug output. But there is no useful information. It always stops with the line who shows the transmission state and with 0% and 0KB/s.
I have now spent several days on a solution. I have also already taken it home and everything has suddenly worked smoothly again. But as soon as I mounted it there again, the problem reappeared.
Does anyone know of a bug with the raspberry zero that it can no longer transfer scp files when the data transfer rate is low? One image is about 300kb and my laptop takes about 20 seconds to transfer over the same connection as the one from the Raspberry.
After countless attempts, my simplest solution was to set up a cronjob, which restarts the raspberry shortly before it takes a photo for the webcam. It then searches for a new network and finds it very reliably.
I need to be able to pass some parameters to my virtual machine during it's bootup so it sets itself properly. To do that I either have to bake the info into the image or somehow pass it as parameters to my qemu-kvm command. These parameters are just few, and if it was VMware, we would just pass it as ova paramas and when the VM launches we would call the ova-environment to get these params. But launching it from qemu-kvm I have no such options. I did some homework and found that I could use virtio-9p driver for sharing files across host and guest. Unfortuantely RHEL/Centos has decided not to support 9p.
With no option of rebuilding my RHEL kernel with the 9p options enabled, how do I solve my above problem? Either solution would work, which is, pass/share some kind of json file to the VM(pre-populated on the host), which will read this and do it's setup OR set some kind of "environment variables" which I can query from within the VM to get these params and continue with setup. Any pointers would help.
If your version of QEMU supports it, you could use its -fw_cfg option to pass information to the guest. If that guest is running a Linux kernel with CONFIG_FW_CFG_SYSFS enabled, you will be able to read out the information from sysfs. An example:
If you launch your VM like so:
qemu-system-x86_64 <OPTIONS> -fw_cfg name=opt/com.example.test,string=qwerty
From inside the guest, you can then get the value back from sysfs:
cat /sys/firmware/qemu_fw_cfg/by_name/opt/com.example.test/raw
There appears to be some driver for Windows as well, but I've never used it.
When you boot your guest with -kernel and -initrd you should be able to pass environment variables with -append.
The downside is that you have to keep track of your current kernel and initrd outside of your disk image.
Other possibilities could be a small prepared disk image (as you said) or via network/dhcp or a serial link into your guest or ... this really depends on your environment.
I was just searching to see if this situation had improved and came across this question. Apparently it has not improved.
What I do is output my variable data to a temp file (eg. /tmp/xxFoo). Usually I write text or a tar straight to that file then truncate it to a minimum size and 512 byte multiple like 64K otherwise the disk controller won't configure it. Then the VM starts with a raw drive as that file. After the VM is started the temp file is deleted. From within the guest you can read/cat the raw block device and get the variable data (in BSD use the c partition as the raw drive).
In Windows guests it's tricky to get to the data. In theory you can read \\.\PhysicalDriveN but I have not ever been able to get that to work. Cygwin can do it and it works like Linux. The other option is to make your temp file a partitioned and formatted image but that's a pain to create and update.
As far as sharing a folder I use Samba which works in just about anything. I usually use several instances of smbd running with different configurations.
One option is to create a ISO file and pass as parameter. This works for both host Win and Ubuntu and Guest Win and Ubuntu. You can read the mounted CD ROM inside the guest OS
>>qemu-system-x86_64 -drive file=c:/qemuiso/winlive1.qcow2,format=qcow2 -m 8G -drive file=c:\qemuiso\sample.iso,index=1,media=cdrom
On Guest Linux Mount CDROM in Ubuntu:-
>>blkid //to check if media is there
>>sudo mkdir /mnt/cdrom
>>sudo mount /dev/sr0 /mnt/cdrom //this step can also be put in crontab
>>cd /mnt/cdrom
Scenario:
I'm using ssh to connect to a remote machine. I use the command line and run ssh <pathname>, which connects me to the machine at . I want to edit and run code on that remote machine. So far the only way I know is to create, edit, and run the files in the command window in vi, because my only connection to that machine is that command window.
My Question is:
I'd love to be able to edit my code in VSCode on my own machine, and then use the command line to save that file to the remote machine. Does anyone know if this is possible? I'm using OS X and ssh'ing into a Linux Fedora machine.
Thanks!
Sounds like you're looking for a command like scp. SCP stands for secure copy protocol, and it builds on top of SSH to copy files from one machine to another. So to upload your code to your server, all you'd have to do is do
scp path/to/source.file username#host:path/to/destination.file
EDIT: As #Pam Stums mentioned in a comment below the question, rsync is also a valid solution, and is definitely less tedious if you would like to automatically sync client and server directories.
You could export the directory on the remote machine using nfs or samba and mount it as a share on your local machine and then edit the files locally.
If you're happy using vim, check out netrw (it comes with most vim distributions; :help netrw for details) to let you use macvim locally to edit the remote files.
I am doing an intensive computing project with a super old C program. The program requires a library called Sun Performance Library which is a commercial ware. Instead of purchasing the library by myself, I am running the program by logging onto a Solaris machine in our computer lab with the ssh command, while the working directory to store output data is still on my local Mac.
Now, a problem just occurred: the program uses large amount of disk space to save some intermediate results and the space on my local Mac is quickly filled (50 GB for each user prescribed by the administrator). These results are necessary for the next stage of computing and I cannot delete any of them before it finally produce the output data. Therefore, I have to move the working directory to an external hard drive in order to continue. Obviously,
cd /Volumes/VOLNAME
is not the correct way to do it because the remote machine will give me a prompt saying
/Volumes/VOLNAME: No such file or directory.
So, what is the correct way to do it?
sshfs recently added support for "slave mode" which allows you to do this. Assuming you have sshfs on Solaris (I'm not sure about this), the following command (ran from your Mac) will do what you want: dpipe /usr/lib/openssh/sftp-server = ssh SOLARISHOSTNAME sshfs MACHOSTNAME:/Volumes/VOLNAME MOUNTPOINT -o slave
This will result in the MOUNTPOINT directory on the server being mounted to your local external drive. Note that I'm not sure whether macOS has dpipe. If it doesn't, you can replace it with one of the equivalent solutions at How to make bidirectional pipe between two programs?. Also, if your SFTP server binary is somewhere else, substitute its path.
The common way to mount a remote volume in Solaris is via NFS, but that usually requires root permissions.
Another approach would be to make your application read its data from stdin and output its results to stdout, without using the file system directly. Then you could just redirect the data from/to your local machine through ssh. For instance:
ssh user#host </Volumes/VOLNAME/input.data >/Volumes/VOLNAME/output.data