I'm using Windows Subsystem for Linux (WSL2), and OneDrive mounted using rclone, on a Windows 10 machine.
When using WSL2 in the local directories, I can create a virtual environment for a project:
python -m venv myenv/
However, if I'm in a directory on OneDrive, and I run this command, I get an error:
Error: [Errno 5] Input/output error: 'lib' -> '/home/andrew/onedrive/myproject/venv/lib64'
If I look in the myproject directory, I can see that directory venv has been created. However it is incomplete, in that it only has 'lib' and 'include' subdirectories. When it is created properly (that is, in a directory not on OneDrive), it has 'lib, 'include', 'bin', 'lib64', 'share', and 'pyvenv.cfg'.
'lib64' is a symbolic link pointing to 'lib' in the normal installation. In the error message above, it seems that lib is actually pointing to lib64, so I suppose this is the input/output error?
Is there a way to get venv to work on OneDrive directories when mounted via rclone and using WSL2?
My hunch is that something in the way Rclone works is causing your issue. It purports to be "inspired by rsync", which means that it isn't really "mounting", but is "syncing" the cloud storage.
I did a quick install of rclone in a new WSL instance, and I'm seeing the same "Input/Output error" that you experienced.
I'm guessing that you would see the same problem even if under a "full" Linux installation. In other words, I doubt this has anything to do with the interaction between WSl and rclone. It's probably more about the interaction between Python's venv module and rclone. I don't know for sure, but I have my doubts that rclone is designed for "live" usage. It's more for being able to access (and write) files on cloud-storage providers from Linux.
The good news is that there's likely an alternative. Windows automatically makes your OneDrive folder available in %userprofile%\OneDrive. From WSL, you can access this with:
cd $(powershell.exe -c 'Write-Host -NoNewLine $env:userprofile' | xargs -0 wslpath)/OneDrive
That's usually /mnt/c/Users/<yourusername>/OneDrive.
From there, python3 -m venv myvenv/ worked correctly for me.
That said, I highly recommend against doing this in WSL2, as performance of NTFS files is abysmal. Best to use a WSL1 instance if you really need to do it.
Related
I'm trying to run my static web app using Windows Subsystem for Linux (2), but I can't figure out where on my computer I should store the git repository to be able to run it decently quickly. I have tried storing it on under /mnt/c/{workfolder}, but it takes several minutes to start up (using npm run start), and I have to rerun to see any changes. This is useless when I'm trying to work...
I have also tried to store it in /mnt/wsl/{workfolder}, and in that case it starts up quickly and I can see my changes without rerunning the app. However, it seems to disappears when I restart my computer.
Where should I store the git repository to be able to run the app quickly and see changes without rerunning? I'm assuming there's something I'm not understanding, help me get this it you know.
You'll want it somewhere on the ext4 partition of the WSL distribution. Typically, the best place is going to be under your WSL /home/<username> folder.
I would recommend:
mkdir ~/src
# or
mkdir ~/projects
# or something similar
Then create subdirectories for each project in that directory.
Why the others don't work:
/mnt/c is the Windows C: drive. That drive is mounted into WSL2 using the 9P network file system, and yes, it's (a) slow, and (b) does not support inotify, so apps cannot register for notifications of changes to files.
/mnt/wsl is a tmpfs mount. It's really there for holding things that need to be shared between all running WSL instances. The auto-generated resolv.conf that you see there is one of those things. You can also use it for copying a file from one WSL distribution to another -- Simply copy the file to /mnt/wsl, start another WSL distribution, and copy or move the file out.
But yes, all tmpfs mounts are ephemeral and will terminate when the last WSL2 distribution/instance terminates.
Fed up with struggling with lib install/dependencies problems, I'm starting working with Singularity.
Though, I'm not sure I understand precisely how it works regarding files management in the sandbox mode (not data, programs).
For example, I designed a very simple definition file that is just a "naked" Debian:
Bootstrap: library
From: debian
%post
apt-get update
I create a sandbox with this to add stuff:
sudo singularity build --sandbox Test/ naked_Debian.def
And I try to install a program. But what I don't understand is that I managed to do it, removed the sandbox directory but I think there are still files that were created during the sandbox life (in /dev, /run, /root, etc.). For example, the program that I cloned from git is now in /root of my local (independently of any container).
From what I understood, everything was in the container and should disappear if I remove the sandbox directory. Otherwise, I'm gonna leave a lot of mess with all the tests? And then I can't port the container from system to another.
If I create any new sandbox directory, the program is already there.
Cheers,
Mathieu
By default, singularity mounts $HOME to the container and uses that path as the working directory for singularity shell / exec. Since you're running the sandbox with sudo, /root is being mounted in and that's where any repos you cloned would end up if you didn't cd to a different directory. /tmp is also automatically mounted in, though that is unlikely to cause an issue since it's just temp files.
You have a few options to avoid files ending up in places you don't expect.
Disable automount of home: singularity shell --no-home ...
The default working directory is now / instead of $HOME and files are created directly in sandbox (as opposed to a mounted in directory)
If you want to get files out of the sandbox, you'll either need to copy to /tmp inside the container, and on the host OS from /tmp to the desired location
Set a different location to use as home: singularity shell --home $PWD ...
This mounts in and uses the current directory as $HOME instead of the user's $HOME on the host OS
Simpler to move files between host OS and container, but still creates files in the host OS
Don't mount system directories at all: singularity shell --contain --workdir /some/dir ...
Directories for /tmp and /var/tmp are created inside /some/dir instead of using /tmp and /var/tmp on the host. $HOME has the same path as the host and is used as the working directory, but it is empty and separate from the host OS
Complete separation from host OS, while still allowing some access between container and OS
Additional details on these options can be found in the documentation.
I'm running Ubuntu and have a remote CentOS system which stores (and has access to) various files and network locations. I have SSH access to the CentOS machine and want to be able to work locally on Ubuntu.
I'm trying to mirror a remote directory structure. The remote directory is structured:
/my_data/user/*
And I want to replicate this structure locally (a lot of scripts rely on absolute paths).
However, for reasons of speed, I want a certain subfolder, for example:
/my_data/user/sourcelibs/
To be stored locally on disk. I know the sourcelibs subfolder doesn't change much (but the rest might). So I can comfortably rsync it:
mkdir -p /my_data/user/sourcelibs/
rsync -r remote_user#remote_host:/my_data/user/sourcelibs/ /my_data/user/sourcelibs/
My question is, if I use sshfs to mount /my_data/user:
sudo sshfs -o allow_other,default_permissions, remote_user#remote_host:/my_data/user /my_data/user
Will it overwrite my existing files? Is there a way to have sshfs mount but exclude certain subfolders?
Yes, sshfs will overwrite existing files. I have almost the same use case and just tested this myself. BTW, you'll need to add -o nonempty to your sshfs command since the destination dir /my_data/user already exists.
What I found to work is make a copy of the remote directory excluding the large sub dirs. IDK if keeping 2 copies in sync on the remote machine is feasible for your use case? But if you'll mostly be updating on your local machine and rarely making changes remotely, that could work.
I recently installed WSL version 2 according to the instruction. Everything work greatly. But I notice that the folder /mnt/c is not available any more.
I understand the new WSL using vhdx as file system which gives us better FS performance, but I could not find a way to access files in Windows.
I also tried to check the wsl --help but I could not find anything helpful.
Any suggestions?
After read more, I notice that WSL 2 is actually an optimized VM on Windows 10, which means the running WSL 2 has no idea about where it self. Seems there should be a way to pass some Windows folders while launch it. Haven't figured out how.....
By default, the command prompt in WSL is located at C:\Users\username. WSL considers this /home/user but you can can reference any directory on your Windows machine by starting at the root directory with /../../.
Here is an example command: find "/../../mnt/c/anyFolder/subfolder" -name "test" -type d
As part of our product release we ship Apache HTTP Server binaries that we have compiled on our (UNIX) development machine.
We tell our clients to install the binaries (on their UNIX servers) under the same directory structure that we compiled it under. For some clients this is not appropriate, e.g. where there are restrictions on where they can install software on their servers and they don't want to compile Apache themselves.
Is there a way of compiling Apache HTTP Server so its installation location(s) can be specified dynamically using environment variables ?
I spent a few days trying to sort this out and couldn't find a way to do it. It led me to believe that the Apache binaries were hard coding some directory paths at compilation preventing the portability we require.
Has anyone managed to do this ?
I think the way to do(get around) this problem is to develop a "./configure && make" script that your client uses to install, specify and compile the binaries. That would offcourse require that the client has all the source-code installed on his server or you can make it available on an NFS share.
If you are compiling Apache2 for a particular location but want your clients to be able to install it somewhere else (and I'm assuming they have the same architecture and OS as your build machine) then you can do it but the apachectl script will need some after-market hacking.
I just tested these steps:
Unpacked the Apache2 source (this should work with Apache 1.3 as well though) and ran ./configure --prefix=/opt/apache2
Ran make then sudo make install to install on the build machine.
Switch to the install directory (/opt/apache2) and tar and gzip up the binaries and config files. I used cd /opt/apache2; sudo tar cf - apache2 | gzip -c > ~/apache2.tar.gz
Move the tar file to the target machine. I decided to install in /opt/mynewdir/dan/apache2 to test. So basically, your clients can't use rpm or anything like that -- unless you know how to make that relocatable (I don't :-) ).
Anyway, your client's conf/httpd.conf file will be full of hard-coded absolute paths -- they can just change these to whatever they need. The apachectl script also has hard coded paths. It's just a shell script so you can hack it or give them a sed script to convert the old paths from your build machine to the new path on your clients.
I skipped all that hackery and just ran ./bin/httpd -f /opt/mynewdir/dan/conf/httpd.conf :-)
Hope that helps. Let us know any error messages you get if it's not working for you.
I think the way to do(get around) this problem is to develop a "./configure && make" script that your client uses to install, specify and compile the binaries. That would offcourse require that the client has all the source-code installed on his server or you can make it available on an NFS share.
Not to mention a complete build toolchain. These days, GCC doesn't come default with most major distributions. Wouldn't it be sane to force the client to install it to /opt/my_apache2/ or something like that?
#Hissohathair
I suggest 1 change to #Hissohathair's answer.
6). ./bin/httpd -d <server path> (although it can be overridden in the config file)
In apacheclt there is a variable for HTTPD where you could override to use it.