Getting an installed module to recognize changes to config files - config

I have a package that uses config.json for some settings it uses. I keep the package locally rather than installing it from CPAN. My problem is when I make changes to config.json, the package doesn't recognize the changes since the config file's cached elsewhere, forcing me to run zef install --force-install or delete precomp. How can I ensure that the package always recognizes updates to the config file?

When you install packages using zef, it keeps them in the filesystem, but their names are converted into sha1, something like
/home/jmerelo/.rakudobrew/moar-2018.03/install/share/perl6/site/sources/81436475BD18D66BFD96BBCEE07CCCDC0F368879
zef keeps track of them, however, and you can locate them using zef locate, for instance:
zef locate lib/Zef/CLI.pm6
You can run that from a program, for instance this way:
sub MAIN( Str $file ) {
my $location = qqx/zef locate $file/;
my $sha1 = ($location ~~ /\s+ \=\> \s+ (.+)/);
say "$file → $sha1[0]";
}
which will return pretty much the same, except it will give you the first location of the file you give it at the command line:
lib/Zef/CLI.pm6 → /home/jmerelo/.rakudobrew/moar-2018.03/install/share/perl6/site/sources/81436475BD18D66BFD96BBCEE07CCCDC0F368879
You probably need to install your config.json file in a resources directory (which is the preferred location) and then use something like that.
That said, probably actually installing a module you're testing is not the best strategy. If you're still testing things, it's probably better if you just keep it in the directory you're working with and use perl6 -I<that directory> or else use lib <that directory> is probably a better option. You can just delete that when you release, or keep it, since that only adds another directory to the search path and will not harm the released module.

Related

Singularity Recipe: How to access executable within container?

I am a beginner with Singularity.
What I want to achieve in the long run: I have a programming project with a long lists of dependencies, and I want to be able to give the program to other people in my company without there being bugs caused by missing dependencies, or wrong versions of dependencies.
The idea was now to use Singularity in order to easily provide a working environment.
In order to test this, I wrote a Hello World application which I now want to run in a container. I have a folder HelloWorld/ which contains the source code for a C++ Qt project. Then I wrote the following recipe file:
project.recipe
Bootstrap: docker
From: ubuntu:18.04
%setup
cp -R <some_folder>/HelloWorld ${SINGULARITY_ROOTFS}/HelloWorld
%post
apt update
apt-get install -y qt5-default
apt install -y g++
apt-get install -y build-essential
cd HelloWorld
qmake
make
echo "after build:"
ls
%runscript
echo "before execution:"
ls HelloWorld/
./HelloWorld/HelloWorld
where the echos and directory listings are for my current debugging process.
I can sucessfully build an image file using sudo singularity build --writable project.img project.recipe. (My debugging output shows me that the executable was build successfully.)
The problem is now that if I try to run it using ./project.img, or singularity run project.img, it won't find the executable.
Using my debugging output, I found out that the lines in %runscript use the folders outside of the container.
Tutorials like https://sylabs.io/guides/3.1/user-guide/build_a_container.html made it seem to me as if my recipe was the way to go, but apparently it isn't?
My questions:
Is there some way for me to access my executable? Am I calling it wrong?
Is the way I do it the way it is supposed to be done? Or would one normally do something like getting the executable outside of the container and then use the container to call that outside file? Or is there a different best practice?
If the executable is to be copied outside of the container after compilation, how do I do that? How do I access outside folders when I'm within %post?
Is this the best work process for what I want to achieve? Later on, my idea is that the big project is copied likewise in the container, dependencies are either installed or copied, then the project is compiled and finally its source being deleted. I also considered using a repository, but I can't have the project being in an open repository, and I don't want to store any passwords.
Firstly, use %files, don't use %setup. %setup is run as root and can directly modify the host server. You can very easily and accidentally break things without realizing it. You can get the same effect this way:
%files
some_folder/HelloWorld /HelloWorld
You are calling it wrong. In your %setup (and hopefully now in your %files) steps, you are copying the data to /HelloWorld. In your %runscript your are calling ./HelloWorld/HelloWorld which is the equivalent of $PWD/HelloWorld/HelloWorld. Since singularity automatically mounts in $PWD (as well as $HOME and some other directories), you are not calling what you're trying to call.
You don't copy the executable outside of the container, you just need to make sure what you're executing is where you think it is.
There is no access to the host filesystem in %post, you should have everything you need copied in via %files first.
That's a reasonable workflow. Having a local private repo for the code is probably a good idea for tracking your changes, but that's your call.

How to hack on installed perl6 module source?

I'd like to be able to view and make changes to the source code of installed (via zef) perl6 modules. How can I do that?
On my system, the module sources are under ~/.perl6/sources/ and there's also some kind of metadata file about the modules in ~/.perl6/dist/.
I can also use zef locate ... to show a module's source path, but making changes directly to the source files doesn't seem to have any effects (i.e., use the module from the REPL doesn't show my changes).
I'm guessing it's because the modules were pre-compiled, and perl6 doesn't pick up my changes and re-precompile the modules when I make changes directly to the module source files that way...
UPDATE: Deleting the corresponding pre-compiled files under ~/.perl6/precomp/... seems to work, but I'm not sure how and if that messes up anything.
I'd like to be able to view and make changes to the source code of installed (via zef) perl6 modules. How can I do that?
Please, don't do it that way. Installed modules are supposed to be immutable and as you've found out: if there is a pre-compiled version of a module available, it will not check if the original source file has been updated. That's because it doesn't have to, because it is considered immutable.
If you want to test changes on an installed module, please download the tar file / git clone the module's distribution, make changes you need in there, and then do:
zef install . --force-install
while in the top directory in the distribution. That will re-install the module and handle pre-compilation for you.

How do I install local modules?

For creating and maintaining Perl 5 modules, I use Dist::Zilla. One of my favorite features is being able to install local modules.
However, with Perl 6, I'm not sure how to install local modules. Sure, I can use use lib:
use lib 'relative/path';
use My::Awesome::Module;
But, I'd really like to be able to install My::Awesome::Module, so that all I had to do was use it:
use My::Awesome::Module;
One way to accomplish this, would be setting PERL6LIB, but that still isn't "installing" a module like zef install ./My-Awesome-Module.
Update: Looks like I need to craft an appropriate META6.json file.
To understand how to setup a module to be understood by toolchain utilities, see Preparing the module. Typically this means adding a META6.json file that describes the distribution, including quasi-manifest elements such as which files you really meant to include/provide. Once the META6.json is created the module is ready to be installed:
zef install ./My-Awesome-Module
which (assuming no uninstalled dependencies) is essentially:
my $install-to-repo = CompUnit::RepositoryRegistry.repository-for-name("site");
my $preinstall-dist = Distribution::Path.new("./My-Awesome-Module");
$install-to-repo.install($preinstall-dist);
Starting with rakudo 2019.01 you can, assuming no uninstalled dependencies, install a local distribution without a META6.json -- but this is purely developmental nicety that won't work on complex setups that do not have e.g. namespacing and file structures that can be inferred.
my $read-from-repo = CompUnit::Repository::FileSystem.new(prefix => "./My-Awesome-Module/lib");
my $install-to-repo = CompUnit::RepositoryRegistry.repository-for-name("site");
my $some-module-name = "My::Awesome::Module"; # needed to get at the Distribution object in the next step
my $preinstall-dist = $read-from-repo.candidates($some-module-name).head;
$install-to-repo.install($preinstall-dist);
I'm writing a bin that may help you: http://github.com/FCO/6pm

How to create RPM subpackages using the same paths for different envs?

I would like to use a rpm to build subpackages for different environments (live,testing,developer) but for the same files, so having a package called name-config-live, one called name-config-testing and one called name-config-developer and in them to have the same paths but each with the configs corresponding to the environment it's named after.
as an example
let's say on all environments I have a file called /etc/name.conf and on testing I want it to contain "1", on development "2" and on live "3". Is it possible to do this in the same spec since the subpackage generation only happens last not in the order I enter it. ( and hopefully not with %post -n )
I tried using BuildRoot but it seems that's a global attribute
I don't think there's a native way; I would do a %post like you had noted.
However, I would do this (similar to something I do with an internal-only package I develop for work):
Three separate files /etc/name.conf-developer, /etc/name.conf-live, etc.
Have all three packages provide a virtual package, e.g. name-config
Have main package require name-config
This will make rpm, yum, or whatever require at least one be installed in the same transaction
Have all three packages conflict with each other
Have each config package's %post (and possibly %verify) symlink /etc/name.conf to the proper config
This also helps show the user what is happening
Cons:
It's a little hackish
rpm --whatprovides /etc/name.conf will say it is not owned by any package

Why does the building from Binary files do not require Root access?

When I am in my dept's server, I cannot use commands such as "apt-get install nethack". I have to build the nethack from Binary files to get it working, at least so I have been told. I cannot understand the reason. Why do I need to build things from binaries? Why is the use of the commands, such as "apt-get", forbidden? Why do I not need Root access to build from binaries?
apt-get is a system-level command that installs packages for all users.
If you download and compile, you are only creating local "copies" of the binaries, not system-wide. If you tried to complete the install process with make install this would most likely fail because you do not have sufficient privileges to install the program for all users' access (same reason you can't run apt-get install)
When you compile a program from source, you can give it the '--prefix=~/'. This causes it to install relative to your own home directory (so binary programs typically end up in '~/bin', man pages in '~/man' etc). This poses no problems because you already have permission to write here.
Apt-get on the other hand installs the packages in the global filesystem ('/bin/', '/usr/bin/', etc), which can impact other users and so, quite rightly, require administrative access.
If you want to install some program you can use the command
apt-get source app-name
This will work even if you are not root since it only fetch the source code to the app-name and put it in the current directory, which is easier than having to track down the source and there is a better chance to get it work, since you download the version that should work on your system.
Alternatively you should bug your sysadmin to install the programs you need, since it is his job (and if you need them, chances are that the rest of your team does too).
Because apt-get will install a program system wide.
The locations to which apt-get writes installed files (/bin, /usr/bin, ...) are restricted to root access. I imagine that when you build from source you're not executing the install step of the bulid. You're going to need to set a prefix for the installation such that the packages end up somewhere you can write. This thread talks a bit about setting prefixes for apt-get and you'll probably want to set your prefix to something like
~/software/
and then add the resulting bin directories to your PATH.