I am using Chef and this s3cmd cookbook.
As this tutorial says I use knife to download and tar it. I actually made s3cmd work following the tutorial instructions, but I have problems understanding where exactly the installation of s3cmd is happening?
Can anyone explain to me what Chef is doing when using the s3cmd recipe?
When you run chef-solo locally (or chef-client in a server setup) you are telling Chef to compile a set of resources defined in your cookbooks recipes to then be applied to the node you are running the command on.
A resource defines the tasks and settings for something you want to setup.
The run_list set for the node defines what cookbook recipes will be compiled.
s3cmd
In your case, your run_list will have probably been recipe[s3cmd].
This instructs Chef to look in the s3cmd cookbook and as you didn't give a specific recipe, it loads s3cmd/recipes/default.rb.
If you gave a specific recipe, like recipe[s3_cmd::other] then Chef would load the file s3_cmd/recipes/other.rb.
Chef will compile all the resources defined in the recipe(s) into a list and the run through the list applying changes as required to your system.
What s3cmd::default does
First it installs some packages (via your distributions package manager)
python, python-setuptools, python-distutils-extra, python-dateutil, s3cmd
Note: This is entirely different to what the readme says about how s3cmd is installed! Always check!
Figures out where the config should go.
if node['s3cmd']['config_dir']
home_folder = node['s3cmd']['config_dir']
else
home_folder = node['etc']['passwd'][node['s3cmd']['user']]['dir']
end
Creates the .s3cfg config file from a template in the cookbook.
template "#{home_folder}/.s3cfg" do...
What else
Cookbooks can also define their own resources and providers to create more reusable cookbooks. The resource names and attributes will be defined in cookbook/resources/blah.rb. The code for each resource action will be in cookbook/providers/blah.rb
Code can be packaged in cookbook/libraries/blah.rb and included in other Ruby files.
Run chef-solo with the --log-level DEBUG option and step through the output. Try and identify the run list compilation phase, and then where everything is being applied.
Related
I am a beginner with Singularity.
What I want to achieve in the long run: I have a programming project with a long lists of dependencies, and I want to be able to give the program to other people in my company without there being bugs caused by missing dependencies, or wrong versions of dependencies.
The idea was now to use Singularity in order to easily provide a working environment.
In order to test this, I wrote a Hello World application which I now want to run in a container. I have a folder HelloWorld/ which contains the source code for a C++ Qt project. Then I wrote the following recipe file:
project.recipe
Bootstrap: docker
From: ubuntu:18.04
%setup
cp -R <some_folder>/HelloWorld ${SINGULARITY_ROOTFS}/HelloWorld
%post
apt update
apt-get install -y qt5-default
apt install -y g++
apt-get install -y build-essential
cd HelloWorld
qmake
make
echo "after build:"
ls
%runscript
echo "before execution:"
ls HelloWorld/
./HelloWorld/HelloWorld
where the echos and directory listings are for my current debugging process.
I can sucessfully build an image file using sudo singularity build --writable project.img project.recipe. (My debugging output shows me that the executable was build successfully.)
The problem is now that if I try to run it using ./project.img, or singularity run project.img, it won't find the executable.
Using my debugging output, I found out that the lines in %runscript use the folders outside of the container.
Tutorials like https://sylabs.io/guides/3.1/user-guide/build_a_container.html made it seem to me as if my recipe was the way to go, but apparently it isn't?
My questions:
Is there some way for me to access my executable? Am I calling it wrong?
Is the way I do it the way it is supposed to be done? Or would one normally do something like getting the executable outside of the container and then use the container to call that outside file? Or is there a different best practice?
If the executable is to be copied outside of the container after compilation, how do I do that? How do I access outside folders when I'm within %post?
Is this the best work process for what I want to achieve? Later on, my idea is that the big project is copied likewise in the container, dependencies are either installed or copied, then the project is compiled and finally its source being deleted. I also considered using a repository, but I can't have the project being in an open repository, and I don't want to store any passwords.
Firstly, use %files, don't use %setup. %setup is run as root and can directly modify the host server. You can very easily and accidentally break things without realizing it. You can get the same effect this way:
%files
some_folder/HelloWorld /HelloWorld
You are calling it wrong. In your %setup (and hopefully now in your %files) steps, you are copying the data to /HelloWorld. In your %runscript your are calling ./HelloWorld/HelloWorld which is the equivalent of $PWD/HelloWorld/HelloWorld. Since singularity automatically mounts in $PWD (as well as $HOME and some other directories), you are not calling what you're trying to call.
You don't copy the executable outside of the container, you just need to make sure what you're executing is where you think it is.
There is no access to the host filesystem in %post, you should have everything you need copied in via %files first.
That's a reasonable workflow. Having a local private repo for the code is probably a good idea for tracking your changes, but that's your call.
Two computers are working on the SAME repository but first computer detect the library and work well but second computer not detect it and show "Error 'PhpOffice\Phpspreadsheet\Reader\Xlsx' not found".
In vendor, the library also exist.
composer.json and composer.lock also the same on both computer.
One thing is that by git ignore, I use yii's composer mechanism at 1st computer but at sec computer(err computer), I add library manually.
If you want to use a composer package, you absolutely need to install it using composer. This ensures that the autoloader is generated properly and your class can be found through PHP.
Copying library files into vendor directory is not enough to install it. During installation Composer creates autoload script with information how to find all classes installed by Composer. If you just copy library files, Composer will not even know that it exist and will not able load any class from it.
If you cannot use Composer on server/computer A, you should install all dependencies on different computer (B) and copy the entire vendor directory into server/computer A. Autoload definitions are in vendor so it should work if you copy the whole dorectory.
I successfully installed subversion on my server. When I run the command:
which svn
I get the response:
/usr/bin/svn
I create a repository inside my root directory which is my httpdocs folder with the following command:
svnadmin create ~/svn
I successfully created the following files and folders and configured them and I see the following files and directories inside the svn directory:
conf db format hooks locks README.txt
But I cannot create trunk, branch, and tag directories. In fact I cannot create directories period. Here is the command I have been using:
svn mkdir file:///httpdocs/svn/site
When I make this command the message I get is the following:
svn: Could not use external editor to fetch log message; consider setting the
$SVN_EDITOR environment variable or using the --message (-m) or --file (-F) options
svn: None of the environment variables SVN_EDITOR, VISUAL or EDITOR are set, and no
'editor-cmd' run-time configuration option was found
Also When I type in url:
http://mydomain.com/svn
I get the follwoing message:
404 - Not Found
What do I need to do to configure my repository, check to see if my repository is actually working. Oh and I did run the following command:
svnserve -d
I create a repository inside my root directory which is my httpdocs folder with the following command...
That's not how you get Subversion to work with Apache httpd if that's what you're trying to do. You need the mod_dav and mod_dav_svn modules for Apache. You might have to recompile Apache to get these.
Once you get mod_dav and mod_dav_svn installed, you need to configure your httpd.conf file (or put a configuration file under your httpd's conf.d depending how it's configured on your system) for it to work.
In the end, it's not all that difficult. Most Apache installations have mod_dav and mod_dav_svn added in, and you can see if your http.conf file is configured correctly. The on line Subversion manual will have everything you need to get it working.
svn: Could not use external editor to fetch log message; consider setting the
$SVN_EDITOR environment variable or using the --message (-m) or --file (-F) options
svn: None of the environment variables SVN_EDITOR, VISUAL or EDITOR are set, and no
'editor-cmd' run-time configuration option was found
Read the message. It says Could not use external editor because you didn't specify one. Again the Subversion on line manual explains it in detail.
Basically, when you commit a change in Subversion, you need to create a commit message. This can be done in two ways:
You use the -m option like svn commit -m "This is my commit message".
You set the variables SVN_EDITOR, VISUAL, or EDITOR to the name of the program you want to use. For example, in Windows, you'd say C:> set EDITOR=notepad.exe. On Unix, you'd say something like $ export EDITOR=vi. Subversion first checks the value of the SVN_EDITOR variable, then VISUAL, then EDITOR. If none of them are set, it gives you the error you saw.
Why did you get it on the svn mkdir command?
Because you use the URL form of the command, it will make the directory, then commit the change, thus the need for a commit message. This should work:
svn mkdir -m"Adding the site directory to my repository" file:///httpdocs/svn/site
Now, you need to move your repository out of your HTTPDOCs directory. That's a very bad thing to have because it hurts your web server.
If you don't want to use the file:// protocol, you can use svnserve. This is a built-in Subversion repository server and uses the svn:// protocol. Observe:
$ svnadmin create my_repos #Creates the repository
$ vi my_repos/conf/svnserve.conf #See footnote 1 below
$ vi my_repos/conf/passwd #See footnote 1 below
$ svnserve -r my_repos -d
$ svn mkdir -m "Making basic directory layout svn://localhost/trunk \
> svn://localhost/tags svn://localhost/branches
$ export SVN_EDITOR="vi"
Your repository also becomes accessible to all the machines on your network too. Using svnserve is much simpler than Apache httpd, but there are a few issues:
You can't run multiple instances of svnserve since it wants to use port 3690 and it doesn't like to share. With Apache httpd, you can have multiple repositories.
Security setup with svnserve is limited. With Apache, I can use Windows Active directory or LDAP or I can configure it manually.
Some web based subversion repository browsers don't like svnserve.
By the way, if you haven't, go through the Subversion on line manual. It's one of the best manuals I've seen for any open source project.
1 When you setup your repository using svnserve, you need to edit the svnserve.conf file by enabling the line password-db = passed which is about line #20 in the file. Then you have to edit the passed file (located in the same directory) to configure the users and their passwords. Both are very straight forward, but easy to forget, and if you don't do it, you can't commit anything into your repository.
when you need to trunk branches and tags than you have to need the download import_dirs.copy
than after than cd /location/import_dirs.copy than run than command
cd /data/svn/import_dirs.copy/
svn import file:///location of repos/reposname/ -m "inital message"
this is the example of it.....
cd /home/raj/import_dirs.copy/
svn import file:///srv/svn/nmg/ -m "inital message"
than trunk branches and tags have been done
When I am in my dept's server, I cannot use commands such as "apt-get install nethack". I have to build the nethack from Binary files to get it working, at least so I have been told. I cannot understand the reason. Why do I need to build things from binaries? Why is the use of the commands, such as "apt-get", forbidden? Why do I not need Root access to build from binaries?
apt-get is a system-level command that installs packages for all users.
If you download and compile, you are only creating local "copies" of the binaries, not system-wide. If you tried to complete the install process with make install this would most likely fail because you do not have sufficient privileges to install the program for all users' access (same reason you can't run apt-get install)
When you compile a program from source, you can give it the '--prefix=~/'. This causes it to install relative to your own home directory (so binary programs typically end up in '~/bin', man pages in '~/man' etc). This poses no problems because you already have permission to write here.
Apt-get on the other hand installs the packages in the global filesystem ('/bin/', '/usr/bin/', etc), which can impact other users and so, quite rightly, require administrative access.
If you want to install some program you can use the command
apt-get source app-name
This will work even if you are not root since it only fetch the source code to the app-name and put it in the current directory, which is easier than having to track down the source and there is a better chance to get it work, since you download the version that should work on your system.
Alternatively you should bug your sysadmin to install the programs you need, since it is his job (and if you need them, chances are that the rest of your team does too).
Because apt-get will install a program system wide.
The locations to which apt-get writes installed files (/bin, /usr/bin, ...) are restricted to root access. I imagine that when you build from source you're not executing the install step of the bulid. You're going to need to set a prefix for the installation such that the packages end up somewhere you can write. This thread talks a bit about setting prefixes for apt-get and you'll probably want to set your prefix to something like
~/software/
and then add the resulting bin directories to your PATH.
As part of our product release we ship Apache HTTP Server binaries that we have compiled on our (UNIX) development machine.
We tell our clients to install the binaries (on their UNIX servers) under the same directory structure that we compiled it under. For some clients this is not appropriate, e.g. where there are restrictions on where they can install software on their servers and they don't want to compile Apache themselves.
Is there a way of compiling Apache HTTP Server so its installation location(s) can be specified dynamically using environment variables ?
I spent a few days trying to sort this out and couldn't find a way to do it. It led me to believe that the Apache binaries were hard coding some directory paths at compilation preventing the portability we require.
Has anyone managed to do this ?
I think the way to do(get around) this problem is to develop a "./configure && make" script that your client uses to install, specify and compile the binaries. That would offcourse require that the client has all the source-code installed on his server or you can make it available on an NFS share.
If you are compiling Apache2 for a particular location but want your clients to be able to install it somewhere else (and I'm assuming they have the same architecture and OS as your build machine) then you can do it but the apachectl script will need some after-market hacking.
I just tested these steps:
Unpacked the Apache2 source (this should work with Apache 1.3 as well though) and ran ./configure --prefix=/opt/apache2
Ran make then sudo make install to install on the build machine.
Switch to the install directory (/opt/apache2) and tar and gzip up the binaries and config files. I used cd /opt/apache2; sudo tar cf - apache2 | gzip -c > ~/apache2.tar.gz
Move the tar file to the target machine. I decided to install in /opt/mynewdir/dan/apache2 to test. So basically, your clients can't use rpm or anything like that -- unless you know how to make that relocatable (I don't :-) ).
Anyway, your client's conf/httpd.conf file will be full of hard-coded absolute paths -- they can just change these to whatever they need. The apachectl script also has hard coded paths. It's just a shell script so you can hack it or give them a sed script to convert the old paths from your build machine to the new path on your clients.
I skipped all that hackery and just ran ./bin/httpd -f /opt/mynewdir/dan/conf/httpd.conf :-)
Hope that helps. Let us know any error messages you get if it's not working for you.
I think the way to do(get around) this problem is to develop a "./configure && make" script that your client uses to install, specify and compile the binaries. That would offcourse require that the client has all the source-code installed on his server or you can make it available on an NFS share.
Not to mention a complete build toolchain. These days, GCC doesn't come default with most major distributions. Wouldn't it be sane to force the client to install it to /opt/my_apache2/ or something like that?
#Hissohathair
I suggest 1 change to #Hissohathair's answer.
6). ./bin/httpd -d <server path> (although it can be overridden in the config file)
In apacheclt there is a variable for HTTPD where you could override to use it.