When updating and upgrading pkgsrc and its packages using
sudo pkgin -y update
sudo pkgin -y full-upgrade
sudo pkgin -y autoremove
it suggested that some directories could be deleted:
downloading packages...
p5-Variable-Magic-0.62.tgz 100% 42KB 41.6KB/s 41.6KB/s 00:00
p5-Role-Tiny-2.000006.tgz
...
removing packages to be upgraded...
removing pkgin-0.9.4nb7...
===========================================================================
The following files are no longer being used by pkgin-0.9.4nb7,
and they can be removed if no other packages are using them:
/opt/pkg/etc/pkgin/repositories.conf
===========================================================================
===========================================================================
The following directories are no longer being used by pkgin-0.9.4nb7,
and they can be removed if no other packages are using them:
/opt/pkg/etc/pkgin
/var/db/pkgin
===========================================================================
removing p5-PerlMagick-7.0.7.8nb1...
removing jasper-2.0.12...
...
My question is that how does one determine if no other packages are using the directories suggested above so they can be deteled? Simply ignoring this step and deleting them does not seem safe.
Those messages are, for the most part, really just for the pedants who would worry and fuss over things that they no longer need and might get confused by, even when they don't cause any problems or get in the way of anything. (Such leftovers could possibly be of some concern if one was working on changing the package they were used by.)
In this particular case though the messages are actually misleading since they are about components used by pkgin itself and once pkgin upgrades itself they will in fact be in use again. One might even consider the appearance of those messages in this case to be a bug.
[Solved (at bottom). installed quartz and re-installed with x11 via brew then restarted machine.]
I'm learning Ocaml and am going through these documentations pages and need to install some modules (graphics).
I'm missing a Graphics module in Ocaml. After trying to load it on toplevel (the REPL right?) with:
$ ocaml
OCaml version blahblah
# #load "graphics.cma";;
# open Graphics;;
I get the error message:
Cannot find file graphics.cma.
So I wander over to this question and after not finding the file with the command:
ls `ocamlc -where`/graphics*`
I read that this means that:
Graphics is not installed and you have to reinstall OCaml compiler
enabling Graphics.
Does this mean I have to recompile Ocaml every time I need a new module? I'm not sure what he meant by that.
Then, I tried to install Graphics with: opam install graphics.
I got this error:
This package relies on external (system) dependencies that may be missing. `opam depext lablgl.1.05' may help you find the correct installation for your system.
So I did opam depext lablgl.1.05
After this, I tried opam install graphics again, but it failed with this error:
#=== ERROR while installing graphics.1.0 ======================================#
# opam-version 1.2.2
# os darwin
# command ocamlc -custom graphics.cma -o test
# path /Users/alexanderkleinhans/.opam/system/build/graphics.1.0
# compiler system (4.02.2)
# exit-code 2
# env-file /Users/alexanderkleinhans/.opam/system/build/graphics.1.0/graphics-24451-7afd23.env
# stdout-file /Users/alexanderkleinhans/.opam/system/build/graphics.1.0/graphics-24451-7afd23.out
# stderr-file /Users/alexanderkleinhans/.opam/system/build/graphics.1.0/graphics-24451-7afd23.err
### stderr ###
# File "_none_", line 1:
# Error: Cannot find file graphics.cma
=-=- Error report -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
The following actions failed
∗ install graphics 1.0
No changes have been performed
=-=- graphics.1.0 troubleshooting -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
=> This package checks whether the Graphics library was compiled.
This error says Cannot find file graphics.cma which brings me back to this question and what the steps are to get graphics.cma (and other modules as I might need them).
I though opam was a package manager for ocaml (this install modules right?)
EDIT:
I did brew info ocaml and I did install with x11 so I though this meant I should have it...
ocaml: stable 4.04.1 (bottled), devel 4.05.0+beta3, HEAD
General purpose programming language in the ML family
https://ocaml.org/
/usr/local/Cellar/ocaml/4.04.1 (1,730 files, 194.4MB) *
Poured from bottle on 2017-06-13 at 15:23:43
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/ocaml.rb
==> Requirements
Optional: x11 ✘
==> Options
--with-flambda
Install with flambda support
--with-x11
Install with the Graphics module
--devel
Install development version 4.05.0+beta3
--HEAD
Install HEAD version
EDIT 2:
Running
brew install Caskroom/cask/xquartz
brew reinstall ocaml --with-x11
Allowed me to compile, but running gave me this fatal exception. Seems to be an X11 thing?
Fatal error: exception Graphics.Graphic_failure("Cannot open display ")
Solved
So I think the two steps that were necessary were to make sure ocaml was installed with X11. Note that `brew info ocaml` seemed to give wrong information (said it was installed with X11 but reinstall was necessary). On OSX, I also needed to install quarts.
brew install Caskroom/cask/xquartz
brew reinstall ocaml --with-x11
After this I COULD compile, but got an error on execution. This was simply solved by restarting which I read was necessary after installation of xquartz.
After that I could run fine.
The graphics module is an optional part of the base OCaml install, not an external module. This explains why you can't install it using OPAM. The OPAM module that you show is only testing whether it is installed in the current OCaml system. It can't (and hence doesn't try to) install graphics as a separate module.
For this reason, installing graphics (when it's not already installed) is unusually tricky. There's no need to recompile OCaml for installing most (if not all) other modules.
For what it's worth, I am running macOS 10.12.4, and I used "opam switch" to switch my OCaml system to the 4.03.0 release. In the resulting environment, the Graphics module is installed, and I have no trouble running the examples at the website you mention. (For the first, I see concentric red and yellow circles, for example.)
You might try doing "opam switch" to switch to a recent version of the compiler, and see if this gets things going for you. In the past I have had trouble getting Graphics to work, but it is working great for me now.
I'm starting with Ansible, and I found that there is a module called command which lets me execute any command in a remote node.
I saw a couple of example where initial setups are solved by using command instead of specific modules. For example, as far as I know, both of these do the same task:
- name: Install git using apt module
apt:
name: git
state: present
- name: Install git using command
command: apt-get install git
So, my question is: is there any difference or any reason to use a module instead of command?
The difference in short is that using a specific module will give you playbook's idempotence and provide better portability and readability.
What I mean by idempotence? When you run:
- name: Install git using apt module
apt:
name: git
state: present
It will install git package only if it is not yet installed on the target system and after playbook run this task will be reported in green colour (OK) if git had been already installed.
2nd approach with the command module:
- name: Install git using command
command: apt-get install git
Above command will always report status as changed (yellow colour) when in fact nothing changed (assuming git package had been already installed). There are ways to make tasks that use the command module idempotent as well but it costs you some more work.
Best practice is to always use a specific module before command in playbooks.
Ansible is all about describing and managing system state. When you run a playbook on a certain target system it can be very misleading to see a task reporting a changed state while in fact nothing has been changed.
Think declaratively about describing desired state, not about low level commands needed to get a system to this state.
Below article will also provide some explanation around differences and consequences of using command vs specific module:
Ansible Best Practices: The Essentials
There are probably numerous reasons but here are a few:
Intrinsic idempotence (does not execute task every time without extra effort)
Superior readability (much clearer what you are trying to do)
More concise tasks (much fewer words to describe the task)
Platform-agnostic execution (works on all OS instead of just one without extra effort)
I'm having a hell of a time deploying matplotlib on AWS Elastic Beanstalk. I gather that my issue comes from some dependencies and the way that EB deploys packages installed with PIP, and have attempted to follow the instructions here on SO for resolving the issue.
I first tried incrementally deploying, as suggested in the linked answer, by adding pieces of the matplotlib package stack to my requirements.txt file in stages. But this takes forever (for each stage) and is prone to failure and timing out (which seems to leave build directories behind that stall subsequent package installations).
So the simple solution mentioned off-handedly at the end of the answer appeals to me: just eb ssh, activate the virtialenv with
source /opt/python/run/venv/bin/activate
and pip install packages manually. But I can't get this to work either. First I'm often confronted with left-beind build directories (as mentioned above)
pip can't proceed with requirement 'xxxx' due to a pre-existing build directory.
location: /opt/python/run/venv/build/xxxx
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
Please delete it and try again.
But even after removing these, I consistently get
Exception:
Traceback (most recent call last):
File "/opt/python/run/venv/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/opt/python/run/venv/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/opt/python/run/venv/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files
do_download,
File "/opt/python/run/venv/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url
self.session,
File "/opt/python/run/venv/lib/python2.7/site-packages/pip/download.py", line 582, in unpack_http_url
unpack_file(temp_location, location, content_type, link)
File "/opt/python/run/venv/lib/python2.7/site-packages/pip/util.py", line 625, in unpack_file
untar_file(filename, location)
File "/opt/python/run/venv/lib/python2.7/site-packages/pip/util.py", line 533, in untar_file
os.makedirs(location)
File "/opt/python/run/venv/lib64/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/opt/python/run/venv/build/xxxx'
in response to pip install xxxx (and sudo pip fails with sudo: pip: command not found).
What can I do to get this working on AWS-EB? In particular, what do I need to do to get the simple SSH+PIP approach working; or is there some other better — simpler! — approach I should try.
FWIW, I have a .ebextensions/software.config with
packages:
yum:
gcc-c++: []
gcc-gfortran: []
python-devel: []
atlas-sse3-devel: []
lapack-devel: []
libpng-devel: []
freetype-devel: []
zlib-devel: []
and a requirements.txt that ends with
pytz==2014.10
pyparsing==2.0.3
python-dateutil==2.4.0
nose==1.3.4
six>=1.8.0
mock==1.0.1
numpy==1.9.1
matplotlib==1.4.2
After about 4 hours, I've gotten far as numpy (as reported by pip list in the EB virtualenv).
And (in case it matters) the user who is SSHing is part in a group with the policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"elasticbeanstalk:*",
"ec2:*",
"elasticloadbalancing:*",
"autoscaling:*",
"cloudwatch:*",
"s3:*",
"sns:*",
"cloudformation:*",
"rds:*",
"sqs:*",
"iam:PassRole"
],
"Resource": "*"
}
]
}
I have used many approaches to build and deploy numpy/scipy/matplotlib, on Windows as well as Linux systems. I have used system-provided package managers (aptitude, rpm), 3rd-party package managers (pypm), Python package managers (easy_install, pip), source releases, used different build environments/tools (GCC, but also Intel MKL, OpenMP). While doing so, I have run into many many quite annoying situations, but have also learned a lot about the pros and cons of each approach.
I have no experience with Elastic Beanstalk (EB), but I have experience with EC2. I see that you can SSH into an instance and poke around. So, what I suggest further below is based on
above-stated experiences and on
the more or less obvious boundary conditions regarding Beanstalk and on
your application scenario, described in another question here on SO and on
the fact that you just want to get things running, quickly
My suggestion: start off with not building these things yourself. Do not use pip. If possible, try to use the package manager of the Linux distribution in place and let it handle the installation of everything required for you, with a single command (e.g. sudo apt-get install python-matplotlib).
Disadvantages:
possibly old package versions, depending on the Linux distro in use
non-optimized builds (e.g. not built against e.g. Intel MKL or not leveraging OpenMP features or not using special instruction sets)
Advantages:
it quickly downloads, because packages are most likely cached near your machine
it quickly installs (these packages are pre-built, no compilation involved)
it just works
So, I hope you can just use aptitude or rpm or whatever on these machines and inherit the great work that the distribution package maintainers do for you, behind the scenes.
Once you are confident in your application and identified some bottleneck or issue, you might have reason to use a newer version of numpy/matplotlib/... or you might have reason to have a faster version of these, by creating an optimized build.
Edit: EB-related details of outlined approach
In the meantime we have learned that EB by default runs Amazon Linux which is based on Red Hat Enterprise Linux. Likewise, it uses yum as package manager and packages are in RPM format.
Amazon provides documentation about available packages. In Amazon Linux 2014.09, these packages are available: http://aws.amazon.com/de/amazon-linux-ami/2014.09-packages/
In this list we find
numpy-1.7.2
python-matplotlib-0.99.1.2
This version of matplotlib is very old, according to the changelog it is from September 2009: "2009-09-21 Tagged for release 0.99.1".
I did not anticipate it to be so old, but still, it might be sufficient for your needs. So we proceed with our plan (but I'd understand if that's a blocker).
Now, we have learned that system Python and EB Python are isolated from each other. That does not mean that EB Python cannot access system Python site packages. We just need it to tell so. A simple and clean method is to set up a proper directory structure with the packages that should be accessible to EB Python, and to communicate this directory to EB Python via sys.path.
Clearly, we need to customize the bootstrapping phase of EB containers. The available tools are documented here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
Obviously, we want to make use of the packages approach, and tell EB to install the numpy and python-matplotlib packages via yum. So the corresponding config file section should contain:
packages:
yum:
numpy: []
python-matplotlib: []
Explicitly mentioning numpy might not be necessary, it likely is a dependency of python-matplotlib.
Also, we need to make use of the commands section:
You can use the commands key to execute commands on the EC2 instance.
The commands are processed in alphabetical order by name, and they run
before the application and web server are set up and the application
version file is extracted.
The following three commands create above-mentioned directory, and set up symbolic links to the numpy/mpl installation paths (these paths hopefully are available in the moment these commands become executed):
commands:
00-create-dir:
command: "mkdir -p /opt/py26-selected-site-packages"
01-link-numpy:
command: "ln -s /usr/lib64/python2.6/site-packages/numpy /opt/py26-selected-site-packages/numpy"
02-link-mpl:
command: "ln -s /usr/lib64/python2.6/site-packages/matplotlib /opt/py26-selected-site-packages/matplotlib"
Two uncertainties: the AWS docs to not clarify that packages are processed before commands are executed. You have to try. It it does not work, use container_commands. Secondly, it is just an educated guess that /usr/lib64/python2.6/site-packages/matplotlib is available after installing python-matplotlib. It should be installed to this place, but it may end up somewhere else. Needs to be tested. Numpy should end up where specified as inferred from this article.
[UPDATE FROM SEB]
AWS documentation says "The cfn-init helper script processes these configuration sections in the following order: packages, groups, users, sources, files, commands, and then services."
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html
So, your approach is safe
[/UPDATE]
The crucial step, as pointed out in the comments to this answer, is to tell your Python app where to look for packages. Direct modification of sys.path before attempting to import is a reliable method to take control of this. The following code adds our special directory to the selection of directories in which Python looks out for packages, and then attempts to import matplotlib:
sys.path.append("/opt/py26-selected-site-packages")
from matplotlib import pyplot
The order in sys.path defines priorities, so in case there is any other matplotlib or numpy package available in one of the other directories, it might be a better idea to
sys.path.insert(0, "/opt/py26-selected-site-packages")
However, this should not be necessary if our whole approach was well thought-through.
To add to Jan-Philip Answer :
AWS Elastic Beanstalk is using Amazon Linux distribution (except for .Net environments). Amazon Linux uses the yum package manager. MatPlotLib is available in Amazon's software repository.
[ec2-user#ip-1-1-1-174 ~]$ yum list | grep matplot
python-matplotlib.x86_64 0.99.1.2-1.6.amzn1 amzn-main
If this version is the one you need for your application, I would try to simply modify your .ebextensions/software.config file and to add the package to the yum section of it:
packages:
yum:
python-matplotlib: []
python-devel: []
atlas-sse3-devel: []
lapack-devel: []
libpng-devel: []
freetype-devel: []
zlib-devel: []
A last note about AWS Elastic BeansTalk and SSH.
While Amazon gives you the possibility to SSH to your Elastic Beanstalk instances, you should use this possibility only for debugging purposes, to understand why your app failed or is not installing as suggested.
Other than that, your deployment must be 100% automatic. When Elastic Beanstalk (Auto Scaling to be precise) will scale out your infrastructure (add more instances) or scale it in (terminate instances) depending on your application workload, all your manual configuration will be lost.
Best practices is to not install SSH keys on your production environment, it further reduces the surface of attacks.
I might be a bit late to this question, but as AWS and a lot of the cloud service providers are moving into Docker and taking into consideration that you haven't specified the platform . I have a fast solution to your question:
Use the generic docker platform.
I created some images with Python, Numpy, Scipy and Matplotlib preinstalled, so you can directly pull and start using them with one line of code.
Python 2.7(This one also has the versions that you were specifying for numpy and matplotlib)
sudo docker pull chuseuiti/pynuscimat2.7
Python 3.4
sudo docker pull chuseuiti/pynusci
However you can create your own image or modify existing images.
In case you want to automate your instances, you can pass a Dockerfile to AWS with the definition of your image.
Tip, in case you don't know about docker:
It is need to login before been able to pull:
sudo docker login
After pulling the image, you can generate and work in a container created from an image with the next code:
sudo docker run -i -t chuseuiti/pynuscimat2.7 bash
PS. At least with the free tier AWS is always complaining about running out of time with scipy and matplotlib, it takes too much time to install them, that is why I use this option.
I'm trying to build apache from source on debian. The only reason I'm not using spt-get install is because in the apache cookbook, they recommend installing from source.I get the following error when I ./configure:
configure: error: invalid variable name: ' --with-mpm'
I also saw some warnings when I ./buildconf Is this something I should be concerned about? This is my first attempt at compiling from source, and I'd really appreciate any help.
I'm using the ./configure arguments directly from the apache cookbook:
./configure --prefix=/usr/local/apache --with-layout=Apache --enable-modules=most --enable-mods-shared=all \ --with-mpm=prefork
I'm running a minimum debian install in virtual box to train myself for deploying in the rackspace cloud soon.
EDIT: I'm building Apache 2.2.16
I suspect you are typing that entire build line you provided on one line, complete with the '\' in the middle.
You should get rid of '\', which in bash either treats the following as part of the same string, but the slash has to immediately follow a non-whitespace character. It is also used for special escape sequences, which I think is the case here and generating that message.
This should be the correct line in your case.
./configure --prefix=/usr/local/apache --with-layout=Apache --enable-modules=most --enable-mods-shared=all --with-mpm=prefork
On a side note, doesn't the Apache Cookbook say that building from source is one possibility for installing it, in addition to installing from a pre-packaged build like you can get from Debian's repositories? I suppose if you really wanted a far newer build or a more repeatable process to ensure consistency across a variety of distributions, building from scratch will do that for you, but otherwise I would try to utilize the distribution's package management as much as possible. Building from source removes you from the security patches and ease-of-upgrade path that Debian APT gives you.