custom compile an egg and install console scripts - buildout

Question

I'm trying to get the following buildout working. The buildout is meant to be self contained so I don't want to install any system packages
[s3ql]
recipe = zc.recipe.egg:scripts
eggs =
${s3ql-build:egg}
[s3ql-build]
recipe = zc.recipe.egg:custom
egg = s3ql
find-links =
https://bitbucket.org/nikratio/s3ql/downloads/s3ql-1.18.1.tar.bz2#egg=s3ql-1.18.1
http://security.ubuntu.com/ubuntu/pool/universe/s/s3ql/s3ql_1.16.orig.tar.bz2#egg=s3ql
https://github.com/wilkinson/s3ql/archive/master.zip#egg=s3ql
https://github.com/rogerbinns/apsw/releases/download/3.8.4.3-r1/apsw-3.8.4.3-r1.zip
include-dirs =
${liblzma:location}/include
${sqlite:location}/include
library-dirs =
${liblzma:location}/lib
${sqlite:location}/lib
rpath =
${liblzma:location}/lib
${sqlite:location}/lib
[liblzma]
recipe = zc.recipe.cmmi
url = http://tukaani.org/xz/xz-5.0.5.tar.gz
[pysqlite]
recipe = zc.recipe.egg:custom
egg = pysqlite
include-dirs = ${sqlite:location}/include
library-dirs = ${sqlite:location}/lib
rpath = ${sqlite:location}/lib
[sqlite]
recipe = zc.recipe.cmmi
url = http://www.sqlite.org/2014/sqlite-autoconf-3080403.tar.gz
and I get the following
Updating liblzma.
Updating sqlite.
Updating s3ql-build.
Installing s3ql.
Getting distribution for 'pyliblzma>=0.5.3'.
/bin/sh: 1: pkg-config: not found
/bin/sh: 1: pkg-config: not found
In file included from src/liblzma.c:1:0:
src/liblzma.h:24:18: fatal error: lzma.h: No such file or directory
compilation terminated.
error: Setup script exited with error:
command 'gcc' failed with exit status 1
An error occurred when trying to install pyliblzma 0.5.3. Look above this message for any errors that were output by easy_install.
While:
Installing s3ql.
Getting distribution for 'pyliblzma>=0.5.3'.
Error: Couldn't install: pyliblzma 0.5.3
It seems like it tries to compile it again.

Answer 1

You need to install liblzma-dev on your OS.
gottfried#dell-xt2:~/virtualenvs/liblzma$ sudo apt-get install liblzma-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
liblzma-doc
The following NEW packages will be installed:
liblzma-dev
0 upgraded, 1 newly installed, 0 to remove and 22 not upgraded.
Need to get 159 kB of archives.
After this operation, 645 kB of additional disk space will be used.
Get:1 http://de.archive.ubuntu.com/ubuntu/ saucy/main liblzma-dev amd64 5.1.1alpha+20120614-2ubuntu1 [159 kB]
Fetched 159 kB in 1s (111 kB/s)
Selecting previously unselected package liblzma-dev:amd64.
(Reading database ... 272326 files and directories currently installed.)
Unpacking liblzma-dev:amd64 (from .../liblzma-dev_5.1.1alpha+20120614-2ubuntu1_amd64.deb) ...
Setting up liblzma-dev:amd64 (5.1.1alpha+20120614-2ubuntu1) ...
gottfried#dell-xt2:~/virtualenvs/liblzma$ bin/easy_install pyliblzma
Searching for pyliblzma
Reading https://pypi.python.org/simple/pyliblzma/
Best match: pyliblzma 0.5.3
Downloading https://pypi.python.org/packages/source/p/pyliblzma/pyliblzma-0.5.3.tar.bz2#md5=500f61116ee1ab4063b49c121786863a
Processing pyliblzma-0.5.3.tar.bz2
Writing /tmp/easy_install-kAg4nj/pyliblzma-0.5.3/setup.cfg
Running pyliblzma-0.5.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-kAg4nj/pyliblzma-0.5.3/egg-dist-tmp-TmcZm_
/bin/sh: 1: pkg-config: not found
/bin/sh: 1: pkg-config: not found
zip_safe flag not set; analyzing archive contents...
Adding pyliblzma 0.5.3 to easy-install.pth file
Installed /home/gottfried/virtualenvs/liblzma/lib/python2.7/site-packages/pyliblzma-0.5.3-py2.7-linux-x86_64.egg
Processing dependencies for pyliblzma
Finished processing dependencies for pyliblzma

Answer 2

ok, I figured it out. Yes using a combination of zc.recipe.egg:scripts and zc.recipe.egg:custom is the right approach. But it's confusing as zc.recipe.egg:custom doesn't build the dependencies so it appears like it wasn't picking up the built s3ql.
Here is the solution that built for me.
[s3ql]
recipe = zc.recipe.egg:scripts
eggs =
${pyliblzma:egg}
${llfuse:egg}
${apsw:egg}
${s3ql-build:egg}
[s3ql-build]
recipe = zc.recipe.egg:custom
egg = s3ql
find-links =
https://bitbucket.org/nikratio/s3ql/downloads/s3ql-1.18.1.tar.bz2#egg=s3ql-1.18.1
# need this patch to build on osx
# https://bitbucket.org/nikratio/s3ql/commits/39352e8e4cb48521dd9ff589e93fc2e4ae390f9d/raw/
include-dirs =
${liblzma:location}/include
${sqlite:location}/include
library-dirs =
${liblzma:location}/lib
${sqlite:location}/lib
rpath =
${liblzma:location}/lib
${sqlite:location}/lib
[pyliblzma]
recipe = zc.recipe.egg:custom
egg = pyliblzma
include-dirs = ${liblzma:location}/include
library-dirs = ${liblzma:location}/lib
rpath = ${liblzma:location}/lib
[llfuse]
recipe = zc.recipe.egg:custom
egg = llfuse
# NOTE: requires apt-get install libfuse-dev pkg-config libattr1-dev
# fuse has to go into the kernal so we can't compile it into the buildout
[apsw]
recipe = zc.recipe.egg:custom
egg = apsw
find-links =
https://github.com/rogerbinns/apsw/releases/download/3.8.4.3-r1/apsw-3.8.4.3-r1.zip
# includes, libraries etc is ignored by apsw setup.py
environment = apsw-env
[apsw-env]
CPPFLAGS= -I${sqlite:location}/include -I${liblzma:location}/lib
LDFLAGS= -L${sqlite:location}/lib -Wl,-rpath=${sqlite:location}/lib -L${liblzma:location}/include -Wl,-rpath=${liblzma:location}/lib
PKG_CONFIG_PATH=${sqlite:location}/lib/pkgconfig:${liblzma:location}/lib/pkgconfig
[liblzma]
recipe = zc.recipe.cmmi
url = http://tukaani.org/xz/xz-5.0.5.tar.gz
[sqlite]
recipe = zc.recipe.cmmi
url = http://www.sqlite.org/2014/sqlite-autoconf-3080403.tar.gz