Multiple issues trying to install PyBOSSA - redis

I am trying to set up PyBOSSA on an AWS EC2 instance running Ubuntu 18.04 LTS. I am following the official instructions and have encountered three errors so far.
sudo apt-get install -y git postgresql postgresql-all postgresql-server-dev-all libpq-dev python-psycopg2 libsasl2-dev libldap2-dev libssl-dev python-virtualenv python-dev build-essential libjpeg-dev libssl-dev libffi-dev dbus libdbus-1-dev libdbus-glib-1-dev libldap2-dev libsasl2-dev python-pip python3-pip redis-server
cd ~
git clone --recursive https://github.com/Scifabric/pybossa
cd pybossa
virtualenv -p python3 env (I'm using Python3 explicitly as my system also has Python 2.7 installed).
source env/bin/activate
pip install -U pip
pip install -r ~/pybossa/requirements.txt
At this point, I start getting error messages... I have copied the stdout and stderr into a file, which I have uploaded here.
I'm not sure if the errors there are what have caused my later errors, but I pushed on through the instructions anyway in hopes it'd work...
cp settings_local.py.tmpl settings_local.py
cp alembic.ini.template alembic.ini
redis-server contrib/sentinel.conf --sentinel
I noted that the Redis server version was 4.0.9 (the instructions say it needs to be v2.6 or greater).
The output from starting the Redis server was as follows:
30284:X 30 Mar 03:09:22.004 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
30284:X 30 Mar 03:09:22.004 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=30284, just started
30284:X 30 Mar 03:09:22.004 # Configuration loaded
...I gather that's ok...
rqscheduler --host 127.0.0.1
This command wasn't installed on my system. I tried to use apt to install it, but there was nothing there. I also tried apt install rq rqscheduler rq-scheduler - nothing found. I then Googled and found the website for rq-scheduler, and found that I could install it by running pip install rq-scheduler
That installed correctly. Nonetheless, running the command rqscheduler --host 127.0.0.1 in the terminal still failed: rqscheduler: command not found.
Knowing that it was a Python package, I wondered if maybe I needed to prepend python3 onto the start of the command: python3 rqscheduler --host 127.0.0.1. Response: python3: can't open file 'rqscheduler': [Errno 2] No such file or directory.
I also tried pip3 install rq-scheduler (which installed fine) and then running the command, but encountered the same error.
I would appreciate knowing how to get that running, but for the purpose of this test, I skipped setting up Regis and the scheduler, and continued with the PyBOSSA instructions:
sudo su postgres
createuser -d -P pybossa
(Password set)
createdb pybossa -O pybossa
exit
python3 cli.py db_create
...and then I got this error:
File "cli.py", line 162
'''SELECT id, created FROM task_run WHERE created LIKE ('\x%')''')
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 54-55: truncated \xXX escape
I instead tried python cli.py db_create, just in case it'd work, and got a different error:
python cli.py db_create
ValueError: invalid \x escape
So I'm seeing three separate issues:
Installing the PyBOSSA-required Python packages.
The issue with the rqscheduler command.
The error when starting the PyBOSSA server.
What do these errors mean?

1 ) For the installation, try this:
virtualenv env
source env/bin/activate
sudo apt install python3-pip
pip3 install -r requirements.txt
Which ended with no error.
2) Try :
pip install rq-scheduler==0.9.1
or
pip3 install rq-scheduler==0.9.1
3) The \ char need to be escaped (like \\) in python.
So you may alter the cli.py line 162 (using text editor) from:
'''SELECT id, created FROM task_run WHERE created LIKE ('\x%')''')
To:
'''SELECT id, created FROM task_run WHERE created LIKE ('\\x%')''')
But it will be better to be fixed by dev on github ...
CONCLUSION
According to official documentation,
PYBOSSA for python 3 We’ve finally migrated PYBOSSA to python 3. We’re
not going to merge into master until we test it in production a bit
more, so please, help us by testing it. All you have to do is
basically, check out the python3 branch (migrate-python3) and run it.
Then, any bug, issue you find, you just report it and we will be happy
to help you.
The PYBOSSA python3 version is freshly migrated so finaly is not very stable ... I expect that it will be better to use the PYBOSSA python2.7 branch and follow exactly the documentation.
And according to official github account they try to make money with support (?...)
Get professional support You can hire us to help you with your PYBOSSA
project or server (specially for python 2.7). Go to our website, and
contact us.

The issue has now been fixed for the master branch (https://github.com/Scifabric/pybossa/pull/1986). You can fetch the new code and use it.

Related

Installing tensorboard built from source

This is about a tensorboard which is built from source, not about pip-installed one.
I could successfully build it.
$ git clone https://github.com/tensorflow/tensorboard.git
$ cd tensorboard/
$ bazel build //tensorboard
tensorflow/tensorboard$ bazel build //tensorboard
Starting local Bazel server and connecting to it...
......................................
: (log messages here)
Target //tensorboard:tensorboard up-to-date:
bazel-bin/tensorboard/tensorboard
INFO: Elapsed time: 326.553s, Critical Path: 187.92s
INFO: 619 processes: 456 linux-sandbox, 12 local, 151 worker.
INFO: Build completed successfully, 1268 total actions
Then yes I can run it as documented in tensorboard/README.md, and it works.
$ ./bazel-bin/tensorboard/tensorboard --logdir path/to/logs
The problem is, I'd like to run it as if installed via pip like this:
$ tensorboard --logdir path/to/logs
But as far as I looked for, no script provided to create .whl file so that we can locally-pip-install it, unlike tensorflow provides one like this.
$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
$ sudo pip install /tmp/tensorflow_pkg/tensorflow-1.8.0-py2-none-any.whl
So... can anybody show how to do that? Making packaging script would solve this, but it should exist somewhere as long as tensorboard is provided via pip anyway. :)
My workaround so far is not clean enough:
$ ln -s /my/build/folder/tensorboard/bazel-bin/tensorboard/tensorboard ~/bin
$ ln -s /my/build/folder/tensorboard/bazel-bin/tensorboard/tensorboard.runfiles ~/bin
I appreciate your suggestions, thanks!
Update July-21:
Thanks to W JC, I found instruction is already there in tensorboard/pip_package/BUILD.
# rm -rf /tmp/tensorboard
# bazel run //tensorboard/pip_package:build_pip_package
# pip install -U /tmp/tensorboard/*py2*.pip
Though unfortunately it shows error in my environment, and I guess it's local issue maybe because I'm using anaconda.
But basically the problem was resolved. It should basically work as long as running on supported environment.
It seems there exists an script in the /tensorboard/pip_packages try to build wheels
bazel run //tensorboard/pip_package:build_pip_package ./ did generate the wheel out, but in the folder where bazel-bin points to. In my case, it's generated at ~/.cache/bazel/_bazel_peijia/b64ba42719633ff75eec6880decefcd3/execroot/org_tensorflow_tensorboard/bazel-out/k8-fastbuild/bin/tensorboard/pip_package/build_pip_package.runfiles/org_tensorflow_tensorboard/tensorboard-2.10.0a0-py3-none-any.whl

Installation of sql relay fails

I just created a docker container and tried to install SQL Relay inside it.
I've checked the prerequisites here and followed the installation documents here.
However, at the end of make install of sqlrelay, I saw an error like this:
update-rc.d: /etc/init.d/sqlrelay: file does not exist
update-rc.d: /etc/init.d/sqlrcachemanager: file does not exist
make[1]: *** [install] Error 1
make[1]: Leaving directory `/sqlrelay-0.66.0/init'
make: *** [install-init] Error 2
What might be wrong with my installation?
Here's the docker file I used to start my installation:
FROM ubuntu:trusty
RUN apt-get update && \
apt-get install libxml2-dev libpcre3 libpcre3-dev libmysqld-dev -y
RUN apt-get install mysql-server libmysqlclient-dev -y
# sql relay prerequisites
RUN apt-get install g++ make perl php5-dev python-dev ruby-dev \
tcl-dev openjdk-7-jdk erlang-dev nodejs-dev node-gyp mono-devel \
libmariadbclient-dev libpq-dev firebird-dev libfbclient2 libsqlite3-dev \
unixodbc-dev freetds-dev mdbtools-dev -y
COPY rudiments-0.56.0.tar.gz /
COPY sqlrelay-0.66.0.tar.gz /
EXPOSE 80
Here are the outputs of ./configure, make, and make install inside sqlrelay-0.66.0 folder:
configure_log
make_log
make_install_log
If you need more information of my installation process, just let me know. I can provide it.
I think you should use ADD instead of COPY in your lines such as
COPY rudiments-0.56.0.tar.gz /
Your COPY just copies the .tar.gz, but does not unpack them
as with ADD
If the <src> parameter of ADD is an archive in a recognised compression format, it will be unpacked
This is extracted from
What is the difference between the `COPY` and `ADD` commands in a Dockerfile?
I have recently hit the same issue. The issue I found was that the init Makefile was incorrectly detecting the use of systemctl on Ubuntu Trusty and putting the scripts there. Later on the script would try to find the scripts in init.d and fail.
The solution is to edit the Makefile: sqlrelay-X.X.X/init/Makefile
Replace:
install:
if ( test -d "/lib/systemd/system" ); \
With:
install:
if ( test -d "/lib/systemd/system_x" ); \
Make a similar change to the uninstall option later in the script and it will now correctly install on Ubuntu.

Cannot upgrade Arch Linux (pacman -Syu not working)

I am running sudo pacman -Syu on my Arch Linux and I am getting the following:
cristian#localhost:~$ sudo pacman -Syu
:: Synchronizing package databases...
core is up to date
extra is up to date
community is up to date
multilib is up to date
xenlism-arch is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
error: failed to prepare transaction (could not satisfy dependencies)
:: package-query: requires pacman<4.3
What's the solution to fix this?
UPDATE
I have tried both solutions sugested by #jham. I have completely removed yaourt and package-query. At pacman -Qi pacman at 'required by' I have none. I also commented multilib and xenlism-arch from pacman.conf. When I do pacman -Syu I get the following:
:: Proceed with installation? [Y/n]
(244/244) checking keys in keyring [###################################] 100%
(244/244) checking package integrity [###################################] 100%
error: confuse: signature from "Thorsten Töpper <atsutane#freethoughts.de>" is unknown trust
:: File /var/cache/pacman/pkg/confuse-2.8-2-x86_64.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n]
error: failed to commit transaction (invalid or corrupted package)
Errors occurred, no packages were upgraded.
I just had this very same error. The problem seems to be that there are new keys in the archlinux-keyring package, and new packages (confuse) signed with those keys. Since both packages are updated in the same transaction, well the new keys cannot be used until the update is finished, but the transaction will not start until the packages are checked...
The solution would be to update the archlinux-keyring by itself:
pacman -S archlinux-keyring
And then do the rest:
pacman -Su
If that fails, you could try running through the keys manually, with:
pacman-key --populate
but usually, it is not necessary.
I just happened to have the same problem, and solved it the following way:
$ sudo pacman -Rdd package-query # Purge the conflicting package-query
$ sudo pacman -Syu # There it works
# Now reinstall package-query
$ git clone https://aur.archlinux.org/package-query.git
$ cd package-query && makepkg -si
For anyone else coming in here that didn't find the solution by rorido working, try users Bernhard Fürst's or Jham's answer of just pacman -S package-query which worked for me without issues.
Also, if you are still getting issues like this with libalpm.so.8: cannot open shared object file: No such file or directory then you have to manually reinstall package-query and yaourt.
sudo pacman-db-upgrade
yaourt -R package-query yaourt
git clone https://aur.archlinux.org/package-query.git
cd package-query
makepkg -si
cd ..
git clone https://aur.archlinux.org/yaourt.git
cd yaourt
makepkg -si
cd ..
If you still have an error. Try this:
sudo find /var/cache/pacman/pkg/ -iname "*.part" -exec rm {} \;
It removes .part files, which causes the "invalid or corrupted package" error. After you remove them, run this:
sudo pacman -Syyu
This should solve the problem, if there is no any missing key.
I am using Manjaro and after a long search, I was able to fix the issue by following these simple commands.
NOTE: You must run pacman-key --init before first using pacman; the local
keyring can then be populated with the keys of all official Manjaro Linux
packagers with pacman-key --populate archlinux manjaro.
failed to prepare transaction (invalid or corrupted database)
Here it was due to the actual (faulty) mirror servers delivering junk.
comment out those standard servers and use a quality server e.g.
ftp://ftp5.gwdg.de/pub/linux/archlinux/community/os/x86_64/
Too bad that /etc/pacman.conf is so poorly commented, as if to deliberatly be unhelpful and feckless, always being vague, never concretely helpful.
this was the last error in an seemingly endless ordain of errors from pacman. keys are unmanagable, servers are a mess, libs spell chaos. Took me more than 1 day to get through this horrific Arch experience.
The following is for ArchLinux however applies to other Linux distros too.
To correct an invalid KEY one needs to do the following:
rm -fr /etc/pacman.d/gnupg
pacman-key --init
pacman-key --populate archlinux
say the key throwing an error is in Blackarch then is also needed to :
sudo pacman-key --populate blackarch
and finally
sudo pacman -Sy archlinux-keyring
sudo pacman-key --populate archlinux
sudo pacman-key --refresh-keys

how to configure gearmand with libdrizzle on linux?

I have been trying to work with persistent job queues of gearman. When I try to use libdrizzle like-
gearmand -q libdrizzle --libdrizzle-host=127.0.0.1 --libdrizzle-user=gearman --libdrizzle-password=secret --libdrizzle-db=some_db --libdrizzle-table=gearman_queue --libdrizzle-mysql
It gives me error like-
gearmand: unknown option libdrizzle-host
Also a strange thing is that when I do man gearmand it does not have the libdrizzle options. What should I do? I want the persistent queues in mysql. I had tried using gearman_udf_mysql but it did not work too. I have posted the problem. see Where does mysql save the path for gearman mysql udf files?
Gearman must be compiled with libdrizzle/mysql support. You can check if it's been compiled in by running the command gearmand --help or on older versions man gearmand.
If libdrizzle/mysql support is available, you will see the Drizzle/Mysql options in the help. If it's not there, you will need to recompile gearmand. Here are the commands to install Gearman 1.1.12 on Ubuntu 12.04 LTS (tested on a clean Vagrant precise64 box):
vagrant#precise64:~$ sudo apt-get update
vagrant#precise64:~$ sudo apt-get install libboost-all-dev gperf libevent-dev uuid-dev libmysqld-dev
vagrant#precise64:~$ wget https://launchpad.net/gearmand/1.2/1.1.12/+download/gearmand-1.1.12.tar.gz
vagrant#precise64:~$ tar zxvf gearmand-1.1.12.tar.gz
vagrant#precise64:~$ cd gearmand-1.1.12/
vagrant#precise64:~/gearmand-1.1.12$ ./configure
At this point, look for the following lines at the end of the ./configure output, which indicates Mysql support is installed:
* Building with libdrizzle yes
* Building with libmysql yes
Then continue with the installation:
vagrant#precise64:~/gearmand-1.1.12$ make
vagrant#precise64:~/gearmand-1.1.12$ sudo make install
Gearmand is now configured with libdrizzle and will store jobs in a Mysql database.

How to install wkhtmltopdf on a linux based (shared hosting) web server

I have tried in all ways to get wkhtmltopdf installed on our web server but unfortunately it is not getting installed. I cannot access user/bin folder as stated in a tutorial on installation.
On the server in public_html folder there is a sub folder _vti_bin, I copied the file wkhtmltopdf-i386 from wkhtmltopdf-0.9.1-static-i386, but I am not able to execute it.
How to install wkhtmltopdf on (shared hosting) web server and get it working?
I've managed to successfully install wkhtmltopdf-amd64 on my shared hosting account without root access.
Here's what i did:
Downloaded the relevant static binary v0.10.0 from here: http://code.google.com/p/wkhtmltopdf/downloads/list
EDIT: The above has moved to here
via ssh on my shared host typed the following:
$ wget {relavant url to binary from link above}
$ tar -xvf {filename of above wget'd file}
you'll then have the binary on your host and will be able to run it regardless of if its in the /usr/bin/ folder or not. (or at least i was able to)
To test:
$ ./wkhtmltopdf-amd64 http://www.example.com example.pdf
Note remember that if you're in the folder in which the executable is, you should probably preface it with ./ just to be sure.
Worked for me anyway
If you have sudo access...
Ubuntu 14.04 / 15.04 / 18.04:
sudo apt-get install wkhtmltopdf
# or
sudo apt install wkhtmltopdf
Others
Look at the other answers.
If its ubuntu then go ahead with this, already tested.:--
first, installing dependencies
sudo aptitude install openssl build-essential xorg libssl-dev
for 64bits OS
wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.9.9-static-amd64.tar.bz2
tar xvjf wkhtmltopdf-0.9.9-static-amd64.tar.bz2
mv wkhtmltopdf-amd64 /usr/local/bin/wkhtmltopdf
chmod +x /usr/local/bin/wkhtmltopdf
for 32bits OS
wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.9.9-static-i386.tar.bz2
tar xvjf wkhtmltopdf-0.9.9-static-i386.tar.bz2
mv wkhtmltopdf-i386 /usr/local/bin/wkhtmltopdf
chmod +x /usr/local/bin/wkhtmltopdf
Debian 8 Jessie
This works
sudo apt-get install wkhtmltopdf
Chances are that without full access to this server (due to being a hosted account) you are going to have problems. I would go so far as to say that I think it is a fruitless endeavor--they have to lock servers down in hosted environments for good reason.
Call your hosting company and make the request to them to install it, but don't expect a good response--they typically won't install very custom items for single users unless there is a really good reason (bug fixes for example).
Lastly, depending on how familiar you are with server administration and what you are paying for server hosting now consider something like http://www.slicehost.com. $20 a month will get you a low grade web server (256 ram) and you can install anything you want. However, if you are running multiple sites or have heavy load the cost will go up as you need larger servers.
GL!
Latest update for CentOS:
sudo yum install -y libpng libjpeg openssl icu libX11 libXext libXrender xorg-x11-fonts-Type1 xorg-x11-fonts-75dpi
wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
tar -xvf wkhtmltox-0.12.4_linux-generic-amd64.tar
sudo mv wkhtmltox/bin/* /usr/local/bin/
check installation success: wkhtmltopdf -V
rm -rf wkhtmltox
rm -f wkhtmltox-0.12.4_linux-generic-amd64.tar
Place the wkhtmltopdf executable on the server and chmod it +x.
Create an executable shell script wrap.sh containing:
#!/bin/sh
export HOME="$PWD"
export LD_LIBRARY_PATH="$PWD/lib/"
exec $# 2>/dev/null
#exec $# 2>&1 # debug mode
Download needed shared objects for that architecture and place them an a folder named "lib":
lib/libfontconfig.so.1
lib/libfontconfig.so.1.3.0
lib/libfreetype.so.6
lib/libfreetype.so.6.3.18
lib/libX11.so.6 lib/libX11.so.6.2.0
lib/libXau.so.6 lib/libXau.so.6.0.0
lib/libxcb.so.1 lib/libxcb.so.1.0.0
lib/libxcb-xlib.so.0
lib/libxcb-xlib.so.0.0.0
lib/libXdmcp.so.6
lib/libXdmcp.so.6.0.0
lib/libXext.so.6 lib/libXext.so.6.4.0
(some of them are symlinks)
… and you're ready to go:
./wrap.sh ./wkhtmltopdf-amd64 --page-size A4 --disable-internal-links --disable-external-links "http://www.example.site/" out.pdf
If you experience font problems like squares for all the characters, define TrueType fonts explicitly:
#font-face {
font-family:Trebuchet MS;
font-style:normal;
font-weight:normal;
src:url("http://www.yourserver.tld/fonts/Trebuchet_MS.ttf");
format(TrueType);
}
List of stable versions wkhtmltopdf: http://wkhtmltopdf.org/downloads.html
Installing wkhtmltopdf on Debian 8.2 (jessie) x64:
sudo apt-get install xfonts-75dpi
sudo apt-get install xfonts-base
sudo wget http://download.gna.org/wkhtmltopdf/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-jessie-amd64.deb
sudo dpkg -i wkhtmltox-0.12.2.1_linux-jessie-amd64.deb
Shared hosting no ssh or shell access?
Here is how i did it;
Visit https://wkhtmltopdf.org/downloads.html and download the appropriate stable release for Linux. For my case I chose 32-bit
which is wkhtmltox-0.12.4_linux-generic-i386.tar.xz
Unzip to a folder on your local drive.
Upload the folder to public_html (or whichever location fits your need) using an FTP program just like any other file(s)
Change the binary paths in snappy.php file to point the appropriate files in the folder you just uploaded.
Bingo! there you have it. You should be able to generate PDF files.
A few things have changed since the top answers were added. They used to work out for me, but not quite anymore, so I have been hacking around for a bit and came up with the following solution for Ubuntu 16.04. For Ubuntu 14.04, see the comment at the bottom of the answer. Apologies if this doesn't work for shared hosting, but it seems like this is the goto answer for wkhtmltopdf installation instructions in general.
# Install dependencies
apt-get install libfontconfig \
zlib1g \
libfreetype6 \
libxrender1 \
libxext6 \
libx11-6
# TEMPORARY FIX! SEE: https://github.com/wkhtmltopdf/wkhtmltopdf/issues/3001
apt-get install libssl1.0.0=1.0.2g-1ubuntu4.8
apt-get install libssl-dev=1.0.2g-1ubuntu4.8
# Download, extract and move binary in place
curl -L -o wkhtmltopdf.tar.xz https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
tar -xf wkhtmltopdf.tar.xz
mv wkhtmltox/bin/wkhtmltopdf /usr/local/bin/wkhtmltopdf
chmod +x /usr/local/bin/wkhtmltopdf
Test it out:
wkhtmltopdf http://www.google.com google.pdf
You should now have a file named google.pdf in the current working directory.
This approach downloads the binary from the website, meaning that you can use the latest version instead of relying on package managers to be updated.
Note that as of today, my solution includes a temporary fix to this bug. I realize that the solution is really not great, but hopefully it can be removed soon. Be sure to check the status of the linked GitHub issue to see if the fix is still necessary when you read this answer!
For Ubuntu 14.04, you will need to downgrade to a different version of libssl. You can find the versions here. Anyways, be sure to consider the implications of downgrading libssl before doing so on any production server.
I hope this helps someone!
After trying, below command work for me
cd ~
yum install -y xorg-x11-fonts-75dpi xorg-x11-fonts-Type1 openssl git-core fontconfig
wget https://downloads.wkhtmltopdf.org/0.12/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
tar xvf wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
mv wkhtmltox/bin/wkhtmlto* /usr/bin
Version 12.5 of wkhtmltopdf only lists DEB files on their download page now. Being a mac user and not knowing much linux or what DEB files were I couldn't use the solutions posted.
This page helped me get past the knew twist of downloading a DEB file: http://www.g-loaded.eu/2008/01/28/how-to-extract-rpm-or-deb-packages/
Basically what I did was:
Downloaded from https://wkhtmltopdf.org/downloads.html
Unzipped the DEB file.
Unzipped data.tar.xz
Uploaded the binary in the unzipped 'usr' folder from step 3 (usr/local/bin/wkhtmltopdf)
Then I found out that the 'exec' function was disabled on my host. So make sure you can specifically run 'exec' if you're using PHP to run this. "Can I run the wkhtmltopdf binary" isn't specific enough. My fault.