How can I use Gearman with redis for persistent queue?
I am trying to run it for centos7.
I have Gearman working.
On running
gearmand -h
I can see support for
libsqlite3,
Postgres,
libtokyocabinet and
MySQL.
I've installed hiredis using yum as per various forums although i cant figure out how to run it through terminal.
I downloaded tar file for gearman and tried running this command inside the gearman folder to no avail.
./configure --enable-FEATURE sql
To build gearmand with hiredis persistence queue you have to install libhiredis-devel on CentOS or libhiredis-dev on Debian/Ubuntu package.
Insure ./configure --enable-hiredis output contains
checking hiredis/hiredis.h usability... yes
checking hiredis/hiredis.h presence... yes
checking for hiredis/hiredis.h... yes
checking for main in -lhiredis... yes
To run gearmand with redis as queue type, run:
gearmand --verbose DEBUG --queue-type redis
Related
I used to create brokers in Artemis on both Windows, Linux and in WSL. There was never a problem.
Except on one of my machine having Windows and running WSL2.
I did everything the same when installing artemis:
sudo groupadd artemis
sudo useradd -s /bin/false -g artemis -d /opt/artemis artemis
cd /opt
sudo wget https://archive.apache.org/dist/activemq/activemq-artemis/2.12.0/apache-artemis-2.12.0-bin.tar.gz
sudo tar -xvzf apache-artemis-2.12.0-bin.tar.gz
sudo mv apache-artemis-2.12.0 artemis
sudo chown -R artemis: artemis
sudo chmod o+x /opt/artemis/bin/
sudo rm apache-artemis-2.12.0-bin.tar.gz
It installs, but when I try to create my own broker instance:
/opt/artemis/bin/artemis create --user app --password pwd --allow-anonymous test
I've got the following error message:
Cannot initialize queue:Function not implemented
I've tried it several times, even uninstalled artemis and removed the user and group and started the whole process again, but the result was always the same.
I can't figure out what the difference would be or how to fix the problem. Any help would be highly appreciated!
UPDATE 1:
There is not much log, but turning on verbose mode gives the following lines:
Executing org.apache.activemq.artemis.cli.commands.Create create --verbose --user app --password pwd --allow-anonymous test
Home::/opt/artemis, Instance::null
Cannot initialize queue:Function not implemented
As far as I can tell the message "Cannot initialize queue:Function not implemented" comes from the AIO integration layer. I recommend you try creating the instance using --nio to force the broker to use the Java-based NIO storage interface.
I am trying to set up PyBOSSA on an AWS EC2 instance running Ubuntu 18.04 LTS. I am following the official instructions and have encountered three errors so far.
sudo apt-get install -y git postgresql postgresql-all postgresql-server-dev-all libpq-dev python-psycopg2 libsasl2-dev libldap2-dev libssl-dev python-virtualenv python-dev build-essential libjpeg-dev libssl-dev libffi-dev dbus libdbus-1-dev libdbus-glib-1-dev libldap2-dev libsasl2-dev python-pip python3-pip redis-server
cd ~
git clone --recursive https://github.com/Scifabric/pybossa
cd pybossa
virtualenv -p python3 env (I'm using Python3 explicitly as my system also has Python 2.7 installed).
source env/bin/activate
pip install -U pip
pip install -r ~/pybossa/requirements.txt
At this point, I start getting error messages... I have copied the stdout and stderr into a file, which I have uploaded here.
I'm not sure if the errors there are what have caused my later errors, but I pushed on through the instructions anyway in hopes it'd work...
cp settings_local.py.tmpl settings_local.py
cp alembic.ini.template alembic.ini
redis-server contrib/sentinel.conf --sentinel
I noted that the Redis server version was 4.0.9 (the instructions say it needs to be v2.6 or greater).
The output from starting the Redis server was as follows:
30284:X 30 Mar 03:09:22.004 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
30284:X 30 Mar 03:09:22.004 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=30284, just started
30284:X 30 Mar 03:09:22.004 # Configuration loaded
...I gather that's ok...
rqscheduler --host 127.0.0.1
This command wasn't installed on my system. I tried to use apt to install it, but there was nothing there. I also tried apt install rq rqscheduler rq-scheduler - nothing found. I then Googled and found the website for rq-scheduler, and found that I could install it by running pip install rq-scheduler
That installed correctly. Nonetheless, running the command rqscheduler --host 127.0.0.1 in the terminal still failed: rqscheduler: command not found.
Knowing that it was a Python package, I wondered if maybe I needed to prepend python3 onto the start of the command: python3 rqscheduler --host 127.0.0.1. Response: python3: can't open file 'rqscheduler': [Errno 2] No such file or directory.
I also tried pip3 install rq-scheduler (which installed fine) and then running the command, but encountered the same error.
I would appreciate knowing how to get that running, but for the purpose of this test, I skipped setting up Regis and the scheduler, and continued with the PyBOSSA instructions:
sudo su postgres
createuser -d -P pybossa
(Password set)
createdb pybossa -O pybossa
exit
python3 cli.py db_create
...and then I got this error:
File "cli.py", line 162
'''SELECT id, created FROM task_run WHERE created LIKE ('\x%')''')
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 54-55: truncated \xXX escape
I instead tried python cli.py db_create, just in case it'd work, and got a different error:
python cli.py db_create
ValueError: invalid \x escape
So I'm seeing three separate issues:
Installing the PyBOSSA-required Python packages.
The issue with the rqscheduler command.
The error when starting the PyBOSSA server.
What do these errors mean?
1 ) For the installation, try this:
virtualenv env
source env/bin/activate
sudo apt install python3-pip
pip3 install -r requirements.txt
Which ended with no error.
2) Try :
pip install rq-scheduler==0.9.1
or
pip3 install rq-scheduler==0.9.1
3) The \ char need to be escaped (like \\) in python.
So you may alter the cli.py line 162 (using text editor) from:
'''SELECT id, created FROM task_run WHERE created LIKE ('\x%')''')
To:
'''SELECT id, created FROM task_run WHERE created LIKE ('\\x%')''')
But it will be better to be fixed by dev on github ...
CONCLUSION
According to official documentation,
PYBOSSA for python 3 We’ve finally migrated PYBOSSA to python 3. We’re
not going to merge into master until we test it in production a bit
more, so please, help us by testing it. All you have to do is
basically, check out the python3 branch (migrate-python3) and run it.
Then, any bug, issue you find, you just report it and we will be happy
to help you.
The PYBOSSA python3 version is freshly migrated so finaly is not very stable ... I expect that it will be better to use the PYBOSSA python2.7 branch and follow exactly the documentation.
And according to official github account they try to make money with support (?...)
Get professional support You can hire us to help you with your PYBOSSA
project or server (specially for python 2.7). Go to our website, and
contact us.
The issue has now been fixed for the master branch (https://github.com/Scifabric/pybossa/pull/1986). You can fetch the new code and use it.
I have installed redis cluster 3.0.0. But Want to upgrade it to 3.0.7. Can somebody tell me the steps to do it?
I don't want to loose any data. And don't want any downtime either.
Steps I did when upgrading from 2.9.101 to 3.0 release. I hope it will do for upgrading to 3.0.7 too.
Compile 3.0.7 from the source and start several instances with cluster enabled.
Let the 3.0.7 instances replicate the 3.0.0 instances as slave
Connect to each 3.0.7 instance and do a manual failover, then the 3.0.0 masters would become slaves after several seconds.
Wait for your application to connect to the new masters; also check the configuration files, and modify the entries to the new masters on your need
Remove those slaves
UPDATE : Docker approach
As it's probably unable to replacing the binary executable while the process is still alive, you could do it by run some Redis in docker.
First you should install docker on your machine and pull the Redis image, or pull a basic OS image and manually build Redis in it, whatever
Based on this image, you are supposed to
copy your current redis.conf into it
make sure the dir exists in the image (cluster-config-file could be the same for all the containers as they are saved individually in their own fs)
make sure the directory for logfile exists and is not the same as dir (we will later map this directory to the host)
leave port logfile anything you like, as they are specified when a container is started
commit the image as redis-3.0.7
Now launch a containerized Redis. I suppose your logfile is located in /var/log/redis/, this Redis binds :8000, and your config file in the image is /etc/redis/redis.conf
docker run -d --net=host -v /var/log/redis:/var/log/redis \
-p 8000:8000 -t redis-3.0.7 \
/usr/bin/redis-server /etc/redis/redis.conf \
--port 8000 \
--logfile /var/log/redis/redis_8000.log
Now you have a Redis 3.0.7 instance, and are ready to finish the rest steps in the previous part.
I have been trying to work with persistent job queues of gearman. When I try to use libdrizzle like-
gearmand -q libdrizzle --libdrizzle-host=127.0.0.1 --libdrizzle-user=gearman --libdrizzle-password=secret --libdrizzle-db=some_db --libdrizzle-table=gearman_queue --libdrizzle-mysql
It gives me error like-
gearmand: unknown option libdrizzle-host
Also a strange thing is that when I do man gearmand it does not have the libdrizzle options. What should I do? I want the persistent queues in mysql. I had tried using gearman_udf_mysql but it did not work too. I have posted the problem. see Where does mysql save the path for gearman mysql udf files?
Gearman must be compiled with libdrizzle/mysql support. You can check if it's been compiled in by running the command gearmand --help or on older versions man gearmand.
If libdrizzle/mysql support is available, you will see the Drizzle/Mysql options in the help. If it's not there, you will need to recompile gearmand. Here are the commands to install Gearman 1.1.12 on Ubuntu 12.04 LTS (tested on a clean Vagrant precise64 box):
vagrant#precise64:~$ sudo apt-get update
vagrant#precise64:~$ sudo apt-get install libboost-all-dev gperf libevent-dev uuid-dev libmysqld-dev
vagrant#precise64:~$ wget https://launchpad.net/gearmand/1.2/1.1.12/+download/gearmand-1.1.12.tar.gz
vagrant#precise64:~$ tar zxvf gearmand-1.1.12.tar.gz
vagrant#precise64:~$ cd gearmand-1.1.12/
vagrant#precise64:~/gearmand-1.1.12$ ./configure
At this point, look for the following lines at the end of the ./configure output, which indicates Mysql support is installed:
* Building with libdrizzle yes
* Building with libmysql yes
Then continue with the installation:
vagrant#precise64:~/gearmand-1.1.12$ make
vagrant#precise64:~/gearmand-1.1.12$ sudo make install
Gearmand is now configured with libdrizzle and will store jobs in a Mysql database.
I'm trying to get started on libvirt with VirtualBox as a virtualization solution. I installed everything and VirtualBox itself is running when using their VBoxHeadless command.
However, libvirt fails to connect to VirtualBox:
# virsh -c vbox:///session
libvir: error : could not connect to vbox:///session
error: failed to connect to the hypervisor
I could not find any hints in the libvirt documentation that point to whether I have to make any domain specific configuration before using virsh.
Does anyone have a hint? Or even better, maybe a tutorial that works through the way of using libvirt, virsh or it's APIs (my later goal) from the ground up.
If you are doing this on Ubuntu, then the problem is their libvirt package is built without VirtualBox support.
You can rebuild the package with support very easily. Something like:
apt-get source -d libvirt
sudo apt-get build-dep libvirt
dpkg-source -x libvirt*dsc
Go into the libvirt directory and edit debian/rules so that instead of --without-vbox it says --with-vbox. You can add an entry to the top of debian/changelog so the package is compiled as a different version (e.g., append ~local1 to the version).
dpkg-buildpackage -us -uc -b -rfakeroot
You'll get new .debs built in the directory above. Use dpkg -i to install the relevant ones (libvirt0, libvirt0-bin, and whatever else you want).
Double-check whether or not you have write access to /var/run/libvirt/libvirt-sock.
The socket file should have permissions similar to:
$ sudo ls -la /var/run/libvirt/libvirt-sock
srwxrwx--- 1 root libvirtd 0 2010-08-24 14:54 /var/run/libvirt/libvirt-sock
I think it could be helpful also to increase the libvirt logging capabilities by running this in your shell:
export LIBVIRT_DEBUG=1
There is Ubuntu PPA for libvirt with VirtualBox support: https://launchpad.net/~cxl/+archive/ubuntu/libvirt