I am trying to start celery worker and celery beat on startup. celery worker to start with.
using ubuntu 20.04, redis, celery, python 3.8.10, django 4.0.7, virtual env
Followed the link to install redis and secure with password (using requirepass foobared) and disabled dangerous commands
https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-20-04
now to automate the celery worker start on boot following the link
Celery: Start Worker Automatically (on boot)
("worker" is the sudo user, Virutal env is at "/home/worker/Fusion/envFCorp")
my /etc/default/celeryd looks like this
#Where your Celery is present
CELERY_BIN="/home/worker/Fusion/envFCorp/bin/celery"
# App instance to use
CELERY_APP="app.celery"
#CELERY_APP="FusionCorp"
# Where to chdir at start
CELERYD_CHDIR="/home/worker/Fusion/FusionCorp/FusionCorp/"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# A user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
export SECRET_KEY="MYSECRETPASSWORD"
redis-cli ping and pong is working good
(envFCorp) worker#server:~/Fusion$ sudo systemctl enable redis
Failed to enable unit: Refusing to operate on alias name or linked unit file: redis.service
I am able to run celery worker and also able to schedule with beat manually if requirepass is off else it start giving error
celery -A FusionCorp worker --loglevel=info --pool=gevent --concurrency=10
enter code hereenter code here`celery -A FusionCorp.celery beat
Now after activating the Virtual Environment
(envFCorp) worker#server:~/Fusion$ sudo /etc/init.d/celeryd start
celery init v10.1.
Using config script: /etc/default/celeryd
(envFCorp) worker#server:~/Fusion$ sudo /etc/init.d/celeryd status
celery init v10.1.
Using config script: /etc/default/celeryd
celeryd down: no pidfiles found
Please suggest how do i automate to start my celery worker and beat on start up. Please note i see error in step 3).
also tried using supervisor to automate but without luck
Thanks
Related
I've installed Redis version 3.2.12 on one node CentOS 7 of a cluster with Cloudera Manager 6.3 and my redis never stop.
Everything is on default, I just added the password, but that has no effects because I can't restart. Option daemonize is no
My instalation was:
sudo yum -y install redis
sudo service redis start
When I type redis-cli, CLI starts normally at 127.0.0.1:6379. When I try shutdown, the console shows 'not connected', but with lsof -i :6379 I can identify that some jobs die and return with another PID.
If I try to kill the redis jobs, it always return with another PID.
service redis stop Return 'Redirecting to /bin/systemctl stop redis.service' but has no effects.
If I try service redis restart then service redis status it returns:
redis.service: main process exited, code=exited, status=1/FAILURE
Unit redis.service entered failed state.
Someone can please help me as a way to debug or understand what is happening? It's my first time with Redis.
Not sure how is this related to celery...
CentOS 7 uses systemd so I would recommend stop using the service tool and start using the systemctl. First thing you should try is systemctl status redis to check the status of the Redis service. If it shows that for whatever reason it is down, then you should either check Redis logs, or use journalctl tool to look for system logs created by Redis.
I have seen that some installations might have redis as the command-line executable while some might have redis-server. So, please try one of these commands (one will work depending on the redis package):
sudo service redis-server restart
# OR
sudo service redis restart
If you have a newer Cent OS having systemctl installed, then try one of these:
sudo systemctl restart redis-server
# OR
sudo systemctl restart redis
Unable to import redisgraph module redisgraph.so indo redis database.
I successfully compiled redisgraph.so from sources.
redisgraph.so execution rights are set for everyone.
I tried:
$ redis-cli
> shutdown ((stop redis-server))
$ redis-server --loadmodule pathto/redisgraph.so
((System replies:))
# oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
# Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=2407, just started
# Configuration loaded
* Increased maximum number of open files to 10032 (it was originally set to 1024).
# Creating Server TCP listening socket *:6379: bind: Address already in use
$ redis-cli
> module list
(empty list or set)
> module load pathto/redisgraph.so
(error) ERR Error loading the extension. Please check the server logs.
((log file says: *no permission*))
redis database works fine as key-value database.
But I fail to extend it by graph functionality.
So far I am unable to drop commands like "GRAPH.QUERY" (redis replies: "unknown command").
I have no idea why redis-server seems to ignore the import command or redis-cli complains about permission rights.
The error indicates that you already have a running process bound to the same port (probably another redis-server).
Also, you'd be better off using redisgraph with the latest Redis version (i.e. v5).
It's better to have redis managed by systemd and you could configure it as follow:
Inside
update the supervised directive in /etc/redis/redis.conf to use systemd by setting supervised systemd
Creating a redis systemd file /etc/systemd/system/redis.service and set unit, service and install directive:
[Unit]
Description=Redis In-Memory Data Store
After=network.target
[Service]
User=redis
Group=redis
ExecStart=/usr/local/bin/redis-server /etc/redis/redis.conf
ExecStop=/usr/local/bin/redis-cli shutdown
Restart=always
[Install]
WantedBy=multi-user.target
Then start redis
sudo systemctl start redis
sudo systemctl status redis
If you want redis to automatically restart when your server starts then:
Assuming all of these tests worked and that you would like to start Redis automatically when your server boots, enable the systemd service:
sudo systemctl enable redis
I have setup node-red in Raspbian and I want to store the logs coming from node-red client in some storage place like .log file.
Two ways this can be done.
Write a log function for the standard logging module. https://nodered.org/docs/user-guide/logging
Use a third part node-red module to take care of logging to file. Something like - https://flows.nodered.org/node/node-red-contrib-advance-logger
The default install for Node-RED on raspbian will set it up as a service and the logs will be sent to syslog already.
The logs can be accesses withe the node-red-log tool or by using the journalctrl command
The hack to store node-red logs in linux distribution systems just follow the below steps:-
Create a custom node-red service in /etc/systemd/system/
Command to make a .service file nano /etc/systemd/system/node-red-custom.service
[Unit]
Description=Node-RED is a tool for wiring together hardware devices, APIs and online services in new and interesting ways.
After=syslog.target network.target
Documentation=http://nodered.org/
[Service]
#Full Path to Node.js
ExecStart= /usr/bin/node-red
WorkingDirectory=/root/node-red/
# User/Group that launches node-RED (it's advised to create a new user for Node-RED)
# You can do : sudo useradd node-red
# then change the User=root by User=node-red
User=root
Group=root
Nice=10
#SyslogIdentifier=Node-RED
SyslogIdentifier=node-red-custom
StandardOutput=syslog
StandardError=syslog
# Make Node-RED restart if it fails
Restart=on-failure
# Node-RED need a SIGINT to be notified to stop
KillSignal=SIGINT
[Install]
WantedBy=multi-user.target
2.Make configuration file which targets where do you want to store the logs
nano /etc/rsyslog.d/node-red-custom.conf
if $programname == 'node-red-custom' then /var/log/node-red-logs.log
& stop
after creating these two file please run below commands
sudo systemctl restart rsyslog
sudo systemctl enable node-red-custom.service
sudo systemctl start node-red-custom.service
Now your custom node-red service start storing logs in /var/log/node-red-logs.log
Note:- You must kill the running node-red service before enable the custom node-red service as mentioned above.
I have installed it in RHEL 7 and configured it a bit.
It is up and running as a root.
I am trying to run Redis Service as non-root user.
Any pointers would be appreciated.
If the user and group “redis” has not been created,please create it.
useradd redis
Then change the owner of the file named "redis-server" and "redis-cli"(Actually,I advice chang all the files about redis but I do not know the path you installed).
chown redis. "your path"
create the script like this
vim /usr/lib/systemd/system/redis.service
Write the contents
[Unit]
Description=Redis In-Memory Data Store
After=network.target
[Service]
User=redis
Group=redis
Type=forking
ExecStart="the absolute path of redis-server" "ths absolute path of redis.conf"
ExecStop="the absolute path of redis-cli" shutdown
[Install]
WantedBy=multi-user.target
And then you can use the following codes
systemctl status redis
systemctl start redis //start the service
sysyemctl stop redis //stop the service
systemctl enable redia //start the service when system boot
I also paste the config in my machine and it works well for me
Wish this helps!
For those who use docker, you can build your own redis image with non-root user as the following:
FROM redis:6.0.10-alpine
# Create the home directory for the new non-root user.
RUN mkdir -p /home/nonroot
# Create an non-root user so our program doesn't run as root.
RUN adduser -S -h /home/nonroot nonroot
VOLUME /home/nonroot/tmp
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
CMD redis-cli ping
USER nonroot
EXPOSE 6379
Probably also add the working directory to the service since redis does not seem to change to that on its own (at least on my configuration):
WorkingDirectory=/var/lib/redis
I want to run multiple instance of Redis on Centos 7.
Can anyone point me to proper link or post steps here.
I googled for the information but I didn't find any relevant information.
You can run multiple instances of Redis using different ports on a single machine. If this what concerns you then you can follow the below steps.
By installing the first Redis instance, it listens on localhost:6379 by default.
For Second Instance create a new working directory
The default Redis instance uses /var/lib/redis as its working directory, dumped memory content is saved under this directory with name dump.rdb if you did not change it. To avoid runtime conflicts, we need to create a new working directory.
mkdir -p /var/lib/redis2/
chown redis /var/lib/redis2/
chgrp redis /var/lib/redis2/
Generate configurations
Create a new configuration file by copying /etc/redis/redis.conf
cp /etc/redis/redis.conf /etc/redis/redis2.conf
chown redis /etc/redis/redis2.conf
Edit following settings to avoid conflicts
logfile "/var/log/redis/redis2.log"
dir "/var/lib/redis2"
pidfile "/var/run/redis/redis2.pid"
port 6380
Create service file
cp /usr/lib/systemd/system/redis.service /usr/lib/systemd/system/redis2.service
Modify the settings under Service section
[Service]
ExecStart=/usr/bin/redis-server /etc/redis/redis2.conf --daemonize no
ExecStop=/usr/bin/redis-shutdown redis2
Set to start with boot
systemctl enable redis2
Start 2nd Redis
service redis2 start
Check Status
lsof -i:6379
lsof -i:6380
By Following this you can start two Redis servers. If you want more repeat the steps again.
If I set to --daemonize no, Redis will crash when data insert.
ExecStart=/usr/bin/redis-server /etc/redis2.conf --daemonize no
Should change to
ExecStart=/usr/bin/redis-server /etc/redis2.conf --supervised systemd
My Redis is 5.0.7.
FYI.