I am trying to start celery worker and celery beat on startup. celery worker to start with.
using ubuntu 20.04, redis, celery, python 3.8.10, django 4.0.7, virtual env
Followed the link to install redis and secure with password (using requirepass foobared) and disabled dangerous commands
https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-20-04
now to automate the celery worker start on boot following the link
Celery: Start Worker Automatically (on boot)
("worker" is the sudo user, Virutal env is at "/home/worker/Fusion/envFCorp")
my /etc/default/celeryd looks like this
#Where your Celery is present
CELERY_BIN="/home/worker/Fusion/envFCorp/bin/celery"
# App instance to use
CELERY_APP="app.celery"
#CELERY_APP="FusionCorp"
# Where to chdir at start
CELERYD_CHDIR="/home/worker/Fusion/FusionCorp/FusionCorp/"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# A user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
export SECRET_KEY="MYSECRETPASSWORD"
redis-cli ping and pong is working good
(envFCorp) worker#server:~/Fusion$ sudo systemctl enable redis
Failed to enable unit: Refusing to operate on alias name or linked unit file: redis.service
I am able to run celery worker and also able to schedule with beat manually if requirepass is off else it start giving error
celery -A FusionCorp worker --loglevel=info --pool=gevent --concurrency=10
enter code hereenter code here`celery -A FusionCorp.celery beat
Now after activating the Virtual Environment
(envFCorp) worker#server:~/Fusion$ sudo /etc/init.d/celeryd start
celery init v10.1.
Using config script: /etc/default/celeryd
(envFCorp) worker#server:~/Fusion$ sudo /etc/init.d/celeryd status
celery init v10.1.
Using config script: /etc/default/celeryd
celeryd down: no pidfiles found
Please suggest how do i automate to start my celery worker and beat on start up. Please note i see error in step 3).
also tried using supervisor to automate but without luck
Thanks
How do I use SSH Keys (created from cPanel) to connect to the server? And eventually pull a fresh copy and run composer updates and database migrations (a Symfony script)
I get permission denied errors so my ssh example.net.au ls -l /staging.example.net.au is reaching the server, I'm just unsure how to use keys made from cPanel to make an authentication.
bitbucket-pipelines.yml
# This is an example Starter pipeline configuration
# Use a skeleton to build, test and deploy using manual and parallel steps
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: atlassian/default-image:2
pipelines:
default:
- parallel:
- step:
name: 'Build and Test'
script:
- echo "Your build and test goes here..."
- step:
name: 'Lint'
script:
- echo "Your linting goes here..."
- step:
name: 'Security scan'
script:
- echo "Your security scan goes here..."
# The following deployment steps will be executed for each pipeline run. To configure your steps and conditionally deploy see https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/
- step:
name: 'Deployment to Staging'
deployment: staging
script:
- echo "Your deployment to staging script goes here..."
- echo $TESTVAR
- ssh example.net.au ls -l /staging.example.net.au
- step:
name: 'Deployment to Production'
deployment: production
trigger: 'manual'
script:
- echo "Your deployment to production script goes here..."
I think your SSL set-up may be incorrect. Please try the following to ensure both servers trust each other:
==Part 1==
Step 1. SSH into cPanel server (use PuTTY or your preferred SSH client), and run the following commands to generate a new key:
ssh-keygen
eval $(ssh-agent)
ssh-add
cat ~/.ssh/id_rsa.pub
Step 2. Copy the resulting key from the 'cat' command above, into: Bitbucket -> your repo -> Settings -> Access keys
==Part 2==
Step 3. In Bitbucket, go to your repo -> settings -> SSH keys -> Generate key
Step 4. Back on your cPanel server's SSH connection, copy the key from Step 3 above into the authorized keys file. Save when you are done:
nano ~/.ssh/authorized_keys
Right click to paste (usually)
CNRL+O to save
CNRL+X to exit
Step 5. In the same Bitbucket screen from Step 3, fetch and add host's fingerprint. You will need to enter the URL or IP address of your cPanel server here. Some cPanels servers use non-default ports. If port 22 is not the correct port, be sure to specify like so:
example.com:2200
(Port 443 is usually reserved for HTTPS and it is unlikely the correct port for an SSH connection. If in doubt, try the default 22 and common alternative 2200 ports first.)
Let me know if you have any questions and I am happy to assist you further.
Our Spinnaker is deployed on Ubuntu 18, Spinnaker version is 1.20.3. The only way we can view the logs is to run journalctl -u $microservice there are no logs on /var/log/spinnaker.
Is this normal?
Yes. The preferred way of installation for Spinnaker is in Kubernetes. A quick and easy way for you to get started and easily migrate is to backup all you config with halyard, export the pipelines as json and run Minnaker in any Ubuntu 18 Compute box
Then import your old spinnaker data and pipelines.
The Ubuntu18 debian deploy flavor that you are running could be useful to debug cloud driver issues or for development purposes.
I suggest that you perform the migration to a Kubernetes cluster.
The reason why none of the Spinnaker microservices output any logs to their log file directories in /var/log/spinnaker is because the preferred method of installation for Spinnaker is to use Kubernetes.
If the microservices were to create log files in /var/log/spinnaker, there is a good chance that the Kubernetes pods would die due to running out of storage, hence they all output their logs to STDOUT, and can be retrieved from Kubernetes by running:
kubectl -n spinnaker logs POD_NAME > my_logfile_name.log
If you prefer to run Spinnaker on a VM rather than in Kubernetes and want to enable the log files so that you can debug a specific issue instead of using journalctl, you can edit the systemd service file for the particular microservice, for example Clouddriver, and add the following line in the [Service] section:
StandardOutput=append:/var/log/spinnaker/clouddriver/clouddriver.log
Then you reload the systemctl daemon and restart the service and it will then output its logs to the specified log file instead of STDOUT, for example:
sudo systemctl daemon-reload
sudo systemctl restart clouddriver.service
I have installed the latest version of RabbitMQ on a VPS Debian Linux box. Tried to get login through guest/guest but returned with the message login failed. I did a little research and found that for security reason its prohibited to get login via guest/guest remotely.
I also have tried enabling guest uses on this version to get logged in remotely by creating a rabbitmq.config file manually (because the installation didn't create one) and placing the following entry only
[{rabbit, [{loopback_users, []}]}].
after restart the rabbitmq with the following command.
invoke-rc.d rabbitmq-server stop -- to stop
invoke-rc.d rabbitmq-server start -- to start
It still doesn't logged me in with guest/guest. I also have tried installing RabbitMQ on Windows VPS and tried to get log in via guest/guest through localhost but again i get the same message login failed.
Also provide me a source where I could try installing the old version of RabbitMQ that does support logging remotely via guest/guest.
I had the same Problem..
I installed RabbitMQ and Enabled Web Interface also but still couldn't sign in with any user i newly created, this is because you need to be administrator to access this.
Do not create any config file and mess with it..
This is what i did then,
Add a new/fresh user, say user test and password test:
rabbitmqctl add_user test test
Give administrative access to the new user:
rabbitmqctl set_user_tags test administrator
Set permission to newly created user:
rabbitmqctl set_permissions -p / test ".*" ".*" ".*"
That's it, enjoy :)
I tried on Debian the same configuration with the following steps:
Installed RabbitMQ.
Enabled the web-management plug-in (not necessary).
When I tried to login I had the same error:
So I created a rabbitmq.config file (classic configuration file) inside the /etc/rabbitmq directory with the following content (notice the final dot):
[{rabbit, [{loopback_users, []}]}].
Alternatively, one can create instead a rabbitmq.conf file (new configuration file) inside the same directory with the following content:
loopback_users = none
Then I executed the invoke-rc.d rabbitmq-server start command and both the console and the Java client were able to connect using the guest/guest credentials:
So I think you have some other problem if this procedure doesn't work. For example your RabbitMQ might be unable to read the configuration file if for some reason you have changed the RABBITMQ_CONFIG_FILE environment variable.
This is a new features since the version 3.3.0. You can only login using guest/guest on localhost. For logging from other machines or on ip you'll have to create users and assign the permissions. This can be done as follows:
rabbitmqctl add_user test test
rabbitmqctl set_user_tags test administrator
rabbitmqctl set_permissions -p / test ".*" ".*" ".*"
Adding the below line in the config file and restarting the server worked for me. Kindly try in your setup.
loopback_users.guest = false
I got this line from the example RabbitMQ config file from Github as linked here.
notice: check your PORT is 15672 ! (version > 3.3 ) if 5672 not works
First of all, check the "choosen answer above":
rabbitmqctl add_user test test
rabbitmqctl set_user_tags test administrator
rabbitmqctl set_permissions -p / test ".*" ".*" ".*"
and if still can't make connection work, check if your port is correct!
for me, this command works:
$ rabbitmqadmin -H 10.140.0.2 -P 15672 -u test -p test list vhosts
+------+----------+
| name | messages |
+------+----------+
| / | |
+------+----------+
for the completed ports , check this:
What ports does RabbitMQ use?
to verify your rabbit mq server, check this: Verify version of rabbitmq
p.s.
For me, after I created the "test" user and run set_user_tags, set_permissions , I can't connect to rabbitmq via port 5672. but I can connect via 15672.
However, port 15672 always gives me a "blank response". and my code stop working.
so about 5 minutes later, I switched to 5672, everything worked!
Very wired problem. I have no time to dig deeper. so I wrote it down here for someone meeting the same problems.
for other guys which use Ansible for RabbitMQ provisioning, what I missed for rabbitmq_user module was tags: administrator
here is my working Ansible configuration to recreate "guest" user (for development environment purpose, don't do that in production environment):
- name: Create RabbitMQ user "guest"
become: yes
rabbitmq_user:
user: guest
password: guest
vhost: /
configure_priv: .*
read_priv: .*
write_priv: .*
tags: administrator
force: yes # recreate existing user
state: present
and I also had to setup a file /etc/rabbitmq/rabbitmq.config containing the following:
[{rabbit, [{loopback_users, []}]}].
in order to be able to log using "guest"/"guest" from outside of localhost
#Create rabbitmq.conf file with
rabbitmq.conf
loopback_users = none
Dockerfile:
FROM rabbitmq:3.7-management
#Rabbitmq config
COPY rabbitmq.conf /etc/rabbitmq/rabbitmq.conf
#Install vim (edit file)
RUN ["apt-get", "update"]
RUN ["apt-get", "-y", "install", "vim"]
#Enable plugins rabbitmq
RUN rabbitmq-plugins enable --offline rabbitmq_mqtt rabbitmq_federation_management rabbitmq_stomp
Run:
$ docker build -t my-rabbitmq-image .
$ docker run -d --hostname my-rabbit --name some-rabbit -p 8080:15672 my-rabbitmq-image
Check that the rabbitmq.conf file has been copied correctly.
$ docker exec -it my_container_id /bin/bash
$ vim /etc/rabbitmq/rabbitmq.conf
I had the same problem. I tried what was suggested by Gas and ran "invoke-rc.d rabbitmq-server start" it didn't start. I tried to reboot the server and the webui worked with the guest user. Maybe after adding the rabbitmq.config file, something else also needed to started.
I used rabbitmq version 3.5.3.
One more thing to note: if you're using AWS instance then you need to open inbound port 15672. (The port for RabbitMQ versions prior to 3.0 is 55672.).
Students and I stared at this problem for an hour. Be sure you've named your files correctly. In the /etc/rabbitmq directory, there are two distinct files. There is an /etc/rabbitmq/rabbitmq.config file which you should edit to get the loopback users as described, but there is another file called rabbitmq-env.conf file. Many folks were using tab completion and just adding "ig", which isn't the right file. Double check!
sometimes you don't need the comma , which is there in the configuration file by default , if nothing else is configured below rabbit tag , while starting broker
we will get a crash
like
{loopback_users, []} , I spend many times hours forgetting this and later removing the comma , it is applicable for all other configurations including SSL
Try restart your rabbitmq and login again, for me work.
For a slightly different use, but might be useful for anyone dealing with accessing the API for monitoring purposes:
I can confirm the answer given by #Oliboy50 works well, however make sure you enable it for each vhost you want the user to be able to monitor, such as:
permissions:
- vhost: "{{item.name}}"
configure_priv: .*
write_priv: .*
read_priv: .*
state: present
tags: management
with_items: "{{user_system_users}}"
With this loop I was able to get past the "401 Unauthorized" error when using the API for any vhost.
By default, the guest user is prohibited from connecting from remote hosts; it can only connect over a loopback interface (i.e. localhost). This applies to connections regardless of the protocol. Any other users will not (by default) be restricted in this way.
It is possible to allow the guest user to connect from a remote host
by setting the loopback_users configuration to none
# DANGER ZONE!
#
# allowing remote connections for default user is highly discouraged
# as it dramatically decreases the security of the system. Delete the user
# instead and create a new one with generated secure credentials.
loopback_users = none
Or, in the classic config file format (rabbitmq.config):
%% DANGER ZONE!
%%
%% Allowing remote connections for default user is highly discouraged
%% as it dramatically decreases the security of the system. Delete the user
%% instead and create a new one with generated secure credentials.
[{rabbit, [{loopback_users, []}]}].
See at "guest" user can only connect from localhost
TIP: It is advisable to delete the guest user or at least change its password to reasonably secure generated value that won't be known to the public.
If you will check the log file under info report you will get this.
`config file(s) : /etc/rabbitmq/rabbitmq.config (not found)`.
Change the config file permission using below command then login using guest , it will work
sudo chmod 777 /etc/rabbitmq/rabbitmq.config
I have setup a basic redis image based on the following instructions: http://docs.docker.io/en/latest/examples/running_redis_service/
With my snapshot I have also edited the redis.conf file with requirepass.
My server runs fine and I am able to access it remotely using redis-cli however the authentication isn't working. I am wondering if the config file isn't being used but when I try starting the container with:
docker run -d -p 6379:6379 jwarzech/redis /usr/bin/redis-server /etc/redis/redis.conf
the container immediately crashes.
the default config of redis is set to be a daemon. You can't run a daemon within a docker container, otherwise, lxc will lose track of it and will destroy the namespace.
I just tried doing this within the container:
$>redis-server - << EOF
requirepass foobared
EOF
Now, I can connect to it and I will get a 'ERR operation not permitted'. When I connect with redis-cli -a foobared, then it works fine.