I am using salt-stack to manage my production machine.
The minions run Raspbian and my and I have configured a systemd service. The services config file is located at /lib/systemd/system/my_service.service
When I run the following command:
sudo salt my_minion service.stop my_service
The following error is returned:
ERROR: Unable to run command ['/etc/init.d/my_service', 'stop'] with the context {'with_communicate': True, 'shell': False, 'env': {'LANG': 'en_GB.UTF-8', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'C'}, 'stdout': -1, 'close_fds': True, 'stdin': None, 'stderr': -2, 'cwd': '/root'}, reason: [Errno 2] No such file or directory
I understand that salt tries to use sysvinit instead of systemd.
Is there any way to tell salt to use systemd?
EDIT:
Tried adding
providers:
service: systemd
to /etc/salt/minion as suggested by Eric. Still getting the same error
EDIT 2
The issue was fixed by using Erics suggestion + upgrading salt-minion to 2015.8.8 from 2015.8.3
This is almost certainly because newer Raspbian is based off of Debian 8, and Salt's systemd execution module does not properly detect newer Raspbian as needing systemd. Can the OP please reply to this message with the output from sudo salt my_minion grains.items? Please redact any grains which you feel have personally-identifiable information, I'm mainly interested in the grains that deal with OS name and version.
EDIT: One more thing. Please confirm that /run/systemd/system exists on the Raspbian box. What I think is happening here is two modules are both claiming to be the ones to provide the service module.
https://github.com/saltstack/salt/pull/32421 should fix this, but you can work around this immediately (without waiting for a new Salt release) by adding the following to /etc/salt/minion on your Raspbian minions:
providers:
service: systemd
Related
I'm new to Elm. and I'm not good at English. So, if any ambiguous or wrong thing is there, please let me correct it.
----------- edit -----------
All my problem below is on WSL. when I'm trying on windows, all work fine. then... why elm install doesn't work on WSL? did you have any idea?
-------- problem --------
when I try to elm-test init, it doesn't work like below
$ elm-test init
Here is my plan:
Add:
elm/random 1.0.0
elm-explorations/test 1.2.2
Would you like me to update your elm.json accordingly? [Y/n]:
-- PROBLEM DOWNLOADING PACKAGE -------------------------------------------------
I was trying to download the source code for elm/random 1.0.0, so I tried to
fetch:
https://github.com/elm/random/zipball/1.0.0/
But my HTTP library is giving me the following error message:
ConnectionTimeout
Are you somewhere with a slow internet connection? Or no internet? Does the link
I am trying to fetch work in your browser? Maybe the site is down? Does your
internet connection have a firewall that blocks certain domains? It is usually
something like that!
but my Browser(Chrome) is working beautifully, and even in WSL (the environment that I run elm-test init command at) is too.
$ curl https://github.com/elm/random/zipball/1.0.0/
https://codeload.github.com/elm/random/legacy.zip/1.0.0<body>You are being redirected.</body></html>
then I also try again to redirect the URL
$ curl https://codeload.github.com/elm/random/legacy.zip
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
I think there's no Network Problem. My PC can connect with the repo, and I think it will be downloaded well.
I lastly tried just elm install, and that throws the same error too. it seems like Elm can't connect with the repo, even if My PC can.
$ elm install elm/random
Here is my plan:
Add:
elm/random 1.0.0
Would you like me to update your elm.json accordingly? [Y/n]:
-- PROBLEM DOWNLOADING PACKAGE -------------------------------------------------
I was trying to download the source code for elm/random 1.0.0, so I tried to
fetch:
https://github.com/elm/random/zipball/1.0.0/
But my HTTP library is giving me the following error message:
ConnectionTimeout
Are you somewhere with a slow internet connection? Or no internet? Does the link
I am trying to fetch work in your browser? Maybe the site is down? Does your
internet connection have a firewall that blocks certain domains? It is usually
something like that!
Please help me, what should I do?
I recently had reason to use WSL for elm development. It wasn't much fun and I'm glad to be back on Mac! What I found was that certain disk operations on WSL seemed to go very slowly and that could lead a variety of weird issues.
I was using Webstorm, which does not handle WSL well, so in the end I did everything on C: drive (rather than in /home/...) so that webstorm could run the windows version of elm-format, while my node development environment was run on the linux layer.
That's not a precise answer to your question but just to say that it can be done, but its not an ideal way to write Elm code in my experience
I had same issue and it was solved.
It was due to DNS server settings.
Create a file /etc/resolv.conf and write the following line.
nameserver 8.8.8.8
Then WSL will refer to Google Public DNS and works fine.
However, when restart WSL, the settings revert back.
Therefore, the following settings are also required.
Create a file /etc/wsl.conf and write the following line.
[network]
generateResolvConf = false
wsl --shutdown and restart WSL.
Reference link
https://github.com/microsoft/WSL/issues/4285#issuecomment-522201021
On my staging server I had my celery worker(4.3.0) up and running with celery beat as daemons via systemd with RabbitMQ as broker. Everything was alright for few weeks just to the one moment 4 days ago when there was some sort of connection error between celery and amqp through kombu. [Errno 104] Connection reset by peer after started
I wasn't paying much of an attention to the server logs, since the project is in WiP stage, however when I tried to deploy newest version of the code, I realized that something is wrong with the worker.
I googled for the issue and that's what popped out:
https://github.com/celery/celery/issues/4867
The easy solution was to downgrade celery to 4.1.1 and wait till fix in future stable releases.
I removed celery, amqp, billiard and kombu from my venv, installed celery.4.1.1, which installed above packages in appropriate versions.
Atm services of celery and celerybeat are active, celerybeat sends the tasks to the celery worker, however celery logs shows me error message (please see error code of celery after downgrade). It is weird, because I haven't changed anything in task declarations or my settings ( which may be the issue here).
The weirdest thing is that if I shut down systemd services and run them with the commend:
celery -A celery_cfg:app worker -B --loglevel=DEBUG
All current tasks are being proceed as the past ones. So the celery and celerybeat configs as they are seems to be working.
Few pointed approaches I tried:
1) Made sure to import all modules without relatives imports.
2) In the past encountered issue with missing packages in venv --> they are up to date
3) Rebooted celery/celerybeat/gunicorn/systemd/rabbitmq and server itself
4) Double checked the paths in systemd services (however maybe I am debugging this to long and I just cant see the typo or something)
5) Tried with developing version 4.4.0rc2, (celery worker won't stand up)
6) Installed apps contains all required apps
Error message after downgrade of celery version
`2019-06-16 19:35:00,092: ERROR/MainProcess] Received unregistered task of type 'apps.mailing.tasks.execute_sending_system_mail'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you're using relative imports?
Please see
http://docs.celeryq.org/en/latest/internals/protocol.html
for more information.
The full contents of the message body was:
'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b)
Traceback (most recent call last):
File "/home/user/apps/venv/loans/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 557, in on_task_received
strategy = strategies[type_]
KeyError: 'apps.mailing.tasks.execute_sending_system_mail'
Celery Service Systemd Code
Description=Celery Service
After=network.target
[Service]
Type=forking
User=<user>
Group=<user>
EnvironmentFile=/etc/default/celery
WorkingDirectory=/home/<user>/apps/loans
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
Celery Beat Service Systemd Code
Description=Celery Beat Service
After=network.target
[Service]
Type=simple
User=user
Group=user
EnvironmentFile=/etc/default/celery
WorkingDirectory=/home/user/apps/loans
ExecStart=/bin/sh -c '${CELERY_BIN} beat \
-A ${CELERY_APP} --pidfile=${CELERYBEAT_PID_FILE} \
--logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}'
[Install]
WantedBy=multi-user.target
Conf file for variables
CELERYD_NODES="w1"
CELERY_BIN="/home/user/apps/venv/loans/bin/celery"
CELERY_APP="celery_cfg:app"
CELERYD_MULTI="multi"
CELERYD_OPTS=""
CELERYD_PID_FILE="/home/user/apps/pids/celery/%n.pid"
CELERYD_LOG_FILE="/home/user/apps/logs/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
CELERYBEAT_PID_FILE="/home/user/apps/pids/celery/beat.pid"
CELERYBEAT_LOG_FILE="/home/user/apps/logs/celery/beat.log"
celery_cfg file
app = Celery('loans_apps')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.set_default()
# <====CELERY BEAT PERIODIC TASKS ====>
app.conf.beat_schedule = {
'execute_sending_system_mail': {
'task': 'apps.mailing.tasks.execute_sending_system_mail',
'schedule': crontab(minute='*/5'),
'args': (),
},
}
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
minor cut of settings containing celery cfg variables
BROKER_URL = 'amqp://localhost//',
CELERY_ENABLE_UTC = True
I know I can try setting celery and celerybeat without systemd, however I treat this as the last resort solution. I'd like to keep the conf as it was, even though I've no clue what's is wrong up there.
EDIT
By the mistake and guided by my friend I just found out, that both celery and celerybeat services seems to be working fine on user root, which is obviously not the solution but narrows down the number of possible flaws
It would be rude to leave the question unanswered, even though the answer comes from me, here it is:
If someone will ever encounter such issue, after following step pointed by me above, try to check for the permissions of directories which celery and celerybeat uses - You might have created them with root permissions, which may ends up with mentioned issue. Good luck to everyone in the future !
I use Anaconda Navigator for my machine learning experiments. Recently, I have started getting this strange SSL error no matter what I try to install using conda prompt. The error is SSL error. I am posting the error message that I get
Solving environment: failed
CondaHTTPError: HTTP 000 CONNECTION FAILED for url
https://repo.anaconda.com/pkgs/free/linux-64/repodata.json.bz2
Elapsed: -
An HTTP error occurred when trying to retrieve this URL. HTTP errors
are often intermittent, and a simple retry will get you on your way.
If your current network has https://www.anaconda.com blocked, please
file a support request with your network engineering team.
SSLError(MaxRetryError('HTTPSConnectionPool(host=\'repo.anaconda.com\',
port=443): Max retries exceeded with url:
/pkgs/free/linux-64/repodata.json.bz2 (Caused by
SSLError(SSLError("bad handshake: Error([(\'SSL routines\',
\'ssl3_get_server_certificate\', \'certificate verify
failed\')],)",),))',),)
I have an institutional proxy server; the credentials, apt.conf file etc are all OK.
Even then I am not able to install any package using conda.
Interestingly, I have installed numerous packages using conda prompt and I never faced any such error until recently. Also, I am able to follow the links (using the browser) given in the error message which means that the proxy is not blocking the site.
I was asked to use
conda config --set ssl_verify True
but even this did not help. Any help ?
I even entered the proxy server details as given in HTTP error while using conda for installation of any packages
Try copying these files from Anaconda3/Library/bin to Anaconda3/DLLs :
libcrypto-1_1-x64.dll
libssl-1_1-x64.dll
When I had an SSL error related to conda on my work laptop behind a firewall, I had to set ssl_verify to false to get it to work.
This should do the job.
conda config --set ssl_verify False
This is due to .dll error
go to location where you've install anaconda3>Library>bin. search and copy following dll files
libcrypto-1_1-x64.dll
libssl-1_1-x64.dll
and paste to anaconda3>DLLs.
then restart your pc.
issue will get resolved.
The following steps worked for me:
Use this command to see proxy_servers
conda config --show
This will remove saved proxy servers from anaconda configuration.
conda config --remove-key proxy_servers
conda clean --source-cache
Hope this helps you.
This is driving me nuts.
I'm setting up airflow in a cloud environment. I have one server running the scheduler and the webserver and one server as a celery worker, and I'm using airflow 1.8.0.
Running jobs works fine. What refuses to work is logging.
I've set up the correct path in airflow.cfg on both servers:
remote_base_log_folder = s3://my-bucket/airflow_logs/
remote_log_conn_id = s3_logging_conn
I've set up s3_logging_conn in the airflow UI, with the access key and the secret key as described here.
I checked the connection using
s3 = airflow.hooks.S3Hook('s3_logging_conn')
s3.load_string('test','test',bucket_name='my-bucket')
This works on both servers. So the connection is properly set up. Yet all I get whenever I run a task is
*** Log file isn't local.
*** Fetching here: http://*******
*** Failed to fetch log file from worker.
*** Reading remote logs...
Could not read logs from s3://my-bucket/airflow_logs/my-dag/my-task/2018-02-15T21:46:47.577537
I tried manually uploading the log following the expected conventions and the webserver still can't pick it up - so the problem is on both ends. I'm at a loss at what to do, everything I've read so far tells me this should be working. I'm close to just installing the 1.9.0 which I hear changes logging and see if I'm more lucky.
UPDATE: I made a clean install of Airflow 1.9 and followed the specific instructions here.
Webserver won't even start now with the following error:
airflow.exceptions.AirflowConfigException: section/key [core/remote_logging] not found in config
There is an explicit reference to this section in this config template.
So I tried removing it and just loading the S3 handler without checking first and I got the following error message instead:
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/lib64/python3.6/logging/config.py", line 384, in resolve:
self.importer(used)
ModuleNotFoundError: No module named
'airflow.utils.log.logging_mixin.RedirectStdHandler';
'airflow.utils.log.logging_mixin' is not a package
I get the feeling that this shouldn't be this hard.
Any help would be much appreciated, cheers
Solved:
upgraded to 1.9
ran the steps described in this comment
added
[core]
remote_logging = True
to airflow.cfg
ran
pip install --upgrade airflow[log]
Everything's working fine now.
I try to use celery. I have installed rabbit-mq by command from celery tutorial:
sudo apt-get install rabbitmq-server
And all worked well while I write my code in one file and run it to test functionality. But when I tried to add my code in Django views, and then to do concurrent requests to my views, I got this kind of exception:
File "/home/kinmanz/PycharmProjects/GitFace/myvenv/lib/python3.5/site-packages/amqp/connection.py", line 464, in drain_events
return self.blocking_read(timeout)
File "/home/kinmanz/PycharmProjects/GitFace/myvenv/lib/python3.5/site-packages/amqp/connection.py", line 468, in blocking_read
frame = self.transport.read_frame()
File "/home/kinmanz/PycharmProjects/GitFace/myvenv/lib/python3.5/site-packages/amqp/transport.py", line 251, in read_frame
'Received {0:#04x} while expecting 0xce'.format(ch))
amqp.exceptions.UnexpectedFrame: Received 0x00 while expecting 0xce
I think that problem may be in concurrency of request, and I should somehow to make queue concurrent safe.
I use Python 3.5, Celery 4.0.0, RabbitMQ 3.5.7
Actually problem in amqplib see answer below.
May be for someone who has the same problem, I will list possible solutions that I have managed to find. If you know better solution please add your answer or comment mine.
If you are using Python 2.x then see that issue https://github.com/celery/celery/issues/922
problem actually in amqplib if it change on librabbitmq all should be working, it's quite easy to do, see:
Framing Errors in Celery 3.0.1
But if you are using Python 3.x you can't solve that problem in that way, because there is no Python 3-compatible librabbitmq available, see that issue https://github.com/celery/celery/issues/2066 but in that case you can change your result backend on redis for example:
1) Install redis server:
$ sudo aptitude install redis-server
2) Change your app configuration
app = Celery('tasks', backend='redis://localhost', broker='pyamqp://')
Some useful links about installation redis: Setting up an asynchronous task queue for django using celery redis and Celery-redis quick guide
Also for Python 3 you can try to run celery worker in Python 2.7 while your app is working on Python 3, in that case don't forget install librabbitmq instead of amqplib. (This way seems to be inconvenient)