I am trying to run Airflow Celery worker with redis backend but cant get the connection between the worker and redis to work.
Here is how the configuration looks like:
broker_url = redis://:<my-password>#localhost:6379/0
I am getting error:
ValueError: invalid literal for int() with base 10: 'my-password-string'
Anyone know any fix for this?
That's because there're special characters in your password, you can simply do this:
e.g. your password is my#pw#
do url-encoding, then you get my%23pw%23
append another % before % to allow airflow read the config successfully
then the final password is my%%23pw%%23, so e.g. the redis broker url in airflow.cfg should be like this:
broker_url = redis://:my%%23pw%%23#redis-host:6379/0
So i solved it now.
My password was generated from random characters piped through base64 and it was causing problems.
I changed my password to bit shorter and simplier one and airflow worker now runs normally.
Related
I am using Apache VFS to upload a file to an SFTP server, if the file is newer than the file on the server or doesn't exist there yet. The server connection uses SSH Keys for Authentication.
I am using the following java code (plus error handling etc.) to connect to the server and check the file modification date-time:
DefaultFileSystemManager manager = new DefaultFileSystemManager();
manager.addProvider("sftp", new SftpFileProvider());
manager.init();
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(server.sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts, identityInfo);
remoteFileObject = manager.resolveFile(new URI("sftp",server.UserName,server.HostName,server.Port,remoteFilePath,null,null).toString(), createDefaultOptions(server.Key));
FileContent content = remoteFileObject.getContent();
return content.getLastModifiedTime();
The SSH key is in the format -----BEGIN RSA PRIVATE KEY----- etc.; as exported by puttyGen under Conversions -> Export OpenSSH Key (i.e. the old format of OpenSSH key, not the new one).
I have tested this code on Windows, with a locally hosted SFTP server (i.e. also on the same Windows machine), and it works successfully.
I am now wanting to use this in a Linux environment (RHEL), connecting to an AWS Transfer SFTP server, secured using SSH keys as described.
I can connect successfully using the SFTP command from the Linux OS shell:
sftp -oIdentityFile=/path/to/test.ppk USER#xxx.xxx.xxx.xxx
But, when I try to run the java code, the code hangs on the call to manager.resolveFile.
After half an hour (I think - this might not be related), I get the following in /var/log/messages:
systemd-logind[1297]: Session 115360 logged out. Waiting for processes to exit.
systemd[1]: session-115360.scope: Succeeded.
systemd-logind[1297]: Removed session 115360.
I don't have SELinux enabled, so I don't think that's interfering in any way.
Can anyone help suggest what might be causing this?
There were a couple of things, as it turns out:
Timeout
The timeout can be set when you configure the SftpFileSystemConfigBuilder, by using the .setSessionTimeout(FileSystemOptions, Duration) method call. This reduces the timeout which, if nothing else, makes the issue easier to debug.
The Session comments in the messages log were not related to the issue. Instead, the issue happened because the exec channel is disabled on the SFTP server, but VFS is trying to use it. At a simple level, this can be disabled using setDisableDetectExecChannel on the SftpFileSystemConfigBuilder object - but you should know the implications of this before doing so.
The server error is:
but when I use pika to connect to the rabbitmq server with the same parameter such as same usernameļ¼ password, virtual host, the pika can connect to the server.
the Error code is blow:
transport, protocol = await aioamqp.connect(host='localhost',
virtualhost='/', port=5672, ssl=False, insist=True,
login_method='AMQPLAIN') # use default parameters
How can I solve it?
I have fixed the error, and the solution is below:
1\ the master branch code did not correspond to aioamqp pypy code.
2\ so I git clone the master branch code to my project and make some test for it, then id did well for me.
I have setup Spark SQL on Jypterhub using Apache Toree SQL kernel. I wrote a Python function to update Spark configuration options in the kernel.json file for my team to change configuration based on their queries and cluster configuration. But I have to shutdown the running notebook and re-open or restart the kernel after running Python function. In this way, I'm forcing the Toree kernel to read the JSON file to pick up the new configuration.
I thought of implementing this shutdown and restart of kernel in a programmatic way. I got to know about the Jupyterhub REST API documentation and am able implement it by invoking related API's. But the problem is, the single user server API port is set randomly by the Spawner object of Jupyterhub and it keeps changing every time I spin up a cluster. I want this to be fixed before launching the Jupyterhub service.
Here is a solution I tried based on Jupyterhub docs:
sudo echo "c.Spawner.port = 35289
c.Spawner.ip = '127.0.0.1'" >> /etc/jupyterhub/jupyterhub_config.py
But this did not work as the port was again set by the Spawner randomly. I think there is a way to fix this. Any help on this would be greatly appreciated. Thanks
I am trying to send my logs from filebeat into redis and then from redis into logstash and elasticsearch.
I am getting the error as
( Failed to RPUSH to redis list with ERR wrong number of arguments for 'rpush' command
2018-05-22T11:15:43.063+0530 ERROR pipeline/output.go:92
Failed to publish events: ERR wrong number of arguments for 'rpush' command).
What configuration is to be made in my filebeat.yml so that the output is redis
Also what configuartion is to be made in my logstash.yml when my input is redis.
Also I am getting error as Redis Connection problem when running logstash conf file
Please provide solution for this.
Thank you.
I'm new to Fabric, so this might have a simple answer I've missed due to bad search terminology.
I'm trying to start a new ubuntu EC2 instance in AWS, then connect to it with Fabric and have it execute several commands. However, it seems there is a problem with Fabric's SSH connection, maybe I'm defining some env variable wrong?
#task //starts new EC2 instance and sets env variables
def prep_deploy():
//code to start new EC2 instance, named buildhost
env.hosts=[buildhost.public_dns_name]
env.user = "ubuntu"
env.key_filename = ".../keypair.pem"
env.port = 22
#task
def deploy():
run("echo $HOME") //code fails here
....
I run fab prep_deploy deploy, since I read you need a new task for the new env variables to take effect.
I get
Fatal error: Timed out trying to connect to ...amazonaws.com (tried 1 time)
Underlying exception: timed out
The security groups for the instance are open to SSH: I can connect through Putty. In fact, if I empty the `env.host_string' variable at the start of deploy(), when it prompts me to manually input a host, I can write in "ubuntu#...amazonaws.com:22", with the host name exactly as seen from output at the task start, and it will connect to the instance. But I can't figure how to manipulate the environment variables so that it understands the host name.
It looks like your fabric settings are correct with the use of variables. I was able to use the code you provided to connect to my Ubuntu VM. I am wondering if you are having a connection issue due to the amazon Instance not being fully booted and ready for connections when your script runs the second task. I have run into that issue on a different VM hosts.
I added the following code to check and try the connection again. This might help you
import socket
import time
def waitforssh():
s=socket.socket()
address=env.host_string
port=22
while True:
time.sleep(5)
try:
s.connect((address,port))
return
except Exception,e:
print "failed to connec to %s:%s %(address,port)
pass
insert the function call into your deploy task
def deploy():
waitforssh()
This should test the connection. If the port does not respond, it will wait 5 seconds and try again.
That could explain why your second attempt to connect works.