Currently I am trying to use Apache Airflow with Celery executor. For this I have Redis service from IBM cloud. This service has the TLS connection type which means it has the redis protocol as rediss://. Side not: I am using puckel's airflow dockerfile.
I have set redis parameters and my broker url is in the form of rediss://username:password#hostname:port/virtual_host. While I try to run for example, Flower I get these errors:
Secure redis scheme specified (rediss) with no ssl options, defaulting to insecure SSL behaviour.
Steps that I have done till now:
I have added the following lines to the entrypoint.sh:
: "${AIRFLOW__CELERY__SSL_ACTIVE:="True"}"
: "${AIRFLOW__CELERY__BROKER_USE_SSL:="{'ssl_cert_reqs': ssl.CERT_REQUIRED, 'ssl_ca_certs': '/6be25d73-0600-11ea-9bce-eaebe975ceba.crt'}"}"
AIRFLOW__CELERY__BROKER_USE_SSL="${AIRFLOW__CELERY__BROKER_USE_SSL}"
AIRFLOW__CELERY__SSL_ACTIVE="${AIRFLOW__CELERY__SSL_ACTIVE}"
export AIRFLOW__CELERY__SSL_ACTIVE
export AIRFLOW__CELERY__BROKER_USE_SSL
I have tried to use redis:// with the same broker url, but flower even did not started.
Yes it does, but you need a fairly recent version of Kombu and redis-py. We simply have a broker URL that looks like rediss://:BABADEDAuikxWx0oPZYfPE3IXJ9BVlSC#broker.example.com:6379/12?ssl_cert_reqs=CERT_OPTIONAL and it "just works".
Related
I installed and set up Airflow 2.0.1 with Celery, Rabbitmq and Postgresql9.6 on RHEL7, using the constraints https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.7.txt.
So I am not using Docker container, and in fact am building a cluster with 3 nodes.
I created DB and username for PSQL, and created user and set permission for Rabbitmq and am able to access its WebUI in 15672 port.
I am able to run airflow webserver and scheduler and access airflow WebUI with no problem.
The issue arises when I try to start airflow worker (whether from master node or worker nodes). Even though airflow.cfg is set to point out to rabbitmq, I get the error that says:
ImportError: Missing redis library (pip install redis)
Because it is trying to access redis instead of rabbitmq, but I have no idea why.
I checked airflow.cfg line by line and there is not a single line with redis, so am I missing a configuration or what?
My airflow.cfg configuration:
sql_alchemy_conn = postgresql+psycopg2://airflow_user:airflow_pw#10.200.159.59:5432/airflow
broker_url= amqp://rabbitmq_user:rabbitmq_pw#10.200.159.59:5672/airflow_virtual_host
celery_result_backend = db+postgresql://airflow_user:airflow_pw#10.200.159.59:5432/airflow
dags_are_paused_at_creation = True
load_examples = False
Why does my airflow worker try to reach redis when it is configured for rabbitmq?
I found the problem after spending many hours on such a simple, silly issue.
Airflow still tried to connect to redis, which isthe default Airflow config despite my rabbitmq configuration in airflow.cfg because I had written all of the configs under [core] section, wheras lines must be written to related parts in airflow.cfg.
I moved broker_url and result_backend to under [celery] and issue was resolved.
I want to expose Redis HA service running in kubernetes to clients running outside the cloud. For this, I'm trying to setup envoy which supports Redis.
I'm using ambassador which is a wrapper around envoy for kubernetes. Followed this doc for the initial setup. I'm new to envoy and kubernetes.
How can I configure ambassador to act as proxy for my Redis service?
I'm guessing there is someplace to specify address of the Redis service in the proxy. Finding it hard to get this info.This page refers to Redis proxy in envoy documentation but I don't follow where to make the changes.
Also, I'm interested only in the edge proxy feature, not the service proxy feature of envoy for my use case.
I'd focus on your first sentence rather than your own conclusions which follow.
You want to expose Redis to the public network.
How you ended up with Envoy is beyond me; you probably only need a Kubernetes service with type set to LoadBalancer.
This is a terrible idea because Redis is unauthenticated by default, and the connection is in clear-text, don't say you haven't been warned ;-)
As for Envoy, sure, it does support Redis, but Ambassador has nothing to do with it, and if I understand your requirement correctly, is an entire overkill which seems to mostly distract you rather than help you get the job done.
https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/
client must know all brokers using Failover Transport, right? Like that,
failover:(tcp://broker1:61616,tcp://broker2:61616,tcp://broker3:61616)
Is there optimization,so that the client does not have to know the existence of each broker ?
Put a TCP load balancer in front of the brokers. Only forward requests to the master broker. The LB can ping who's online or not by checking the "Slave" attribute of the broker via Jolokia/JMX.
A standalone approach would be to provide an URL to a comma separated list of broker URLs to try in case of failure. Can be done using the updateURIsURL option in the failover URI.
There is also some possibilities to auto-discover brokers using Multicast or by querying an LDAP directory, but that requires certain infrastructure in place. Read more about it here.
I am using Kafka Version 0.10.2.0. Is there a way to secure communication between Zookeper Client i.e ZkClient and zookeper server with SSL. I found some way to do through SASL but i want it through SSL.
Zookeeper 3.5 includes SSL support but it is still in alpha so Kafka doesn't yet support it. The highest supported version is 3.4 which only includes sasl.
Ref: https://issues.apache.org/jira/browse/ZOOKEEPER-1000
This task can still be achieved by a simple workaround mentioned in the steps below;
Install zookeeper-3.5.1-alpha (to use the .jar files. version 3.5+ can be used)
Replace default zookeeper*.jar with /zookeeper-3.5.1-alpha/zookeeper-3.5.1-alpha.jar in <kafka-installation-folder>\libs
Copy /zookeeper-3.5.1-alpha/lib/netty-3.7.0.Final.jar into <kafka-installation-folder>\libs
Relevant changes to enable SSL on Zookeeper (https://cwiki-test.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide)
ActiveMQ broker setup:
Broker is running on machine: hostA
Clients from different host can connect to my broker instance running on hostA, there can be any number of client from any host.
Is there a way to find out how many clients are connected to broker and also list which tell me how many connection from each host is there to my broker.
I want to do this without making assumption about number of hosts.
I can do this by using lsof command and some parsing over output, but I am in situation where I can not use this.
Is there any feature provided by ActiveMQ command line utility activemq-admin.
You can get to pretty much any Mbean attribute ActiveMQ exposes via the activemq-admin. There are no attributes or operations that give you a quick count of connections from specific clients. You will have to do some work on your end to get all the details you want, but all the raw data is there.
Examples:
Broker Stats:
activemq-admin query --objname type=Broker,brokerName=localhost
Connection Stats
activemq-admin query --objname type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=<transport connector name>,connectionViewType=clientId,connectionName=*
See full doc here.
NOTE: Documentation as of this writting has not be updated to take into account the Mbean changes made in AMQ. References to Object names in examples are not correct.
You can get the object name (or example sytax) from JMX (using jconsole or visual vm for example) from the MBeanInfo. Each object name wills stat something like org.apache.activemq:type. For the script, remove the "org.apache.activemq:" and you should be in business for any thing you need from JMX via the script.
I think you may also look into using Jolokia with your broker. Although not compatible with the activemq-admin script, you can reach everything you can from the activemq-admin script, but also have access to all of the operations. In the past I've heavily used the activemq-admin script for local monitoring/command line administration of the broker, but have started converting everything to hit the Jolokia service. But again, activemq-admin will give you a way to access what you are looking for here.