ActiveMQ built-in cluster discovery mechanisms are basically based on multicast (excepting LDAP here).
Openshift v3 / Kubernetes don't support well multicast as it could be quite bad or misfunctioning on a public cloud infrastructure.
Is there any existing option to enable network of activemq brokers discovery within Openshift v3 ?
I saw the project jboss-openshift/openshift-ping enabling discovery for JGroups members on Openshift. I am looking for an equivalent for ActiveMQ.
fabric8 is a project that has a number of value-adds for OS3 / kubernetes platforms
http://fabric8.io/
There is clustered ActiveMQ out of the box
http://fabric8.io/guide/fabric8MQ.html
As the project is in development, you may get best help on irc chat on #fabric8 on freenode - all the guys hang out there.
Related
I have a Kafka cluster that is running on K8S. I am using the confluent kafka image as and I have an EXTERNAL listeners that is working.
How can I add SSL encryption? Should I use an ingress? Where can I find good documentation?
Thank you
You have a manual way in this gist, which does not use the confluent image.
But for Confluent and its Helm chart (see "Confluent Operator: Getting Started with Apache Kafka and Kubernetes" from Rohit Bakhshi), you can follow:
"Encryption, authentication and external access for Confluent Kafka on Kubernetes" from Ryan Morris
Out of the box, the helm chart doesn’t support SSL configurations for encryption and authentication, or exposing the platform for access from outside the Kubernetes cluster.
To implement these requirements, there are a few modifications to the installation needed.
In summary, they are:
Generate some private keys/certificates for brokers and clients
Create Kubernetes Secrets to provide them within your cluster
Update the broker StatefulSet with your Secrets and SSL configuration
Expose each broker pod via an external service
I recommend using Strimzi kafka operator to deploy Kafka to Kubernetes. I'm using it in production for a year now.
It supports SSL, external load balancers, kafka exporter, etc
Strimzi Kafka Operator
does anyone of you have some experience using OpenWhisk together with an AMQP feed? I have a RabbitMQ infrastructure with multiple queues and events which should feed into Openwhisk triggers and execute my docker containers.
Sadly I did not found much on the Internet about OpenWhisk in connection with AMQP/RabbitMQ
Many Thanks
An example feed provider for AMQP brokers has recently been published:
https://github.com/cliffjansen/amqp-wsk-feed
This is an open-source feed provider and not currently included in the core platform. You will have to manage and run the feed provider service to use this functionality.
Another option would be to enable the webhooks plugin for RabbitMQ: https://github.com/jbrisbin/rabbitmq-webhooks
Webhooks could be configured to invoke Web Actions on the platform.
For a large online application, use k8s to run it. The scale maybe daily activity user 500,000.
The application inside k8s need messaging feature - Pub/Sub, there are these options:
Kafka
RabbitMQ
Redis
Kafka
It needs zookeeper and good to run on os depends on disk I/O. So if install it into k8s cluster, how? The performance will be worse?
And, if keep Kafka outside of the k8s cluster, connect Kafka from application inside the k8s cluster, how about that performance? They are in the different layer, won't be slow?
RabbitMQ
It's slow than Kafka, but for a daily activity user 500,000 application, is it good enough? If so, maybe it's a good choice.
Redis
It's another option. Maybe the most simple one. But from the internet I got that it will lose message sometimes. If true, that's terrible.
So, the most important thing is, use Kafka(also with zookeeper) on k8s, good or not in this use case?
Yes, running Kafka on Kubernetes is great. Check out this example: https://github.com/Yolean/kubernetes-kafka. It includes ZooKeeper and Kafka as StatefulSets.
PS. Running any of the services in your question on Kubernetes will be pleasant. You can Google the name of the service and "kubernetes" and find example manifests. Many examples here: https://github.com/kubernetes/charts.
For Kafka, you can find some suggestion here. Kubernetes 1.7+ supports local persistent volume, which may be good for Kafka deployment.
You can also take a look to the following project :
https://github.com/EnMasseProject/barnabas
It's about running Kafka on Kubernetes and OpenShift as well. It provides deploying with StatefulSets with persistent volumes or just in memory (for developing or just testing purpose). It provides deploying for Kafka Connect and Prometheus metrics as well.
Another simple configuration of Kafka/Zookeeper on Kubernetes in DigitalOcean with external access:
https://github.com/StanislavKo/k8s_digitalocean_kafka
You can connect to Kafka from outside of AWS/DO/GCE by regular binary protocol. Connection is PLAINTEXT or SASL_PLAINTEXT (user/password).
Kafka cluster is StatefulSet, so you can scale cluster easily.
So I am currently trying to deploy the Redis CF service onto Cloud Foundry which is on an Openstack IAAS and getting stuck with deploying the Redis Service Broker. Cloud Foundry is all set up (deployed using BOSH), but how to I deploy the broker? Is it pushed like an application? Do I need to change any files/manifests manually?
The documentation seemed a little light on how I would do this.
Googling "Cloud Foundry Redis service broker", the first result I find is https://github.com/pivotal-cf/cf-redis-release. This appears to document the deployment and configuration steps pretty well in its README. You will deploy the Redis service as a BOSH deployment, and this service includes the broker. You can drive that broker to register it with the Cloud Foundry service broker API as documented in that same README.
There are a couple ways to handle this.
Run open source Cloud Foundry on top of Openstack and then deploy open source Redis (or Redis Cloud, Redis Enterprise, etc.) and connect to it using User-Provided Service Instances
The easiest way is with a Service Broker (aka Pivotal Tile) as Amit describes in the previous answer. Run Pivotal Cloud Foundry on top of Openstack and then deploy a Redis Service Broker, such as Redis Enterprise Service Broker for Pivotal Cloud Foundry and connect to it as documented.
There are other Redis service providers in the The Foundry: such as a9s, IBM, Redis Labs, Pivotal and Swisscom
I want to create (2) broker clusters connected by network of brokers in JBoss Fuse 6.2; each cluster has 2 master/slave pairs.
It's a small cluster, so we don't intend to use Fabric/Zookeeper; everything will be statically configured, no auto discovery.
Questions
Is it possible to use fabric profiles to build the topology, but
avoid using fabric at runtime?
Can we use Git, or something similar, for centrally managing container config files, again, without fabric?
We tried creating profiles using fabric:mq-create, but the command is not available unless a fabric is first created, which defeats the purpose.
No fabric profiles requires using fabric. You can use git to store files, but you cannot have JBoss Fuse automatic use it such as it does with fabric. You would need to use git manually.
The AMQ broker in JBoss Fuse is just standard Apache ActiveMQ so you can configure it manually/static as a network of brokers. It just not very easy to do if you haven't done that before.
See the JBoss A-MQ documentation as that covers the broker: http://www.jboss.org/products/amq/overview/
for example at: https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_A-MQ/6.2/html/Using_Networks_of_Brokers/index.html