I have a Kafka cluster that is running on K8S. I am using the confluent kafka image as and I have an EXTERNAL listeners that is working.
How can I add SSL encryption? Should I use an ingress? Where can I find good documentation?
Thank you
You have a manual way in this gist, which does not use the confluent image.
But for Confluent and its Helm chart (see "Confluent Operator: Getting Started with Apache Kafka and Kubernetes" from Rohit Bakhshi), you can follow:
"Encryption, authentication and external access for Confluent Kafka on Kubernetes" from Ryan Morris
Out of the box, the helm chart doesn’t support SSL configurations for encryption and authentication, or exposing the platform for access from outside the Kubernetes cluster.
To implement these requirements, there are a few modifications to the installation needed.
In summary, they are:
Generate some private keys/certificates for brokers and clients
Create Kubernetes Secrets to provide them within your cluster
Update the broker StatefulSet with your Secrets and SSL configuration
Expose each broker pod via an external service
I recommend using Strimzi kafka operator to deploy Kafka to Kubernetes. I'm using it in production for a year now.
It supports SSL, external load balancers, kafka exporter, etc
Strimzi Kafka Operator
Related
Currently hazelcast is using cloud discovery for communication.
So if there are 4 kubernetes pods and each of them is having in-memory hazelcast. whenever hazelcast cache is updated in one of the pod, it gets updated in one of the other pod. but in case both of these pods get downscaled and get terminated, the data which is only in these 2 pods is lost. Can we have something like redis where we can provide server, port of the hazelcast cluster and it will be independent of kubernetes pod
Please check the following Blog Post ("Scale without Data Loss!" section) to read how to scale Hazelcast cluster on Kubernetes to avoid data loss.
Also, you can check the official README of hazelcast/hazelcast-kubernetes plugin. There is a section dedicated to scaling there.
Objective
I want to access the redis database in kubernetes, from a function inside ibm functions using javascript.
Question
How do I get the right URI, when redis is running on a Pod in Kubernetes?
Situation
I used this sample to setup the redis database in kubernetes This is the link to the sample in Kubernetes
I run Kuberentes inside IBM Cloud.
Findings
I was not able to find a answer to my question on the redis documentation
As far as I understand by default no password configured.
Is this assumption right?
redis://[USER]:[PASSWORD]#[CLUSTER-PUBLIC-IP]:[PORT]
Thanks for help ... I know this is maybe a to simple question, but currently I do not see the tree in the woods ;-)
As far as I understand by default no password configured.
Yes, there is no default password in that image with Redis, you are right.
If you following the instruction you mentioned, you will use a kubectl proxy, which will forward port of your Redis in cluster to your local machine by call kubectl port-forward redis-master 6379:6379.
So in that case, Redis will be available on redis://localhost:6379 on your PC.
If you want to make it available directly from ouside of the cluster, you need to create Service with NodePort, Service with LoadBalancer (if you in Cloud) or simply Service with Ingress.
Inside a cluster, you can create Service with Cluster IP (which is actually simply Service, because it always has Cluster IP) for your Redis pod and will be available on:
redis://[USER]:[PASSWORD]#[SERVICE-IP]:[PORT]
Here is a good official documentation about connecting applications with service.
For a large online application, use k8s to run it. The scale maybe daily activity user 500,000.
The application inside k8s need messaging feature - Pub/Sub, there are these options:
Kafka
RabbitMQ
Redis
Kafka
It needs zookeeper and good to run on os depends on disk I/O. So if install it into k8s cluster, how? The performance will be worse?
And, if keep Kafka outside of the k8s cluster, connect Kafka from application inside the k8s cluster, how about that performance? They are in the different layer, won't be slow?
RabbitMQ
It's slow than Kafka, but for a daily activity user 500,000 application, is it good enough? If so, maybe it's a good choice.
Redis
It's another option. Maybe the most simple one. But from the internet I got that it will lose message sometimes. If true, that's terrible.
So, the most important thing is, use Kafka(also with zookeeper) on k8s, good or not in this use case?
Yes, running Kafka on Kubernetes is great. Check out this example: https://github.com/Yolean/kubernetes-kafka. It includes ZooKeeper and Kafka as StatefulSets.
PS. Running any of the services in your question on Kubernetes will be pleasant. You can Google the name of the service and "kubernetes" and find example manifests. Many examples here: https://github.com/kubernetes/charts.
For Kafka, you can find some suggestion here. Kubernetes 1.7+ supports local persistent volume, which may be good for Kafka deployment.
You can also take a look to the following project :
https://github.com/EnMasseProject/barnabas
It's about running Kafka on Kubernetes and OpenShift as well. It provides deploying with StatefulSets with persistent volumes or just in memory (for developing or just testing purpose). It provides deploying for Kafka Connect and Prometheus metrics as well.
Another simple configuration of Kafka/Zookeeper on Kubernetes in DigitalOcean with external access:
https://github.com/StanislavKo/k8s_digitalocean_kafka
You can connect to Kafka from outside of AWS/DO/GCE by regular binary protocol. Connection is PLAINTEXT or SASL_PLAINTEXT (user/password).
Kafka cluster is StatefulSet, so you can scale cluster easily.
I am sort of new to cloud foundry. I have some queries -
Can I use REDIS as a service in Cloud Foundry , if yes , how. Do we need service broker as well for that.
Manifest file for deploying Redis on Cloud foundry in openstack Neutron.
Can I do HA of Redis service in CF.
I have been through these links as well
https://github.com/pivotal-cf/cf-redis-release
https://github.com/cloudfoundry-community/redis-boshrelease
and deployed redis with a dedicated node and broker but not sure how it will work with an app.
Yes, you can use Redis as a service in CF, and yes, you'll need to make sure that there is a service broker -- in fact, having a service broker is the definition of something being a CF Service (if you can write a service broker for it, you can use it as a service). Here's an overview of the CF Service Broker API. Once you have your Redis cluster and service broker set up, you'll need to do the following:
Register your service broker with cf create-service-broker redis-broker <username> <password> <url to service broker>.
Create a service instance: cf create-service redis <redis-plan-name> myRedis
Bind your app to the service instance: cf bind-service myApp myRedis
Building a manifest file depends on which Redis release you use. The cloudfoundry-community/redis-boshrelease has a template for generating an openstack manifest. Unfortunately, that release doesn't have a service broker so you can't use that redis as a service in CF. The pivotal-cf/cf-redis-release, on the other hand, does have a service broker. Maybe you can use the Openstack-specific properties from the cloudfoundry-community/redis-boshrelease to make an Openstack manifest for pivotal-cf/cf-redis-release?
I don't know too much about HA Redis. You'll have to get some help from Redis experts, but I do know that there's a piece of software called Sentinel that's meant to get Redis to HA. You should take a look at that and see if you can extend the release to include Sentinel.
Hope that helps!
ActiveMQ built-in cluster discovery mechanisms are basically based on multicast (excepting LDAP here).
Openshift v3 / Kubernetes don't support well multicast as it could be quite bad or misfunctioning on a public cloud infrastructure.
Is there any existing option to enable network of activemq brokers discovery within Openshift v3 ?
I saw the project jboss-openshift/openshift-ping enabling discovery for JGroups members on Openshift. I am looking for an equivalent for ActiveMQ.
fabric8 is a project that has a number of value-adds for OS3 / kubernetes platforms
http://fabric8.io/
There is clustered ActiveMQ out of the box
http://fabric8.io/guide/fabric8MQ.html
As the project is in development, you may get best help on irc chat on #fabric8 on freenode - all the guys hang out there.