From both the official Kafka docs, as well as an ocean of blogs that churned up during the course of my travels, it looks like I can spin up a Kafka broker whose server.properties config file contains:
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:Bob;User:Alice
This defines two superusers (Bob + Alice) who can then produce messages to, and consume messages from, my broker's topics.
But how do I to leverage these users from the client-side? If I have a Java client that needs to send messages to my Kafka broker, how does that client "authenticate" itself as 'Bob' (or 'Alice', or any other superuser)?
And where are the super user passwords defined/used?!?
I did some digging this week and it looks like "basic auth"-style (username + password) credentials are not supported in Kafka proper.
It looks like you can set up Kerberos or a similar solution (JAAS/SASL, etc.) to create a ticket service that works with Kafka, which is what these ACLs seem to be for. I think the gist is that you would first authenticate against, say, Kerberos, at which point you would be granted a ticket/token. You would then present your username/principle along with your ticket to Kafka, and Kafka would work with Kerberos to ensure the ticket was still valid. I think that's how it works, based on some obscure/vague blogs I was able to get my hands on.
I also see evidence that Kafka currently, or plans on, having some kind of integration-layer support with LDAP, and so you might be able to hook your Kafka cluster up to AD or similar.
The best way to manage Kafka authentication, weirdly enough, seems to be the Yahoo! Kafka Manager tool, which seems to be a very popular, well-maintained project rife with recent updates and community support. This is likely what I will run with, at least for the time being. HTH.
Related
I need to handle advisories messages from the ActiveMQ.Advisory.Connection topic from one broker to another broker that is using a network connectors.
On broker A, that is linked broker B, if a client Ca is connecting, I would like the client Cb to receive the advisoiry message of A.
Is it possible ?
As I did not tried by myself, I firstly though the Advisory topics are treated as other topics but it seems not to be the case as explained in this post
The documentation on the Advisory topics does not mentions this and the above post is written by an ActiveMQ developer which seems to be a reliable source.
When connecting to Advisory topics, you have to be careful if you have authorization in place: you must give read/write access to all clients to the Advisory Topics like said in the security doc. This SO question relates exactly the issue you can have by not configuring properly the security.
On your side, I would recommend to create a network of broker with e.g. a Docker Compose and test it locally.
Further useful readings:
https://activemq.apache.org/networks-of-brokers.html#NetworksofBrokers-Networksofbrokersandadvisories
https://dzone.com/articles/how-monitor-activemq-networks
https://dzone.com/articles/active-mq-network-brokers
https://blog.christianposta.com/activemq/from-inside-the-code-activemq-network-bridges-and-demand-forward-subscriptions
https://activemq.apache.org/advisory-message.html
https://activemq.apache.org/components/cms/tutorials/handling-advisory-messages
https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/ensuring-effective-amazon-mq-performance.html#network-of-brokers-configure-correctly
Amqp brokers have persistence settings that allow guaranteed delivery - but that only works if the message actually reaches the broker. If there is a network failure and a subsequent client crash/reboot messages could be lost. Is there some way in rabbitmq or activemq or some other messaging framework for the client (producer) to persist messages to disk so that in the event the client crashes or is rebooted any unsent messages will not be lost?
I have seen people run a broker locally in order to get around this issue. That seems like an unnecessary amount of work, especially if you don't have much control over the deployment of your client.
In reality you've answered your own question pretty well. Many people looking for client side persistence turn to embedded brokers because it's actually a very good solution. Having a local broker that can store and forward gives you a lot more flexibility than just an built in persistence layer in each client, all local clients can share one broker instance which can allow you to move storage as needed in cases where you find that your stored local messages are building up due to unforeseen remote downtime.
There are of course some client implementations that do offer storage but finding one depends on your chosen broker / protocol and of course your willingness to shell out the money to buy support or licensing if that client happens to not be from say an open source implementation. The MQTT Paho client does I think have a local storage option as do some others.
I'm attempting to set up a pub/sub system. My technical director has suggested using Rabbitmq and STOMP for the project I've been assigned.
In this system I want exchanges to be created on the server side and passed to the user to subscribe to. I want a user to subscribe to an exchange and receive information as it becomes available. I do not want the client to have the ability to subscribe to arbitrary exchanges (or routing keys for topic exchanges, I'm not sure what system I want to use yet).
For example, if someone works for a company widgetInk, when they logged in to our website they would receive a connection to the widgetInk.whatever exchange and AllUsers.whatever exchange, but that's it. if the Client side attempted to subscribe to something else they'd get an error.
I've gone through all of the RabbitMQ tutorials and I've looked through their how to section. I found this basic article on access control, but having read it three times now I still don't know if rabbitmq is a good fit for my requirements.
Can Rabbitmq be configured for my requirements? What resources can I use to learn about Rabbitmq's permission system? Has anyone build or used a system like this? Would it be in my best interest to switch to ZeroMQ or ActiveMQ?
I'd like to mention another platform you might use: Autobahn (Open-Source) or WebMQ (based on Autobahn).
WebMQ provides these features (besides a lot of other out of the box):
PubSub over WebSocket (WAMP)
Authentication of client sessions (WAMP-CRA)
fine-grained configurable authorization for topics
If you want to go the Open-source/build-your-own road, here are some tutorials. If you want an integrated, commercially supported product with Web UI for configuration/administration, please get in contact.
Disclaimer: I am creator of Autobahn/WAMP and work for Tavendo.
(Ha! see what I did there?)
I have a system whereby a server pushes information from a central DB out to many client DBs (cross-domain via internet), and periodically they call services on the server. This has to withstand intermittent connections, ie queue messages.
I've created a development version using duplex MSMQ that I'm trying to apply transport security. From the reading I've done, it appears that:
MSMQ uses AD Windows Security, which is irrelevant cross-domain.
Due to the nature of duplex, each client is effectively a server as well. That means I need to pay $1200 every time I install the system with another client if I want to use SSL.
Are these facts correct? Am I really the only person who needs to secure services that are queued AND cross-domain AND duplex?
"MSMQ uses AD Windows Security, which is irrelevant cross-domain."
No, MSMQ uses Windows security which includes local accounts and, if available, domain accounts. MSMQ also uses certificates, if available.
"Due to the nature of duplex, each client is effectively a server as
well."
MSMQ doesn't use a client-server model. All MSMQ machines are effectively peers, sending messages between each other. For the $1,200 payment, are you referring to the certificate needed by the web service for sending MSMQ over HTTPS?
This is the first time I've seen anyone want to push secure messages over HTTPS to multiple destinations.
You may, in fact, be the only person in the world right now who wants to do this.
Let me embellish.
Not many companies are using MSMQ (in the grand scheme of things).
Of those that are, the vast majority are using only private queues, a small minority only use public queues.
Of those that are, only a handful are using it across the internet.
Of those that are, perhaps one is using it to exchange messages in both directions (that would be yours).
But that aside, it seems to me your main challenge will be using MSMQ as a secure transport layer over the internet. Although I have never had to do this, here are a couple of articles:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms701477(v=vs.85).aspx
http://msdn.microsoft.com/en-us/magazine/cc164041.aspx
Sorry couldn't be of more help.
I am having a hard time understanding the ZeroMQ messaging system, so before I dive in, I wanted to see if anyone knew if what I want to do is even possible.
I want to setup a pubsub server with ZeroMQ that will publish certain streams of data and to subscribe to some of those streams, a user must authenticate to see if they have access to those streams. Everything I have seen has the subscribing taking place with the zmq.SUBSCRIBE, command.
Can this be modified to authenticate? Does it support it out of the box?
No, there is no such functionality out of the box. ZeroMQ operates on lower level and it is likely that auth-features will never be in the core.
Since pubsub is implemented on top of IP-multicast, I can suggest to write an auth-server that will control a network router and forbid all multicast traffic to the client by IP/port until this client will not be authorized. You're free to choose auth method in this case, of course.
If you can sacrifice ZeroMQ’s stability and performance to the development cost, just take ActiveMQ. It has authentication features.