I am interested in having a mix of encrypted and plaintext topics on the same Kafka cluster. Some topics would be accesible through plaintext protocol while others would (must) not.
I have been going through Kafka encryption with SSL documentation and several other entries through the web, and I have the impression that a per topic security configuration might not be supported out of the box with simple configuration, but would like to validate this impression with more experienced people.
All related documentation I find shows how to enable secured and plaintext protocols for the whole cluster, but I could not find any way to do so per topic. Is that currently possible? Perhaps by implementing a custom Authorizer class, although ideally I was looking for something more "out of the box", like a simple topic configuration which would specifically indicate which protocols this topic would support.
Thanks in advance.
#JPS
i think you might mix things up here. What exactly do you want to achive?
The link from your initial post refers to a documentation, how to set up ssl-in-flight-encryption and/or authentication with your kafka-cluster. But this does not alter the data, that is written into your topics. Kafka does not provide an out-of-the-box "data-at-rest-encryption"-solution. So, what is it exactly?
Related
Sorry for what may seem a simple question to some, but.
Currently use RMQ for pretty simple client/Queue/Consumer type transactions. Some use return message queues, while others are just simple 'jobs'.
Looking to distribute between 'sites' and for use-cases of RMQ-Clustering with nodes that are not co-located, ie: on WAN.
Has anyone done such a thing, or should I bite-the-bullet and move to ActiveMQ/Artemis.
Thank you for any insights.
ActiveMQ 5's network connectors are designed to support the "long link" or WAN connectivity pattern (along with others). The messaging pattern is known as 'store-and-forward'. It supports one-way push, bi-directional and pull approaches.
Network of Brokers
ref: https://activemq.apache.org/networks-of-brokers
I am new to DDS domain and need to have the below understanding.
how to publish common topics between two vendors to achieve interoperability in DDS?
The Scenario is :
Suppose there are two vendor products V1 and V2. V1 has a publisher which publishes on topic T1. V2 wants to subscribe for this topic.How will the Subscriber(V2) know that there exists a Topic T1?
I have a similar doubt on Domain level.how will a subscriber know to which domain it has to participate in?
I am using OpenDDS.
Thanks
Interoperability between vendors is possible, and regularly tested/demonstrated by the main vendors.
You will need to configure your DDS implementation to use RTPS ( I think RTPS2 currently), rather than any proprietary transport that vendors may use. This might be enabled by default.
In terms of which domain to participate in, you programmatically create a domain participant in a particular domain (which domain it connects to might be controlled by a config file) and all further entities (publishers, subscribers, etc) that you create then belong to that domain participant and therefore operate in that domain
To build on #rcs's answer a bit... the actual amount of work you have to do can depend on DDS implementations (OpenDDS, RTI, Prismtech...) because they'll have different defaults. If you use the same on both ends, then your configuration becomes a lot simpler since defaults should line up for things like domain and RTPS.
You will need to make sure the following match:
Domain ID
Domain Partion
Transport (I recommend RTPS, FWIW version difference between 2.1 and 2.2 hasn't mattered in my experience)
TCP or UDP
Discovery port and data port - this will matter more or less depending which implementations you use and whether or not you're using the same one on both ends of the connection, if use using the same, they'll have the same defaults.
Make sure the topic one publishes matches the topic the other subscribes to, this will apply to the Topic and the Type (see more here)
Serialization of the data
Discovery (unicast vs. multicast, make sure
whatever setup you choose is valid, ex: both devices are in the same
multicast group)
QoS settings will need to line up, though I think defaults will likely work (read more here)
Get the Shapes demo working between the machines you're working on first, this does some basic sanity checking to know that it is possible with the given configuration and network setup. Every vendor/implementation that I've seen has a shapes demo to run, for example, here is RTI's.
That's all I can think of right now, hope that helps. I have found DDS documentation to be really good, especially if you know when you can (and when you can't) use any vendor's documentation's answer for your implementation (ex: answer found on RTI's doc or forum and whether or not it works for your OpenDDS application). Often the solutions are similar, but you'll find RTI supports the most and RTI + Prismtech have some of the best documentation.
The DDS RTPS protocol exchanges discovery information so that different applications participating in the same domain (!) know who is out there, and what they are offering/requesting. You need to make sure that the two applications are using the same domain ID (specified on the domain participant). Also, as some implementations allow for different transport options, make sure to use RTPS (sometimes called DDSI) networking.
The RTPS specification contains a mapping from domain ID to port numbers, so if applications from different vendors use the same ID it should just work. Implementations might however override portnumbers with configuration.
To maximize the chance that the applications communicate properly, ensure they use the same IDL datamodel. Vendors have different approaches to type evolution / mapping types that don't exactly match, and not all of them implement the XTypes specification (yet).
Also, as some implementations are stricter than others, ensure that you stay within bounds of the specification. This means that a topic name should only contain alphanumerical characters (I sometimes see ':' to indicate scoping, that is not allowed).
Things that will definitely not work between vendors is TRANSIENT/PERSISTENT durability or communication over TCP, as both have not been standardized yet. TRANSIENT_LOCAL should work. The difference between TRANSIENT_LOCAL and TRANSIENT is that with TRANSIENT_LOCAL, data is no longer aligned after a publisher (writer) leaves the system, whereas with TRANSIENT that data will still be available.
Also note that for API-level interoperability between vendors, your best chance is to use the new isocpp API, since that one has been implemented pretty consistently across the vendor implementations I've seen.
Hope that helps!
When you read about WCF Message Security and compare it to Transport security, one of the drawbacks that they always mention is that transport security is point-to-point and can't secure a message routed through intermediaries.
What is an example of these intermediaries. When would you use one?
All my experience with services is with point-to-point communication so I'm trying to build a context for when you might encounter a SOAP intermediary or router or proxy.
There are other questions on SO that beat around what I'm getting at but don't directly answer my question. For example, in this question:
Does SSL provide point-to-point security?
the answer says:
intermediate system', I think that quote means a system that must
access the message in the middle (intentionally or not)... not just a
router, but something actually decrypting, viewing and/or modifying
the message.
My question is: What would be an example of a system that need to view/decrypt/modify the message and why would it need to do that?
I just found a partial answer in this document from MS:
http://msdn.microsoft.com/en-us/library/ff647097.aspx
Part of it says:
Message layer security that uses X.509 certificates is flexible enough to provide point-to-point or end-to-end security. This allows messages to be persisted in a secure state for short periods for queue-based processing or for longer periods in an archived state.
So an example would be if a message were queued or stored off to disk. It would stay encrypted through the time it was processed and deleted from disk, presumably.
Edit: Here is a really good article on building a WCF Service Router that explains some of the reasons you might want to use one: http://msdn.microsoft.com/en-us/magazine/cc500646.aspx
See also: http://msdn.microsoft.com/en-us/library/ms731081.aspx
I found this on ActiveMQ tutorial: http://activemq.apache.org/how-can-i-get-a-list-of-the-topics-and-queues-in-a-broker.html
But I really cannot figure out how to get the topics. Can you give me an example, I mean, programmatically?
The answer depends a bit on what you are looking to do here. ActiveMQ has extensive JMX APIs for monitoring an managing the broker, there's a ton of articles out there on using them to list queues and topics, a quick google search found this one. You can also take advantage of ActiveMQ's Advisory Topics if you don't want to use JMX.
I'd like to create a web service that an application server can contact to add itself to a list of servers implementing the application. Clients could then contact the service to get a list of servers. Something similar to how minecraft's heartbeats work for adding your server to the main server list.
I could implement it myself pretty easily, but I'm hoping someone has already created something like this.
Advanced features would be useful. Things like:
Allowing a client to perform queries on application-specific properties like the number of users currently connected to the server
Distributing the server list across more than one machine
Timing out a server's entry in the list if it hasn't sent a heartbeat within some amount of time
Does anyone know of a service like this? I know there are open protocols and servers for doing local-LAN service discovery, but this would be a WAN service.
The protocols I could find that had any relevance to your intended application are these:
XRDS (eXtensible Resource Descriptor Sequence).
XMPP Service Discovery protocol.
The XRDS documentation is obtuse, but you may be able to push service descriptions in XML format. The service type specification might be generic, but I get a headache from trying to decipher committee-speak.
The XMPP Service Discovery protocol (part of the protocol Formerly Known As Jabber) also looked promising, but it seems that even though you could push your service description, they expect it to be one of the services mentioned on this list. Extending it would make it nonstandard.
Finally, I found something called seap (SErvice Announcement Protocol). It's old, it's rickety, the source may be propriety, it's written in C and Perl, it's a kludge, but it seems to do what you want, kind-of.
It seems like pushing a service announcement pulse is such an application-specific and trivial problem, that almost nobody has considered solving the general case.
My advice? Read the protocols and sources mentioned above for inspiration (I'd start with seap), and then write, implement, and publish a generic (probably xml-based) protocol yourself. All the existing ones seem to be either application-specific, incomprehensible, or a kludge.
Basically, you can write it yourself though I am not aware if anyone has one for public (I wrote one over 10 yrs ago, but for a company).
database (TableCols: auto-counter, svr_name, svr_ip, check_in_time, any-other-data)
code to receive heartbeat (http://<you-app.com>?svr_name=XYZ&svr_ip=P.Q.R.S)
code to list out servers within certain check_in_time
code to do some housecleaning once a while (eg: purge old records)
To send a heartbeat out, you only need to send a http:// call, on Linux use wget* with crontab, on windows use wget.exe with task scheduler.
It is application specific, so even if you wrote one yourself, others can't use it without modifying the source code.