I wated to know is there any Kafka Spout which supports Secure Kafka Brokers.
KafkaSpout from apache storm is not having any support for SSL Kafka.
public KafkaSpout(SpoutConfig spoutConf) {
_spoutConfig = spoutConf;
}
Below mentioned Kafka not taking in any parameters for SSL Kafka producer/consumer support.
public KafkaConfig(BrokerHosts hosts, String topic) {
this(hosts, topic, kafka.api.OffsetRequest.DefaultClientId());
}
public KafkaConfig(BrokerHosts hosts, String topic, String clientId) {
this.hosts = hosts;
this.topic = topic;
this.clientId = clientId;
}
Please let me know is there any way we can achive the secure Kafka Message Stream Processing with storm topology.
Storm's old kafka-spout uses simple consumer API for which security is not supported. If you are looking to use SSL with kafka consumer you should be using
https://github.com/apache/storm/tree/master/external/storm-kafka-client
Here are the details
https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
To enable SSL its similar to what you would do regular kafka-consumer, you can follow the details in here
http://kafka.apache.org/documentation.html#security_ssl
Related
Would it be possible to create a server that publishes a client's request to Kafka's topic, and the Consumer that is subscribed to the topic will respond directly to the client?
client -> server -> kafka -> consumer -> client
I guess you are looking for an HTTP bridge, able to bridge the Apache Kafka protocol over HTTP.
I would suggest to take a look at the Strimzi HTTP - Kafka bridge here: https://github.com/strimzi/strimzi-kafka-bridge
It's open source and Apache 2.0 licensed. It's not just when you have Kafka running on Kubernetes via the Strimzi project but even if you have a bare metal Kafka installation. Details on the official documentation: https://strimzi.io/docs/bridge/latest/
Another one is the Confluent REST proxy here: https://github.com/confluentinc/kafka-rest
Still open source but with a custom Confluent Community License. Official documentation https://docs.confluent.io/platform/current/kafka-rest/index.html
In one of our spring boot apps used in Springcloud dataflow streams, we are currently using HapiContext to construct a new HL7 client and establish a connection out of it to send HL7 messages to a TCP host and port.
#Autowired
HapiContext context;
Connection connection = context.newClient(host, Integer.parseInt(port), false);
// The initiator which will be used to transmit our message
Initiator initiator = connection.getInitiator();
Message response = initiator.sendAndReceive(adtMessage);
Currently we are not using SSL/TLS for this connection and call. but we now have a requirement such that the call should be changed to an SSL based one.
I have tried doing a lot of searches on the Internet, but I am not able to find any documentation on how to achieve this.
Is there anyway to get this done?
How are you creating the HapiContext?
The DefaultHapiContext seems to provide for creating a client with a tls parameter.
lookup for the ca.uhn.hl7v2.hoh.sockets.CustomCertificateTlsSocketFactory, this should have createClientSocket which will add the SSL context necessary
I'm looking a way to declare an alternate exchange upon a exchange using spring-cloud-stream. We use 3.2.2 and this fonctionnality does not exist.
It would be used in case the producer send its first messages before the receiver creates its queue. The producer must declare an alternate-exchange on the exchange and a queue associated to the exchange to retrieve messages that has not been routed to an existing queue.
There is tons of applications related to alternate-exchange that looks not address by the spring cloud stream lib right now. https://www.rabbitmq.com/ae.html
Thanks
I think this has nothing to do with Spring Cloud Stream.
According that RabbitMQ docs, we just can do on the client side:
Map<String, Object> args = new HashMap<String, Object>();
args.put("alternate-exchange", "my-ae");
channel.exchangeDeclare("my-direct", "direct", false, false, args);
channel.exchangeDeclare("my-ae", "fanout");
channel.queueDeclare("routed");
channel.queueBind("routed", "my-direct", "key1");
channel.queueDeclare("unrouted");
channel.queueBind("unrouted", "my-ae", "");
So, technically you just declare respective beans according Spring AMQP API: https://docs.spring.io/spring-amqp/docs/current/reference/html/#broker-configuration
Or as that docs points out: use policies on the broker to modify an exchange for this alternate-exchange feature.
If you still see a value in some high-level API for this, feel free to raise a GH issue against Spring AMQP project: https://github.com/spring-projects/spring-amqp/issues.
Or in Spring Cloud Stream if you find existing Spring AMQP capabilities as enough, but still think that Spring Cloud Stream destination provisioner could be improved with some extra properties: https://github.com/spring-cloud/spring-cloud-stream
In Amazon MQ, when we connect from Active MQ client to Amazon MQ broker, we just use connection URL as ssl://<broker>:61617 but nowhere in whole AWS documentation it mentioned if this client-broker communication is secure or not and which version of TLS does client-broker connection use. Below is sample snippet of code to connect Amazon MQ as provide by AWS here.
// Create a connection factory.
final ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(wireLevelEndpoint);
// Pass the username and password.
connectionFactory.setUserName(activeMqUsername);
connectionFactory.setPassword(activeMqPassword);
// Create a pooled connection factory.
final PooledConnectionFactory pooledConnectionFactory = new PooledConnectionFactory();
pooledConnectionFactory.setConnectionFactory(connectionFactory);
pooledConnectionFactory.setMaxConnections(10);
// Establish a connection for the producer.
final Connection producerConnection = pooledConnectionFactory.createConnection();
producerConnection.start();
This article says it uses SSL but no mention, which version of TLS/SSL it uses as old SSL is no more secure.
Which TLS/SSL version does Amazon MQ broker - client connection use? Do we have any othe way to use latest version of TLS with Amazon MQ broker - client?
Amazon recommend TLS 1.2 or later.
https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/data-protection.html
As per my last question and with the help I was successfully able to look up an ejb deployed in websphere using the thin client.
Unable to lookup if SSL is enabled in Websphere 8.5
I had also made the SSL-Required for both the inbound and outbound communication.
To conclude my testing I thought of capturing the traffic using the Wireshark to ensure that all the communication is happening over SSL instead of TCP/IP but to my surprise when I seen the packets in wireshark it is still using the TCP/IP and all the data was transferring in text format.
My understanding is once the "SSL-Required" is enabled at the transport layer all the communication and handshake should be happening over SSL rather than TCP/IP. Is my understanding correct ?
Server Configuration :
Sample Client:
public static void main(String args[]) throws NamingException {
Properties ejbProps = new Properties();
ejbProps.put("org.omg.CORBA.ORBClass", "com.ibm.CORBA.iiop.ORB");
ejbProps.put(Context.INITIAL_CONTEXT_FACTORY,
"com.ibm.websphere.naming.WsnInitialContextFactory");
ejbProps.put(Context.PROVIDER_URL, "corbaloc:iiop:160.XX.XX.XX:2809");
InitialContext ffmContext = new InitialContext(ejbProps);
Object remoteObject = ffmContext
.lookup("ejb/MyAppEar-CLUSTER/MyAppEJB.jar/BatchIdTrackingBean#indi.nh.business.framework.bos.di.BatchIdTrackingBeanRemote");
BatchIdTrackingBeanRemote serviceTester = (BatchIdTrackingBeanRemote) PortableRemoteObject
.narrow(remoteObject, BatchIdTrackingBeanRemote.class);
System.out.println(serviceTester);
}
Wireshark capture :
Here , X.X.X.126 is my client location from where I am executing thin client and X.X.X.241 is my websphere app server where i am doing the lookup.
Update :
On analyzing more , i found that when I had enabled the "SSL-Required" , all the data is transferring over TCP/IP but it is encrypted , my understanding is it should be displayed as SSL instead of TCP in wireshark , isn't it ? . And when I made the transport type as only "TCP/IP" , then data transfers over GIOP and I can see the all the data in Text format.
Can anybody please confirm on this behavior ?