I have a SQL 2008 R2 database that i would like to integrate with Kafka
so essentially I want to use Change data capture to capture changes in my table and put them on a Kafka Queue - this is for the front end Devs to read the data off Kafka. Has anyone done this before or have any tips on how to go about it?
Kafka Connectors will solve this problem now in particular the JDBC connector.
The JDBC connector allows you to import data from any relational
database with a JDBC driver into Kafka topics. By using JDBC, this
connector can support a wide variety of databases without requiring
custom code for each one.
Source: http://docs.confluent.io/3.0.0/connect/connect-jdbc/docs/jdbc_connector.html
See also:
Kafka Connect JDBC Connector source code on GitHub
Kafka Connect Documentation
There is no way you can do it directly from Sql server. You have to write your own producer that will pull day from Sql, and push to Kafka queue. We are currently doing the same thing via background services that pushed data to Kafka
Related
I want to stream data from on-premise to Cloud(S3) using Kafka. For which I need to intsall kafka on source machine and also on cloud. But I don't want to intsall it on cloud. I need some S3 connector through which I can connect with kafka and stream data from on-premise to cloud.
If your data is in Avro or Json format (or can be converted to those formates), you can use the S3 connector for Kafka Connect. See Confluent's docs on that
Should you want to move actual (bigger) files via Kafka, be aware that Kafka is designed for small messages and not for file transfers.
There is a kafka-connect-s3 project consisting of both sink and source connector from Spreadfast, which can handle text format. Unfortunately it is not really updated, but works nevertheless
i am trying to connect mainframe from MuleESB we have CICS regions but i am not sure how useful CICS regions to connect and do we need to connect MQ to intgrate with Mainframe. is thr any way with out connecting MQ can we connect mainframes
CICS itself is capable of being connected to using many different transports, including MQ and HTTP. Within those transports, CICS also supports many data formats, including SOAP for Web Services, JSON, binary, and so on.
It'll depend on your exact setup at your organisation as to which have been enabled, so you'll need to find out which transports are available for you to use and which data formats they're talking.
If you have IBM's WebSphere MQ on your mainframe, you will find it easy to communicate to your mainframe CICS transactions using the standard JMS component in Mule...we do this all the time using ActiveMQ, which is very familiar to any Mule developer. You will need a JMS "bridge" to connect Active MQ to WebSphere MQ - see ActiveMQ bridge connector to WebSphereMQ without using XML config.
Once you have connectivity, there are a lot of alternatives as to the various data formats and message payloads. As Ben Cox says, you have a bewildering array of choices, from raw application objects to XML, SOAP, JSON and so forth. Most of what you use at this level will probably depend on whether you're connecting to existing applications, or building new software that you can control.
If you're comfortable extending Mule using it's Connector Factory APIs, you should be able to encapsulate most of the information in a way that's Mule-friendly. We do this today in several large systems and it works quite well overall.
I am trying to create a transformation using Kafka producer and consumer in Pentaho Data Integration. Is there any example available in Pentaho for Apache Kafka Producer and Consumer? or can you please let me know how to create the transformation?
You need to download a plugin to enable your apache kafka. This DOC
shows you how to use Apache Kafka in PDI. Hope it helps :)
For example, since our server is using TIBCO EMS, would I be able to connect to it using OpenJMS or WeblogicJMS?
JMS standardizes the API, but not the wire-protocol. So all JMS implementations are based on the same API interfaces, but you will require different implementation libraries/jar-files in your class-path that match the server you're connecting to. In the TIBCO EMS case, if you're connecting to a EMS, you'll need tibjms.jar and possibly other of this jars; you cannot use something from OpenJMS etc. instead since they use different wire-protocols.
JMS is pretty much the same as JDBC in this regard.
Is there any good way to integrate OpenLDAP or ApacheDS servers (or maybe another open-source LDAP server) with JMS to propagate LDAP database modification to another service?
Basically I need to have LDAP server cluster (several instances with master to master replication) and another standalone Java application, connected via a JMS server (e.g. ActiveMQ), so that:
All changes to LDAP data structure are sent to the Java app.
The Java app. can send messages to the LDAP database via JMS server to update LDAP data
I found out that there is a way to set up JMS replication for ApacheDS (https://cwiki.apache.org/DIRxSRVx11/replication-requirements.html#ReplicationRequirements-GeneralRequirements), but I am in doubt whether it will work in case we have a cluster of several ApacheDS masters + one JMS replication node to send all modifications to the cluster.
UPDATE: The page describing JMS replication for ApacheDS turned out to be 5 ears old, so currently the only way of replication in ApacheDS, I know about, is LDAP protocol based replication.
There IDM products that will perform what you are asking about.
I know NetIQs IDM products works well with JMS.
OpenLDAP and ApacheDS have a changeLog that you could use to determine the changes made.
You could then write some code to send the changes to JMS Queue.
I can't speak for ApacheDS, but OpenLDAP already contains a full-blown replication system, with about six different ways to configure it; in other words, you can do it perfectly well, and much more efficiently, without Java and JMS.