I am looking for a serverless framework(free) , where i can create a kafka trigger and when triggered a kube function is to be invoked (python)
I have tried nuclio but the problem is that i have kafka version higher and they do not support higher than 2.4.
I want something like:
apiVersion: "nuclio.io/v1beta1"
kind: "NuclioFunction"
spec:
runtime: "python:3.6"
handler: NuclioKafkaHandler:consumer
minReplicas: 1
maxReplicas: 1
triggers:
myKafkaTrigger:
kind: kafka-cluster
attributes:
initialOffset: earliest
topics:
- nuclio
brokers:
- kafka-bootstrap:9092
consumerGroup: Consumer
And a kube function like:
def consumer(context, event):
context.logger.debug(event.body)
print(event.trigger.kind)
As simple as these two files and i have already existing kafka cluster so i just want to have trigger on that.
what are the possible alternatives apart from nuclio? I looked into kubeless seemed complicated. Fission does not support python.
I don't know much about Nuclio but the scenario you described looks possible with Knative.
Simplest way, you can create a Knative Service for your consumer. For the Kafka part, you can use a KafkaSource to get the events into Knative Eventing system. In your KafkaSource, you can tell it to call the Knative Service when there's an event coming from Kafka.
Above is the simplest way. If you need more advanced features, there is also support for filtering based on event types or having multiple consumers subscribed to events and more features.
Red Hat's Knative Tutorial has a section for serverless eventing with Kafka.
The exact same use case is possible with Fission which is an open source serverless framework for Kubernetes.
You can create a Message Queue trigger for Kafka and associate it with a serverless function like this:
fission mqt create --name kafkatest --function consumer --mqtype kafka --mqtkind keda --topic request-topic --resptopic response-topic --errortopic error-topic
This would trigger a function called consumer whenever there's a message in the request-topic queue of Kafka.
You can also associate meta data like authentication information as secrets or flags like pollingintervals, max retries etc.
Reference: https://fission.io/docs/usage/triggers/message-queue-trigger-kind-keda/kafka/
Related
Using ASP.NET Core microservices, both API and worker roles, running in Azure Service Fabric.
We use Service Bus to do inter-microservice communication.
Consider the following situation;
Each microservice holds a local, in-mem copy of cached objects of type X.
One worker role is responsible for processing a message that would result in a rebuild of this cache for all instances.
We are having multiple nodes, and thus multiple instances of each microservice in Service Fabric.
What would be the best approach to trigger this update?
I though of the following approaches:
Calling SF for all service replica's and firing an HTTP POST on each replica to trigger the update
This however does not seem to work as worker roles don't expose any APIs
Creating a specific 'broadcast' topic where each instance registers a subscription for, and thus using pub/sub mechanism
I fail to see how I can make sure each instance has it's own subscription, but also I don't end up with ghost subscriptions when something happens like a crash
You can use the OSS library Service Fabric Pub Sub for this.
Every service partition can create its own subscription for messages of a given type.
It uses the partition identifier for subscriptions, so crashes and moves won't result in ghost subscriptions.
It uses regular SF remoting, so you won't need to expose API's for messaging.
I am integrating a NestJS application with RabbitMQ and i followed this tutorial
From the tutorial i can see that message consumer can connect to the rabbitMQ broker using a queue name, but actually i want to create a temporary queue that connects to an exchange. I cannot find an option to specify the exchange name in the queue options specified in the tutorial. can someone please point to the right configuration to use to specify an exchange instead of queue?
is it possible to specify an exchange name inside the 'queueOptions' structure?
From what I understand from your tutorial (I don't know NestJS), createMicroservice only connects to a queue (or create it if needed). In other words, it performs assertQueue operation:
var queue = 'task_queue';
channel.assertQueue(queue, {
durable: false
});
If you want to bind a queue to an existing exchange, you need to perform bindQueue operation:
channel.bindQueue(queue, exchange, '');
Thus you can't bind to an exchange using queueOptions. Take a look at the official Rabbitmq documentation, especially the "Temporary queues" and following "Bindings" sections.
I haven't been able to find much information about this online. I'm wondering if its possible to build a Flink app that can dynamically consume all topics matching a regex pattern and sync those topics to s3. Also, each topic being dynamically synced would have Avro messages and the Flink app would use Confluent's Schema Registry.
So lucky man! Flink 1.4 just released a few days ago and this is the first version that provides consuming Kafka topics using REGEX. According to java docs here is how you can use it:
FlinkKafkaConsumer011
public FlinkKafkaConsumer011(PatternsubscriptionPattern,DeserializationSchema<T> valueDeserializer,Properties props)
Creates a new Kafka streaming source consumer for Kafka 0.11.x. Use
this constructor to subscribe to multiple topics based on a regular
expression pattern. If partition discovery is enabled (by setting a
non-negative value for
FlinkKafkaConsumerBase.KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS in the
properties), topics with names matching the pattern will also be
subscribed to as they are created on the fly.
Parameters:
subscriptionPattern - The regular expression for a pattern of topic names to subscribe to.
valueDeserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.
props - The properties used to configure the Kafka consumer client, and the ZooKeeper client.
Just notice that running Flink streaming application, it fetch topic data from Zookeeper at intervals specified using the consumer config :
FlinkKafkaConsumerBase.KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS
It means every consumer should resync the metadata including topics, at some specified intervals.The default value is 5 minutes. So adding a new topic you should expect consumer starts to consume it at most in 5 minutes. You should set this configuration for Flink consumer with your desired time interval.
Subscribing to Kafka topics with a regex pattern was added in Flink 1.4. See the documentation here.
S3 is one of the file systems supported by Flink. For reliable, exactly-once delivery of a stream into a file system, use the flink-connector-filesystem connector.
You can configure Flink to use Avro, but I'm not sure what the status is of interop with Confluent's schema registry.
For searching on these and other topics, I recommend the search on the Flink doc page. For example: https://ci.apache.org/projects/flink/flink-docs-release-1.4/search-results.html?q=schema+registry
My Cleint is having 2 instances and I am using below snippet to rename the queue and can see testExchange.testQueue is created
under which i can see 2 consumers i.e. my client instances but while /bus/refresh I can see only single instance is getting refreshed and
I am not getting Cloud Bus feature viz on /bus/refresh all instances should get refreshed, please let me know if I am missing any
configuration to rename the queue in readable format.
spring:
cloud:
stream:
bindings:
springCloudBusInput:
destination: testExchange
group: testQueue
config:
bus:
enabled: true
uri: https://Config-Server-offshore.com/
name: ClientApp
With spring-cloud-stream, using group creates competing consumers on the same queue.
If you remove the group each instance will get its own queue.
You can use a place holder in the group to make it unique...
spring.cloud.stream.bindings.input.group=${instanceIndex}
instanceIndex=1
...if you are running on cloud foundry, you can use the vcap instance index.
I have multiple application instances inside of Amazon EC2, each running several worker processes. What I want is each worker process to be subscribed to some notification(e.g. configuration change). This notification should be basically broadcast message, so that once it is sent - every worker receives it.
I know SQS does not support messages broadcast. Looking through similar questions/threads I see the suggestions to use SNS instead of SQS. I'm not sure this will work for me due to the following reasons:
application instances are part of autoscaling group so they can be dynamically added and removed. In this case I don't see any clear way to unsubscribe every worker(I have multiple workers per instance) once instance gets terminated, which means I'll end up with the mess of dead subscribers after some time.
protocol to use for subscription is also not clear. HTTP endpoint looks like the only option, which means my every worker should run HTTP server on its own port. It also looks I should listen only on instance public IP, which adds one more layer of complexity and insecurity.
At the moment I have a solution based on third party - I'm using 0MQ pub/sub server. But I'm looking for some out-of-box solutions AWS provides.
Thanks,
Vovan
The out-of-the-box AWS solution that comes to mind would be to create one SNS topic, and then for each instance, when the instance boots up, it would create its own SQS queue and subscribe the queue to the SNS topic, so that each individual queue gets a broadcast copy of each message you publish to SNS.
You'd want unsubscribe and delete these queues on instance termination, which could be done with lifecycle hooks.
If you didn't want to use a server to manage the processing of the lifecycle hooks (which publish the launch or termination events to SNS or SQS) you could create an AWS API Gateway endpoint to fire an AWS Lambda function, then subscribe the API Gateway endpoint to the SNS topic using https, to handle the cleanup tasks in Lambda, with no server needed.
That's several services working together and may sound a little complicated, but would be very inexpensive and require little maintenance or attention.
One more solution I've figured out is to use Amazon Kinesis.The implication here is that each subscriber has to maintain it's own checkpoint to receive only most recent notifications.
I realize this is an old thread, but I'd like to share my experience with this. Kinesis has a 5 reads/sec throttle. So if you have 10 nodes polling for events in the stream 1/sec, you're going to be in a constant state of throttling.
Kinesis looks to be primarily for massive writes with just a few readers, which doesn't quite fit a broadcast to many nodes use-case.
Redis is handy solution for broadcasting a message to all subscribers on a topic. It is convenient because it can be used as a docker container for rapid prototyping, but is also offered by AWS as a managed service for multi-node clusters.