How to get container registry pubsub notifications inside code (java or any other lang) - notifications

My aim is to get notifications from google container registry in code whenever any image is updated/inserted/deleted from the registry.
I am following tutorial - https://cloud.google.com/container-registry/docs/configuring-notifications
I am able to pull notification messages from the registry using the google console using command - gcloud alpha pubsub subscriptions pull SUBSCRIPTION
But I want these notification messages to be delivered in code (in java).
If someone can give me any reference to any article or tutorial that will help.
After comment from dsesto i have added following code. This code gave me some messages when i run first. But after that i kept application running and tried to delete/insert images from container registry but it did not gave any message.
Any suggestions.
package com.avaya.ipoffice.mcm.googleconnect;
import org.springframework.stereotype.Service;
import com.google.cloud.pubsub.v1.AckReplyConsumer;
import com.google.cloud.pubsub.v1.MessageReceiver;
import com.google.cloud.pubsub.v1.Subscriber;
import com.google.pubsub.v1.ProjectSubscriptionName;
import com.google.pubsub.v1.PubsubMessage;
#Service
public class RecieveMessagesUtil {
public static void main(String... args) throws Exception {
String projectId = "xxxxx";
String subscriptionId = "prashantsub";
ProjectSubscriptionName subscriptionName = ProjectSubscriptionName.of(projectId, subscriptionId);
// Instantiate an asynchronous message receiver
MessageReceiver receiver = new MessageReceiver() {
#Override
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
// handle incoming message, then ack/nack the received message
System.out.println("Id : " + message.getMessageId());
System.out.println("Data : " + message.getData().toStringUtf8());
consumer.ack();
}
};
Subscriber subscriber = null;
try {
// Create a subscriber for "my-subscription-id" bound to the message receiver
subscriber = Subscriber.newBuilder(subscriptionName, receiver).build();
subscriber.startAsync();
// ...
} catch (Exception e) {
System.out.println("Exception while subscribing" + e);
} finally {
// stop receiving messages
if (subscriber != null) {
subscriber.stopAsync();
}
}
}
}

From your question I understand that you have already successfully setup the notifications system of Container Registry using Pub/Sub topics and subscriptions, given that you said that you are already able to retrieve the messages from your Pub/Sub subscription using command gcloud pubsub subscriptions pull. Therefore, it looks like your concern is mostly related to pulling messages from a Subscription programatically.
First of all, I would recommend you to have a look at this documentation page about Subscribers in Pub/Sub. Especially, have a look at the Pull vs. Push comparison, where you will have a better idea of the possibilities available, and you should first decide whether to work with a Pull Subscription (such as the one you are using when calling the gcloud command, where the application initiates requests) or with a Push Subscription (where Pub/Sub initiates the requests to a subscriber application, such as an App Engine application).
Once that is clear (and assuming that you go for the Pull Subscription, which you are already using with gcloud), you can have a look at the documentation on how to perform Asynchronous Pull operations, with a Java-based example. Additionally, you can have a look at the complete subscriber example available in GitHub.
Finally, you should have a look at the Client Libraries documentation, more specifically the Pub/Sub Java reference, where you will find the complete documentation for the Pub/Sub Client Libraries used to work with Pub/Sub programatically.

I would recommend looking at the Cloud Pub/Sub documentation, particularly, the subscriber guide, which gives you the code necessary to receive messages in Java.

Related

Access message properties from Apache Flink RabbitMQ source connector

I am using Apache Flinke 1.7.2 RabbitMQ connector: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/rabbitmq.html
I want to access the message_id in the amqp message properties sent along with the body of the amqp message. i want to be able to group by that message ID. The problem is that i only get the body of the message out from the source after i build it.
Is there an easy way that doesn't require me to rewrite the source class from scratch ?
I guess this is not possible. Looking at the source-code of the connector you can see that they extract only the body of the RMQ message:
#Override
public void run(SourceContext<OUT> ctx) throws Exception {
while (running) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
synchronized (ctx.getCheckpointLock()) {
OUT result = schema.deserialize(delivery.getBody());
// ....
ctx.collect(result);
}
}
}
I guess you have to find another connector (3rd party) or implement this by your own. Sorry for the bad news!

Kafka Error handling : Processor.output().send(message, kafkaTimeoutInMS) always returns true and its async

May be this issue is already reported and resolved .I didn't find the solution and any open issues which talk about this, so creating new one.
I am trying to handle error while publishing data to kafka topic.
With kafka spring steam we are pushing to kafka by using this
if (processor.output().send(messsage , kafkaTimeoutInMS) && acknowledgment != null)
{
LOGGER.debug("Acknowledgment provided");
LOGGER.info("Sending to Kafka successful");
acknowledgment.acknowledge();
}
else
{
LOGGER.error("Sending to Kafka failed", message);
}
Send() method always returns true, I tried stopping kafka manual while running in debug mode, but still it returns true. I have that read it is asynchronous.
I Tried setting
bindings: output: producer: sync: true
This didnt help.
But I see some error which I cant use in my logic to decide whether there is failure or success.
We are manually acknowledging hence we are only supposed to acknowledge when its sent to topic successfully and we need to log all failed messages.
Any suggestions?
I believe you've misinterpreted on how spring-cloud-stream works.
As a framework there is certain contract between the user and the framework and when it comes to messaging the acks, retries, DLQ and many more aspects are handled automatically to ensure the user doesn't have to be exposed to this manually (as you are trying to do).
Consider spending a little time and going through the user guide - https://docs.spring.io/spring-cloud-stream/docs/Fishtown.M3/reference/htmlsingle/
Also, here is the very basic example that will demonstrates a typical interaction of user(developer) with the framework. As you can see, all you're doing is implementing a simple handler which receives and returns a piece of data. The rest (the actual receive from Kafka and send to Kafka or any other messaging system) is handled by the framework provided binders.
#SpringBootApplication
#EnableBinding(Processor.class)
public class ProcessorApplication {
public static void main(String[] args) {
SpringApplication.run(ProcessorApplication.class);
}
#StreamListener(Processor.INPUT)
#SendTo(Processor.OUTPUT)
public String echo(String message) {
return message;
}
}

How to invoke a web service after redeliveries exhausted in Apache Camel?

I have failed to find an enterprise integration pattern or recipe that promotes a solution for this problem:
After the re-delivery attempts have been exhausted, I need to send a web service request back to the originating source, to notify the sender of a failed delivery.
Upon exhaustion of all re-delivery attempts, should I move the message to a dead letter queue? Then create a new consumer listening on that DL queue? Do I need a unique dead letter queue for each of my source message queues? Should I add a message header, noting the source queue, before I move it to the dead letter queue? If all messages go to a single dead letter queue, how should my consumer know where to send the web service request?
Can you point me to a book, blog post, or article? What is the prescribed approach?
I'm working in a really old version of Fuse ESB but I expect that solutions in ServiceMix to be equally applicable.
Or maybe, what I'm asking for is an anti-pattern or code-smell. Please advise.
If you are new to Camel and really want to get an in-depth knowledge of it, I would recommend Camel in Action, a book by Claus Ibsen. There's a second edition in the works, with 14 out of 19 chapters already done so you may also give that a shot.
If that's a bit too much, online documentation is pretty okay, you can find out the basics just fine from it. For error handling I recommend starting with the general error handling page then moving on to error handler docs and exception policy documentation.
Generally, dead letter channel is the way to go - Camel will automatically send to DLC after retries have been exhausted, you just have to define the DLC yourself. And its name implies, it's a channel and doesn't really need to be a queue - you can write to file, invoke a web-service, submit a message to a message queue or just write to logs, it's completely up to you.
// error-handler DLC, will send to HTTP endpoint when retries are exhausted
errorHandler(deadLetterChannel("http4://my.webservice.hos/path")
.useOriginalMessage()
.maximumRedeliveries(3)
.redeliveryDelay(5000))
// exception-clause DLC, will send to HTTP endpoint when retries are exhausted
onException(NetworkException.class)
.handled(true)
.maximumRedeliveries(5)
.backOffMultiplier(3)
.redeliveryDelay(15000)
.to("http4://my.webservice.hos/otherpath");
I myself have always preferred having a message queue and then consuming from there for any other recovery or reporting. I generally include failure details like exchange ID and route ID, message headers, error message and sometimes even stacktrace. The resulting message, as you can imagine, grows quite a bit but it tremendously simplifies troubleshooting and debugging, especially in environments where you have quite a number of components and services. Here's a sample DLC message from one my projects:
public class DeadLetterChannelMessage {
private String timestamp = Times.nowInUtc().toString();
private String exchangeId;
private String originalMessageBody;
private Map<String, Object> headers;
private String fromRouteId;
private String errorMessage;
private String stackTrace;
#RequiredByThirdPartyFramework("jackson")
private DeadLetterChannelMessage() {
}
#SuppressWarnings("ThrowableResultOfMethodCallIgnored")
public DeadLetterChannelMessage(Exchange e) {
exchangeId = e.getExchangeId();
originalMessageBody = e.getIn().getBody(String.class);
headers = Collections.unmodifiableMap(e.getIn().getHeaders());
fromRouteId = e.getFromRouteId();
Optional.ofNullable(e.getProperty(Exchange.EXCEPTION_CAUGHT, Exception.class))
.ifPresent(throwable -> {
errorMessage = throwable.getMessage();
stackTrace = ExceptionUtils.getStackTrace(throwable);
});
}
// getters
}
When consuming from the dead letter queue, route ID can tell you where the failure originated from so you can then implement routes that are specific for handing errors coming from there:
// general DLC handling route
from("{{your.dlc.uri}}")
.routeId(ID_REPROCESSABLE_DLC_ROUTE)
.removeHeaders(Headers.ALL)
.unmarshal().json(JsonLibrary.Jackson, DeadLetterChannelMessage.class)
.toD("direct:reprocess_${body.fromRouteId}"); // error handling route
// handle errors from `myRouteId`
from("direct:reprocess_myRouteId")
.log("Error: ${body.errorMessage} for ${body.originalMessageBody}");
// you'll probably do something better here, e.g.
// .convertBodyTo(WebServiceErrorReport.class) // requires a converter
// .process(e -> { //do some pre-processing, like setting headers/properties })
// .toD("http4://web-service-uri/path"); // send to web-service
// for routes that have no DLC handling supplied
onException(DirectConsumerNotAvailableException.class)
.handled(true)
.useOriginalMessage()
.removeHeaders(Headers.ALL)
.to({{my.unreprocessable.dlc}}); // errors that cannot be recovered from

How can I create a "Health Check" for an Azure Storage Queue

So when developing an app utilizing Azure Storage Queues and a Web Job, I feel like I need some sort of health check (via API) to ensure my Azure Storage Queue is properly configured for each environment up to prod. I don't have access (directly) to view the Dashboard or Kudu.
My thought thus far was to just create an API route that returns a bool that tells me if I was able to create the queue if it doesn't exist, and peek at a message (even if one doesn't exist), like :
public async Task<bool> StorageQueueHealthCheck()
{
return await _queueManager.HealthCheck();
}
And the implementation:
public async Task<bool> HealthCheck()
{
try
{
CloudQueue queue = _queueClient.GetQueueReference(QueueNames.reportingQueue);
queue.CreateIfNotExists();
CloudQueueMessage peek = await queue.PeekMessageAsync();
return true; // as long as we were able to peek at messages
}
catch (Exception ex)
{
return false;
}
}
Is this a bad approach? Is there another way to "health check" certain Azure functionality when the dashboard is abstracted away? If I absolutely needed I would be able to view the Kudu but would rather just use an API and hit it via Swagger.
Looks good. You can also try CloudQueue.FetchAttributeAsync() since the payload would be smaller when the message size is large.
This is a good approach, please just make sure you do have a retry mechanism so that your healthcheck does not just return false for intermittent failures.
Second Approach,
Instead of api which will perform the job only is trigger there should be a console app (webjob) which does this task on regular interval (1min) and based on some logic lets say all 'creates' in last 10mins threw error sends an email. This can be used in all environments.

Which ServiceControl.Contracts messages should IHandle?

I'm trying to make sure that 3rd party dependencies are running, and built a service to do this based on the Monitoring 3rd party Sample Application, which emits ServiceControl CheckResult messages.
This works fine; ServicePulse alerts me when I stop/start my local and remote windows services, Databases, Flux Capacitors, etc.
I now want to build a windows service / nServiceBus Endpoint, like ServicePulse, but with logic that can attempt recovery, send emails etc. I don't really want to put this code into the 3rdParty monitor.
I followed the servicecontrol/external-integrations and servicecontrol/contracts tutorials, and created my MendStuffOrEmail endpoint - But it doesn't work; It doesn't receive any messages.
I was going to ask "what am I doing wrong?", but I think I know; I'm using IHandleMessages<ServiceControl.Contracts.MessageFailed> which is for failed messages.
I need to listen for the "CheckResult" type messages - but what are they? I have looked through the ServiceControl and ServicePulse code, but cannot work out what is being sent/received. How can I find this out, or has anyone else actually done this and already knows?
UPDATE
After more extensive rummaging, I also subscribed to CustomCheckFailed and CustomCheckSucceeded messages. I implemented IHandle interfaces for them, but I'm still not getting any messages. The log shows autosubscriber has taken out a subscription to them. What should I check for next?
I compared my code to Sean's posted
example and found the mistake:
I had implemented two of the interfaces, IConfigureThisEndpoint and AsA_Server in the wrong class (a 2am cut 'n' paste error).
The example listens for failed messages, but for anyone else trying to do this, you do need to subscribe to CustomCheckFailed and CustomCheckSucceeded messages (nuget ServiceControl.Contracts).
public partial class MessageHandler : IHandleMessages<CustomCheckFailed>,
IHandleMessages<CustomCheckSucceeded>
{
public void Handle(CustomCheckFailed message)
{
this.HandleImplementation(message);
}
partial void HandleImplementation(CustomCheckFailed message);
public void Handle(CustomCheckSucceeded message)
{
this.HandleImplementation(message);
}
partial void HandleImplementation(CustomCheckSucceeded message);
public IBus Bus { get; set; }
}
then the logic to do something with the messages. (I left in my original test - sending email - but our system has a library with all sorts of recovery & notification methods. You'll need something similar to stop an email flood):
public partial class MessageHandler
{
partial void HandleImplementation(CustomCheckFailed message)
{
var messageBody = string.Format("Message with id {0} failed with reason {1}", message.CustomCheckId, message.FailureReason);
MailMessageFactory.sendEmail("Failure Notification", messageBody);
Console.Out.WriteLine(messageBody);
}
}
And a similar file with the logic for recovery messages (CustomCheckSucceeded). You probably want a check in there to detect it is actually recovering from a failure, not just passing the test.
So anyway, fixed - on my dev pc.
The next problem was making it work on the server, which took a support call. It turns out ServiceControl ALSO needs a licence, available as part of the "Advanced", "Enterprise", and "Ultimate" editions - Not part of the platform on the standard licence.