Poison Queue content into main queue - azure-queues

I am trying to move poison messages into the main queue. I am not facing any problem in moving the messages, but looks like message is getting converted to some other encryption level.
<bound method DictMixin.values of {'id': '389a834e-48af-41b5-be36-5f61ad7c2232', 'inserted_on':
datetime.datetime(2020, 6, 30, 3, 13, tzinfo=datetime.timezone.utc),
'expires_on': datetime.datetime(2020, 7, 7, 3, 13, tzinfo=datetime.timezone.utc),
'dequeue_count': 0, 'content':
'eyJjYWxsX2JhY2tfdXJpIjogImh0dHBzOi8vcG9zdG1hbi1lY2hvLmNvbS9wb3N0IiwgInBpcGVsaW5lX3J1bl
9pZCI6ICI3OTY1MGU3Zi01NmFmLTRiYzgtOWE3NC0yYTk3YWRhOWRhNWUiLCAiZXhlY
3V0aW9uX2lkIjogImRmYzcwMjAwLTY3MzgtNDNkMy1',
'pop_receipt': None, 'next_visible_on': None}>
how can I convert content into the message queue ?

As version 1.4.1 of the Microsoft Azure Storage Explorer doesn’t have the ability to move messages from one Azure queue to another.
Here is a simple solution to transfer poison messages back to the originating queue. Obviously, you'll need to have fixed the error that caused the messages to end up in the poison message queue!
You’ll need to add a NuGet package reference to Microsoft.NET.Sdk.Functions :
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Queue;
void Main()
{
const string queuename = "MyQueueName";
string storageAccountString = "xxxxxx";
RetryPoisonMesssages(storageAccountString, queuename);
}
private static int RetryPoisonMesssages(string storageAccountString, string queuename)
{
CloudQueue targetqueue = GetCloudQueueRef(storageAccountString, queuename);
CloudQueue poisonqueue = GetCloudQueueRef(storageAccountString, queuename + "-poison");
int count = 0;
while (true)
{
var msg = poisonqueue.GetMessage();
if (msg == null)
break;
poisonqueue.DeleteMessage(msg);
targetqueue.AddMessage(msg);
count++;
}
return count;
}
private static CloudQueue GetCloudQueueRef(string storageAccountString, string queuename)
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageAccountString);
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
CloudQueue queue = queueClient.GetQueueReference(queuename);
return queue;
}

Related

How to move messages from one queue to another in RabbitMQ

In RabbitMQ,I have a failure queue, in which I have all the failed messages from different Queues. Now I want to give the functionality of 'Retry', so that administrator can again move the failed messages to their respective queue. The idea is something like that:
Above diagram is structure of my failure queue. After click on Retry link, message should move into original queue i.e. queue1, queue2 etc.
If you are looking for a Java code to do this, then you have to simply consume the messages you want to move and publish those messages to the required queue. Just look up on the Tutorials page of rabbitmq if you are unfamiliar with basic consuming and publishing operations.
It's not straight forward consume and publish. RabbitMQ is not designed in that way. it takes into consideration that exchange and queue both could be temporary and can be deleted. This is embedded in the channel to close the connection after single publish.
Assumptions:
- You have a durable queue and exchange for destination ( to send to)
- You have a durable queue for target ( to take from )
Here is the code to do so:
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.QueueingConsumer;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.amqp.rabbit.connection.CachingConnectionFactory;
public object shovelMessage(
String exchange,
String targetQueue,
String destinationQueue,
String host,
Integer port,
String user,
String pass,
int count) throws IOException, TimeoutException, InterruptedException {
if(StringUtils.isEmpty(exchange) || StringUtils.isEmpty(targetQueue) || StringUtils.isEmpty(destinationQueue)) {
return null;
}
CachingConnectionFactory factory = new CachingConnectionFactory();
factory.setHost(StringUtils.isEmpty(host)?internalHost.split(":")[0]:host);
factory.setPort(port>0 ? port: Integer.parseInt(internalPort.split(":")[1]));
factory.setUsername(StringUtils.isEmpty(user)? this.user: user);
factory.setPassword(StringUtils.isEmpty(pass)? this.pass: pass);
Channel tgtChannel = null;
try {
org.springframework.amqp.rabbit.connection.Connection connection = factory.createConnection();
tgtChannel = connection.createChannel(false);
tgtChannel.queueDeclarePassive(targetQueue);
QueueingConsumer consumer = new QueueingConsumer(tgtChannel);
tgtChannel.basicQos(1);
tgtChannel.basicConsume(targetQueue, false, consumer);
for (int i = 0; i < count; i++) {
QueueingConsumer.Delivery msg = consumer.nextDelivery(500);
if(msg == null) {
// if no message found, break from the loop.
break;
}
//Send it to destination Queue
// This repetition is required as channel looses the connection with
//queue after single publish and start throwing queue or exchange not
//found connection.
Channel destChannel = connection.createChannel(false);
try {
destChannel.queueDeclarePassive(destinationQueue);
SerializerMessageConverter serializerMessageConverter = new SerializerMessageConverter();
Message message = new Message(msg.getBody(), new MessageProperties());
Object o = serializerMessageConverter.fromMessage(message);
// for some reason msg.getBody() writes byte array which is read as a byte array // on the consumer end due to which this double conversion.
destChannel.basicPublish(exchange, destinationQueue, null, serializerMessageConverter.toMessage(o, new MessageProperties()).getBody());
tgtChannel.basicAck(msg.getEnvelope().getDeliveryTag(), false);
} catch (Exception ex) {
// Send Nack if not able to publish so that retry is attempted
tgtChannel.basicNack(msg.getEnvelope().getDeliveryTag(), true, true);
log.error("Exception while producing message ", ex);
} finally {
try {
destChannel.close();
} catch (Exception e) {
log.error("Exception while closing destination channel ", e);
}
}
}
} catch (Exception ex) {
log.error("Exception while creating consumer ", ex);
} finally {
try {
tgtChannel.close();
} catch (Exception e) {
log.error("Exception while closing destination channel ", e);
}
}
return null;
}
To requeue a message you can use the receiveAndReply method. The following code will move all messages from the dlq-queue to the queue-queue:
do {
val movedToQueue = rabbitTemplate.receiveAndReply<String, String>(dlq, { it }, "", queue)
} while (movedToQueue)
In the code example above, dlq is the source queue, { it } is the identity function (you could transform the message here), "" is the default exchange and queue is the destination queue.
I also have implemented something like that, so I can move messages from a dlq back to processing. Link: https://github.com/kestraa/rabbit-move-messages
Here is a more generic tool for some administrative/supporting tasks, the management-ui is not capable of.
Link: https://github.com/bkrieger1991/rabbitcli
It also allows you to fetch/move/dump messages from queues even with a filter on message-content or message-headers :)

How to create a durable publisher/subscriber topic using the AMQP.Net Lite library in .NET Core with clientId and subscriber name and topic name

I am new to ActiveMQ, but I tried and am able to create a durable publisher, but I am not able to set Client Id, because I am not finding any properties with client Id and am even unable to find in Google. It will be great help if I will get some sample code.
Note:
Not with the NMS protocol. I am using AMQP.Net Lite with ActiveMQ in the .NET Core Web API for creating a durable publisher/subscriber with ClientId.
In order to create a durable subscription to ActiveMQ or ActiveMQ Artemis your client needs to do a couple things.
Set a unique "client-id" for the client using the AMQP 'ContainerId' property which can be seen in the code below. The client must use that same container ID every time it connects and recovers it's durable subscription.
Create a new Session.
Create a new Receiver for the address (in this case Topic) that you want to subscribe to. The Source of a durable subscription need to have the address set to a Topic address (in ActiveMQ this is topic://name). The Source also needs the expiray policy set to NEVER, the Source must also have the terminus durability state set to UNSETTLED_STATE, and the distribution mode set to COPY.
Once the Receiver is created then you can either set an onMessage handler in start or call receive to consume messages (assuming you've granted credit for the broker to send you any).
using System;
using Amqp;
using Amqp.Framing;
using Amqp.Types;
using Amqp.Sasl;
using System.Threading;
namespace aorg.apache.activemq.examples
{
class Program
{
private static string DEFAULT_BROKER_URI = "amqp://localhost:5672";
private static string DEFAULT_CONTAINER_ID = "client-1";
private static string DEFAULT_SUBSCRIPTION_NAME = "test-subscription";
private static string DEFAULT_TOPIC_NAME = "test-topic";
static void Main(string[] args)
{
Console.WriteLine("Starting AMQP durable consumer example.");
Console.WriteLine("Creating a Durable Subscription");
CreateDurableSubscription();
Console.WriteLine("Attempting to recover a Durable Subscription");
RecoverDurableSubscription();
Console.WriteLine("Unsubscribe a durable subscription");
UnsubscribeDurableSubscription();
Console.WriteLine("Attempting to recover a non-existent durable subscription");
try
{
RecoverDurableSubscription();
throw new Exception("Subscription was not deleted.");
}
catch (AmqpException)
{
Console.WriteLine("Recover failed as expected");
}
Console.WriteLine("Example Complete.");
}
// Creating a durable subscription involves creating a Receiver with a Source that
// has the address set to the Topic name where the client wants to subscribe along
// with an expiry policy of 'never', Terminus Durability set to 'unsettled' and the
// Distribution Mode set to 'Copy'. The link name of the Receiver represents the
// desired name of the Subscription and of course the Connection must carry a container
// ID uniqure to the client that is creating the subscription.
private static void CreateDurableSubscription()
{
Connection connection = new Connection(new Address(DEFAULT_BROKER_URI),
SaslProfile.Anonymous,
new Open() { ContainerId = DEFAULT_CONTAINER_ID }, null);
try
{
Session session = new Session(connection);
Source source = CreateBasicSource();
// Create a Durable Consumer Source.
source.Address = DEFAULT_TOPIC_NAME;
source.ExpiryPolicy = new Symbol("never");
source.Durable = 2;
source.DistributionMode = new Symbol("copy");
ReceiverLink receiver = new ReceiverLink(session, DEFAULT_SUBSCRIPTION_NAME, source, null);
session.Close();
}
finally
{
connection.Close();
}
}
// Recovering an existing subscription allows the client to ask the remote
// peer if a subscription with the given name for the current 'Container ID'
// exists. The process involves the client attaching a receiver with a null
// Source on a link with the desired subscription name as the link name and
// the broker will then return a Source instance if this current container
// has a subscription registered with that subscription (link) name.
private static void RecoverDurableSubscription()
{
Connection connection = new Connection(new Address(DEFAULT_BROKER_URI),
SaslProfile.Anonymous,
new Open() { ContainerId = DEFAULT_CONTAINER_ID }, null);
try
{
Session session = new Session(connection);
Source recoveredSource = null;
ManualResetEvent attached = new ManualResetEvent(false);
OnAttached onAttached = (link, attach) =>
{
recoveredSource = (Source) attach.Source;
attached.Set();
};
ReceiverLink receiver = new ReceiverLink(session, DEFAULT_SUBSCRIPTION_NAME, (Source) null, onAttached);
attached.WaitOne(10000);
if (recoveredSource == null)
{
// The remote had no subscription matching what we asked for.
throw new AmqpException(new Error());
}
else
{
Console.WriteLine(" Receovered subscription for address: " + recoveredSource.Address);
Console.WriteLine(" Recovered Source Expiry Policy = " + recoveredSource.ExpiryPolicy);
Console.WriteLine(" Recovered Source Durability = " + recoveredSource.Durable);
Console.WriteLine(" Recovered Source Distribution Mode = " + recoveredSource.DistributionMode);
}
session.Close();
}
finally
{
connection.Close();
}
}
// Unsubscribing a durable subscription involves recovering an existing
// subscription and then closing the receiver link explicitly or in AMQP
// terms the close value of the Detach frame should be 'true'
private static void UnsubscribeDurableSubscription()
{
Connection connection = new Connection(new Address(DEFAULT_BROKER_URI),
SaslProfile.Anonymous,
new Open() { ContainerId = DEFAULT_CONTAINER_ID }, null);
try
{
Session session = new Session(connection);
Source recoveredSource = null;
ManualResetEvent attached = new ManualResetEvent(false);
OnAttached onAttached = (link, attach) =>
{
recoveredSource = (Source) attach.Source;
attached.Set();
};
ReceiverLink receiver = new ReceiverLink(session, DEFAULT_SUBSCRIPTION_NAME, (Source) null, onAttached);
attached.WaitOne(10000);
if (recoveredSource == null)
{
// The remote had no subscription matching what we asked for.
throw new AmqpException(new Error());
}
else
{
Console.WriteLine(" Receovered subscription for address: " + recoveredSource.Address);
Console.WriteLine(" Recovered Source Expiry Policy = " + recoveredSource.ExpiryPolicy);
Console.WriteLine(" Recovered Source Durability = " + recoveredSource.Durable);
Console.WriteLine(" Recovered Source Distribution Mode = " + recoveredSource.DistributionMode);
}
// Closing the Receiver vs. detaching it will unsubscribe
receiver.Close();
session.Close();
}
finally
{
connection.Close();
}
}
// Creates a basic Source type that contains common attributes needed
// to describe to the remote peer the features and expectations of the
// Source of the Receiver link.
private static Source CreateBasicSource()
{
Source source = new Source();
// These are the outcomes this link will accept.
Symbol[] outcomes = new Symbol[] {new Symbol("amqp:accepted:list"),
new Symbol("amqp:rejected:list"),
new Symbol("amqp:released:list"),
new Symbol("amqp:modified:list") };
// Default Outcome for deliveries not settled on this link
Modified defaultOutcome = new Modified();
defaultOutcome.DeliveryFailed = true;
defaultOutcome.UndeliverableHere = false;
// Configure Source.
source.DefaultOutcome = defaultOutcome;
source.Outcomes = outcomes;
return source;
}
}
}

Processing message from rabbitmq at specified rate

We have been trying to make listener read messages from rabbitmq at a certain rate 1 msg/2 seconds. We did not find any such utility with rabbit mq so far. So thought of doing this with DB i.e. listener will read the messages and store it into DB and later a scheduler will process at that desired rate from DB. If there is any better way of doing this, please suggest. We are developing our application in Spring. Thanks in advance.
You can't do it with a listener, but you can do it with a RabbitTemplate ...
#SpringBootApplication
public class So40446967Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So40446967Application.class, args);
RabbitAdmin admin = context.getBean(RabbitAdmin.class);
AnonymousQueue queue = new AnonymousQueue();
admin.declareQueue(queue);
RabbitTemplate template = context.getBean(RabbitTemplate.class);
for (int i = 0; i < 10; i++) {
template.convertAndSend(queue.getName(), "foo" + i);
}
String out = (String) template.receiveAndConvert(queue.getName());
while (out != null) {
System.out.println(new Date() + " " + out);
Thread.sleep(2000);
out = (String) template.receiveAndConvert(queue.getName());
}
context.close();
}
}
Of course you can use something more sophisticated like a task scheduler or a Spring #Async method rather than sleeping.
Inspired on the Gary Russel answer:
you can use something more sophisticated like a task scheduler or a Spring #Async
You can also get a number of determined message per minute and simulate the same limit rate:
private final RabbitTemplate rabbitTemplate;
#Scheduled(fixedDelay = 60000) // 1 minute
public void read() {
List<String> messages = new ArrayList<>();
String message = getMessageFromQueue();
while(message != null && messages.size() < 30) { // 30 messages in 1 minute = 1 msg / 2 seconds
messages.add(message);
message = getMessageFromQueue();
}
public String getMessageFromQueue() {
return (String) rabbitTemplate.receiveAndConvert(QUEUE_NAME);
}
}

How to consume an object from azure service bus topic subscription

I got this error upon receving an object from a subscription in azure service bus.
An exception of type 'System.Runtime.Serialization.SerializationException' occurred in System.Runtime.Serialization.dll but was not handled in user code
I've tried some deserialization code but nothing works.
This is how I send a message. Please tell me how to receive it.
public void SendMessage()
{
BrokeredMessage message = new BrokeredMessage(new TestMessage() {
MsgNumber = 1, MsgContent = "testing message" }, new DataContractSerializer(typeof(TestMessage)));
// Send message to the topic
TopicClient topicClient = TopicClient.CreateFromConnectionString(cn, topicNamespace);
topicClient.Send(message);
}
public string ReceiveMessage(){
//??????
}
To receive a single message, you need to get the SubscriptionClient :
public void ReceiveMessage(string connectionString, string topicPath, string subscriptionName)
{
var subscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionString, topicPath, subscriptionName);
var brokeredMessage = subscriptionClient.Receive();
var message = brokeredMessage.GetBody<TestMessage>();
}

Send message has been delivering successfully, But when i'm trying to get messages from agent no messages be presented in agent

I'm new to JMS+OPenMq + Glassfish , please give me up by make successful send message and receive messages....
I have created two different servlet programs and i have deployed in galssfish server....Here i'm sending message successfully , but consumer has not able to consuming messages ......
producer :
Properties p = new Properties();
p.put("java.naming.factory.initial","com.sun.enterprise.naming.SerialInitContextFactory");
p.put("java.naming.factory.url.pkgs","com.sun.enterprise.naming");
p.put("java.naming.provider.url", "iiop://localhost:3700");
InitialContext jndiContext = new InitialContext(p);
TopicConnectionFactory connectionFactory = (TopicConnectionFactory) jndiContext.lookup("jms/HQTapicConnectionFactory");
Topic topic = (Topic) jndiContext.lookup("jms/HqDestTopic");
System.out.println(topic.getTopicName());
TopicConnection connection = (TopicConnection) connectionFactory.createTopicConnection();
System.out.println(connection.toString());
TopicSession session = connection.createTopicSession(true, Session.AUTO_ACKNOWLEDGE); //createSession(false, Session.AUTO_ACKNOWLEDGE);
TopicPublisher publisher = session.createPublisher(topic);
ObjectMessage message = session.createObjectMessage();
ArrayList<Employee> employeeList= new ArrayList<Employee>();
Employee employee = null;
for (int i = 0; i < 5; i++) {
employee = new Employee();
employee.setEmpid((100+i));
employee.setName("devid"+i);
employeeList.add(employee);
}
System.out.println(employeeList.size());
message.setObject(employeeList);
publisher.send(message);
Consumer:
public void onMessage(Message message){
ObjectMessage objectMessage= (ObjectMessage) message;
try{
System.out.println("Received the following message: ");
Object object = objectMessage.getObject();
if(object instanceof ArrayList){
ArrayList arrayList = (ArrayList)object;
for (int i = 0; i < arrayList.size(); i++) {
Object object1 = arrayList.get(i);
if(object1 instanceof Employee){
Employee employee = (Employee)object1;
System.out.println(employee.getEmpid());
System.out.println(employee.getName());
System.out.println();
}
}
}
}
catch (JMSException e)
{
e.printStackTrace();
}
}
I'm not able to receiving messages ,
Please help me to proper configure for broker in glassfish server.
,...appreciate for your replay
If your consumer is in a servlet, it will only catch the messages which are sent in the same moment of time (quite unlikely) - you are using topics which do not buffer by default.
Either use queues (instead of topics) or write a stand-alone program which is permanently running (and thus listening/eceiving). Normally topic listeners do not make much sense in a servlet.