Just want to know the meaning of the parameters in worker.py file:
def callback(ch, method, properties, body):
print " [x] Received %r" % (body,)
What do ch, method, and properties mean?
ch
"ch" is the "channel" over which the communication is happening.
Think of a RabbitMQ connection in two parts:
the TCP/IP connection
channels within the connection
the actual TCP/IP connection is expensive to create, so you only want one connection per process instance.
A channel is where the work is done with RabbitMQ. a channel exists within an connection, and you need to have the channel reference so you can ack/nack messages, etc.
method
i think "method" is meta information regarding the message delivery
when you want to acknowledge the message - tell RabbitMQ that you are done processing it - you need both the channel and the delivery tag. the delivery tag comes from the method parameter.
i'm not sure why this is called "method" - perhaps it is related to the AMQP spec, where the "method" is meta-data about which AMQP method was executed?
properties
the "properties" of the message are user-defined properties on the message. you can set any arbitrary key / value pair that you want in these properties, and possibly get things like routing key used (though this may come from "method")
properties are often uses for bits of data that your code needs to have, but aren't part of the actual message body.
for example, if you had a re-sequencer process to make sure messages are processed in order, the "properties" would probably contain the message's sequence number.
Related
I've searched for a good explanation for the difference between these two,
but didn't really find one.
What I know till now is that:
correlation id is a string (Guid which was converted to string), and delivery tag is an int.
correlation id is unique for each message, and delivery tag is unique only in
the channel (The channel is the scope).
That's fine....but what is the difference in the purposes? why do we need two identifiers for a message?
The two identifiers exist at two different conceptual layers of communication, and have different properties that are useful in each case. While a protocol could probably be designed that had one identifier serving both purposes, keeping them separate makes both implementations simpler.
Delivery tags
Part of the AMQP communication layer, built into RabbitMQ itself.
Example use: a consumer process can acknowledge that a message has been processed and can be permanently discarded on the broker (RabbitMQ server).
Automatically assigned within the open channel for every message delivered.
Must be unique within that channel in order for the protocol to function correctly. Does not need to be unique across different channels, so a simple incrementing integer is simple to implement.
The same message may be delivered at different times with different delivery tags, or even exist on multiple queues and be delivered simultaneously to different consumers.
Correlation IDs
Part of the logic of the application that is using RabbitMQ, not the broker itself.
Example use: using a matching correlation ID and "reply to" on two separate messages, which the application wants to treat as a request and a response in an RPC pattern.
Needs to be manually added when the message is first created, and is optional.
Not guaranteed to be unique by the protocol, which just sees it as an arbitrary string. It is up to an application to generate in a way that is sufficiently unlikely to collide for its use case, such as an appropriate form of UUID.
Will stay the same every time a message is delivered, no matter how many times it is forwarded or duplicated into multiple queues.
Correlation ID is generally used in the context of RabbitMQ when I want to see a synchronous behavior in which a message is sent and in response to it another sender will send a response but will have the correlationID in the reply-to tag . The common pattern which is replicated in RabbitMQ is the RPC call which is more like a Synchronous messaging.
Delivery Tag is however an indicator of the delivery of the message per channel and generally comes in scope when Acknowledged Delivery model is being followed.
Both have completely different purpose and are not message identifier as such.
I have a program that uses log4Net using both text and smtp appenders.
When the program runs several logging are captured by the smtp appenders and are buffered waiting for the program end before sending mail.
When the program is nearly completed I might discover that I do not need to send emails, and actually I log something like "Nothing to do".
I know that it is possible to manipulate by code the appenders configuration so I could suppress the mail setting the Threshold to off.
I would like to know if it is possible to get the same result using just log4net configuration: the smtp appender should not send an email if a specific string is logged at any time. If instead that string is not logged it should behave normally and send all the lines that match the defined filters.
TL;DR;
create a custom ITriggeringEventEvaluator
configure your smtp appender with this evaluator: decide what must happen when the buffer is full (discard messages, send them)
use the auto-flush of your appender to send the log events forward
There are two properties in a BufferingAppenderSkeleton in log4net that may interest you here. When you configure such an appender to be Lossy, Evaluator and LossyEvaluator are used to determine if messages should be sent to the next appenders.
The Evaluator property lets you define a class inheriting from ITriggeringEventEvaluator; when a message is logged and the buffer is full, the evaluator is called and if it returns true on the IsTriggeringEvent method the event buffer is processed.
The LossyEvaluator works in the same way, except it is called to decide whether the oldest event that will disappear from the buffer because it is full must be logged or not.
Since your SmtpAppender is a BufferingAppenderSkeleton, you can use the Evaluator property in order to define what triggers or not the email being sent (ie a logging event where you say "no log", or whatever)
However if you expect your appender to decide on its own whether or not to send the event logs when it is closed (ie should it auto-flush or not) it is the LossyEvaluator that is used.
Now for the bad news: there is only one instance of a ITriggeringEventEvaluator implementation in log4net, which evaluates log events based on their level. So you will have to code your own trigger in order to recognize special messages sent to the appender by your application.
What is the recommended way to deal with message versioning? The main schools of thought appear to be:
Always create a new message class as the message structure changes
Never use (pure) serialized objects as a message. Always use some kind of version header field and a byte stream body field. In this way, the receiver can always accept the message and check the version number before attempting to read the message body.
Never use binary serialized objects as a message. Instead, use a textual form such as JSON. In this way, the receiver can always accept the message, check the version number, and then (when possible) understand the message body.
As I want to keep my messages compact I am considering using Google Protocol Buffers which would allow me to satisfy both 2 & 3.
However I am interested in real world experiences and advice on how to handle versioning of messages as their structure changes?
In this case "version" will be basically some metadata about the message, And these metadata are some instruction/hints to the processing algorithm. So I willsuggest to add such metadata in the header (outside of the payload), so that consumer can read the metadata first before trying to read/understand and process the message payload. For example, if you keep the version info in the payload and due to some reason your (message payload is corrupted) then algorithm will fail parse the message, then it can not event reach the metadata you have put there.
You may consider to have both version and type info of the payload in one header.
I have a program that uses log4Net using both text and smtp appenders.
When the program runs several logging are captured by the smtp appenders and are buffered waiting for the program end before sending mail.
When the program is nearly completed I might discover that I do not need to send emails, and actually I log something like "Nothing to do".
I know that it is possible to manipulate by code the appenders configuration so I could suppress the mail setting the Threshold to off.
I would like to know if it is possible to get the same result using just log4net configuration: the smtp appender should not send an email if a specific string is logged at any time. If instead that string is not logged it should behave normally and send all the lines that match the defined filters.
TL;DR;
create a custom ITriggeringEventEvaluator
configure your smtp appender with this evaluator: decide what must happen when the buffer is full (discard messages, send them)
use the auto-flush of your appender to send the log events forward
There are two properties in a BufferingAppenderSkeleton in log4net that may interest you here. When you configure such an appender to be Lossy, Evaluator and LossyEvaluator are used to determine if messages should be sent to the next appenders.
The Evaluator property lets you define a class inheriting from ITriggeringEventEvaluator; when a message is logged and the buffer is full, the evaluator is called and if it returns true on the IsTriggeringEvent method the event buffer is processed.
The LossyEvaluator works in the same way, except it is called to decide whether the oldest event that will disappear from the buffer because it is full must be logged or not.
Since your SmtpAppender is a BufferingAppenderSkeleton, you can use the Evaluator property in order to define what triggers or not the email being sent (ie a logging event where you say "no log", or whatever)
However if you expect your appender to decide on its own whether or not to send the event logs when it is closed (ie should it auto-flush or not) it is the LossyEvaluator that is used.
Now for the bad news: there is only one instance of a ITriggeringEventEvaluator implementation in log4net, which evaluates log events based on their level. So you will have to code your own trigger in order to recognize special messages sent to the appender by your application.
In our scenario I'm thinking of using the pub sub technique. However I don't know which is the better option.
1 ########
A web service of ours will publish a message that something has happened when it is called externally, ExternalPersonCreatedMessage!
This message will contain a field that represents the destinations to process the message into (multiple allowed).
Various subscribers will subscribe. These subscribers will filter the message to see if any action is required by checking the destination field.
2 ########
A web service of ours will parse the incoming call and publish specific types of messages depending on the destinations supplied in the field. i.e. many Destination[n]PersonCreatedMessage messages would be created.
Subscribers will subscribe to only the specific message they care for. i.e. not having to filter any messages
QUESTIONS
Which of the above is the better option and why? And how do I stop myself from making RequestMessages. From what I've read/seen I should be trying to structure this in a way of PersonCreated, PersonDeleted i.e. SOMETHING HAS HAPPENED and NOT in the REQUEST SOMETHING TO HAPPEN form such as CreatePerson or DeletePerson
Are my thoughts correct? I've been looking for guidance on how to structure messages and making sure I don't go down a wrong path but have found no guidance out there on do's and dont's. Can any one help and guide? I want to try and get this correct from the off :)
Based on the integration scenario in the referenced article, it appears to me that you may need a Saga to complete the workflow of accept message -> operate on message -> send confirmation. In the case that the confirmation is sent immediately after the operation, you could use NSBs message handler pipeline feature which allows you to chain handlers in a specified sequence such as...
First<FilterHandler>.Then<DoWorkHandler>().AndThen<SendConfirmationHandler>();
In terms of the content filtering, you can do this although you incur some transport overhead, meaning the queue will have to accept the message and the process will always call the first handler on every message(you can short-circuit the above pipeline at any point). It may be the case that what you really want is a Distributor/Worker setup where all Workers are the same and you can handle some load.
If you truly have different endpoints with completely different logic, then I would have the Publisher process(only accepts and Publishes message) do the work of translating the inbound message to something else a Subscriber can then be interested in. If then you find that a given Published message only ever has 1 Subscriber, then you don't need to Publish at all, you need to just Bus.Send() to the correct endpoint.
The way NServiceBus handles pub-sub is more like your option two.
A publisher service has an input queue and a subscription store.
A subscriber service has an input queue
The subscriber, on start-up will send a subscription message to the input queue of the publisher
The subscription message contains the type of message subscriber is interested in and the subscribers queue address
The publisher records the subscription in the subscription store.
The publisher receives a message.
The publisher evaluates the message type against the list of subscriptions
For each match found the publisher sends the message to the queue address.
In my opinion, you should stop thinking about destinations. Messages are messages. They should not have any inherent destination information in them. The subscription mechanism defines the addressing/routing requirements for the solution.