if you look at the code of RedisLock.cs class you can see that it is reading the lock value to validate timeout itself outside Redis and it is also using Watch and Unwatch to overwrite timeout value if nobody touched it.
In another words, what are the exact points of using watch and unwatch and commit while we can use Redis internal set timeouts which are even more reliable?
ExecUtils.RetryUntilTrue(() =>{
//This pattern is taken from the redis command for SETNX http://redis.io/commands/setnx
//Calculate a unix time for when the lock should expire
var realSpan = timeOut ?? new TimeSpan(365, 0, 0, 0); //if nothing is passed in the timeout hold for a year
var expireTime = DateTime.UtcNow.Add(realSpan);
var lockString = (expireTime.ToUnixTimeMs() + 1).ToString();
//Try to set the lock, if it does not exist this will succeed and the lock is obtained
var nx = redisClient.SetValueIfNotExists(key, lockString);
if (nx)
return true;
//If we've gotten here then a key for the lock is present. This could be because the lock is
//correctly acquired or it could be because a client that had acquired the lock crashed (or didn't release it properly).
//Therefore we need to get the value of the lock to see when it should expire
redisClient.Watch(key);
var lockExpireString = redisClient.Get<string>(key);
if (!long.TryParse(lockExpireString, out var lockExpireTime))
{
redisClient.UnWatch(); // since the client is scoped externally
return false;
}
//If the expire time is greater than the current time then we can't let the lock go yet
if (lockExpireTime > DateTime.UtcNow.ToUnixTimeMs())
{
redisClient.UnWatch(); // since the client is scoped externally
return false;
}
//If the expire time is less than the current time then it wasn't released properly and we can attempt to
//acquire the lock. The above call to Watch(_lockKey) enrolled the key in monitoring, so if it changes
//before we call Commit() below, the Commit will fail and return false, which means that another thread
//was able to acquire the lock before we finished processing.
using (var trans = redisClient.CreateTransaction()) // we started the "Watch" above; this tx will succeed if the value has not moved
{
trans.QueueCommand(r => r.Set(key, lockString));
return trans.Commit(); //returns false if Transaction failed
}
},timeOut);
Because StackExchange.Redis Library equivalent only use string.set capabilities. despite the fact that that ServiceStackRedis locking mechanism is more advanced and will wait and retry until get a lock but still not using timeouts of REDIS SET command itself is not justified.
public bool LockTake(RedisKey key, RedisValue value, TimeSpan expiry, CommandFlags flags = CommandFlags.None) {
if (value.IsNull) throw new ArgumentNullException(nameof(value));
return StringSet(key, value, expiry, When.NotExists, flags);
}
Related
Our web app allows the end-user to set the queue of recurring jobs on the UI. (We create a queue for each server (use server name) and allow users to choose server to run)
How the job is registered:
RecurringJob.AddOrUpdate<IMyTestJob>(input.Id, x => x.Run(), input.Cron, TimeZoneInfo.Local, input.QueueName);
It worked properly, but sometimes we check the log on Production and found that it runs on the wrong queue (server). We don't have more access to Production so that we try to reproduce at Development but it's not happened.
To temporarily fix this issue, we need to get the queue name when the job running, then compare it with the current server name and stop it when they are diferent.
Is it possible and how to get it from PerformContext?
Noted: We use HangFire version: 1.7.9 and ASP.NET Core 3.1
You may have a look at https://github.com/HangfireIO/Hangfire/pull/502
A dedicated filter intercepts the queue changes and restores the original queue.
I guess you can just stop the execution in a very similar filter, or set a parameter to cleanly stop execution during the IElectStateFilter.OnStateElection phase by changing the CandidateState to FailedState
Maybe your problem comes from an already existing filter which messes up with the queues.
Here is the code from the link above :
public class PreserveOriginalQueueAttribute : JobFilterAttribute, IApplyStateFilter
{
public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
var enqueuedState = context.NewState as EnqueuedState;
// Activating only when enqueueing a background job
if (enqueuedState != null)
{
// Checking if an original queue is already set
var originalQueue = JobHelper.FromJson<string>(context.Connection.GetJobParameter(
context.BackgroundJob.Id,
"OriginalQueue"));
if (originalQueue != null)
{
// Override any other queue value that is currently set (by other filters, for example)
enqueuedState.Queue = originalQueue;
}
else
{
// Queueing for the first time, we should set the original queue
context.Connection.SetJobParameter(
context.BackgroundJob.Id,
"OriginalQueue",
JobHelper.ToJson(enqueuedState.Queue));
}
}
}
public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
{
}
}
I have found the simple solution: since we have known the Recurring Job Id, we can get its information from JobStorage and compare it with the current queue (current server name):
public bool IsCorrectQueue()
{
List<RecurringJobDto> recurringJobs = Hangfire.JobStorage.Current.GetConnection().GetRecurringJobs();
var myJob = recurringJobs.FirstOrDefault(x => x.Id.Equals("My job Id"));
var definedQueue = myJob.Queue;
var currentServerQueue = string.Concat(Environment.MachineName.ToLowerInvariant().Where(char.IsLetterOrDigit));
return definedQueue == "default" || definedQueue == currentServerQueue;
}
Then check it inside the job:
public async Task Run()
{
//Check correct queue
if (!IsCorrectQueue())
{
Logger.Error("Wrong queue detected");
return;
}
//Job logic
}
We are using the StackExchange.Redis ConnectionMultiplexer class as follows:
private void InitializeConnection()
{
_logger.Info("Initializing a connection to the Redis cluster. ");
bool isReconnectionAttempt = false;
if (_connectionMultiplexer != null)
{
Debug.WriteLine("disposing " + _connectionMultiplexer.GetHashCode());
_connectionMultiplexer.ConnectionFailed -= HandleConnectionFailedEvent;
// test this change.....
_connectionMultiplexer.Close();
isReconnectionAttempt = true;
_logger.Info("This is reconnection attempt to the Redis cluster.");
}
_connectionMultiplexer = ConnectionMultiplexer.Connect(_connectionString);
_needConnect = !_connectionMultiplexer.IsConnected;
_connectionMultiplexer.ConnectionFailed += HandleConnectionFailedEvent;
When I simulate a network issue, the ConnectionFailed event is fired as expected. When this happens, we attempt to dispose the object and create a new one. However, even after the previous _connectionMultiplexer instance was closed/disposed, we still get several ConnectionFailed events fired by the previous instance.
The document, however, indicates that we only get a single ConnectionFailed event when the network goes down. The next time we get such an event is when the network goes down again. But this is not what I am experiencing. Ideas?
I am receiving messages on UDP in multiple threads. After each reception I raise MessageReceived.OnNext(message).
Because I am using multiple threads the messages raised unordered which is a problem.
How can I order the raise of the messages by the message counter?
(lets say there is a message.counter property)
Must take in mind a message can get lost in the communication (lets say if we have a counter hole after X messages that the hole is not filled I raise the next message)
Messages must be raised ASAP (if the next counter received)
In stating the requirement for detecting lost messages, you haven't considered the possibility of the last message not arriving; I've added a timeoutDuration which flushes the buffered messages if nothing arrives in the given time - you may want to consider this an error instead, see the comments for how to do this.
I will solve this by defining an extension method with the following signature:
public static IObservable<TSource> Sort<TSource>(
this IObservable<TSource> source,
Func<TSource, int> keySelector,
TimeSpan timeoutDuration = new TimeSpan(),
int gapTolerance = 0)
source is the stream of unsorted messages
keySelector is a function that extracts an int key from a message. I assume the first key sought is 0; amend if necessary.
timeoutDuration is discussed above, if omitted, there is no timeout
tolerance is the maximum number of messages held back while waiting for an out of order message. Pass 0 to hold any number of messages
scheduler is the scheduler to use for the timeout and is supplied for test purposes, a default is used if not given.
Walkthrough
I'll present a line-by-line walkthrough here. The full implementation is repeated below.
Assign Default Scheduler
First of all we must assign a default scheduler if none was supplied:
scheduler = scheduler ?? Scheduler.Default;
Arrange Timeout
Now if a time out was requested, we will replace the source with a copy that will simply terminate and send OnCompleted if a message doesn't arrive in timeoutDuration.
if(timeoutDuration != TimeSpan.Zero)
source = source.Timeout(
timeoutDuration,
Observable.Empty<TSource>(),
scheduler);
If you wish to send a TimeoutException instead, just delete the second parameter to Timeout - the empty stream, to select an overload that does this. Note we can safely share this with all subscribers, so it is positioned outside the call to Observable.Create.
Create Subscribe handler
We use Observable.Create to build our stream. The lambda function that is the argument to Create is invoked whenever a subscription occurs and we are passed the calling observer (o). Create returns our IObservable<T> so we return it here.
return Observable.Create<TSource>(o => { ...
Initialize some variables
We will track the next expected key value in nextKey, and create a SortedDictionary to hold the out of order messages until they can be sent.
int nextKey = 0;
var buffer = new SortedDictionary<int, TSource>();
Subscribe to the source, and handle messages
Now we can subscribe to the message stream (possibly with the timeout applied). First we introduce the OnNext handler. The next message is assigned to x:
return source.Subscribe(x => { ...
We invoke the keySelector function to extract the key from the message:
var key = keySelector(x);
If the message has an old key (because it exceeded our tolerance for out of order messages) we are just going to drop it and be done with this message (you may want to act differently):
// drop stale keys
if(key < nextKey) return;
Otherwise, we might have the expected key, in which case we can increment nextKey send the message:
if(key == nextKey)
{
nextKey++;
o.OnNext(x);
}
Or, we might have an out of order future message, in which case we must add it to our buffer. If we do this, we must also ensure our buffer hasn't exceeded our tolerance for storing out of order messages - in this case, we will also bump nextKey to the first key in the buffer which because it is a SortedDictionary is conveniently the next lowest key:
else if(key > nextKey)
{
buffer.Add(key, x);
if(gapTolerance != 0 && buffer.Count > gapTolerance)
nextKey = buffer.First().Key;
}
Now regardless of the outcome above, we need to empty the buffer of any keys that are now ready to go. We use a helper method for this. Note that it adjusts nextKey so we must be careful to pass it by reference. We simply loop over the buffer reading, removing and sending messages as long as the keys follow on from each other, incrementing nextKey each time:
private static void SendNextConsecutiveKeys<TSource>(
ref int nextKey,
IObserver<TSource> observer,
SortedDictionary<int, TSource> buffer)
{
TSource x;
while(buffer.TryGetValue(nextKey, out x))
{
buffer.Remove(nextKey);
nextKey++;
observer.OnNext(x);
}
}
Dealing with errors
Next we supply an OnError handler - this will just pass through any error, including the Timeout exception if you chose to go that way.
Flushing the buffer
Finally, we must handle OnCompleted. Here I have opted to empty the buffer - this would be necessary if an out of order message held up messages and never arrived. This is why we need a timeout:
() => {
// empty buffer on completion
foreach(var item in buffer)
o.OnNext(item.Value);
o.OnCompleted();
});
Full Implementation
Here is the full implementation.
public static IObservable<TSource> Sort<TSource>(
this IObservable<TSource> source,
Func<TSource, int> keySelector,
int gapTolerance = 0,
TimeSpan timeoutDuration = new TimeSpan(),
IScheduler scheduler = null)
{
scheduler = scheduler ?? Scheduler.Default;
if(timeoutDuration != TimeSpan.Zero)
source = source.Timeout(
timeoutDuration,
Observable.Empty<TSource>(),
scheduler);
return Observable.Create<TSource>(o => {
int nextKey = 0;
var buffer = new SortedDictionary<int, TSource>();
return source.Subscribe(x => {
var key = keySelector(x);
// drop stale keys
if(key < nextKey) return;
if(key == nextKey)
{
nextKey++;
o.OnNext(x);
}
else if(key > nextKey)
{
buffer.Add(key, x);
if(gapTolerance != 0 && buffer.Count > gapTolerance)
nextKey = buffer.First().Key;
}
SendNextConsecutiveKeys(ref nextKey, o, buffer);
},
o.OnError,
() => {
// empty buffer on completion
foreach(var item in buffer)
o.OnNext(item.Value);
o.OnCompleted();
});
});
}
private static void SendNextConsecutiveKeys<TSource>(
ref int nextKey,
IObserver<TSource> observer,
SortedDictionary<int, TSource> buffer)
{
TSource x;
while(buffer.TryGetValue(nextKey, out x))
{
buffer.Remove(nextKey);
nextKey++;
observer.OnNext(x);
}
}
Test Harness
If you include nuget rx-testing in a console app, the following will run given you a test harness to play with:
public static void Main()
{
var tests = new Tests();
tests.Test();
}
public class Tests : ReactiveTest
{
public void Test()
{
var scheduler = new TestScheduler();
var xs = scheduler.CreateColdObservable(
OnNext(100, 0),
OnNext(200, 2),
OnNext(300, 1),
OnNext(400, 4),
OnNext(500, 5),
OnNext(600, 3),
OnNext(700, 7),
OnNext(800, 8),
OnNext(900, 9),
OnNext(1000, 6),
OnNext(1100, 12),
OnCompleted(1200, 0));
//var results = scheduler.CreateObserver<int>();
xs.Sort(
keySelector: x => x,
gapTolerance: 2,
timeoutDuration: TimeSpan.FromTicks(200),
scheduler: scheduler).Subscribe(Console.WriteLine);
scheduler.Start();
}
}
Closing comments
There's all sorts of interesting alternative approaches here. I went for this largely imperative approach because I think it's easiest to follow - but there's probably some fancy grouping shenanigans you can employ to do this to. One thing I know to be consistently true about Rx - there's always many ways to skin a cat!
I'm also not entirely comfortable with the timeout idea here - in a production system, I would want to implement some means of checking connectivity, such as a heartbeat or similar. I didn't get into this because obviously it will be application specific. Also, heartbeats have been discussed on these boards and elsewhere before (such as on my blog for example).
Strongly consider using TCP instead if you want reliable ordering - that's what it's for; otherwise, you'll be forced to play a guessing game with UDP and sometimes you'll be wrong.
For example, imagine that you receive the following datagrams in this order: [A, B, D]
When you receive D, how long should you wait for C to arrive before pushing D?
Whatever duration you choose you may be wrong:
What if C was lost during transmission and so it will never arrive?
What if the duration you chose is too short and you end up pushing D but then receive C?
Perhaps you could choose a duration that heuristically works best, but why not just use TCP instead?
Side Note:
MessageReceived.OnNext implies that you're using a Subject<T>, which is probably unnecessary. Consider converting the async UdpClient methods into observables directly instead, or convert them by writing an async iterator via Observable.Create<T>(async (observer, cancel) => { ... }).
I'm attempting to implement DLM using the locking mechanisms provided by the ServiceStack-Redis library and described here, but I'm finding that the API seems to present a race condition which will sometimes grant the same lock to multiple clients.
BasicRedisClientManager mgr = new BasicRedisClientManager(redisConnStr);
using(var client = mgr.GetClient())
{
client.Remove("touchcount");
client.Increment("touchcount", 0);
}
Random rng = new Random();
Action<object> simulatedDistributedClientCode = (clientId) => {
using(var redisClient = mgr.GetClient())
{
using(var mylock = redisClient.AcquireLock("mutex", TimeSpan.FromSeconds(2)))
{
long touches = redisClient.Get<long>("touchcount");
Debug.WriteLine("client{0}: I acquired the lock! (touched: {1}x)", clientId, touches);
if(touches > 0) {
Debug.WriteLine("client{0}: Oh, but I see you've already been here. I'll release it.", clientId);
return;
}
int arbitraryDurationOfExecutingCode = rng.Next(100, 2500);
Thread.Sleep(arbitraryDurationOfExecutingCode); // do some work of arbitrary duration
redisClient.Increment("touchcount", 1);
}
Debug.WriteLine("client{0}: Okay, I released my lock, your turn now.", clientId);
}
};
Action<Task> exceptionWriter = (t) => {if(t.IsFaulted) Debug.WriteLine(t.Exception.InnerExceptions.First());};
int arbitraryDelayBetweenClients = rng.Next(5, 500);
var clientWorker1 = new Task(simulatedDistributedClientCode, 1);
var clientWorker2 = new Task(simulatedDistributedClientCode, 2);
clientWorker1.Start();
Thread.Sleep(arbitraryDelayBetweenClients);
clientWorker2.Start();
Task.WaitAll(
clientWorker1.ContinueWith(exceptionWriter),
clientWorker2.ContinueWith(exceptionWriter)
);
using(var client = mgr.GetClient())
{
var finaltouch = client.Get<long>("touchcount");
Console.WriteLine("Touched a total of {0}x.", finaltouch);
}
mgr.Dispose();
When running the above code to simulate two clients attempting the same operation within short succession of one another, there are three possible outputs. The first one is the optimal case where the Mutex works properly and the clients proceed in the proper order. The second case is when the 2nd client times out waiting to acquire a lock; also an acceptable outcome. The problem, however, is that as arbitraryDurationOfExecutingCode approaches or exceeds the timeout for acquiring a lock, it is quite easy to reproduce a situation where the 2nd client is granted the lock BEFORE the 1st client releases it, producing output like this:
client1: I acquired the lock! (touched: 0x)
client2: I acquired the lock! (touched: 0x)
client1: Okay, I released my lock, your turn now.
client2: Okay, I released my lock, your turn now.
Touched a total of 2x.
My understanding of the API and its documentation is that the timeOut argument when acquiring a lock is meant to be just that -- the timeout for getting the lock. If I have to guess at a timeOut value that is high enough to always be longer than the duration of my executing code just to prevent this condition, that seems pretty error prone. Does anyone have a work around other than passing null to wait on locks forever? I definitely don't want to do that or I know I'll end up with ghost locks from crashed workers.
The answer from mythz (thanks for the prompt response!) confirms that the built-in AcquireLock method in ServiceStack.Redis doesn't draw a distinction between the lock acquisition period versus the lock expiration period. For our purposes, we have existing code that expected the distributed locking mechanism to fail quickly if the lock was taken, but allow for long-running processes within the lock scope. To accommodate these requirements, I derived this variation on the ServiceStack RedisLock that allows a distinction between the two.
// based on ServiceStack.Redis.RedisLock
// https://github.com/ServiceStack/ServiceStack.Redis/blob/master/src/ServiceStack.Redis/RedisLock.cs
internal class RedisDlmLock : IDisposable
{
public static readonly TimeSpan DefaultLockAcquisitionTimeout = TimeSpan.FromSeconds(30);
public static readonly TimeSpan DefaultLockMaxAge = TimeSpan.FromHours(2);
public const string LockPrefix = ""; // namespace lock keys if desired
private readonly IRedisClient _client; // note that the held reference to client means lock scope should always be within client scope
private readonly string _lockKey;
private string _lockValue;
/// <summary>
/// Acquires a distributed lock on the specified key.
/// </summary>
/// <param name="redisClient">The client to use to acquire the lock.</param>
/// <param name="key">The key to acquire the lock on.</param>
/// <param name="acquisitionTimeOut">The amount of time to wait while trying to acquire the lock. Defaults to <see cref="DefaultLockAcquisitionTimeout"/>.</param>
/// <param name="lockMaxAge">After this amount of time expires, the lock will be invalidated and other clients will be allowed to establish a new lock on the same key. Deafults to <see cref="DefaultLockMaxAge"/>.</param>
public RedisDlmLock(IRedisClient redisClient, string key, TimeSpan? acquisitionTimeOut = null, TimeSpan? lockMaxAge = null)
{
_client = redisClient;
_lockKey = LockPrefix + key;
ExecExtensions.RetryUntilTrue(
() =>
{
//Modified from ServiceStack.Redis.RedisLock
//This pattern is taken from the redis command for SETNX http://redis.io/commands/setnx
//Calculate a unix time for when the lock should expire
lockMaxAge = lockMaxAge ?? DefaultLockMaxAge; // hold the lock for the default amount of time if not specified.
DateTime expireTime = DateTime.UtcNow.Add(lockMaxAge.Value);
_lockValue = (expireTime.ToUnixTimeMs() + 1).ToString(CultureInfo.InvariantCulture);
//Try to set the lock, if it does not exist this will succeed and the lock is obtained
var nx = redisClient.SetEntryIfNotExists(_lockKey, _lockValue);
if (nx)
return true;
//If we've gotten here then a key for the lock is present. This could be because the lock is
//correctly acquired or it could be because a client that had acquired the lock crashed (or didn't release it properly).
//Therefore we need to get the value of the lock to see when it should expire
string existingLockValue = redisClient.Get<string>(_lockKey);
long lockExpireTime;
if (!long.TryParse(existingLockValue, out lockExpireTime))
return false;
//If the expire time is greater than the current time then we can't let the lock go yet
if (lockExpireTime > DateTime.UtcNow.ToUnixTimeMs())
return false;
//If the expire time is less than the current time then it wasn't released properly and we can attempt to
//acquire the lock. This is done by setting the lock to our timeout string AND checking to make sure
//that what is returned is the old timeout string in order to account for a possible race condition.
return redisClient.GetAndSetEntry(_lockKey, _lockValue) == existingLockValue;
},
acquisitionTimeOut ?? DefaultLockAcquisitionTimeout // loop attempting to get the lock for this amount of time.
);
}
public override string ToString()
{
return String.Format("RedisDlmLock:{0}:{1}", _lockKey, _lockValue);
}
public void Dispose()
{
try
{
// only remove the entry if it still contains OUR value
_client.Watch(_lockKey);
var currentValue = _client.Get<string>(_lockKey);
if (currentValue != _lockValue)
{
_client.UnWatch();
return;
}
using (var tx = _client.CreateTransaction())
{
tx.QueueCommand(r => r.Remove(_lockKey));
tx.Commit();
}
}
catch (Exception ex)
{
// log but don't throw
}
}
}
To simplify use as much as possible, I also expose some extension methods for IRedisClient to parallel the AcquireLock method, along these lines:
internal static class RedisClientLockExtensions
{
public static IDisposable AcquireDlmLock(this IRedisClient client, string key, TimeSpan timeOut, TimeSpan maxAge)
{
return new RedisDlmLock(client, key, timeOut, maxAge);
}
}
Your question highlights the behavior of Distributed Locking in ServiceStack.Redis, if the timeout specified is exceeded, the timed-out clients treats it as an invalid lock and will attempt to auto-recover the lock. If there was no auto-recovery behavior a crashed client would never release the lock and no further operations waiting on that lock would be allowed through.
The locking behavior for AcquireLock is encapsulated in the RedisLock class:
public IDisposable AcquireLock(string key, TimeSpan timeOut)
{
return new RedisLock(this, key, timeOut);
}
Which you can take a copy of and modify to suit the behavior you'd prefer:
using (new MyRedisLock(client, key, timeout))
{
//...
}
If I'm connected to RabbitMQ and listening for events using an EventingBasicConsumer, how can I tell if I've been disconnected from the server?
I know there is a Shutdown event, but it doesn't fire if I unplug my network cable to simulate a failure.
I've also tried the ModelShutdown event, and CallbackException on the model but none seem to work.
EDIT-----
The one I marked as the answer is correct, but it was only part of the solution for me. There is also HeartBeat functionality built into RabbitMQ. The server specifies it in the configuration file. It defaults to 10 minutes but of course you can change that.
The client can also request a different interval for the heartbeat by setting the RequestedHeartbeat value on the ConnectionFactory instance.
I'm guessing that you're using the C# library? (but even so I think the others have a similar event).
You can do the following:
public class MyRabbitConsumer
{
private IConnection connection;
public void Connect()
{
connection = CreateAndOpenConnection();
connection.ConnectionShutdown += connection_ConnectionShutdown;
}
public IConnection CreateAndOpenConnection() { ... }
private void connection_ConnectionShutdown(IConnection connection, ShutdownEventArgs reason)
{
}
}
This is an example of it, but the marked answer is what lead me to this.
var factory = new ConnectionFactory
{
HostName = "MY_HOST_NAME",
UserName = "USERNAME",
Password = "PASSWORD",
RequestedHeartbeat = 30
};
using (var connection = factory.CreateConnection())
{
connection.ConnectionShutdown += (o, e) =>
{
//handle disconnect
};
using (var model = connection.CreateModel())
{
model.ExchangeDeclare(EXCHANGE_NAME, "topic");
var queueName = model.QueueDeclare();
model.QueueBind(queueName, EXCHANGE_NAME, "#");
var consumer = new QueueingBasicConsumer(model);
model.BasicConsume(queueName, true, consumer);
while (!stop)
{
BasicDeliverEventArgs args;
consumer.Queue.Dequeue(5000, out args);
if (stop) return;
if (args == null) continue;
if (args.Body.Length == 0) continue;
Task.Factory.StartNew(() =>
{
//Do work here on different thread then this one
}, TaskCreationOptions.PreferFairness);
}
}
}
A few things to note about this.
I'm using # for the topic. This grabs everything. Usually you want to limit by a topic.
I'm setting a variable called "stop" to determine when the process should end. You'll notice the loop runs forever until that variable is true.
The Dequeue waits 5 seconds then leaves without getting data if there is no new message. This is to ensure we listen for that stop variable and actually quit at some point. Change the value to your liking.
When a message comes in I spawn the handling code on a new thread. The current thread is being reserved for just listening to the rabbitmq messages and if a handler takes too long to process I don't want it slowing down the other messages. You may or may not need this depending on your implementation. Be careful however writing the code to handle the messages. If it takes a minute to run and your getting messages at sub-second times you will run out of memory or at least into severe performance issues.