Cleanup files when using NServiceBus FileShareDataBus - nservicebus

Question

My question is kind of similar to this question but I think the response is not answering the question at all.
To elaborate,
I have the following code piece:
Configuration:
BusConfiguration busConfiguration = new BusConfiguration();
busConfiguration.EndpointName("Samples.DataBus.Sender");
busConfiguration.UseSerialization<JsonSerializer>();
busConfiguration.UseDataBus<FileShareDataBus>().BasePath(BasePath);
busConfiguration.UsePersistence<InMemoryPersistence>();
busConfiguration.EnableInstallers();
using (IBus bus = Bus.Create(busConfiguration).Start())
....
Message:
[TimeToBeReceived("00:01:00")]
public class MessageWithLargePayload : ICommand
{
public string SomeProperty { get; set; }
public DataBusProperty<byte[]> LargeBlob { get; set; }
}
This works fine (it creates queues, sends messages in the queue, creates a file for the LargeBlob property and stores it in the base path, receiver takes the message and handles it).
My question is: Is there any way to remove the created files (LargeBlob) after the message has been handled or taken out of the queue or after it lands in the error-queue.
The Documentation clearly states that files are not cleaned up, but I think this is kind of a messy behaviour, can anyone help?

Answer 1

Is there any way to remove the files after the message has been handled or taken out of the queue or after it lands in the error-queue.
After message handled
After taken out of the queue
After it is in error queue
I'm not really sure what you're after? You want to remove the files, but you're not sure when?
NServiceBus has no way to figure out when the file should be deleted. Perhaps you're deferring a message for the file to be processed later. Or you're giving the task to another handler. Which means that if the file is removed, there's no way that other handler can process the file. So removing the file depends on your functional needs.
When the message is in the error queue, it is most likely that you want to try and process it again. Why else put the message in an error queue, instead of just removing the message altogether?
Besides that, the file system isn't transactional. So there's no way for any software to tell if messages got processed correctly and the file should be deleted. And when the outbox has been enabled in NServiceBus, the message is removed from the queuing storage, but it's not been processed yet. If the file would've been removed by then, it also can't be processed anymore.
As you can tell, there are a large number of scenarios where removing the file can pose a problem. The only one who actually knows when which file could be removed, is you as a developer. You'll have to come up with a strategy to remove the files.
The sample has a class Program with a static field BasePath. Make it public so your handler can access it. Then in the handler you can obtain the file location like this:
public void Handle(MessageWithLargePayload message)
{
var filename = Path.Combine(Program.BasePath, message.LargeBlob.Key);
Console.WriteLine(filename);
UPDATE
Added some documentation about a possible cleanup strategy. We have some plans for a really good solution, but it'll take time. So for now perhaps this can help.

Answer 2

I solved the file cleanup challenge by creating a service that removes the files after a configurable number of hours. I will say that with a large number of bus files, you will be more successful if you do a rolling delete vs trying to do it once a day.
There are two options out on GitHub that are already coded that can be used:
PowerShell Script:
https://gist.github.com/smartindale/50674db76bd2e36cc94f
Windows Service: https://github.com/bradjolicoeur/VapoDataBus
The PowerShell script worked with low volumes, but I had issues with running reliably with larger volumes of files.
The Windows Service is reliable on large volumes and is more efficient.