I want export channel messages to ftp server or external drive. I thing we can export messages via rest API. Could you any one help on this..
If you want to send messages to a REST API, you can use the HTTP Sender destination connector type.
If your REST API Endpoint requires any special headers or authentication, you will need to configure this appropriately (such as by setting variables in the Destination Transformer). Don't forget to put something in the "Content" box at the bottom of the screen - this usually has a value such as ${message.transformedData} or ${message.rawData}.
If you want to send messages to an FTP server, you can use the File Writer destination connector type. Again, make sure you put something such as ${message.transformedData} in the "Template" field.
The POST /channels/{channelId}/messages/_export endpoint is to export messages to files on the server filesystem. When the client does an export to the local file system, it basically writes the results of GET /channels/{channelId}/messages with one file per message and attachments included. See Source.
Possibly the most effective way to get all your processed messages offsite is to just take a database backup.
The data pruner also has an option to archive messages to disk as they are pruned, and those files could be picked up and sent offsite if desired.
Related
I am building an application where I need to generate a csv file which would be very large. So I want to process it through Queues ( RabbitMQ in my case).
Here's the requirement:
After user click on download, send the message to the queue.
Process and upload the csv to S3.
Send a notification to user that file upload has been done.
I am stuck with how should the user be notified. I just need the logic of implementation. If it is some generic topic/design , would be glad to know what it is called. Couldnt find anything relevant from my seartch queries.
I have this service that posts JSON messagest to a given url, the requests cannot be modified and because of that I cannot add all the headers I need to post to a queue in Azure.
I have been researching on this but seems like the only way to make a post other from the authorization headers is allowing a range of IPs to post to the Q; something that I cannot do either because the sender can change IPs and I could loose data, not good.
What I'm trying to find out is if Microsoft has a way (or some different service from queues) that I can use to prevent data loss in case my app is down (this is where the queue comes in), or if there's some way I can allow this external provider to post to my queue without all the security or with a minimum of it.
Thanks in advance.
In ActiveMQ while using blob messages we use this as broker
String broker1 = "tcp://localhost:7005?jms.blobTransferPolicy.UploadUrl=http://localhost:7005/fileserver/"
Can anybody explain what is UploadUrl and why we need to configure for blob messages(we don't need to configure for text messages). Why it doesn't allow tcp protocol?
So plain text messages are good and easy to use, but needs to be in memory at all times. It works well with a KBs of data, or even a few MB. However, sending very large files, such as initial data loads, large media files or BI data, is not nice to keep around in memory. There may still be a need to pass the message around, route/filter based on message properties, use transactions and similar.
Blob messages is an attempt to solve the need to pass around GBs of data through the semantics of messaging. The trade off is that you have to define a streaming based server somewhere that both sender and receiver can reach. It can be HTTP, FTP, a local file, WebDAV or similar. ActiveMQ comes with a HTTP based fileserver if you have no other file area around.
I am currently researching the possibility of using NServiceBus for one of our applications. The current application takes large text files and parses the details into a database. The users perform various operations on the file content, approve the changes, and finally release the updated file. When the file is released, various other services need to do something with that file data (drop file in ftp folder, email customer, bill customer).
From what I read services should be autonomous and not share data, except via messages. I like that concept, however in my case I am wondering if it is practical. Some of these files can contain up to a million records.
So my question is, should each service (operations, billing, emailer) all have their own database and table for storing this file data, and move the data via the DataBus? Or should I be more pragmatic and only send the fileID in the message which references a central file table?
Thanks for any guidance you can offer.
There are a couple of things that one should not do with a service bus:
move masses of data
perform queries
large ETL operations
You are certainly able to do all these things but you will probably be left disappointed. The messaging to enable some of these operations is fine, though. Your idea of sending the FileID is definitely the way to go.
As an example: I have previously implemented an e-mail sending service. This service can send attachments but these can be large. So instead of including the attachments in the messages I stored the attachments on a shared folder and sent a SendEMailCommand message that also included the unique attachment ids that need to be sent with the e-mail. The e-mail service would then pick up the attachments from the shared folder. After the service successfully sent the mail an EMailSentEvent message would be published.
What are the essential differences between publishing a message using Bus.Publish and sending a message using Bus.Send? I am looking to understand how they differ and also when I should choose to use one over the other.
Publishing is used to notify multiple Subscribers of a particular event. A Publishing endpoint will have subscription storage to identify where to send messages to. Sending is typically used to issue a command to an endpoint. A command is telling the endpoint to do something and should not expect a reply(although you sometimes do want a reply and NSB supports this).
The reason you do not see a destination for Send() is that you specify the destination via configuration. In your app.config you will map message types(a whole assembly or a class) to a destination. When you do so, you do not have to provide the destination.
Bus.Publish: used when you don't know where the message is going (0 to many subscribers).
Bus.Send: when you are sending a message to a specific handler (client to server).
ususally Context.Publish() is for publishing Event Type and Context.Send() is for Command Type