I'm looking for a way of implementing a file transfer service over HTTPS which uses chunking to cope with intermittent connectivity loss and to reduce the large timeouts required by using Streaming. Because the client may be behind firewalls, the Chunking Channel sample on MSDN isn't suitable.
There is an old discussion about this on the Microsoft Forums but not a complete answer, or at least not one that I have the know-how to implement.
There is a sample of a resumable download service here: http://cid-8d29fb569d8d732f.skydrive.live.com/self.aspx/.Public/WCF/Resume%5E_Download%5E_WCF%5E_1%20%5E52%5E6.zip
This sample uses a custom WCF binding. It looks like it works by getting a segment of the file at a time, with the possibility to get any remaining segments when the system is back on line.
Related
I am very new on RabbitMQ World and Microservices Architecture. I've watched some tutorials on youtube how to use rabbitMQ but there are some questions I would like to ask.
1.) If the client post the data to my api controller and my api controller publish the data into queue, what is the proper way to respond the post request while the data is being processed on rabbitMQ? Because from the tutorial, It just returns "Ok (Http Code 200)" even though the data is not still completed yet.
2.) Can consumer subscribe more than 1 queue? If yes, is there any configuration sample on startup.cs?
3.) Is there any sample using rabbitMQ on .NetCore for "Real World" cases? Please let me know.
Thanks
It's perfectly okey to return 200. That's just one of many tradeoffs for microservices architecture. E.g. from the performance perspective, it's efficient that you can return early with 200 and propagate all the changes asynchronously through the rest of the distributed system. On the other hand, it adds another type of complexity that you need to embrace - Eventual Consistency. This concept kind of describes what you asked about. Let say your client received 200, but if it immediately calls another microservice the client may not see changes introduced by the previous request, because there is a probability that the changes haven't propagated yet. You need to decide whether it's acceptable in your project or not. If not maybe you should redesign how you split your business domain into microservices, trying to group transactionally close to each other entities together in order to mitigate such problems. If you can't really tolerate Eventual Consistency maybe you should give up on microservices for the particular project.
Yes it can, you could for example create an implementation of IHostedService for each queue listening for messages and run them in parallel with your asp net core app by registering them in the starutp.cs
You'll find this in the repository from the below's links. They use RabbitMQ. Although, there's a bit of abstraction which can make it harder to grasp, it's a great implementation with a bonus of being documented in the free ebook.
https://github.com/dotnet-architecture/eShopOnContainers/ - I can't stress enough how this repository helped me with understanding microservices. There's also a free ebook from Microsoft docs about this repo: https://learn.microsoft.com/en-us/dotnet/architecture/microservices/ . They tackle concepts such as Eventual Consistency and asynchronous communication. It's exactly what you look for.
FYI: This will be my first real foray into Async/Await; for too long I've been settling for the familiar territory of BackgroundWorker. It's time to move on.
I wish to build a WCF service, self-hosted in a Windows service running on a remote machine in the same LAN, that does this:
Accepts a request for a single .ZIP archive
Creates the archive and packages several files
Returns the archive as its response to the request
I have to support archives as large as 10GB. Needless to say, this scenario isn't covered by basic WCF designs; we must take additional steps to meet the requirement. We must eliminate timeouts while the archive is building and memory errors while it's being sent. Both of these occur under basic WCF designs, depending on the size of the file returned.
My plan is to proceed using task-based asynchronous WCF calls and streaming mode.
I have two concerns:
Is this the proper approach to the problem?
Microsoft has done a nice job at abstracting all of this, but what of the underlying protocols? What goes on 'under the hood?' Does the server keep the connection alive while the archive is building (could be several minutes) or instead does it close the connection and initiate a new one once the operation is complete, thereby requiring me to properly route the request through the client machine firewall?
For #2, clearly I'm hoping for the former (keep-alive). But after some searching I'm not easily finding an answer. Perhaps you know.
You need streaming for big payloads. That is the right approach. This has nothing at all to do with asynchronous IO. The two are independent. The client cannot even tell that the server is async internally.
I'll add my standard answers for whether to use async IO or not:
https://stackoverflow.com/a/25087273/122718 Why does the EF 6 tutorial use asychronous calls?
https://stackoverflow.com/a/12796711/122718 Should we switch to use async I/O by default?
Each request runs over a single connection that is kept alive. This goes for both streaming big amounts of data as well as big initial delays. Not sure why you are concerned about routing. Does your router kill such connections? That's a problem.
Regarding keep alive, there is nothing going over the wire to do that. TCP sessions can stay open indefinitely without any kind of wire traffic.
basically need to transfer large file between wcf service and java client,
can someone give directions please?
Basically I need to a create wcf service which needs to read blob content(actually a file content stored in db column) and pass it to a java web application(being a client to wcf).
File size may vary from 1kb to 20MB in size.
By now I have already researched/checked below options but still not able to finalize which one i should go with, which is feasible and which is not,
could someone guide me please.
pass file content as byte[]:
I understand it will increase data size passed to client as it will encode data into base 64 format and embed the base 64 encoding into soap message itself and hence makes communication slower and have performance issues.
But this works for sure, but I am not sure if it is advisable to go by this approach.
Share a NetworkDrive/FTPFolder accessible to both client and wcf service App:
Per this File needed by client will first get stored there by wcf and then client needs to use java I/O OR FTP options to read it.
This looks good from data size/bandwidth point of view, but has extra processing at both service and client side (as need to store/read via NetworkShared/FTP folder)
Streaming:
This one I am not sure will be feasible with a Java client, but my understanding is that streaming is supported for Non .net clients but how to go about it i am not sure???
I understand for streaming i need to use basichttp binding, but do i need to use DataContract or MessageContract or any will work, and then what is to be done at java client side, that i am not sure about.
Using MTOM approach for passing large data in soap requests:
This looks actually having support specifically designed to solve large data transfer in web service calls, but have to investigate further on this, as of now I don’t have much idea on this. Does anyone of you have some suggestion on this?
I understand question is bit lengthier but i had to put all 4 options i have tried and my concerns/findings with each, so that you all can suggest among these or new option may be, also you will know what i have already tried and so can direct me in more effective way.
I am in the same position as yourself and I can state from experience that option 1 is a poor choice for anything more than a couple of MB.
In my own system, times to upload increase exponentially, with 25MB files taking in excess of 30mins to upload.
I've run some timings and the bulk of this is in the transfer of the file from the .NET client to the Java Web Service. Our web service is a facade for a set of 3rd party services; using the built in client provided by the 3rd party (not viable in the business context) is significantly faster - less than 5mins for a 25MB file. Upload to our client application is also quick.
We have tried MTOM and, unless we implemented it incorrectly, didn't see huge improvements (under 10% speed increase).
Next port of call will be option 2 - file transfers are relatively quick so by uploading the file directly to one of the web service hosts I'm hoping this will speed things up dramatically - if I get some meaningful results I will add them to my post.
I need to streaming a large content with WCF and deferred execution (using NetTcpBinding), in other words, return a list of person (can be anything) from a database through a WCF service without consume all memory in the server side.
I have tried the solution in this post: Streaming large content with WCF and deferred execution
Using BasicHttpBinding it works like a charm, but when using NetTcpBinding....well...not working.
Can anyone help me with this??
Tks!
Project here: WCF Streaming IEnumerable
Here is guidance on transferring large data with streaming.
http://msdn.microsoft.com/en-us/library/ms733742.aspx
At the bottom of the page is a link to some samples, one of which shows using TCP.
http://msdn.microsoft.com/en-us/library/ms789010.aspx
If this doesn't help, then perhaps you could provide more detail on how it is failing (error message)?
I have a need for an application to access reporting data from a remote database. We currently have a WCF service that handles the I/O for this database. Normally the application just sends small messages back and forth between the WCF service and itself, but now we need to run some historical reports on that activity. The result could be several hundred to a few thousand records. I came across http://msdn.microsoft.com/en-us/library/ms733742.aspx which talks about streaming, but it also mentions segmenting messages, which I didn't find any more information on. What is the best way to send large amounts of data such as this from a WCF service?
It seems my options are streaming or chunking. Streaming restricts other WCF features, message security being one (http://msdn.microsoft.com/en-us/library/ms733742.aspx). Chunking is breaking up a message into pieces then putting those pieces back together at the client. This can be done by implementing a custom Channel which MS has provided an example of here: http://msdn.microsoft.com/en-us/library/aa717050.aspx. This is implemented below the security layer so security can still be used.