We're using WCF to build a simple web service which our product uses to upload large files over a WAN link. It's supposed to be a simple HTTP PUT, and it's working fine for the most part.
Here's a simplified version of the service contract:
[ServiceContract, XmlSerializerFormat]
public interface IReplicationWebService
{
[OperationContract]
[WebInvoke(Method = "PUT", UriTemplate = "agents/{sourceName}/epoch/{guid}/{number}/{type}")]
ReplayResult PutEpochFile(string sourceName, string guid, string number, string type, Stream stream);
}
In the implementation of this contract, we read data from stream and write it out to a file. This works great, so we added some error handling for cases when there's not enough disk space to store the file. Here's roughly what it looks like:
public ReplayResult PutEpochFile(string sourceName, string guid, string number, string type, Stream inStream)
{
//Stuff snipped
try
{
//Read from the stream and write to the file
}
catch (IOException ioe)
{
//IOException may mean no disk space
try
{
inStream.Close();
}
// if instream caused the IOException, close may throw
catch
{
}
_logger.Debug(ioe.ToString());
throw new FaultException<IOException>(ioe, new FaultReason(ioe.Message), new FaultCode("IO"));
}
}
To test this, I'm sending a 100GB file to a server that doesn't have enough space for the file. As expected this throws an exception, but the call to inStream.Close() appeared to hang. I checked into it, and what's actually happening is that the call to Close() made its way through the WCF plumbing until it reached System.ServiceModel.Channels.DrainOnCloseStream.Close(), which according to Reflector allocates a Byte[] buffer and keeps reading from the stream until it's at EOF.
In other words, the Close call is reading the entire 100GB of test data from the stream before returning!
Now it may be that I don't need to call Close() on this stream. If that's the case I'd like an explanation as to why. But more importantly, I'd appreciate it if anyone could explain to me why Close() is behaving this way, why it's not considered a bug, and how to reconfigure WCF so that doesn't happen.
.Close() is intended to be a "safe" and "friendly" way of stopping your operation - and it will indeed complete the currently running requests before shutting down - by design.
If you want to throw down the sledgehammer, use .Abort() on your client proxy (or service host) instead. That just shuts down everything without checking and without being nice about waiting for operations to complete.
Related
Problem Statement
Context
I'm a Software Engineer in Test running order permutations of Restaurant Menu Items to confirm that they succeed order placement w/ the POS
In short, this POSTs a JSON payload to an endpoint which then validates the order w/ a POS to define success/fail/other
Where POS, and therefore Transactions per Second (TPS), may vary, but each Back End uses the same core handling
This can be as high as ~22,000 permutations per item, in easily manageable JSON size, that need to be handled as quickly as possible
The Network can vary wildly depending upon the Restaurant, and/or Region, one is testing
E.g. where some have a much higher latency than others
Therefore, the HTTPClient should be able to intelligently negotiate the same content & endpoint regardless of this
Direct Problem
I'm using Apache's HTTP Client 5 w/ PoolingAsyncClientConnectionManager to execute both the GET for the Menu contents, and the POST to check if the order succeeds
This works out of the box, but sometimes loses connections w/ Stream Refused, specifically:
org.apache.hc.core5.http2.H2StreamResetException: Stream refused
No individual tuning seems to work across all network contexts w/ variable latency, that I can find
Following the stacktrace seems to indicate it is that the stream had closed already, therefore needs a way to keep it open or not execute an already-closed connection
if (connState == ConnectionHandshake.GRACEFUL_SHUTDOWN) {
throw new H2StreamResetException(H2Error.PROTOCOL_ERROR, "Stream refused");
}
Some Attempts to Fix Problem
Tried to use Search Engines to find answers but there are few hits for HTTPClient5
Tried to use official documentation but this is sparse
Changing max connections per route to a reduced number, shifting inactivity validations, or connection time to live
Where the inactivity checks may fix the POST, but stall the GET for some transactions
And that tuning for one region/restaurant may work for 1 then break for another, w/ only the Network as variable
PoolingAsyncClientConnectionManagerBuilder builder = PoolingAsyncClientConnectionManagerBuilder
.create()
.setTlsStrategy(getTlsStrategy())
.setMaxConnPerRoute(12)
.setMaxConnTotal(12)
.setValidateAfterInactivity(TimeValue.ofMilliseconds(1000))
.setConnectionTimeToLive(TimeValue.ofMinutes(2))
.build();
Shifting to a custom RequestConfig w/ different timeouts
private HttpClientContext getHttpClientContext() {
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(Timeout.of(10, TimeUnit.SECONDS))
.setResponseTimeout(Timeout.of(10, TimeUnit.SECONDS))
.build();
HttpClientContext httpContext = HttpClientContext.create();
httpContext.setRequestConfig(requestConfig);
return httpContext;
}
Initial Code Segments for Analysis
(In addition to the above segments w/ change attempts)
Wrapper handling to init and get response
public SimpleHttpResponse getFullResponse(String url, PoolingAsyncClientConnectionManager manager, SimpleHttpRequest req) {
try (CloseableHttpAsyncClient httpclient = getHTTPClientInstance(manager)) {
httpclient.start();
CountDownLatch latch = new CountDownLatch(1);
long startTime = System.currentTimeMillis();
Future<SimpleHttpResponse> future = getHTTPResponse(url, httpclient, latch, startTime, req);
latch.await();
return future.get();
} catch (IOException | InterruptedException | ExecutionException e) {
e.printStackTrace();
return new SimpleHttpResponse(999, CommonUtils.getExceptionAsMap(e).toString());
}
}
With actual handler and probing code
private Future<SimpleHttpResponse> getHTTPResponse(String url, CloseableHttpAsyncClient httpclient, CountDownLatch latch, long startTime, SimpleHttpRequest req) {
return httpclient.execute(req, getHttpContext(), new FutureCallback<SimpleHttpResponse>() {
#Override
public void completed(SimpleHttpResponse response) {
latch.countDown();
logger.info("[{}][{}ms] - {}", response.getCode(), getTotalTime(startTime), url);
}
#Override
public void failed(Exception e) {
latch.countDown();
logger.error("[{}ms] - {} - {}", getTotalTime(startTime), url, e);
}
#Override
public void cancelled() {
latch.countDown();
logger.error("[{}ms] - request cancelled for {}", getTotalTime(startTime), url);
}
});
}
Direct Question
Is there a way to configure the client such that it can handle for these variances on its own without explicitly modifying the configuration for each endpoint context?
Fixed w/ Combination of the below to Assure Connection Live/Ready
(Or at least is stable)
Forcing HTTP 1
HttpAsyncClients.custom()
.setConnectionManager(manager)
.setRetryStrategy(getRetryStrategy())
.setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_1)
.setConnectionManagerShared(true);
Setting Effective Headers for POST
Specifically the close header
req.setHeader("Connection", "close, TE");
Note: Inactivity check helps, but still sometimes gets refusals w/o this
Setting Inactivity Checks by Type
Set POSTs to validate immediately after inactivity
Note: Using 1000 for both caused a high drop rate for some systems
PoolingAsyncClientConnectionManagerBuilder
.create()
.setValidateAfterInactivity(TimeValue.ofMilliseconds(0))
Set GET to validate after 1s
PoolingAsyncClientConnectionManagerBuilder
.create()
.setValidateAfterInactivity(TimeValue.ofMilliseconds(1000))
Given the Error Context
Tracing the connection problem in stacktrace to AbstractH2StreamMultiplexer
Shows ConnectionHandshake.GRACEFUL_SHUTDOWN as triggering the stream refusal
if (connState == ConnectionHandshake.GRACEFUL_SHUTDOWN) {
throw new H2StreamResetException(H2Error.PROTOCOL_ERROR, "Stream refused");
}
Which corresponds to
connState = streamMap.isEmpty() ? ConnectionHandshake.SHUTDOWN : ConnectionHandshake.GRACEFUL_SHUTDOWN;
Reasoning
If I'm understanding correctly:
The connections were being un/intentionally closed
However, they were not being confirmed ready before executing again
Which caused it to fail because the stream was not viable
Therefore the fix works because (it seems)
Given Forcing HTTP1 allows for a single context to manage
Where HttpVersionPolicy NEGOTIATE/FORCE_HTTP_2 had greater or equivalent failures across the spectrum of regions/menus
And it assures that all connections are valid before use
And POSTs are always closed due to the close header, which is unavailable to HTTP2
Therefore
GET is checked for validity w/ reasonable periodicity
POST is checked every time, and since it is forcibly closed, it is re-acquired before execution
Which leaves no room for unexpected closures
And otherwise the potential that it was incorrectly switching to HTTP2
Will accept this until a better answer comes along, as this is stable but sub-optimal.
According to Microsoft's samples, here's how one would go about streaming a file throuhg WCF:
// Service class which implements the service contract
public class StreamingService : IStreamingSample
{
public System.IO.Stream GetStream(string data)
{
//this file path assumes the image is in
// the Service folder and the service is executing
// in service/bin
string filePath = Path.Combine(
System.Environment.CurrentDirectory,
".\\image.jpg");
//open the file, this could throw an exception
//(e.g. if the file is not found)
//having includeExceptionDetailInFaults="True" in config
// would cause this exception to be returned to the client
try
{
FileStream imageFile = File.OpenRead(filePath);
return imageFile;
}
catch (IOException ex)
{
Console.WriteLine(
String.Format("An exception was thrown while trying to open file {0}", filePath));
Console.WriteLine("Exception is: ");
Console.WriteLine(ex.ToString());
throw ex;
}
}
...
Now, how do I know who's responsible for releasing the FileStream when the transfer is done?
EDIT: If the code is put inside a "using" block the stream gets shut down before the client receives anything.
The service should clean up and not the client. WCF's default for OperationBehaviorAttribute.AutoDisposeParameters seems to be true, therefore it should do the disposing for you. Although there doesn't seem to be a fixed answer on this.
You could try using the OperationContext.OperationCompleted Event:
OperationContext clientContext = OperationContext.Current;
clientContext.OperationCompleted += new EventHandler(delegate(object sender, EventArgs args)
{
if (fileStream != null)
fileStream.Dispose();
});
Put that before your return.
Check this blog
Short answer: the calling code, via a using block.
Long answer: sample code should never be held up as an exemplar of good practice, it's only there to illustrate one very specific concept. Real code would never have a try block like that, it adds no value to the code. Errors should be logged at the topmost level, not down in the depths. Bearing that in mind, the sample becomes a single expression, File.OpenRead(filePath), that would be simply plugged into the using block that requires it.
UPDATE (after seeing more code):
Just return the stream from the function, WCF will decide when to dispose it.
The stream needs to be closed by party who is responsible to read it. For example, if service returns the stream to client, it's client application responsibility close the stream as Service doesn't know or have control when client finishes reading stream. Also, WCF will not take care of closing the stream again because of the fact that it doesn't know when receiving party has finished reading. :)
HTH,
Amit Bhatia
I am confusing with a problem about upload blobs asynchronously, hopes find answer here.
Please take a look at my code snippet first,
public void UploadMultipleBlobs(List<string> filelocations, string containerName, AsyncCallback callback = null, string path = null)
{
try
{
Parallel.ForEach(filelocations, fileLocation =>
{
//File to Stream
MemoryStream str = new MemoryStream();
byte[] file = File.ReadAllBytes(fileLocation);
str.Write(file, 0, file.Length);
str.Seek(0, SeekOrigin.Begin);
//Operations
if (callback == null)
callback = new AsyncCallback(OnUploadCompleted);
BlobRequestOptions blobRequestOptions = new BlobRequestOptions();
blobRequestOptions.Timeout = new TimeSpan(1, 0, 0);
blobRequestOptions.RetryPolicy = retry;
CloudBlob currentBlob = container.GetBlobReference(blobName);
var result = currentBlob.BeginUploadFromStream(str, blobRequestOptions, callback, new Object[] { currentBlob, str });
currentBlob.EndUploadFromStream(result);
});
}
catch
{
throw;
}
}
private void OnUploadCompleted(IAsyncResult result)
{
try
{
// Get array passed to callback
Object[] states = (Object[])result.AsyncState;
var blob = (CloudBlob)states[0];
var stream = (MemoryStream)states[1];
// End the operation
//blob.EndUploadFromStream(result);
// Close the stream
stream.Close();
}
catch
{
throw;
}
}
I need to upload mutil files to Azure blob, number of files may be 10-50,000, each file is about 10KB-50KB. The code snippet works fine for me currently. However, if I call EndUploadFromStream in callback, it always throw an exception when uploading over 2,000 files. I mean if I remove EndUploadFromStream in upload method and call EndUploadFromStream in callback(OnUploadCompleted method), the exception happens. The exception message as below:
Unable to read data from the transport connection: The connection was closed., StackTrace: at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.get_Result()
at Microsoft.WindowsAzure.StorageClient.CloudBlob.EndUploadFromStream(IAsyncResult asyncResult)
I don't know why it happens...hopes got answer from you guys.
Thanks.
The Begin/End code looks OK. I notice that you're not doing anything to wait for the asynchronous operations to complete, so the problem may be related to that. eg, if you're running this from a console application then the application may exit before all the uploads have completed and then give you those errors. That would not be a problem if the EndUploadFromStream() call is inside the Parallel.ForEach() because it will cause the Parallel.ForEach() call to block until all the uploads have completed.
So try adding code to wait for all the uploads to complete and see if that fixes it. A simple way would be a counter that is initialized to the total number of uploads, and then decremented (using Interlocked.Decremement() for thread safety) inside of the callback. Another options would be use Task.FromAsync() to get an array of Task objects, then using Task.WaitAll() to wait for them to complete.
As an aside, using both Parallel.ForEach() and Begin/End methods at the same time is usually redundant - Begin/End is asynchronous already so there's usually no point using multiple threads to invoke it. Since you have such a big list of items it might make some difference in this case, but probably not much. You're probably better off using a simple foreach loop instead of Parallel.ForEach() unless you've actually measured a significant difference.
We have a simple wpf application that connects to a service running on the local machine. We use a named pipe for the connection and then register a callback so that later the service can send updates to the client.
The problem is that with each call of the callback we get a build up of memory in the client application.
This is how the client connects to the service.
const string url = "net.pipe://localhost/radal";
_channelFactory = new DuplexChannelFactory<IRadalService>(this, new NetNamedPipeBinding(),url);
and then in a threadpool thread we loop doing the following until we are connected
var service = _channelFactory.CreateChannel();
service.Register();
service.Register looks like this on the server side
public void Register()
{
_callback = OperationContext.Current.GetCallbackChannel<IRadalCallback>();
OperationContext.Current.Channel.Faulted += (sender, args) => Dispose();
OperationContext.Current.Channel.Closed += (sender, args) => Dispose();
}
This callback is stored and when new data arrives we invoke the following on the server side.
void Sensors_OnSensorReading(object sender, SensorReadingEventArgs e)
{
_callback.OnReadingReceived(e.SensorId, e.Count);
}
Where the parameters are an int and a double. On the client this is handled as follows.
public void OnReadingReceived(int sensorId, double count)
{
_events.Publish(new SensorReadingEvent(sensorId, count));
}
But we have found that commenting out _event.Publish... makes no difference to the memory usage. Does anyone see any logical reason why this might be leaking memory. We have used a profiler to track the problem to this point but cannot find what type of object is building up.
Well I can partially answer this now. The problem is partially caused by us trying to be clever and getting the connection to be opened on another thread and then passing it back to the main gui thread. The solution was to not use a thread but instead use a dispatch timer. It does have the downside that the initial data load is now on the GUI thread but we are not loading all that much anyway.
However this was not the entire solution (actually we don't have an entire solution). Once we moved over to a better profiler we found out that the objects building up were timeout handlers so we disabled that feature. That's OK for us as we are running against the localhost always but I can imagine for people working with remote services it would be an issue.
I'm finding mixed answers to my question out in the web. To elaborate on the question:
Should I instantiate a service client proxy once per asynchronous invocation, or once per Silverlight app?
Should I close the service client proxy explicitly (as I do in my ASP.NET MVC application calling WCF services synchronously)?
I've found plenty of bloggers and forum posters out contradicting each other. Can anyone point to any definitive sources or evidence to answer this once and for all?
I've been using Silverlight with WCF since V2 (working with V4 now), and here's what I've found. In general, it works very well to open one client and just use that one client for all communications. And if you're not using the DuplexHttBinding, it also works fine to do just the opposite, to open a new connection each time and then close it when you're done. And because of how Microsoft has architected the WCF client in Silverlight, you're not going to see much performance difference between keeping one client open all the time vs. creating a new client with each request. (But if you're creating a new client with each request, make darned sure you're closing it as well.)
Now, if you're using the DuplexHttBinding, i.e., if you want to call methods on the client from the server, it's of course important that you don't close the client with each request. That's just common sense. However, what none of the documentation tells you, but which I've found to be absolutely critical, is that if you're using the DuplexHttBinding, you should only ever have one instance of the client open at once. Otherwise, you're going to run into all sorts of nasty timeout problems that are going to be really, really hard to troubleshoot. Your life will be dramatically easier if you just have one connection.
The way that I've enforced this in my own code is to run all my connections through a single static DataConnectionManager class that throws an Assert if I try to open a second connection before closing the first. A few snippets from that class:
private static int clientsOpen;
public static int ClientsOpen
{
get
{
return clientsOpen;
}
set
{
clientsOpen = value;
Debug.Assert(clientsOpen <= 1, "Bad things seem to happen when there's more than one open client.");
}
}
public static RoomServiceClient GetRoomServiceClient()
{
ClientsCreated++;
ClientsOpen++;
Logger.LogDebugMessage("Clients created: {0}; Clients open: {1}", ClientsCreated, ClientsOpen);
return new RoomServiceClient(GetDuplexHttpBinding(), GetDuplexHttpEndpoint());
}
public static void TryClientClose(RoomServiceClient client, bool waitForPendingCalls, Action<Exception> callback)
{
if (client != null && client.State != CommunicationState.Closed)
{
client.CloseCompleted += (sender, e) =>
{
ClientsClosed++;
ClientsOpen--;
Logger.LogDebugMessage("Clients closed: {0}; Clients open: {1}", ClientsClosed, ClientsOpen);
if (e.Error != null)
{
Logger.LogDebugMessage(e.Error.Message);
client.Abort();
}
closingIntentionally = false;
if (callback != null)
{
callback(e.Error);
}
};
closingIntentionally = true;
if (waitForPendingCalls)
{
WaitForPendingCalls(() => client.CloseAsync());
}
else
{
client.CloseAsync();
}
}
else
{
if (callback != null)
{
callback(null);
}
}
}
The annoying part, of course, is if you only have one connection, you need to trap for when that connection closes unintentionally and try to reopen it. And then you need to reinitialize all the callbacks that your different classes were registered to handle. It's not really all that difficult, but it's annoying to make sure it's done right. And of course, automated testing of that part is difficult if not impossible . . .
You should open your client per call and close it immediately after. If you in doubt browse using IE to a SVC file and look at the example they have there.
WCF have configuration settings that tells it how long it should wait for a call to return, my thinking is that when it does not complete in the allowed time the AsyncClose will close it. Therefore call client.AsyncClose().