My application is querying my server every few seconds for updates.
After leaving it running for about 3 days I observed that the app crashed with the following stack trace.
As you may know, when getting an exception in a working thread, it can't be caught, hence my app crashed.
System.Net.WebException: The request was canceled
System.Net.ServicePointManager.FindServicePoint(Uri address, IWebProxy proxy, ProxyChain& chain, HttpAbortDelegate& abortDelegate, Int32& abortState)
System.Net.HttpWebRequest.FindServicePoint(Boolean forceFind)
System.Net.AuthenticationState.PrepareState(HttpWebRequest httpWebRequest)
System.Net.AuthenticationState.ClearSession(HttpWebRequest httpWebRequest)
System.Net.HttpWebRequest.ClearAuthenticatedConnectionResources()
System.Net.HttpWebRequest.Abort(Exception exception, Int32 abortState)
System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
System.Threading.ThreadPoolWorkQueue.Dispatch()
I've seen many similar threads around the web. But all who had the same stack did not receive any help.
I've also seen that many recommended to set the property of my HttpWebRequest with KeepAlive=false, however, that may hurt my performance and is unacceptable.
This is, in fact, a known bug in Microsoft's framework, ad mentioned here:
https://support.microsoft.com/en-us/kb/2750147
The weird issue about this is that my application is running with .NET4.0 and not .NET4.5
After talking with Microsoft's Support, it seems that if .NET4.5 is installed on the machine, then the application's behavior is changed. So it in fact does make sense.
The evidence could be found in MS's source code. From http://referencesource.microsoft.com/#q=httpwebrequest :
// TimeoutCallback - Called by the TimerThread to abort a request. This just posts ThreadPool work item - Abort() does too
// much to be done on the timer thread (timer thread should never block or call user code).
private static void TimeoutCallback(TimerThread.Timer timer, int timeNoticed, object context)
{
ThreadPool.UnsafeQueueUserWorkItem(s_AbortWrapper, context);
}
private void Abort(Exception exception, int abortState)
{
GlobalLog.ThreadContract(ThreadKinds.Unknown, "HttpWebRequest#" + ValidationHelper.HashString(this) + "::Abort()");
if (Logging.On) Logging.Enter(Logging.Web, this, "Abort", (exception == null? "" : exception.Message));
if(Interlocked.CompareExchange(ref m_Aborted, abortState, 0) == 0) // public abort will never drain streams
{
GlobalLog.Print("HttpWebRequest#" + ValidationHelper.HashString(this) + "::Abort() - " + exception);
NetworkingPerfCounters.Instance.Increment(NetworkingPerfCounterName.HttpWebRequestAborted);
m_OnceFailed = true;
CancelTimer();
WebException webException = exception as WebException;
if (exception == null)
{
webException = new WebException(NetRes.GetWebStatusString("net_requestaborted", WebExceptionStatus.RequestCanceled), WebExceptionStatus.RequestCanceled);
}
else if (webException == null)
{
webException = new WebException(NetRes.GetWebStatusString("net_requestaborted", WebExceptionStatus.RequestCanceled), exception, WebExceptionStatus.RequestCanceled, _HttpResponse);
}
try
{
#if DEBUG
bool setResponseCalled = false;
try
{
#endif
// Want to make sure that other threads see that we're aborted before they set an abort delegate, or that we see
// the delegate if they might have missed that we're aborted.
Thread.MemoryBarrier();
HttpAbortDelegate abortDelegate = _AbortDelegate;
#if DEBUG
m_AbortDelegateUsed = abortDelegate == null ? (object)DBNull.Value : abortDelegate;
#endif
if (abortDelegate == null || abortDelegate(this, webException))
{
// We don't have a connection associated with this request
#if DEBUG
setResponseCalled = true;
#endif
SetResponse(webException);
}
else
{
// In case we don't call SetResponse(), make sure to complete the lazy async result
// objects. abortDelegate() may not end up in a code path that would complete these
// objects.
LazyAsyncResult writeAResult = null;
LazyAsyncResult readAResult = null;
if (!Async)
{
lock (this)
{
writeAResult = _WriteAResult;
readAResult = _ReadAResult;
}
}
if (writeAResult != null)
writeAResult.InvokeCallback(webException);
if (readAResult != null)
readAResult.InvokeCallback(webException);
}
if (!Async)
{
LazyAsyncResult chkConnectionAsyncResult = ConnectionAsyncResult;
LazyAsyncResult chkReaderAsyncResult = ConnectionReaderAsyncResult;
if (chkConnectionAsyncResult != null)
chkConnectionAsyncResult.InvokeCallback(webException);
if (chkReaderAsyncResult != null)
chkReaderAsyncResult.InvokeCallback(webException);
}
if (this.IsWebSocketRequest && this.ServicePoint != null)
{
this.ServicePoint.CloseConnectionGroup(this.ConnectionGroupName);
}
#if DEBUG
}
catch (Exception stressException)
{
t_LastStressException = stressException;
if (!NclUtilities.IsFatal(stressException)){
GlobalLog.Assert(setResponseCalled, "HttpWebRequest#{0}::Abort|{1}", ValidationHelper.HashString(this), stressException.Message);
}
throw;
}
#endif
}
catch (InternalException)
{
}
}
if(Logging.On)Logging.Exit(Logging.Web, this, "Abort", "");
}
As you can see, TimeoutCallback is calling the abort method in a new thread, meaning its not exception-proof.
In addition, Abort may throw an exception in some scenarios. In theory, this could be easily reproduced.
Related
I'm trying to write an infinite length response body and detect when a client disconnects so I can stop writing. I'm used to getting socket exceptions or similar when a client closes the connection but that doesn't seem to be happening when writing directly to Response.Body. I can close the client applications and the server side just keeps on writing. I've included the relevant code below. It's entirely possible there is a better way to do it but this came to mind. Basically I have a live video feed which should go on forever. I'm writing to ResponseBody as chunked content (No content length, flushing after each video frame). The video frames are received via an event callback from elsewhere in the program so I'm subscribing to the events in the controller method and then forcing it to stay open with the await Task.Delay loop so the Response stream isn't closed. The callback for H264PacketReceived is formatting the data as a streaming mp4 file and writing it to the Response Stream. This all seems to work fine, I can play the live stream with ffmpeg or chrome, but when I close the client application I don't get an exception or anything. It just keeps writing to the stream without any errors.
public class LiveController : ControllerBase
{
[HttpGet]
[Route("/live/{cameraId}/{stream}.mp4")]
public async Task GetLiveMP4(Guid cameraId, int stream)
{
try
{
Response.StatusCode = 200;
Response.ContentType = "video/mp4";
Response.Headers.Add("Cache-Control", "no-store");
Response.Headers.Add("Connection", "close");
ms = Response.Body;
lock (TCPVideoReceiver.CameraStreams)
{
TCPVideoReceiver.CameraStreams.TryGetValue(cameraId, out cameraStream);
}
if (this.PacketStream == null)
{
throw new KeyNotFoundException($"Stream {cameraId}_{stream} not found");
}
else
{
connected = true;
this.PacketStream.H264PacketReceived += DefaultStream_H264PacketReceived;
this.PacketStream.StreamClosed += PacketStream_StreamClosed;
}
while(connected)
{
await Task.Delay(1000);
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
finally
{
connected = false;
this.PacketStream.H264PacketReceived -= DefaultStream_H264PacketReceived;
this.PacketStream.StreamClosed -= PacketStream_StreamClosed;
}
}
private bool connected = false;
private PacketStream PacketStream;
private Mp4File mp4File;
private Stream ms;
private async void PacketStream_StreamClosed(PacketStream source)
{
await Task.Run(() =>
{
try
{
Console.WriteLine($"Closing live stream");
connected = false;
ms.Close();
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
});
}
private async void DefaultStream_H264PacketReceived(PacketStream source, H264Packet packet)
{
try
{
if (mp4File == null && packet.IsIFrame)
{
mp4File = new Mp4File(null, packet.sps, packet.pps);
var _p = mp4File.WriteHeader(0);
await ms.WriteAsync(mp4File.buffer, 0, _p);
}
if (mp4File != null)
{
var _p = mp4File.WriteFrame(packet, 0);
var start = mp4File._moofScratchIndex - _p;
if (_p > 0)
{
await ms.WriteAsync(mp4File._moofScratch, start, _p);
await ms.FlushAsync();
}
}
return;
}
catch (Exception ex)
{
connected = false;
Console.WriteLine(ex.ToString());
}
}
Answering my own question.
When the client disconnects mvc core sets the cancellation token HttpContext.RequestAborted
By monitoring and/or using that cancellation token you can detect a disconnect and clean everything up.
That said, the entire design can be improved by creating a custom stream which encapsulates the event handling (producer/consumer). Then the controller action can be reduced to.
return File(new MyCustomStream(cameraId, stream), "video/mp4");
The File Method already monitors the cancellation token and everything works as you'd expect.
I have a list of 350 downloadable image urls. I download 10 images parallely at one shot by running multiple tasks. But after downloading N number of images suddenly my code throws the following exception.
Exception: "An error occurred while sending the request."
InnerException: "The request was aborted: Could not create SSL/TLS
secure channel."
StackTrace: "at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)\r\n at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)\r\n at
System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\r\n ...
I have created a sample project to reproduce this exception. I have 2 test-cases in my hand. You can download the running test project from My Sky Drive Here. Right click on the file HTTPClientTestCases1and2.zip and download.
Case 1: Using a single instance HttpClient for all image Download.
In this case I'm sending parallel request to 10 urls using same HttpClient. In this case download is successful for most of the time. After last successful download of an image wait for minimum 40 seconds (Max 1 minute 40 seconds) to send the next parallel download request for the next batch. One image will definitely fail due to this exception. But so many places its written and suggested to use single HttpClient for multiple request.
public async void DownloadUsingSingleSharedHttpClient(Int32 imageIndex)
{
Uri url = new Uri(ImageURLs[imageIndex]);
UnderDownloadCount++;
try
{
Byte[] contentBytes = null;
try
{
// Exception IS THROWN AT LINE BELOW
HttpResponseMessage response = await _httpClient.GetAsync(url);
contentBytes = await response.Content.ReadAsByteArrayAsync();
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine("Download Failed at GetAsync() :" + ex.Message);
throw ex;
}
DownloadedCount++;
if (OnSuccess != null)
OnSuccess(this, new DownloadSuccessEventArgs() { Index = imageIndex, Data = contentBytes });
}
catch (HttpRequestException hre)
{
DownloadFailedCount++;
if (OnFailed != null)
OnFailed(hre, null);
}
catch (TaskCanceledException hre)
{
DownloadFailedCount++;
if (OnFailed != null)
OnFailed(hre, null);
}
catch (Exception e)
{
DownloadFailedCount++;
if (OnFailed != null)
OnFailed(e, null);
}
}
Case 2: Creating new instance of HttpClient for every image Download
In this case it just fails very frequently due to same exception while downloading images parallely.
public async void DownloadUsingCreatingHttpClientEveryTime(Int32 imageIndex)
{
Uri url = new Uri(ImageURLs[imageIndex]);
UnderDownloadCount++;
try
{
Byte[] contentBytes = null;
using (HttpClientHandler _handler = new HttpClientHandler())
{
_handler.AllowAutoRedirect = true;
_handler.MaxAutomaticRedirections = 4;
using (HttpClient httpClient = new HttpClient(_handler))
{
httpClient.DefaultRequestHeaders.ExpectContinue = false;
httpClient.DefaultRequestHeaders.Add("Keep-Alive", "false");
try
{
// Exception IS THROWN AT LINE BELOW
contentBytes = await httpClient.GetByteArrayAsync(url.OriginalString);
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine("Download Failed :" + ex.Message);
throw ex;
}
}
_handler.Dispose();
}
DownloadedCount++;
if (OnSuccess != null)
OnSuccess(this, new DownloadSuccessEventArgs() { Index = imageIndex, Data = contentBytes });
}
catch (HttpRequestException hre)
{
DownloadFailedCount++;
if (OnFailed != null)
OnFailed(hre, null);
}
catch (TaskCanceledException hre)
{
DownloadFailedCount++;
if (OnFailed != null)
OnFailed(hre, null);
}
catch (Exception e)
{
DownloadFailedCount++;
if (OnFailed != null)
OnFailed(e, null);
}
}
Please edit the following function in MainPage.xaml.cs to check two cases
private void Send10DownloadRequestParallel()
{
for (Int32 index = 0; index < 10; index++)
{
Task.Run(() =>
{
Int32 index1 = rand.Next(0, myImageDownloader.ImageURLs.Count - 1);
UpdateDownloadProgress();
// Case 1: Download Using Single Shared HttpClient
// myImageDownloader.DownloadUsingSingleSharedHttpClient(index1);
// OR
// Case 2: Download Using Creating Http Client Every Time
myImageDownloader.DownloadUsingCreatingHttpClientEveryTime(index1);
});
}
}
My Question: What I'm doing wrong? What is the best way of implementing parallel downloader in WinRT by overcoming this exception.
I ran your sample application and am only get errors in a couple of scenarios:
When the image that your app is requesting does not exist, the .NET HTTP client throws an exception. Your handler doesn't quite handle this case, as the inner exception is NULL. I had to tweak that code just a little:
async void myImageDownloader_OnFailed(object sender, EventArgs e)
{
await App.CurrentDispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, delegate
{
TimeSpan time =(DateTime.Now -dateTimeSuccess);
String timeGap = "Ideal For:" + time.ToString() + "\n";
ErrorListBox.Text += "\n Failed When: " + DownloadInfo.Text + "\n";
ErrorListBox.Text += timeGap;
// CX - added null check for InnerException, as these are NULL on HTTP result status 404
var ex = sender as Exception;
if (ex.InnerException != null)
ErrorListBox.Text += ex.InnerException.Message;
else
ErrorListBox.Text += "Inner Exception null - Outer = (" + ex.ToString() + ")";
});
}
The only time I got your other error Could not create SSL/TLS secure channel in Windows 8 Metro App, is when I was using a HTTP debugging proxy (Fiddler). If I don't user Fiddler, which intercepts all HTTP(S) calls, then I have no problems downloading. I even started multiple downloads in rapid succession (by clicking the blue download area multiple times within one second). The result was that all items were downloaded (except for the 404 errors, as mentioned above).
Here is a screenshot of the successful downloads (again except for the 404s). This screenshot is running test case #2 (multiple instances of HttpClient). I did run test case #1 (single instance of HttpClient) and the results were also successful.
In short, I did not see the problems that you are experiencing. The only thing I can think of is for you to try your app from a different machine or location.
I have a UI view lossreport.xaml in that below code is there
LossReportTowGlassServiceClient wcf = new LossReportTowGlassServiceClient();
wcf.HouseholdSearchCompleted += (o, ev) =>
{
string a = errorMessg.ToUpper();
//Code to work with ev
};
wcf.HouseholdSearchAsync(lossDate, txtPolicyNumber.Text, errorMessg);
in service.svc page
try
{
policyinq.retrieveHouseHoldPoliciesCompleted += new retrieveHouseHoldPoliciesCompletedEventHandler(policyinq_retrieveHouseHoldPoliciesCompleted);
policyinq.retrieveHouseHoldPoliciesAsync(reqh, searchCriteria, lossdate, true, string.Empty, string.Empty);
break;
}
catch (Exception ex)
{
Logger.Exceptions("", "HouseholdSearch", ex);
errorToSend = "Household error";
}
void policyinq_retrieveHouseHoldPoliciesCompleted(object sender, retrieveHouseHoldPoliciesCompletedEventArgs e)
{
{
if (e.transactionNotification != null && e.transactionNotification.transactionStatus == TransactionState.S)
{
}
else
{
ErrorHandling.ErrorSend(e.transactionNotification, "HouseHold");
}
};
}
now before retrieveHouseHoldPolicies is completed HouseholdSearchCompleted event is fired.How to make it wait
You have an architectural issue here, The service should not invoke async request unless you go ta good reason (maybe invoke some paralleled stuff. Just invoke your server side code synchronously.
A service entry point got it's own handler thread, it should be the one who starts and end the request response process on service side. what you do is call an async method on service side making the thread that handle the request finish his job. So you either make this thread wait or execute the entire logic on him without calling async method, kapish?
using System.Threading;
ManualResetEvent _wait = new ManualResetEvent(false);
_wait.Set();//In completed event
_wait.WaitOne();//After the event is completed WaitOne will wait untill the _wait is set with value
After successfully opening Pop3Folder, and retrieving messages from it, I then sometimes get to the point, when folder.isOpen returns false. At the same time, when looking at the Pop3Folder's fields in debug mode, I see that the field opened set to true.
Could somebody give me a hint, what might go wrong here?
Here is the code:
public void popMail(MessageProcessor messageProcessor) throws MessagingException {
Folder inboxFolder = null;
Store store = null;
try {
store = mailSession.getStore();
store.connect(mailSession.getProperty("mail.user"),
mailSession.getProperty("mail.password"));
// OK. Connected to POP3 Store.
inboxFolder = store.getFolder("inbox");
inboxFolder.open(Folder.READ_WRITE);
// The folder is successfully opened.
Message[] msgs = inboxFolder.getMessages();
// Messages are successfully retrieved.
if (msgs != null && msgs.length > 0) {
for (Message msg : msgs) {
if (messageProcessor != null) {
// Calling custom listener to process message
messageProcessor.processMessage(msg);
}
msg.setFlag(Flag.DELETED, true);
}
}
} finally {
// Oops, inboxFolder.isOpen returns false.
// Meanwhile I see in debug mode that inboxFolder#opened is set to true
if (inboxFolder != null && inboxFolder.isOpen()) {
try {
inboxFolder.close(true);
} catch (MessagingException e) {
log.warn("Error while closing folder");
}
} if (store != null) {
try {
store.close();
} catch (MessagingException e) {
log.warn("Error while closing store");
}
}
}
}
The server may be timing out the connection if your processMessage method takes too long. Turn on Session debugging and examine the protocol trace for clues.
I have a WCF service that is processing a call, sending that processed data onto another service, and alerting the caller and any other instances of that application by firing a callback. Originally the callbacks were being called at the end but I found that if the second service was not running that there would be a twenty second delay while we attempted to discover it. Only then were the callbacks called. I moved the callback notification before the call to the second service but it still had the delay. I even tried firing the callbacks on a background process but that didn't work either. Is there a way to get around this delay, outside of changing the timeout of the discovery? Here is a code snippet.
// Alert the admins of the change.
if (alertPuis)
{
ReportBoxUpdated(data.SerialNumber);
}
// Now send the change to the box if he's online.
var scope = new Uri(string.Format(#"net.tcp://{0}", data.SerialNumber));
var boxAddress = DiscoveryHelper.DiscoverAddress<IAtcBoxService>(scope);
if (boxAddress != null)
{
var proxy = GetBoxServiceProxy(boxAddress);
if (proxy != null)
{
proxy.UpdateBox(boxData);
}
else
{
Log.Write("AtcSystemService failed on call to update toool Box: {0}",
data.SerialNumber);
}
}
else if (mDal.IsBoxDataInPendingUpdates(data.SerialNumber) == false)
mDal.AddPendingUpdate(data.SerialNumber, null, true, null);
}
and
private static void ReportBoxUpdated(string serialNumber)
{
var badCallbacks = new List<string>();
Action<IAtcSystemServiceCallback> invoke = callback =>
callback.OnBoxUpdated(serialNumber);
foreach (var theCallback in AdminCallbacks)
{
var callback = theCallback.Value as IAtcSystemServiceCallback;
try
{
invoke(callback);
}
catch (Exception ex)
{
Log.Write("Failed to execute callback for admin instance {0}: {1}",
theCallback.Key, ex.Message);
badCallbacks.Add(theCallback.Key);
}
}
foreach (var bad in badCallbacks) // Clean out any stale callbacks from the list.
{
AdminCallbacks.Remove(bad);
}
}
Have you considered caching the result?