confuse about EndUploadFromStream - azure-storage

Question

I am confusing with a problem about upload blobs asynchronously, hopes find answer here.
Please take a look at my code snippet first,
public void UploadMultipleBlobs(List<string> filelocations, string containerName, AsyncCallback callback = null, string path = null)
{
try
{
Parallel.ForEach(filelocations, fileLocation =>
{
//File to Stream
MemoryStream str = new MemoryStream();
byte[] file = File.ReadAllBytes(fileLocation);
str.Write(file, 0, file.Length);
str.Seek(0, SeekOrigin.Begin);
//Operations
if (callback == null)
callback = new AsyncCallback(OnUploadCompleted);
BlobRequestOptions blobRequestOptions = new BlobRequestOptions();
blobRequestOptions.Timeout = new TimeSpan(1, 0, 0);
blobRequestOptions.RetryPolicy = retry;
CloudBlob currentBlob = container.GetBlobReference(blobName);
var result = currentBlob.BeginUploadFromStream(str, blobRequestOptions, callback, new Object[] { currentBlob, str });
currentBlob.EndUploadFromStream(result);
});
}
catch
{
throw;
}
}
private void OnUploadCompleted(IAsyncResult result)
{
try
{
// Get array passed to callback
Object[] states = (Object[])result.AsyncState;
var blob = (CloudBlob)states[0];
var stream = (MemoryStream)states[1];
// End the operation
//blob.EndUploadFromStream(result);
// Close the stream
stream.Close();
}
catch
{
throw;
}
}
I need to upload mutil files to Azure blob, number of files may be 10-50,000, each file is about 10KB-50KB. The code snippet works fine for me currently. However, if I call EndUploadFromStream in callback, it always throw an exception when uploading over 2,000 files. I mean if I remove EndUploadFromStream in upload method and call EndUploadFromStream in callback(OnUploadCompleted method), the exception happens. The exception message as below:
Unable to read data from the transport connection: The connection was closed., StackTrace: at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.get_Result()
at Microsoft.WindowsAzure.StorageClient.CloudBlob.EndUploadFromStream(IAsyncResult asyncResult)
I don't know why it happens...hopes got answer from you guys.
Thanks.

Answer 1

The Begin/End code looks OK. I notice that you're not doing anything to wait for the asynchronous operations to complete, so the problem may be related to that. eg, if you're running this from a console application then the application may exit before all the uploads have completed and then give you those errors. That would not be a problem if the EndUploadFromStream() call is inside the Parallel.ForEach() because it will cause the Parallel.ForEach() call to block until all the uploads have completed.
So try adding code to wait for all the uploads to complete and see if that fixes it. A simple way would be a counter that is initialized to the total number of uploads, and then decremented (using Interlocked.Decremement() for thread safety) inside of the callback. Another options would be use Task.FromAsync() to get an array of Task objects, then using Task.WaitAll() to wait for them to complete.
As an aside, using both Parallel.ForEach() and Begin/End methods at the same time is usually redundant - Begin/End is asynchronous already so there's usually no point using multiple threads to invoke it. Since you have such a big list of items it might make some difference in this case, but probably not much. You're probably better off using a simple foreach loop instead of Parallel.ForEach() unless you've actually measured a significant difference.