I am using .NET 4.0 to try to leverage the functionality provided by HttpClient. The aim is to upload a particular file around 10.5 MB in size to a remote server using HTTP Verb "PUT". The remote server uses node.js.
I have written a small Console app (.NET 4.0 full not client profile)
The laptop I am using does not has .NET 4.5 or Visual Studio 2012
I installed Added these NuGet packages onto it to enable Async upload features with progress update
<packages>
<package id="Microsoft.AspNet.WebApi.Client" version="4.0.20710.0" targetFramework="net40" />
<package id="Microsoft.Net.Http" version="2.0.20710.0" targetFramework="net40-Client" />
<package id="Newtonsoft.Json" version="4.5.11" targetFramework="net40" />
</packages>
I then wrote a simple function that does a synchronous upload (UploadFile). This function works as expected and uploads the files correctly.
The next method called UploadFileAsync tries to use HttpClient and fails between 55-70% of upload. Visual Studio 2010 throws up a series of exceptions. The first one is usually different ranging from
WebException
"The request was aborted: The request was cancelled"
to various other types. The second exception however is always predictable and same. This is
ObjectDisposedExceptoin
Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
Finally this is the last exception in the chain
IOException
Unable to read data from the transport connection: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'..
As you would notice I have removed all using blocks, close or dispose statements etc. from my code. For the life of me I can not figure out what is going wrong! I have tried PutAsync, and SendAsnyc methods both with HttpClient. The only difference they make is that the first exception changes.
Here is the code block
void UploadFile(string fileToUpload)
{
//string fileToUpload = #"F:\\upload_file.txt";
using (FileStream rdr = new FileStream(fileToUpload, FileMode.Open))
{
HttpWebRequest req = null;
Stream reqStream = null;
try
{
req = (HttpWebRequest)WebRequest.Create(url);
req.Method = "PUT";
req.Headers.Add("x-myheader", "163333.zip");
req.ContentType = "application/octet-stream";
req.ContentLength = rdr.Length;
req.AllowWriteStreamBuffering = false;
req.Timeout = 1000000;
reqStream = req.GetRequestStream();
Console.WriteLine(rdr.Length);
byte[] inData = new byte[rdr.Length];
// Get data from upload file to inData
int bytesRead = rdr.Read(inData, 0, (int)rdr.Length);
// put data into request stream
reqStream.Write(inData, 0, (int)rdr.Length);
rdr.Close();
req.GetResponse();
// after uploading close stream
reqStream.Close();
}
catch (Exception e)
{
}
finally
{
rdr.Close();
req.GetResponse();
// after uploading close stream
reqStream.Close();
}
}
}
void UploadFileAsync(string fileToUpload)
{
//string fileToUpload = #"F:\\upload_file.txt";
FileStream rdr = new FileStream(fileToUpload, FileMode.Open, FileAccess.Read, FileShare.Read, 8192, true);
//newfs rdr = new newfs(fileToUpload, FileMode.Open);
HttpClientHandler clientHandler = new HttpClientHandler();
ProgressMessageHandler progressHandler = new ProgressMessageHandler(clientHandler);
HttpClient client = new HttpClient(progressHandler);
try
{
client.DefaultRequestHeaders.Add("x-myheader", "163333.zip");
HttpRequestMessage requestPayLoad = new HttpRequestMessage(HttpMethod.Put, url);
long totalFileSize = rdr.Length;
Console.WriteLine(totalFileSize);
int? previousPercentage = null;
progressHandler.HttpSendProgress += (sender, args) =>
{
if (!previousPercentage.HasValue || previousPercentage != args.ProgressPercentage)
Console.WriteLine(args.ProgressPercentage);
previousPercentage = args.ProgressPercentage;
};
requestPayLoad.Content = new StreamContent(rdr);
//requestPayLoad.Content = new FileContent(fileToUpload);
//var task = client.PutAsync<byte[]>(url, rdr, new BinaryMediaTypeFormatter());
var task = client.SendAsync(requestPayLoad);
task.Wait();
}
catch (Exception e)
{
}
finally
{
//rdr.Close();
client.Dispose();
//req.GetResponse();
//// after uploading close stream
//reqStream.Close();
}
}
Update
It appears that the file does reach the server OK. Which then basically means that the ProgressHandler is reporting incorrect percentages and the error comes at the very right end of the upload. Fiddler shows HTTP 200 status code.
Related
Sample code below to write a file stream to Response.Body in an ASP.NET Core middleware doesn't work (emits empty response):
public Task Invoke(HttpContext context)
{
context.Response.ContentType = "text/plain";
using (var fs = new FileStream("/valid-path-to-file-on-server.txt", FileMode.Open)
using (var sr = new StreamReader(fs))
{
context.Response.Body = sr.BaseStream;
}
return Task.CompletedTask;
}
Any ideas what could be wrong with this approach of directly setting the context.Response.Body?
Note: any next middleware in the pipeline is skipped for no further processing.
Update (another example): a simple MemoryStream assignment doesn't work either (empty response):
context.Response.Body = new MemoryStream(Encoding.UTF8.GetBytes(DateTime.Now.ToString()));
No. You can never do that directly.
Note that context.Response.Body is a reference to an object (HttpResponseStream) that is initialized before it becomes available in HttpContext. It is assumed that all bytes are written into this original Stream. If you change the Body to reference (point to) a new stream object by context.Response.Body = a_new_Stream, the original Stream is not changed at all.
Also, if you look into the source code of ASP.NET Core, you'll find the Team always copy the wrapper stream to the original body stream at the end rather than with a simple replacement(unless they're unit-testing with a mocked stream). For example, the SPA Prerendering middleware source code:
finally
{
context.Response.Body = originalResponseStream;
...
And the ResponseCachingMiddleware source code:
public async Task Invoke(HttpContext httpContext)
{
...
finally
{
UnshimResponseStream(context);
}
...
}
internal static void UnshimResponseStream(ResponseCachingContext context)
{
// Unshim response stream
context.HttpContext.Response.Body = context.OriginalResponseStream;
// Remove IResponseCachingFeature
RemoveResponseCachingFeature(context.HttpContext);
}
As a walkaround, you can copy the bytes to the raw stream as below:
public async Task Invoke(HttpContext context)
{
context.Response.ContentType = "text/plain";
using (var fs = new FileStream("valid-path-to-file-on-server.txt", FileMode.Open))
{
await fs.CopyToAsync(context.Response.Body);
}
}
Or if you like to hijack the raw HttpResponseStream with your own stream wrapper:
var originalBody = HttpContext.Response.Body;
var ms = new MemoryStream();
HttpContext.Response.Body = ms;
try
{
await next();
HttpContext.Response.Body = originalBody;
ms.Seek(0, SeekOrigin.Begin);
await ms.CopyToAsync(HttpContext.Response.Body);
}
finally
{
response.Body = originalBody;
}
The using statements in the question causes your stream and stream reader to be rather ephemeral, so they will both be disposed. The extra reference to the steam in "body" wont prevent the dispose.
The framework disposes of the stream after sending the response. (The medium is the message).
In net 6 I found I was getting console errors when I tried to do this e.g.:
System.InvalidOperationException: Response Content-Length mismatch: too many bytes written (25247 of 8863).
The solution was to remove the relevant header:
context.Response.Headers.Remove("Content-Length");
await context.Response.SendFileAsync(filename);
I have application that sends requests to same REST server constantly and after some time HttpWebRequest.GetResponse() starts timing out i've noticed that whenever i increase System.Net.ServicePointManager.DefaultConnectionLimit it takes it longer to start timing out again, which should mean that those requests are staying active, but as far as i know i'm closing all of them.
Here is method i'm using for my requests.
Current DefaultConnectionLimit is set to 10.
Also there is 1 request that is going on throughout most of applications lifetime.
I'm using .NET Compact framework and REST server is written using WCF (.NET 4.5)
public static string HttpRequest(string request, string method, string contentType, int timeout)
{
string result = "";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(PodesavanjaManager.TrenutnaPodesavanja.PutanjaServisa + "/" + request);
req.Method = method;
req.ContentType = contentType;
req.Timeout = timeout;
req.KeepAlive = false;
if(method == "POST")
req.ContentLength = 0;
using(Stream stream = req.GetResponse().GetResponseStream())
{
using(StreamReader reader = new StreamReader(stream))
{
result = reader.ReadToEnd();
reader.Close();
}
stream.Close();
stream.Flush();
}
return result;
}
EDIT new version of method:
public static string HttpRequest(string request, string method, string contentType, int timeout)
{
string result = "";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(PodesavanjaManager.TrenutnaPodesavanja.PutanjaServisa + "/" + request);
req.Method = method;
req.ContentType = contentType;
req.Timeout = timeout;
req.KeepAlive = false;
if(method == "POST")
req.ContentLength = 0;
using (HttpWebResponse resp =(HttpWebResponse) req.GetResponse())
{
using (Stream stream = resp.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
result = reader.ReadToEnd();
reader.Close();
}
}
}
GC.Collect();
return result;
}
I agree it does behave like the connection is still in use by a resource that has not been closed. The documentation for HttpWebResponse mentions:
You must call either the Stream.Close or the HttpWebResponse.Close method to close the response and release the connection for reuse. It is not necessary to call both Stream.Close and HttpWebResponse.Close, but doing so does not cause an error.
I was hoping for a more straightforward description like "you must either close the stream returned by GetResponseStream or call the HttpWebResponse.Close method - but if my interpretation of the documentation is correct, your code is fine.
We use HttpWebRequest in our CE applications as well, and always put the response in a using block as well - you could try this:
using(HttpWebResponse response = (HttpWebResponse)req.GetResponse())
using(Stream stream = response.GetResponseStream())
{
// ...
}
Also have you checked your code for other HttpWebRequest usages, just to be sure?
How do I post GZip data using RestSharp. I have the following code but it isn't working as I would expect:
var restRequest = new RestRequest(url, Method.POST)
{
Timeout = Constants.DefaultTimeoutMilliseconds
};
var dataStream = new MemoryStream();
using (var zipStream = new GZipStream(dataStream, CompressionMode.Compress))
{
using (var writer = new StreamWriter(zipStream))
{
writer.Write(new DotNetXmlSerializer().Serialize(content));
}
}
var compressedBytes = dataStream.ToArray();
restRequest.AddParameter("application/x-gzip", compressedBytes, ParameterType.RequestBody);
return _restClient.Execute<TResponseData>(restRequest);
When I run this and check the wireshark trace, the compressedBytes variable is posted as
'System.Byte[]' - as if ToString() has been called on it despite the parameter being a system.object.
If I pass the compressed byte array through as a string using both Convert.ToBase64String() and Encoding.Utf8.GetString() then I am unable to decompress the GZip at the server. I simply get 'System.IO.InvalidDataException: The magic number in GZip header is not correct. Make sure you are passing in a GZip'.
Is there any way of posting Gzipped data using RestSharp?
Make sure you've updated to the latest version of RestSharp (like 104.4.0) as this was a bug in a previous version.
I think this was fixed in 104.2 where the PUT or POST of binary data ended up with the System.Byte[] being represented as the string.
Update your NuGet reference and try it again. Good luck!
var body = "some string";
var dataStream = new MemoryStream();
byte[] dataToCompress = Encoding.UTF8.GetBytes(body);
using (var memoryStream = new MemoryStream())
{
using (var gzipStream = new GZipStream(memoryStream, CompressionLevel.Optimal))
{
gzipStream.Write(dataToCompress, 0, dataToCompress.Length);
}
dataStream = memoryStream;
}
var client = new RestClient("url");
var request = new RestRequest("", Method.POST);
var compressedBytes = dataStream.ToArray();
request.AddHeader("Content-Encoding", "gzip");
request.AddParameter("application/x-gzip", compressedBytes, ParameterType.RequestBody);
//client.RemoteCertificateValidationCallback = (sender, certificate, chain, sslPolicyErrors) => true;
IRestResponse response = await client.ExecuteAsync(request);
Console.WriteLine(response.Content);
I am trying to set up file upload example using JAX RS. I could set up the project and successfully upload file in a server location. But i get the following error when file size is more than 10KB (weird!!)
com.sun.jersey.api.client.UniformInterfaceException: POST http://localhost:9090/DOAFileUploader/rest/file/upload returned a response status of 400
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:607)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:507)
at com.sony.doa.rest.client.DOAClient.upload(DOAClient.java:75)
at com.sony.doa.rest.client.DOAMain.main(DOAMain.java:34)
I am new to JAX RS and i'm not sure what exactly the issue is. Do i need to set some parameters client side or server side (like size, timeout etc)?
This is the client side code calling webservice:
public void upload() {
File file = new File(inputFilePath);
FormDataMultiPart part = new FormDataMultiPart();
part.bodyPart(new FileDataBodyPart("file", file, MediaType.APPLICATION_OCTET_STREAM_TYPE));
WebResource resource = Client.create().resource(url);
String response = resource.type(MediaType.MULTIPART_FORM_DATA_TYPE).post(String.class, part);
System.out.println(response);
}
This is the server side code:
#Path("/file")
public class UploadFileService {
#POST
#Path("/upload")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(
#FormDataParam("file") InputStream uploadedInputStream,
#FormDataParam("file") FormDataContentDisposition fileDetail) {
String uploadedFileLocation = "e://uploaded/"
+ fileDetail.getFileName();
writeToFile(uploadedInputStream, uploadedFileLocation);
String output = "File uploaded to : " + uploadedFileLocation;
return Response.status(200).entity(output).build();
}
private void writeToFile(InputStream uploadedInputStream,
String uploadedFileLocation) {
try {
OutputStream out = new FileOutputStream(new File(
uploadedFileLocation));
int read = 0;
byte[] bytes = new byte[16000];
out = new FileOutputStream(new File(uploadedFileLocation));
while ((read = uploadedInputStream.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
out.flush();
out.close();
} catch (IOException e) {
e.printStackTrace();
} } }
Please let me know what settings i have to change for file sizes greater than 10KB?
Thanks!
I use org.apache.commons.fileupload.servlet.ServletFileUpload in a Jersey context, and it works fine., and yes, it set the max file size, sorry I missed this before.
here is a snipet of code I use (this is a multipart form, so there are other fields along with the file)
private LibraryUpload parseLibraryUpload(HttpServletRequest request) {
LibraryUpload libraryUpload;
File libraryZip = null;
String name = null;
String version = null;
ServletFileUpload upload = new ServletFileUpload();
upload.setFileSizeMax(MAX_FILE_SIZE);
FileItemIterator iter;
try {
iter = upload.getItemIterator(request);
while (iter.hasNext()) {
....
if (item.isFormField()) {
....
}else{
BufferedInputStream buffer = new BufferedInputStream(stream);
buffer.mark(MAX_FILE_SIZE);
libraryZip = File.createTempFile("fromUpload", null);
IOUtils.copy(buffer, new FileOutputStream(libraryZip));
...
}
I have encountered the same problem with Jersey. I have activated jersey trace but nothing help me.
I have changed the library by an apache Library and I see than the problem with linked to a repository for temporary files for tomcat. The repository was not exist. For files under 10k, the repository was not used.
So, after the repository creation, I used jersey library and all works fine.
I have a page containing links to some files.
I basically need to access the source of the page for parsing it then and obtaining all the hyperlinks to the files.
My code is something like this (some piece of code I've found in many places on the net ..):
"private static byte[] ReadImageFromUrl(string url)
{
var myReq = (HttpWebRequest)WebRequest.Create(url);
myReq.Timeout = 10000;
WebResponse myResp = myReq.GetResponse();
Stream stream = myResp.GetResponseStream();
List<byte> bytesList = new List<byte>();
using (var br = new BinaryReader(stream))
{
try
{
while (true)
{
var b = br.ReadByte();
bytesList.Add(b);
}
}
catch (Exception)
{}
br.Close();
}
myResp.Close();
return bytesList.ToArray();
}"
Now the problem is I get "System.Net.WebException: The remote server returned an error: (500) Internal Server Error." when calling "myReq.GetResponse()" - examining the error I see that the status is 'ProtocolError'.
The response property of the WebException object contains some server error ..(although when opening it from the browser it opens correctly) ...also when I call this function with the url of one of my files I get the same ProtocolError status, but the 404 error ...
Please give any hint how could I solve it... or any other possibility of accomplishing this task.
Thanks !
My new code after using Fiddler is:
private static byte[] ReadFileFromUrl(string url)
{
var myReq = (HttpWebRequest)WebRequest.Create(url);
myReq.Accept = const_AcceptHeader;
myReq.Headers.Set(const_AcceptLanguageHeaderName, const_AcceptLanguageHeader);
myReq.UserAgent = const_AcceptUserAgentHeader;
myReq.CookieContainer = new CookieContainer();
myReq.KeepAlive = true;
myReq.Timeout = Int32.Parse(ConfigSettings.RequestPageTimeout) * 1000;
WebResponse myResp = null;
List<byte> bytesList = null;
myResp = myReq.GetResponse();
Stream stream = myResp.GetResponseStream();
bytesList = new List<byte>();
using (var br = new BinaryReader(stream))
{
try
{
while (true)
{
var b = br.ReadByte();
bytesList.Add(b);
}
}
catch (Exception ex)
{
throw;
}
br.Close();
}
return bytesList.ToArray();
}
All variables that start with const_ are taken from Fiddler.
Well, I solved that using Fiddler ... I passed to my request object the headers as I have seen them in Fiddler ...& it worked, no error