Background - I'm trying to stream an existing webpage to a separate web application, using HttpWebRequest/HttpWebResponse in C#. One issue I'm striking is that I'm trying to set the file upload request content-length using the file download's content-length, HOWEVER the issue seems to be when the source webpage is on a webserver for which the HttpWebResponse doesn't provide a content length.
HttpWebRequest downloadRequest = WebRequest.Create(new Uri("downloaduri")) as HttpWebRequest;
using (HttpWebResponse downloadResponse = downloadRequest.GetResponse() as HttpWebResponse)
{
var uploadRequest = (HttpWebRequest) WebRequest.Create(new Uri("uripath"));
uploadRequest.Method = "POST";
uploadRequest.ContentLength = downloadResponse.ContentLength; // ####
QUESTION: How could I update this approach to cater for this case (when the download response doesn't have a content-length set). Would it be to somehow use a MemoryStream perhaps? Any sample code would be appreciated.
If you're happy to download the response from the other web server completely, that would indeed make life easier. Just repeatedly write into a MemoryStream as you fetch from the first web server, then you know the length to set for the second request and you can write the data in easily (especially as MemoryStream has a WriteTo method to write its contents to another stream).
The downside of this is that you'll take a lot of memory if it's a large file. Is that likely to be a problem in your situation? Alternatives include:
Writing to a file instead of using a MemoryStream. You'll need to clean up the file afterwards, of course - you're basically using the file system as bigger memory :)
Using a chunked transfer encoding to "read a chunk, write a chunk"; this may be fiddly to get right - it's certainly not something I've done before.
Related
I was wondering how you would generate the swagger UI to upload a file as a stream to the ASP.net Core controller.
Here is a link describing the difference between uploading small files vs. big files.
Here is the link of describing how to implement the upload for a small file, but doesn't elaborate on how to implement a stream.
https://www.janaks.com.np/upload-file-swagger-ui-asp-net-core-web-api/
Thanks,
Derek
I'm not aware of any capability to work with Stream type(s) on the request parameters directly, but the IFormFile interface stipulates the ability to work with streams. So I would keep the IFormFile type in your request params, then either:
Copy it in full to a memory stream OR
Open the stream for read
In my case I wanted the base 64 bytes in full (and my files are only a few hundred kbs) so I used something like this:
string fileBase64 = null;
using (var memStream = new MemoryStream())
{
model.FormFile.CopyTo(memStream);
fileBase64 = Convert.ToBase64String(memStream.ToArray());
}
The MemoryStream is probably not appropriate in your case; as you mentioned large files which you will not want to keep in memory in their entirety.
So I would suggest opening the stream for reading, e.g.:
using (var fileStream = model.FormFile.OpenReadStream())
{
// Do yo' thang
}
I am developing a small application to upload large file using an Rest API. The Rest API accepts multi part form data. The upload should happen in stream. I am using RestSharp to invoke this API. I could upload small files, but when tried to upload 1GB file, it is taking longer time and i could not see any file in the destination. I am not sure, if i am using streaming or loading complete data into memory.
Please find my code below. Please share a C# sample code to upload large file stream using multi part form data. Is there anything i am missing in the below code:
var client = new RestClient("https://xxxxxxxxxxxxx/UploadFile");
RestRequest request = new RestRequest("/", Method.POST);
request.AddHeader("FileName", txtFileName.Text.Trim().ToString());
request.AddHeader("Content-Type", "multipart/form-data");
request.AddFile("fileData", txtFilePath.Text.Trim().ToString(), );
request.ReadWriteTimeout = 2147483647;
request.Timeout = 2147483647;
var response = client.Execute(request);
Thanks,
Rajkumar
I need to invoke a remote REST interface handler and submit it a file in request body. Please note that I don't control the server. I cannot change the request to be multipart, the client has to work in accordance to external specification.
So far I managed to make it work like this (omitting headers etc. for brevity):
byte[] data = readFileCompletely ();
client.target (url).request ().post (Entity.entity (data, "file/mimetype"));
This works, but will fail with huge files that don't fit into memory. And since I have no restriction on filesize, this is a concern.
Question: is it somehow possible to use streams or something similar to avoid reading the whole file into memory?
If possible, I'd prefer to avoid implementation-specific extensions. If not, a solution that works with RESTEasy (on Wildfly) is also acceptable.
ReastEasy as well as Jersey support InputStream out of the box so simply use Entity.entity(inputStream, "application/octet-stream"); or whatever Content-Type header you want to set.
You can go low-level and construct the HTTP request using a library such as the plain java.net.URLConnection.
I have not tried it myself but there is example code which reads a local file and writes it to the request stream without loading it into a byte array.
Upload files from Java client to a HTTP server
Of course this solution requires more manual coding but it should work (unless java.net.URLConnection loads the whole file into memory)
To whom it may concern,
I have a program that records sounds (*.wav(s) converted to *.wma(s) and *.m4a(s)) that I’m trying to upload to SoundCloud (previously the program had burned CD’s of the WMP Playlists and then I added a section that made a *.zip of the Playlist and attached it to an email), now my users want to upload the WMP playlists to their SoundCloud accounts.
I’ve already added a routine to upload the *.zip file to Dropbox and that works fine. Since the oauth2 for Dropbox was very similar to the oauth2 for SoundCloud, I was able to add my App to the SoundCloud list of Apps, and then add a browser control to my program for the user to “allow” my App to “link” to their SoundCloud account (very similar to how Dropbox handles it).
So I’m up to the point of uploading, but this is handled very differently and I’m having a hard time trying to get this to work (find the right HTTP handlers in Visual Basic to do what I need to do).
In my Dropbox section, I simply had to create a POST message, with the Title of the file and the oauth2 access_token in the URL, then I simply out to read the *.zip file (or any file that I was going to upload) and Write its contents to the GetRequestStream. The only stumbling block that I had was the “Content-Type” which had to be “application/zip” or whatever extension the file was and voila, It was uploaded.
I’ve been reading about multi-part forms and I’m getting a little confused (again the mechanics of the HTTP structures in Visual Basic can be a bit different…duh!...then in other languages…but I’ll try and explain).
I came across one example that used another constructor. Rather than the HttpRequest that I had used to build my “request” and “response” communications, this one had a WebClient. The WebClient could add Headers (Accept-Charset, Accept-Encoding, Accept-Language, Content-Type, Content-Disposition), and also a QueryString (this is where I placed the oath_token, track[asset-data], track[title], etc). Finally, the WebClient constructor had a WebClient.UploadFile command (which took the URL and the physical filename…which I believed would then handle the writing of the file contents the Request).
So…after getting through some 400’s, 401’s (have to deal with expiring token stuff), 422’s, a 503, and a bunch of 500’s (which is where I’m currently at), I still can’t upload a simple *.wma file: Track001.wma.
Here’s my code (strFileName is already built with the physical name of
the sound file \|dir|\Track001.wma, and strAccessToken was already
obtained from SoundCloud):
strURL = "https://api.soundcloud.com/tracks.json?oauth_token=" & strAccessToken
Dim myWebClient As New WebClient
myWebClient.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3")
myWebClient.Headers.Add("Accept-Encoding", "gzip,deflate,sdch")
myWebClient.Headers.Add("Accept-Language", "en-US,en;q=0.8,ru;q=0.6")
myWebClient.Headers.Add("Content-Disposition: form-data; name=track[title]")
I also tried this line as well:
myWebClient.Headers.Add("Content-Disposition: form-data; name=" & HttpUtility.UrlEncode(strFileName))
This adds the SoundCloud params
Dim myQueryStringCollection As New NameValueCollection()
myQueryStringCollection.Add("oauth_token", strAccessToken)
myQueryStringCollection.Add("track[asset_data]", strFileName)
myQueryStringCollection.Add("track[title]", "Test Track Upload")
myQueryStringCollection.Add("track[sharing]", "public")
myQueryStringCollection.Add("track[downloadable]", "true")
myQueryStringCollection.Add("track[post_to]", "")
This attaches the params into the Request
myWebClient.QueryString = myQueryStringCollection
Here’s where I try and send it
Dim responseArray As Byte() = myWebClient.UploadFile(strURL, strFileName)
But I’m receiving an error here: {"errors":[{"error_message":"422 -
Unprocessable Entity"}]}
If I set the Content-type to multi-form, I
get: The Content-Type header cannot be set to a multipart type for
this request
I just tried to get a 500 again, but all I’m getting now is the 422 (so maybe it’s fixed on the other end?).
I have a ServiceStack service with a method to handle a GET request. This method returns binary data.
public object Get(DownloadFile request) {
return new HttpResult(new FileInfo("some file"), "application/octet-stream", asAttachment: true);
}
When the host is Windows it works fine but when I'm running it in Linux with Mono+FastCGI the data I download is not the same.
I analyzed the returned bytes for a few files and concluded that there is a pattern. The data is getting wrapped in this way:
original data size + \r\n + original data + \r\n\r\n0\r\n\r\n
Why is this happening and how to fix it?
Edit:
Turns out this is due to chunked transfers which are part of HTTP 1.1.
Knocte's answer pointed me in the right direction and I was able to work around the problem by forcing my request to use HTTP 1.0:
var req = (HttpWebRequest)WebRequest.Create(url);
req.ProtocolVersion = new Version("1.0");
I didn't need to try the patch suggested by knocte but it looks like it's the proper way to fix the problem instead of avoiding it like I did.
I think you're being affected by this bug.
If the patch that is attached to it works for you, then you could clean it up, and propose it as a pull request to mono in github.