My application (MVC) needs to download, zip and return one or many files from Amazon S3. I am using the .NET SDK and GetObject to receive the files, and want to use DotNetZip to then zip them up and return the generated zip file as a file stream result for the user to download.
Can anyone suggest the most efficient way of doing this, I am seeing OutOfMemory exceptions when downloading large files from S3, they could be up to 1gb in size for example.
My code so far;
using (
var client = AWSClientFactory.CreateAmazonS3Client(
"apikey",
"apisecret",
new AmazonS3Config { RegionEndpoint = RegionEndpoint.EUWest1 })
)
{
foreach (var file in files)
{
var request = new GetObjectRequest { BucketName = "bucketname", Key = file };
using (var response = client.GetObject(request))
{
}
}
}
If I copy the response into a memory stream and add that to the zip, all works ok (on small files), but with large files assume I cannot store the entire thing in memory?
Related
I have a legacy system interfacing issue that my team has elected to solve by standing up a .NET 7 Minimal API which needs to accept a file upload. It should work for small and large files (let's say at least 500 MiB). The API will be called from a legacy system using HttpClient in a .NET Framework 4.7.1 app.
I can't quite seem to figure out how to design the signature of the Minimal API and how to call it with HttpClient in a way that totally works. It's something I've been hacking at on and off for several days, and haven't documented all of my approaches, but suffice it to say there have been varying results involving, among other things:
4XX and 500 errors returned by the HTTP call
An assortment of exceptions on either side
Calls that throw and never hit a breakpoint on the API side
Calls that get through but the Stream on the API end is not what I expect
Errors being different depending on whether the file being uploaded is small or large
Text files being persisted on the server that contain some of the HTTP headers in addition to their original contents
On the Minimal API side, I've tried all sorts of things in the signature (IFormFile, Stream, PipeReader, HttpRequest). On the calling side, I've tried several approaches (messing with headers, using the Flurl library, various content encodings and MIME types, multipart, etc).
This seems like it should be dead simple, so I'm trying to wipe the slate clean here, start with an example of something that partially works, and hope someone might be able to illuminate the path forward for me.
Example of Minimal API:
// IDocumentStorageManager is an injected dependency that takes an int and a Stream and returns a string of the newly uploaded file's URI
app.MapPost(
"DocumentStorage/CreateDocument2/{documentId:int}",
async (PipeReader pipeReader, int documentId, IDocumentStorageManager documentStorageManager) =>
{
using var ms = new MemoryStream();
await pipeReader.CopyToAsync(ms);
ms.Position = 0;
return await documentStorageManager.CreateDocument(documentId, ms);
});
Call the Minimal API using HttpClient:
// filePath is the path on local disk, uri is the Minimal API's URI
private static async Task<string> UploadWithHttpClient2(string filePath, string uri)
{
var fileStream = File.Open(filePath, FileMode.Open);
var content = new StreamContent(fileStream);
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri);
var httpClient = new HttpClient();
httpRequestMessage.Content = content;
httpClient.Timeout = TimeSpan.FromMinutes(5);
var result = await httpClient.SendAsync(httpRequestMessage);
return await result.Content.ReadAsStringAsync();
}
In the particular example above, a small (6 bytes) .txt file is uploaded without issue. However, a large (619 MiB) .tif file runs into problems on the call to httpClient.SendAsync which results in the following set of nested Exceptions:
System.Net.Http.HttpRequestException - "Error while copying content to a stream."
System.IO.IOException - "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.."
System.Net.Sockets.SocketException - "An existing connection was forcibly closed by the remote host."
What's a decent way of writing a Minimal API and calling it with HttpClient that will work for small and large files?
Kestrel allows uploading 30MB per default.
To upload larger files via kestrel you might need to increase the max size limit. This can be done by adding the "RequestSizeLimit" attribute. So for example for 1GB:
app.MapPost(
"DocumentStorage/CreateDocument2/{documentId:int}",
[RequestSizeLimit(1_000_000_000)] async (PipeReader pipeReader, int documentId) =>
{
using var ms = new MemoryStream();
await pipeReader.CopyToAsync(ms);
ms.Position = 0;
return "";
});
You can also remove the size limit globally by setting
builder.WebHost.UseKestrel(o => o.Limits.MaxRequestBodySize = null);
This answer is good but the RequestSizeLimit filter doesn't work for minimal APIs, it's an MVC filter. You can use the IHttpMaxRequestBodySizeFeature to limit the size (assuming you're not running on IIS). Also, I made a change to accept the body as a Stream. This avoids the memory stream copy before calling the CreateDocument API:
app.MapPost(
"DocumentStorage/CreateDocument2/{documentId:int}",
async (Stream stream, int documentId, IDocumentStorageManager documentStorageManager) =>
{
return await documentStorageManager.CreateDocument(documentId, stream);
})
.AddEndpointFilter((context, next) =>
{
const int MaxBytes = 1024 * 1024 * 1024;
var maxRequestBodySizeFeature = context.HttpContext.Features.Get<IHttpMaxRequestBodySizeFeature>();
if (maxRequestBodySizeFeature is not null and { IsReadOnly: true })
{
maxRequestBodySizeFeature.MaxRequestBodySize = MaxBytes;
}
return next(context);
});
If you're running on IIS then https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/requestlimits/#configuration
Team,
I have a blazor web assembly app, which upload the file and process it later. However , I would like to know the base path of the file from where it it picked in the machine.
My code goes as follows . Does anyone has idea to get the file path such as "C:\myfile.txt".
With the File object, I cannot achieve the full path, I can access only its memory stream.
<h1>FILE UPLAOD </h1>
<InputFile OnChange="HandleSelection" ></InputFile>
#code
{
string status;
async Task HandleSelection(IFileListEntry[] files)
{
var file = files.FirstOrDefault();
if (file != null)
{
// Just load into .NET memory to show it can be done
// Alternatively it could be saved to disk, or parsed in memory, or similar
var ms = new MemoryStream();
await file.Data.CopyToAsync(ms);
Console.WriteLine(ms);
status = $"Finished loading {file.Size} bytes from {file.Name}";
var content = new MultipartFormDataContent
{
{ new ByteArrayContent(ms.GetBuffer()),"\"upload\"", file.Name}
};
await client.PostAsync("upload", content);
}
}
}
Even if you get the fullpath (C:\myfile.txt") file won't load
by default, all browser has a security mechanism that any local disk file won't be loaded into a website until you disable that security for your website
Is it possible to download a file from AWS s3 without AWS cli? In my production server I would need to download a config file which is in S3 bucket.
I was thinking of having Amazon Systems Manger run a script that would download the config (YAML files) from the S3. But we do not want to install AWS cli on the production machines. How can I go about this?
You would need some sort of program to call the Amazon S3 API to retrieve the object. For example, a PowerShell script (using AWS Tools for Windows PowerShell) or a Python script that uses the AWS SDK.
You could alternatively generate an Amazon S3 pre-signed URL, which would allow a private object to be downloaded from Amazon S3 via a normal HTTPS call (eg curl). This can be done easily using the AWS SDK for Python, or you could code it yourself without using libraries (it's a bit more complex).
In all examples above, you would need to provide the script/program with a set of IAM Credentials for authenticating with AWS.
Just adding notes for any C# code lovers to solve problem with .Net
Firstly write(C#) code to download private file as string
public string DownloadPrivateFileS3(string fileKey)
{
string accessKey = "YOURVALUE";
string accessSecret = "YOURVALUE";;
string bucket = "YOURVALUE";;
using (s3Client = new AmazonS3Client(accessKey, accessSecret, "YOURVALUE"))
{
var folderPath = "AppData/Websites/Cases";
var fileTransferUtility = new TransferUtility(s3Client);
Stream stream = fileTransferUtility.OpenStream(bucket, folderPath + "/" + fileKey);
using (var memoryStream = new MemoryStream())
{
stream.CopyTo(memoryStream);
var response = memoryStream.ToArray();
return Convert.ToBase64String(response);
}
return "";
}
}
Second Write JQuery Code to download string as Base64
function downloadPrivateFile() {
$.ajax({url: 'DownloadPrivateFileS3?fileName=' + fileName, success: function(result){
var link = this.document.createElement('a');
link.download = fileName;
link.href = "data:application/octet-stream;base64," + result;
this.document.body.appendChild(link);
link.click();
this.document.body.removeChild(link);
}});
}
Call downloadPrivateFile method from anywhere of HTML/C#/JQuery -
Enjoy Happy Coding and Solutions of Complex Problems
I do not have much experience with flutter, what I would like to know is if there is any way to download a video of amazon s3 and save the file in the memory of the cell phone, thanks
This is the url of the video "https://s3-us-west-1.amazonaws.com/videos.ecuestre.digital/165-3745-40957-1.mp4"
This is how I downloaded a binary file in flutter :
In your pubspec.yaml, add
path_provider: ^0.2.2
includes :
import 'dart:typed_data';
import 'package:flutter/foundation.dart';
import 'package:path_provider/path_provider.dart';
code :
static var httpClient = new HttpClient();
Future<File> _downloadFile(String url, String filename) async {
var request = await httpClient.getUrl(Uri.parse(url));
var response = await request.close();
var bytes = await consolidateHttpClientResponseBytes(response);
String dir = (await getApplicationDocumentsDirectory()).path;
File file = new File('$dir/$filename');
await file.writeAsBytes(bytes);
return file;
}
The function consolidateHttpClientResponseBytes() comes from the flutter docs :
https://docs.flutter.io/flutter/foundation/consolidateHttpClientResponseBytes.html
Note: Edited to more concise version from Simon Lightfoot
If you are using aws resources you can use the Amplify framework to interact with your environment. This is especially helpful if you need interact with a lot of resources. Check the flutter amplify storage docs
I am bit new to Apache HC API. I am trying to download huge file from server (10 GB) on cloud environment and then I have to upload on to Amazon S3.
As file is too big, it comes up with transfer encoding as chunked and gzip format. Cloud env neither have enough disk space to store this file as temp file nor such file can be accommodated in memory.
Mainly I have 2 interfaces,
ResourceDownloader {
InputStream download(AbstractChannel channel);
}
ResourceUploader{
void upload(AbstractChannel channel, InputStream inputStream);
}
Part1:
While using Apache Httpclient lib, I am seeing return http response with following structure,
ResponseEntityProxy contains >> {
- BasicHttpEntity [WrappedEntity]
- content as ChunkeInputStream
}
So does this response mean whole 10GB would be available in memory byte buffer at client side after completion of client.execute(getMethod) call?
Or is it like as soon as I invoke call to read as below, it would get chunks from server?
[In real case, disk would not be available , but below is just for demo]
try {
FileOutputStream fos = (FileOutputStream) outputStream;
if(inputStream instanceof GZIPInputStream) {
byte[] buffer = new byte[1024];
int len;
while((len = inputStream.read(buffer)) != -1){
fos.write(buffer, 0, len);
}
//close resources
fos.close();
inputStream.close();
}
} catch (IOException e) {
logger.error("Exception occurred while processing file on disk", e);
}
Part2:
I know multi part upload if I have content length or full file available, but in case of chunked input stream, how should we upload it to Amazon S3?
Thanks,
Dharam
HttpClient always streams request and response entities unless specifically instructed to do otherwise.