I have an API with a file upload endpoint using Rails and ActiveStorage with S3 as the fileserver. I would like to upload directly to S3 from my client app but the code provided in the Active Storage docs only shows that using Javascript https://edgeguides.rubyonrails.org/active_storage_overview.html#direct-uploads
Since i am sending a POST request with the file to the Rails API directly, there is no place i can run JS.
Is there a way with Rails API only apps to use direct upload?
In order to solve a similar issue I've followed the approach proposed in AWS documentation
The simple concept is that for each file that I want to upload, I do the following workflow:
Request my server for a S3 presigned_url/public url pair
Send the file via post/put (depending on the presigned you choose) to S3
Once I get the 200 (OK) from the S3 upload, I send a new request to my server with the resource that i'm trying to update and in the params for this resource i include the public URL.
e.g:
GET myserver.com/api/signed_url?filename=<safe_file_name>
1.1. Replies with
{
presigned_url: "https://bucket-name.s3.us-west-1.amazonaws.com/uploads/1bb275c5-0199-41fe-ac40-133601f5efb0?x-amz-acl=public-read...",
public_url: "https://bucket-name.s3.us-west-1.amazonaws.com/uploads/1bb275c5-0199-41fe-ac40-133601f5efb0"
}
PUT <presigned_url>, data: <file_to_upload>, cache: false, processData: false
2.1. Wait for 200 (OK) from S3 direct upload
POST myserver.com/api/document, data: { name: 'new file', document_url: <public_url> }
Related
I'm creating an ASP .Net Core 2.1 Web API. The front end (being written in Angular) will consume this API, which is used for a number of things, one of which is saving and retrieving files (pictures, PDF and Word docs, etc.)
We are storing all these files on Amazon S3. I was watching a tutorial video (https://www.youtube.com/watch?v=eRUjPrMMhCc) where the guy shows how to create a bucket, as well as upload and download a file from Amazon S3 from an ASP .Net Core 2.0 Web API, which I thought was fantastic as it's exactly what I needed.
But then I realized that, although the uploading functionality could be useful, the downloading might not be. The reason being that, if the user requests a file (stored on Amazon S3) via the client web app, and this request goes to the API (as was my original intention), then the API would have to first download this file from S3 (which might take a few seconds) and then send it to the client (another few seconds). So the file is being transmitted twice, and therefore unnecessarily slowing down the process of getting a file from S3 to the client.
Is my thinking correct here? Would it be better if the Angular client retrieved the file directly from S3 instead of going via the API? In terms of speed?
Amazon SDK has a methods to handle all you scenarios the principe here is to get a signed URL from Amazon S3 using SDK and then passe it to your front end
import * as AWS from "aws-sdk/global";
AWS.config.update({
region: env.bucketRegion,
});
let clientParams:any = {
region: env.bucketRegion,
apiVersion: '2006-03-01',
params: {Bucket: env.rekognitionBucket}
};
if (environment.s3_endpoint) {
clientParams.endpoint = env.s3_endpoint;
}
let s3 = new S3(clientParams);
let url = s3.getSignedUrl('getObject', {
Bucket: env.rekognitionBucket,
Key: '1234.txt',
});
Have a SPA with a redux client and an express webapi. One of the use cases is to upload a single file from the browser to the express server. Express is using the multer middleware to decode the file upload and place it into an array on the req object. Everything works as expected when running on localhost.
However when the app is deployed to AWS, it does not function as expected. Deployment pushes the express api to an AWS Lambda function, and the redux client static assets are served by Cloudfront CDN. In that environment, the uploaded file does make it to the express server, is handled by multer, and the file does end up as the first (and only) item in the req.files array where it is expected to be.
The problem is that the file contains the wrong bytes. For example when I upload a sample image that is 2795 bytes in length, the file that ends up being decoded by multer is 4903 bytes in length. Other images I have tried always end up becoming larger by approximately the same factor by the time multer decodes and puts them into the req.files array. As a result, the files are corrupted and are not displaying as images.
The file is uploaded like so:
<input type="file" name="files" onChange={this.onUploadFileSelected} />
...
onUploadFileSelected = (e) => {
const file = e.target.files[0]
var formData = new FormData()
formData.append("files", file)
axios.post('to the url', formData, { withCredentials: true })
.then(handleSuccessResponse).catch(handleFailResponse)
}
I have tried setting up multer with both MemoryStorage and DiskStorage. Both work, both on localhost and in the aws lambda, however both exhibit the same behavior -- the file is a larger size and corrupted in the store.
I have also tried setting up multer as both a global middleware (via app.use) and as a route-specific middleware on the upload route (via routes.post('the url', multerMiddlware, controller.uploadAction). Again, both exhibit the same behavior. Multer middleware is configured like so:
const multerMiddleware = multer({/* optionally set dest: '/tmp' */})
.array('files')
One difference is that on localhost, both the client and express are served over http, whereas in aws, both the client and express are served over https. I don't believe this makes a difference, but I have yet been unable to test -- either running localhost over https, or running in aws over http.
Another peculiar thing I noticed was that when the multer middleware is present, other middlewares do not seem to function as expected. Rather than the next() function moving flow down to the controller action, instead, other middlewares will completely exit before the controller action invocation, and when the controller invocation exits, control does not flow back into the middlware after the next() call. When the multer middleware is removed, other middlewares do function as expected. However this observation is on localhost, where the entire end-to-end use case does function as expected.
What could be messing up the uploaded image file payload when deployed to the cloud, but not on localhost? Could it really be https making the difference?
Update 1
When I upload this file (11228 bytes)
Here is the HAR chrome is giving me for the local (expected) file upload:
"postData": {
"mimeType": "multipart/form-data; boundary=----WebKitFormBoundaryC4EJZBZQum3qcnTL",
"text": "------WebKitFormBoundaryC4EJZBZQum3qcnTL\r\nContent-Disposition: form-data; name=\"files\"; filename=\"danludwig.png\"\r\nContent-Type: image/png\r\n\r\n\r\n------WebKitFormBoundaryC4EJZBZQum3qcnTL--\r\n"
}
Here is the HAR chrome is giving me for the aws (corrupted) file upload:
"postData": {
"mimeType": "multipart/form-data; boundary=----WebKitFormBoundaryoTlutFBxvC57UR10",
"text": "------WebKitFormBoundaryoTlutFBxvC57UR10\r\nContent-Disposition: form-data; name=\"files\"; filename=\"danludwig.png\"\r\nContent-Type: image/png\r\n\r\n\r\n------WebKitFormBoundaryoTlutFBxvC57UR10--\r\n"
}
The corrupted image file that is saved is 19369 bytes in length.
Update 2
I created a text file with the text hello world that is 11 bytes long and uploaded it. It does NOT become corrupted in aws. This is the case even if I upload it with the txt or png suffix, it ends up as 11 bytes in length when persisted.
Update 3
Tried uploading with a much larger text file (12132 bytes long) and had the same result as in update 2 -- the file is persisted intact, not corrupted.
Potential answers:
Found this https://forums.aws.amazon.com/thread.jspa?threadID=252327
API Gateway does not natively support multipart form data. It is
possible to configure binary passthrough to then handle this multipart
data in your integration (your backend integration or Lambda
function).
It seems that you may need another approach if you are using API Gateway events in AWS to trigger the lambda that hosts your express server.
Or, you could configure API Gateway to work with binary payloads per https://stackoverflow.com/a/41770688/304832
Or, upload directly from your client to a signed s3 url (or a public one) and use that to trigger another lambda event.
Until we get a chance to try out different API Gateway settings, we found a temporary workaround: using FileReader to convert the file to a base64 text string, then submit that. The upload does not seem to have any issues as long as the payload is text.
I am trying to upload large files to S3 in chunks. This needs to be done via browser. I was looking at FineUploader as solution. https://fineuploader.com/
It would need a Signing function which gives me the correct Authorization header.
This is specified in FineUploader as :
signature: {
endpoint: "/s3/signtureHandler",
version: 4
}
I saw the current examples have Servlet and PHP. I wanted to to know if we can get the same achieved by a Lambda function exposed by an API Gateway.
Has anyone tried it before or know of any potential pitfalls?
I'm writing a Golang REST API that needs to generate S3 tokens for users so that they can upload files to the specific folder inside a bucket.
This is what I need to implement:
Method: GET
Endpoint: myapp.com/images/:imageid/token
Description: Return 2 tokens so that the user can upload files to the S3
bucket related to the image ID parameter.
I'm using the Golang echo framework.
And I'm not really sure how to implement this functionality.
Should this be done via the AWS SDK or does Amazon offer other ways of programmatically generrating tokens?
To generate tokens for users to upload files directly to s3, you can use pre-signed URLS.
After generating a pre-signed URL, return that to the user and the calling application can use that to upload the file.
This example from the link above should be about what you're looking for:
svc := s3.New(session.New(&aws.Config{Region: aws.String("us-west-2")}))
req, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String("myBucket"),
Key: aws.String("myKey"),
})
str, err := req.Presign(15 * time.Minute)
log.Println("The URL is:", str, " err:", err)
I have created an api in amazon api gateway service with s3 proxy, and created a method post to upload a file to s3 using the document. Deployed the API and then using that url i tried to call the api from postman. But i couldn't post the file and it returns an error 'missing authentication token'.
I set authorization as NONE.
Then it returns an Unexpected "<" error.
Ah, okay. S3 only supports POST from an HTML form, which is why you see the error where it is expecting multipart formdata.
The method you need to use is PUT, instead of POST. PUT requires an item path, so you'll need to change the resource path to have a bucket and key, or get those from other places.
I have some more info on how to set this up in upload binary from api gateway to S3 bucket
It sounds like the document you're uploading isn't JSON. By default, API Gateway expects UTF-8 encoded JSON.
What content type are you sending with your Postman request?