I'm writing a Golang REST API that needs to generate S3 tokens for users so that they can upload files to the specific folder inside a bucket.
This is what I need to implement:
Method: GET
Endpoint: myapp.com/images/:imageid/token
Description: Return 2 tokens so that the user can upload files to the S3
bucket related to the image ID parameter.
I'm using the Golang echo framework.
And I'm not really sure how to implement this functionality.
Should this be done via the AWS SDK or does Amazon offer other ways of programmatically generrating tokens?
To generate tokens for users to upload files directly to s3, you can use pre-signed URLS.
After generating a pre-signed URL, return that to the user and the calling application can use that to upload the file.
This example from the link above should be about what you're looking for:
svc := s3.New(session.New(&aws.Config{Region: aws.String("us-west-2")}))
req, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String("myBucket"),
Key: aws.String("myKey"),
})
str, err := req.Presign(15 * time.Minute)
log.Println("The URL is:", str, " err:", err)
Related
I'm not a developer, but recently have been doing lots of automations at work using VBA, Python and AWS. Recently I started learning and working with API's in AWS.
I've been trying to upload a CSV file through a REST API, but it's just not working as expected.
Basically I have used API Gateway to set up my API that triggers a Lambda Function to upload the file to a S3 bucket.
When I tested my API in Postman, it worked fine, I just had to set Headers as Key: "Content-Type" and Value: "application/csv", then in the Body part I selected "Binary" and browsed my csv file that I want to upload.
But the problem is when I tried to call my api from my application (or from Postman using the raw text instead of binary), I am sending the following Body (Also tried without double quotes and didn't work either):
--data-binary "\file-location\file-name.csv"
The api returns a success code, but when I open the file in S3, then content is "--data-binary "\file-location\file-name.csv"" instead of the actual file's content that I selected.
This is the code I'm using in my VBa to call the API:
Dim ws As Worksheet
Dim URL, env, msg, result As String
Dim objHTTP As Object
Set ws = ThisWorkbook.Worksheets("Sheet1")
URL = "https://api-url"
msg = "--data-binary ""\\file_location\file_name.csv"""
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
objHTTP.Open "POST", URL, False
objHTTP.setRequestHeader "Content-Type", "application/csv"
objHTTP.send (msg)
result = objHTTP.responseText
'getResponse
ws.Range("s3_api_resp").Value = result
Set objHTTP = Nothing
Uploading a file should be done directly to Amazon S3, the reason I say this, is that AWS Lambda has a limit of 6MBs as an invocation payload.
There are several ways to upload to S3, here are three ways:
SDK
CLI
Rest API
There is no SDK for VB, but if you are using VB .Net you could use the .Net SDK. An example is available here. Alternatively you could invoke the AWS CLI via VB, that is not so slick (more of a hack) but possible. Details are available here. Alternatively you can upload a file using the AWS S3 Rest APIs, details available here.
Files can be uploaded to Amazon S3 without credentials, provided the bucket policy allows open buckets. Examples of bucket policies are available here.
If the bucket is locked down (recommended) then you should use temporary credentials to upload to Amazon S3. You can get temporary credentials by either providing pre-signed URLs for PUT or you can grant temporary credentials using a service such as Amazon Cognito. Amazon Cognito comprises of two parts, User Pools to store credentials and Federated/Identity Pools. You can use Amazon Cognito Identity Pools to generate temporary credentials to Amazon Identity and Access Management using authenticated credentials. Authenticated credentials can be sourced from Apple, Google, Amazon, Cognito, SAML, and OpenID. The authenticated credentials are exchanged for IAM credentials. The IAM credentials are attached to a policy, and that policy should have permissions to upload files.
I have an API with a file upload endpoint using Rails and ActiveStorage with S3 as the fileserver. I would like to upload directly to S3 from my client app but the code provided in the Active Storage docs only shows that using Javascript https://edgeguides.rubyonrails.org/active_storage_overview.html#direct-uploads
Since i am sending a POST request with the file to the Rails API directly, there is no place i can run JS.
Is there a way with Rails API only apps to use direct upload?
In order to solve a similar issue I've followed the approach proposed in AWS documentation
The simple concept is that for each file that I want to upload, I do the following workflow:
Request my server for a S3 presigned_url/public url pair
Send the file via post/put (depending on the presigned you choose) to S3
Once I get the 200 (OK) from the S3 upload, I send a new request to my server with the resource that i'm trying to update and in the params for this resource i include the public URL.
e.g:
GET myserver.com/api/signed_url?filename=<safe_file_name>
1.1. Replies with
{
presigned_url: "https://bucket-name.s3.us-west-1.amazonaws.com/uploads/1bb275c5-0199-41fe-ac40-133601f5efb0?x-amz-acl=public-read...",
public_url: "https://bucket-name.s3.us-west-1.amazonaws.com/uploads/1bb275c5-0199-41fe-ac40-133601f5efb0"
}
PUT <presigned_url>, data: <file_to_upload>, cache: false, processData: false
2.1. Wait for 200 (OK) from S3 direct upload
POST myserver.com/api/document, data: { name: 'new file', document_url: <public_url> }
I'm creating an ASP .Net Core 2.1 Web API. The front end (being written in Angular) will consume this API, which is used for a number of things, one of which is saving and retrieving files (pictures, PDF and Word docs, etc.)
We are storing all these files on Amazon S3. I was watching a tutorial video (https://www.youtube.com/watch?v=eRUjPrMMhCc) where the guy shows how to create a bucket, as well as upload and download a file from Amazon S3 from an ASP .Net Core 2.0 Web API, which I thought was fantastic as it's exactly what I needed.
But then I realized that, although the uploading functionality could be useful, the downloading might not be. The reason being that, if the user requests a file (stored on Amazon S3) via the client web app, and this request goes to the API (as was my original intention), then the API would have to first download this file from S3 (which might take a few seconds) and then send it to the client (another few seconds). So the file is being transmitted twice, and therefore unnecessarily slowing down the process of getting a file from S3 to the client.
Is my thinking correct here? Would it be better if the Angular client retrieved the file directly from S3 instead of going via the API? In terms of speed?
Amazon SDK has a methods to handle all you scenarios the principe here is to get a signed URL from Amazon S3 using SDK and then passe it to your front end
import * as AWS from "aws-sdk/global";
AWS.config.update({
region: env.bucketRegion,
});
let clientParams:any = {
region: env.bucketRegion,
apiVersion: '2006-03-01',
params: {Bucket: env.rekognitionBucket}
};
if (environment.s3_endpoint) {
clientParams.endpoint = env.s3_endpoint;
}
let s3 = new S3(clientParams);
let url = s3.getSignedUrl('getObject', {
Bucket: env.rekognitionBucket,
Key: '1234.txt',
});
I am looking for a solution on the best practice that needs to be followed in AWS S3 access by third party who do not have account in S3.
In my case there are REST interface which would need to provide the link of images .This images resides on AWS S3. Based on the identity of the caller is there a way we can give access to the user. I would not want to make the access level of the bucket to public.
Say if we get a call from user X ( may be we ask them to set a new header ) we allow them the access to the bucket.
As this API is enterprise and we have partners using this API we would want only some of the identified callers to have access to the images.
Any pointers will help a lot.
Signed S3 URL's, make the bucket private, only accessible to your API via an IAM role, if the API is running on EC2, lambda etc.
Your API would do the authentication and authorization, then provide the caller a signed s3 url to download the image.
When you create a pre-signed URL for your object, you must provide your security credentials, specify a bucket name, an object key, specify the HTTP method (GET to download the object) and expiration date and time. The pre-signed URLs are valid only for the specified duration.
Anyone who receives the pre-signed URL can then access the object. For
example, if you have a video in your bucket and both the bucket and
the object are private, you can share the video with others by
generating a pre-signed URL.
I have a REST API with API gateway and Lambda.
I wan't to create an endpoint for uploading a profile picture, that passes the file to a Lambda function, where it is been resized, registers it to the database and returning the url path of the new image.
Is there any way to do so with those services?
Couldn't find anything online (the only suggestion I found is uploading directly to S3, which requires IAM permissions, and having an event triggering a Lambda function that resizing the picture).
Thanks
UPDATE
AWS updated APIGATEWAY and know you can send binaries through an endpoint
Thanks to #blue and #Manzo for commenting it
Uploading a file directly to S3 doesn't necessarily require IAM permissions. You would create an API endpoint that returns a pre-signed S3 URL, which could then be used to upload the file directly to S3. The Lambda function behind the API endpoint would be the only thing that needed the correct IAM permissions for the S3 bucket.
Since API Gateway and Lambda don't support natively currently, you can pass the file to a picture in based64 encoded to API Gateway then pass to Lambda function. Your Lambda function can based64 decoded, then resized, registers it to the database and returning the url path of the new image.