I'm not a developer, but recently have been doing lots of automations at work using VBA, Python and AWS. Recently I started learning and working with API's in AWS.
I've been trying to upload a CSV file through a REST API, but it's just not working as expected.
Basically I have used API Gateway to set up my API that triggers a Lambda Function to upload the file to a S3 bucket.
When I tested my API in Postman, it worked fine, I just had to set Headers as Key: "Content-Type" and Value: "application/csv", then in the Body part I selected "Binary" and browsed my csv file that I want to upload.
But the problem is when I tried to call my api from my application (or from Postman using the raw text instead of binary), I am sending the following Body (Also tried without double quotes and didn't work either):
--data-binary "\file-location\file-name.csv"
The api returns a success code, but when I open the file in S3, then content is "--data-binary "\file-location\file-name.csv"" instead of the actual file's content that I selected.
This is the code I'm using in my VBa to call the API:
Dim ws As Worksheet
Dim URL, env, msg, result As String
Dim objHTTP As Object
Set ws = ThisWorkbook.Worksheets("Sheet1")
URL = "https://api-url"
msg = "--data-binary ""\\file_location\file_name.csv"""
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
objHTTP.Open "POST", URL, False
objHTTP.setRequestHeader "Content-Type", "application/csv"
objHTTP.send (msg)
result = objHTTP.responseText
'getResponse
ws.Range("s3_api_resp").Value = result
Set objHTTP = Nothing
Uploading a file should be done directly to Amazon S3, the reason I say this, is that AWS Lambda has a limit of 6MBs as an invocation payload.
There are several ways to upload to S3, here are three ways:
SDK
CLI
Rest API
There is no SDK for VB, but if you are using VB .Net you could use the .Net SDK. An example is available here. Alternatively you could invoke the AWS CLI via VB, that is not so slick (more of a hack) but possible. Details are available here. Alternatively you can upload a file using the AWS S3 Rest APIs, details available here.
Files can be uploaded to Amazon S3 without credentials, provided the bucket policy allows open buckets. Examples of bucket policies are available here.
If the bucket is locked down (recommended) then you should use temporary credentials to upload to Amazon S3. You can get temporary credentials by either providing pre-signed URLs for PUT or you can grant temporary credentials using a service such as Amazon Cognito. Amazon Cognito comprises of two parts, User Pools to store credentials and Federated/Identity Pools. You can use Amazon Cognito Identity Pools to generate temporary credentials to Amazon Identity and Access Management using authenticated credentials. Authenticated credentials can be sourced from Apple, Google, Amazon, Cognito, SAML, and OpenID. The authenticated credentials are exchanged for IAM credentials. The IAM credentials are attached to a policy, and that policy should have permissions to upload files.
Related
I can successfully send the InitiateMultipartUploadRequest and get InitiateMultipartUploadResponse back, but then get Access Denied error when sending the 1st UploadPartRequest.
Note that all of the below cases upload the document successfully:
Exactly the same code (i.e. using multipart upload), but to a different bucket that uses SSE-S3 encryption.
Using low-level API and uploading the document in one go, i.e. creating PutObjectRequest and then calling amazonS3Client.PutObjectAsync(putObjectRequest).
Using high-level API TransferUtility class.
Maybe the encryption key was not forwarded in the call properly.
I am looking for a solution on the best practice that needs to be followed in AWS S3 access by third party who do not have account in S3.
In my case there are REST interface which would need to provide the link of images .This images resides on AWS S3. Based on the identity of the caller is there a way we can give access to the user. I would not want to make the access level of the bucket to public.
Say if we get a call from user X ( may be we ask them to set a new header ) we allow them the access to the bucket.
As this API is enterprise and we have partners using this API we would want only some of the identified callers to have access to the images.
Any pointers will help a lot.
Signed S3 URL's, make the bucket private, only accessible to your API via an IAM role, if the API is running on EC2, lambda etc.
Your API would do the authentication and authorization, then provide the caller a signed s3 url to download the image.
When you create a pre-signed URL for your object, you must provide your security credentials, specify a bucket name, an object key, specify the HTTP method (GET to download the object) and expiration date and time. The pre-signed URLs are valid only for the specified duration.
Anyone who receives the pre-signed URL can then access the object. For
example, if you have a video in your bucket and both the bucket and
the object are private, you can share the video with others by
generating a pre-signed URL.
I have created an api in amazon api gateway service with s3 proxy, and created a method post to upload a file to s3 using the document. Deployed the API and then using that url i tried to call the api from postman. But i couldn't post the file and it returns an error 'missing authentication token'.
I set authorization as NONE.
Then it returns an Unexpected "<" error.
Ah, okay. S3 only supports POST from an HTML form, which is why you see the error where it is expecting multipart formdata.
The method you need to use is PUT, instead of POST. PUT requires an item path, so you'll need to change the resource path to have a bucket and key, or get those from other places.
I have some more info on how to set this up in upload binary from api gateway to S3 bucket
It sounds like the document you're uploading isn't JSON. By default, API Gateway expects UTF-8 encoded JSON.
What content type are you sending with your Postman request?
I have a REST API with API gateway and Lambda.
I wan't to create an endpoint for uploading a profile picture, that passes the file to a Lambda function, where it is been resized, registers it to the database and returning the url path of the new image.
Is there any way to do so with those services?
Couldn't find anything online (the only suggestion I found is uploading directly to S3, which requires IAM permissions, and having an event triggering a Lambda function that resizing the picture).
Thanks
UPDATE
AWS updated APIGATEWAY and know you can send binaries through an endpoint
Thanks to #blue and #Manzo for commenting it
Uploading a file directly to S3 doesn't necessarily require IAM permissions. You would create an API endpoint that returns a pre-signed S3 URL, which could then be used to upload the file directly to S3. The Lambda function behind the API endpoint would be the only thing that needed the correct IAM permissions for the S3 bucket.
Since API Gateway and Lambda don't support natively currently, you can pass the file to a picture in based64 encoded to API Gateway then pass to Lambda function. Your Lambda function can based64 decoded, then resized, registers it to the database and returning the url path of the new image.
What is the proper way of delegating file access authentication from S3 to our authentiation service?
For example: web site's user(he have our session id in headers) sending request to S3 to get file by url. S3 sends request to our authentication service asking if user with such headers can access that file, and if our auth service allow getting that file it will be downloaded.
There are a lot of information about presigned requests but absolutely nothing about s3 quering with "hidden" authentication.
If a file has been made public on S3, then of course anyone can download it, using a direct link to the file.
If the file is not public, then there needs to be some type of authentication. There are really only two ways a file from S3 can be obtained if it is not public, one is via a pre-signed url, and the other is to be an Amazon user who has access to S3. Obviously this is how it works when you yourself want to access an object on S3, you must provide your access key and a signature in the header of the GET request. You can grant other users access to S3 via Amazon IAM, which is more like the 'hidden' authentication you mentioned. Via the IAM route, there are different ways of providing access including Federated Users. Visit this link to learn more:
http://docs.aws.amazon.com/AmazonS3/latest/dev/MakingAuthenticatedRequests.html
If you are simply trying to provide a authenticated user access to a file, the best and easiest way to do that would be to create a pre-signed url with an expiration time. The expiration time can be something short, like 10 minutes or even 1 minute, to prevent the user from passing the link to others.