I'm using a service that puts the data I need on S3 and gives me a list of presigned URLs to download (http://.s3.amazonaws.com/?AWSAccessKeyID=...&Signature=...&Expires=...).
I want to copy those files into my S3 bucket without having to download them and upload again.
I'm using the Ruby SDK (but willing to try something else if it works..) and couldn't write anything like this.
I was able to initialize the S3 object with my credentials (access_key and secret) that grants me access to my bucket, but how do I pass the "source-side" access_key_id, signature and expires parameters?
To make the problem a bit simpler - I can't even do a GET request to the object using the presigned parameters. (not with regular HTTP, I want to do it through the SDK API).
I found a lot of examples of how to create a presigned URL but nothing about how to authenticate using an already given parameters (I obviously don't have the secret_key of my data provider).
Thanks!
You can't do this with a signed url, but as has been mentioned, if you fetch and upload within EC2 in an appropriate region for the buckets in question, there's essentially no additional cost.
Also worth noting, both buckets do not have to be in the same account, but the aws key that you use to make the request have to have permission to put the target object and get the source object. Permissions can be granted across accounts... though in many cases, that's unlikely to be granted.
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
You actually can do a copy with a presigned URL. To do this, you need to create a presigned PUT request that also includes a header like x-amz-copy-source: /sourceBucket/sourceObject in order to specify where you are copying from. In addition, if you want the copied object to have new metadata, you will also need to add the header x-amz-metadata-directive: REPLACE. See the REST API documentation for more details.
Related
Our application only have Front End and Back End. Each user can have a lot of ducument which is uploaded by backend side but need to be display on FE. We trying to use pre-signed URL to let FE download document directly from S3.
BE return the docutments path, and expose an API which can be use to generate pre-signed url for each document.
POST: /app/v1/generate-pre-signed-url/path-to-document
The issue is above generate pre-signed url API need to be verity that current user has access to the document or not, and it's realy hard to map the user permission with a document path.
Any suggestion design or pattern would be appreciate!
As per AWS Docs
When you create a presigned URL for your object, you must provide your security credentials and then specify a bucket name, an object key, an HTTP method (GET to download the object), and an expiration date and time.
So in summary, only object key would not suffice (like in your case).
I've just realized that if I allow the browser to upload a file to my S3 bucket (using a session token from my server), an attacker can use known object keys to use those temporary permissions to overwrite those files (and either replace with malicious or empty content).
Some say the solution would be to use object versioning, but I'm wondering if a lambda function can intercept that PutObject request, check if the key already exists in the bucket, and if so, deny the operation.
The short answer is no.
This is because S3 is eventually consistent. Even if you did something clever like attempting a getObject to see if the file exists, you may well get a false negative under the heavy quick-fire load you're expecting from an attacker.
If you want to ensure that an signed url can be used once and only once, then you'll have to replace the signed url functionality with your own. An example would be to use API Gateway + Lambda + DynamoDB. In this case you would create an 'upload token' which you would save to DynamoDB and return to the user. When the user then uploads a file using the token, it is removed from DynamoDB (which can be made immediately consistent).
I am testing S3 calls using DHC REST client in Chrome. In these tests, the Authorization is all based on my root user credentials.
I can do a GET with //mybucket.s3.amazonws.com, and a list of the items in mybucket is returned.
If I add an item to retrieve (//mybucket.s3.amazonws.com/myitem), I always get 403 Forbidden.
I thought that the root user had automatic access to the objects, but am I wrong about that?
I took screen prints of both tests, which I'll supply if needed.
After some further monkeying around, I found my answer. Yes, the AWS root user can access individual items. But the Authorization header string changes. When you retrieve an object, that object's key participates in the calculation of the auth string. Thus, the same string used to retrieve the bucket list does not work when retrieving an object.
Once you stored contents in S3 and make it public, then everyone have access to it. Is there a way to let only authorized users have access to the content stored in S3? For example, I have a site that let people store their documents. The server stores these documents in S3 and I would like only the user who uploaded the document to have access to it.
I know I can copy the S3 contents to my server and let only authorized users have access, but this will make the server slow. I would like to be able server the contents directly to the client's browser from the S3.
Thanks.
The link given in the above answer is no longer correct -- Amazon had it's documentation reorganized. I think these are the correct pages to read:
http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?RESTAuthentication.html#RESTAuthenticationQueryStringAuth
http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html
You want to read the section called 'Query String Request Authentication Alternative' found here
It explains how to create a time-based expiring link to an S3 object
You would then have to write the code that manages the users (the who owns which object part of your question).
Is there a way to create a different identity to (access key / secret key) to access Amazon S3 buckets via the REST API where I can restrict access (read only for example)?
The recommended way is to use IAM to create a new user, then apply a policy to that user.
Yes, you can. The S3 API documentation describes the Authentication and Access Control services available to you. You can set up a bucket so that another Amazon S3 account can read but not modify items in the bucket.
Check out the details at http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?UsingAuthAccess.html (follow the link to "Using Query String Authentication")- this is a subdocument to the one Greg Posted, and describes how to generate access URLs on the fly.
This uses a hashed form of the private key and allows expiration, so you can give brief access to files in a bucket without allowed unfettered access to the rest of the S3 store.
Constructing the REST URL is quite difficult, it took me about 3 hours of coding to get it right, but this is a very powerful access technique.