S3 spool bucket / anonymous PUT - file-upload

I'd like to use S3 CORS and key expiration to create a 'spool bucket'. This spool bucket should support the following:
Frontend code (say, jQuery) should be able to HTTP PUT arbitrarily named keys to the bucket
Backend code (say, Python/boto) should be able to read and delete these keys, given a key name
keys should expire by themselves after X days
How exactly should this be achieved?

After some mucking about, here's what I found:
Create a bucket
Give everyone the permissions Upload/Delete and Edit Permissions (should be easy with the API, my case was easier with the console).
Set the following (or similar) CORS policy:
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*.example.com</AllowedOrigin>
<AllowedMethod>PUT</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
set an object lifecycle policy to match your wishes (again, easy with the API, easier in my case with the console)
When doing your PUT, add the header x-amz-acl: bucket-owner-full-control. For example, in CoffeeScript and jQuery, your PUT will look like this:
$.ajax http://my_spool_bucket.s3.amazonaws.com,
type: "PUT"
data: "contents of new object"
headers:
"x-amz-acl": "bucket-owner-full-control"
That's it. Bask in the glory of highly available, self-expunging, CORS compatible, cheap and easy HTTP spool space. One important thing to remember: any client can overwrite any key; you should design accordingly (I use cryptographically secure generated keys).

Related

XACML Authzforce PDP configuration in multiple policy files

I'm running XACML using the Authzforce PDP engine and a configuration pdp.xml file, that looks like:
<?xml version="1.0" encoding="UTF-8"?>
<pdp xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://authzforce.github.io/core/xmlns/pdp/6.0"
version="6.0.0">
<rootPolicyProvider id="rootPolicyProvider"
xsi:type="StaticRootPolicyProvider" policyLocation="${PARENT_DIR}/policy.xml" />
</pdp>
Now, the file ${PARENT_DIR}/policy.xml, that is read by the PDP engine through the rootPolicyProvider contains the actual XACML policies and is becoming fairly large. So, I would like to divide the XACML policies in multiple files policy1.xml, policy2.xml, policy3.xml, etc. These files then need to be read by the PDP engine.
Does anyone know whether the PDP engine configuration xml-file is able to specify this using multiple policyProviders or otherwise? It shouldn't be too difficult, but I have not found any solution yet after a few hours of search on the web.
Looking forward to your replies.
Thx, Jack.
For this use case, I recommend to upgrade to AuthzForce Core 14.0.0 or later. Then you have two options (beware the XML schema and namespace have changed a bit):
Multiple 'policyLocation' elements, for example:
<?xml version="1.0" encoding="UTF-8"?>
<pdp xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://authzforce.github.io/core/xmlns/pdp/7.0" version="7.0.0">
<policyProvider id="refPolicyprovider" xsi:type="StaticPolicyProvider">
<policyLocation>${PARENT_DIR}/policy1.xml</policyLocation>
<policyLocation>${PARENT_DIR}/policy2.xml</policyLocation>
</policyProvider>
<rootPolicyRef>policy1</rootPolicyRef>
</pdp>
Use a wildcard pattern as 'policyLocation', for example (including all policy files with '.xml' extension):
<?xml version="1.0" encoding="UTF-8"?>
<pdp xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://authzforce.github.io/core/xmlns/pdp/7.0" version="7.0.0">
<policyProvider id="refPolicyprovider" xsi:type="StaticPolicyProvider">
<policyLocation>${PARENT_DIR}/*.xml</policyLocation>
</policyProvider>
<rootPolicyRef>policy1</rootPolicyRef>
</pdp>
In both cases, the 'rootPolicyRef' identifies the root policy (where the PDP evaluation starts). In this case, the root policy is supposed to combine the other policies, i.e. be a XACML PolicySet with a defined PolicyCombiningAlgId and one or more PolicyIdReferences or PolicySetIdReferences to the other policies.
You can find a full example using the wildcard option on authzforce github.
Also you can find more info about the PDP configuration format (latest version) in the XML schema.

How to set the policy order in Mule API gateway standalone?

I have found a lot of documentation about how to set the order in which the policies are applied to a proxy using API Manager but nothing about how to change it when you are using only the API gateway standalone. Any idea?
There is a more deterministic way of making sure in which order policies are applied than depending on the order of files on the file system.
If you are using online policies (ie policies that are defined on API Manager side), then you have to define the order there. If you rename an online policy, it simply will be removed in the next polling cycle by the runtime.
If you are using offline policies (ie policies that are not defined on API Manager side, and that you have to deploy manually to the policies folder), then you can define the order in which they will be applied by defining the order attribute in the policy tag. For example:
<?xml version="1.0" encoding="UTF-8"?>
<policy
xmlns="http://www.mulesoft.org/schema/mule/policy"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:mule="http://www.mulesoft.org/schema/mule/core"
xmlns:api-platform-gw="http://www.mulesoft.org/schema/mule/api-platform-gw"
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/policy http://www.mulesoft.org/schema/mule/policy/current/mule-policy.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/api-platform-gw http://www.mulesoft.org/schema/mule/api-platform-gw/current/mule-api-platform-gw.xsd"
online="false"
id="12345"
policyName="SimpleLogger"
order="100">
<before>
<mule:logger level="INFO" message="A message" />
</before>
<pointcut>
<api-platform-gw:api-pointcut apiName="your API name" apiVersion="your API version Name"/>
</pointcut>
</policy>
Take into account that even if you define the order, CORS and Throttling/RateLimit policies are always applied first, regardless of the order specified in those or other policies.
If two or more policies have the same order, then the runtime will decide in which order they will be applied after applying all the policies with lower order number defined.
Policies with no order specified are applied after all the policies in which order was specified are applied.
Best regards,
Nahuel.
I was able to change the policy order refactoring the name of the policies files adding a number as a prefix. The number will be used to set the order.
Ex.
000-client-id-enforcement.xml
111-json-thread-protection.xml
222-custom-policy.xml
These policies will be executed in the order
1st - client-id-enforcement
2nd - json-thread-protection
3rd - custom-policy

Ebay API Version overriding?

Even though I'm setting Compavility version in request header (967), when I'm making a call (GeteBayDetails in that case), the response comes with version higher than I need and want (979). These applies to both app I'm currently developing and even API Test Tool. Is there something that I'm missing? Or the Version tag in response isn't related to Compability Level?
Header:
X-EBAY-API-SITEID:212
X-EBAY-API-COMPATIBILITY-LEVEL:967
X-EBAY-API-CALL-NAME:GeteBayDetails
Body:
<?xml version="1.0" encoding="utf-8"?>
<GeteBayDetailsRequest xmlns="urn:ebay:apis:eBLBaseComponents">
<RequesterCredentials>
<eBayAuthToken>...</eBayAuthToken>
</RequesterCredentials>
</GeteBayDetailsRequest>
And the response:
<?xml version="1.0" encoding="UTF-8"?>
<GeteBayDetailsResponse
xmlns="urn:ebay:apis:eBLBaseComponents">
<Timestamp>2016-09-27T11:21:41.341Z</Timestamp>
<Ack>Failure</Ack>
<Errors>
<ShortMessage>Nieznany błąd.</ShortMessage>
<LongMessage>Nieznany błąd.</LongMessage>
<ErrorCode>17460</ErrorCode>
<SeverityCode>Error</SeverityCode>
<ErrorClassification>RequestError</ErrorClassification>
</Errors>
<Version>979</Version>
<Build>E979_INTL_API_18061441_R1</Build>
</GeteBayDetailsResponse>
PS. As far as I know, request fails because of the newer version of the API. And worked before like a charm. Thats why I want to stick to 967.
What you are seeing is normal behavior in that the response will always contain the most recent API schema that could service your request. I encounter many calls for which there are no applicable execution differences between the requested schema and the performing schema, for a given particular request. Also this returning "latest schema version that could service the API request" behavior is how you can determine if you can move up your compatibility level safely, as support drops off periodically.
Of course when the response has a lower schema than the latest in the release notes for the API, then you know you are in a situation where at some point you have to change your code to reflect what has been deprecated or changed before the support for the last schema that can service your particular request ends.
This eBay DTS article mentions this Information in the API Response
as well as going over the eBay API schema versioning process.
Also, be sure on XML POST requests to specify the API schema version in the request itself using the Version tag, not just the HTTP header as with the example call code for the GeteBayDetails API documentation:
<?xml version="1.0" encoding="utf-8"?>
<GeteBayDetailsRequest xmlns="urn:ebay:apis:eBLBaseComponents">
<!-- Call-specific Input Fields -->
<DetailName> DetailNameCodeType </DetailName>
<!-- ... more DetailName values allowed here ... -->
<!-- Standard Input Fields -->
<ErrorLanguage> string </ErrorLanguage>
<MessageID> string </MessageID>
<Version> string </Version>
<WarningLevel> WarningLevelCodeType </WarningLevel>
</GeteBayDetailsRequest>
Hope this helps

AWS S3 SSE GetObject requires secret key

The idea was to generate a random key for every file being uploaded, pass this key to S3 in order to encrypt it and store the key in the database. Once the user wants to access the file, the key is read from the database and passed to S3 once again.
The first part works. My objects are uploaded and encrypted successfully, but I have issues with retrieving them.
Retrieving files with request headers set:
When setting the request headers such as x-amz-server-side-encryption-customer-algorithm etc. when performing the GET request to the resource, works, and I am able to access it. But since I want to these resources as src to an <img>-Tag, I cannot perform GET requests which require headers to be set.
Thus, I thought about:
Pre signing urls:
To create a pre signed url, I built the HMAC SHA1 of the required string and used it as a signature. The calculated signature is accepted by S3 but I get the following error when requesting the pre signed URL:
Requests specifying Server Side Encryption with Customer provided keys must provide an appropriate secret key.
The URL has the form:
https://s3-eu-west-1.amazonaws.com/bucket-id/resource-id?x-amz-server-side-encryption-customer-algorithm=AES256&AWSAccessKeyId=MyAccessKey&Expires=1429939889&Signature=GeneratedSignature
The reason why the error is shown seems to be pretty clear to me. At no point in the signing process was the encryption key used. Thus, the request cannot work. As a result, I added the encryption key as Base64, and Md5 representation as parameters to the URL. The URL now has the following format:
https://s3-eu-west-1.amazonaws.com/bucket-id/resource-id?x-amz-server-side-encryption-customer-algorithm=AES256&AWSAccessKeyId=MyAccessKey&Expires=1429939889&Signature=GeneratedSignature&x-amz-server-side-encryption-customer-key=Base64_Key&x-amz-server-side-encryption-customer-key-MD5=Md5_Key
Although the key is now present (imho), I do get the same error message.
Question
Does anyone know, how I can access my encrypted files with a GET request which does not provide any headers such as x-amz-server-side-encryption-customer-algorithm?
It seems intuitive enough to me that what you are trying should have worked.
Apparently, though, when they say "headers"...
you must provide all the encryption headers in your client application.
— http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html#sse-c-how-to-programmatically-intro
... they do indeed actually mean headers and S3 doesn't accept these particular values when delivered as part of the query string, as you would expect, since S3 sometimes is somewhat flexible in that regard.
I've tested this, and that's the conclusion I've come to: doing this isn't supported.
A GET request with x-amz-server-side-encryption-customer-algorithm=AES256 included in the query string (and signature), along with the X-Amz-Server-Side-Encryption-Customer-Key and X-Amz-Server-Side-Encryption-Customer-Key-MD5 headers does work as expected... as I believe you've discovered... but putting the key and key-md5 in the query string, with or without including it in the signature seems like a dead end.
It seemed somewhat strange, at first, that they wouldn't allow this in the query string, since so many other things are allowed there... but then again, if you're going to the trouble of encrypting something, there seems little point in revealing the encryption key in a link... not to mention that the key would then be captured in the S3 access logs, leaving the encryption seeming fairly well pointless all around -- and perhaps that was their motivation for requiring it to actually be sent in the headers and not the query string.
Based on what I've found in testing, though, I don't see a way to use encrypted objects with customer-provided keys in hyperlinks, directly.
Indirectly, of course, a reverse proxy in front of the S3 bucket could do the translation for you, taking the appropriate values from the query string and placing them into the headers, instead... but it's really not clear to me what's to be gained by using customer-provided encryption keys for downloadable objects, compared to letting S3 handle the at-rest encryption with AWS-managed keys. At-rest encryption is all you're getting either way.

Amazon S3 ACL for read-only and write-once access

I'm developing a web application and I currently have the following ACL assigned to the AWS account it uses to access its data:
{
"Statement": [
{
"Sid": "xxxxxxxxx", // don't know if this is supposed to be confidential
"Action": [
"s3:*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::cdn.crayze.com/*"
]
}
]
}
However I'd like to make this a bit more restrictive so that if our AWS credentials were ever compromised, an attacker could not destroy any data.
From the documentation, it looks like I want to allow just the following actions: s3:GetObject and s3:PutObject, but I specifically want the account to only be able to create objects that don't exist already - i.e. a PUT request on an existing object should be denied. Is this possible?
This is not possible in Amazon S3 like you probably envisioned it; however, you can work around this limitation by Using Versioning which is a means of keeping multiple variants of an object in the same bucket and has been developed with use cases like this in mind:
You might enable versioning to prevent objects from being deleted or
overwritten by mistake, or to archive objects so that you can retrieve
previous versions of them.
There are a couple of related FAQs as well, for example:
What is Versioning? - Versioning allows you to preserve, retrieve, and restore every version of every object stored in an Amazon S3 bucket. Once you enable Versioning for a bucket, Amazon S3 preserves existing objects anytime you perform a PUT, POST, COPY, or DELETE operation on them. By default, GET requests will retrieve the most recently written version. Older versions of an overwritten or deleted object can be retrieved by specifying a version in the request.
Why should I use Versioning? - Amazon S3 provides customers with a highly durable storage infrastructure. Versioning offers an additional level of protection by providing a means of recovery when customers accidentally overwrite or delete objects. This allows you to easily recover from unintended user actions and application failures. You can also use Versioning for data retention and archiving. [emphasis mine]
How does Versioning protect me from accidental deletion of my objects? - When a user performs a DELETE operation on an object, subsequent default requests will no longer retrieve the object. However, all versions of that object will continue to be preserved in your Amazon S3 bucket and can be retrieved or restored. Only the owner of an Amazon S3 bucket can permanently delete a version. [emphasis mine]
If you are really paramount about the AWS credentials of the bucket owner (who can be different than the accessing users of course), you can take that one step further even, see How can I ensure maximum protection of my preserved versions?:
Versioning’s MFA Delete capability, which uses multi-factor authentication, can be used to provide an additional layer of
security. [...] If you enable Versioning with MFA Delete
on your Amazon S3 bucket, two forms of authentication are required to
permanently delete a version of an object: your AWS account
credentials and a valid six-digit code and serial number from an
authentication device in your physical possession. [...]
If this is accidental overwrite you are trying to avoid, and your business requirements allow a short time window of inconsistency, you can do the rollback in the Lambda function:
Make it a policy that "no new objects with the same name". Most of the time it will not happen. To enforce it:
Listen for S3:PutObject events in an AWS Lambda function.
When the event is fired, check whether more than one version is present.
If there is more than one version present, delete all but the newest one.
Notify the uploader what happened (it's useful to have the original uploader in x-amz-meta-* of the object. More info here).
You can now lock versions of objects with S3 Object Lock. It's a per-bucket setting, and allows you to place one of two kinds of WORM locks.
"retention period" - can't be changed
"legal hold" - can be changed by the bucket owner at any time
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html
As mentioned by #Kijana Woodard below, this does not prevent creation of new versions of objects.
Edit: Applicable if you came here from this question.
Object Locks only work in versioned buckets. If you can not enable versioning for your bucket, but can tolerate brief inconsistencies where files are presumed to exist while DELETE-ing them is still in-flight (S3 is only eventually-consistent) possibly resulting in PUT-after-DELETE failing intermittently if used in a tight-loop, or conversely, successive PUTs falsely succeeding intermittently, then the following solution may be appropriate.
Given the object path, read the Object's Content-Length header (from metadata, HeadObject request). Write the object only if the request succeeds, and where applicable, if length is greater than zero.