This is a S3 issue so I am posting this here and not in the Salesforce stackexchange.
Basically my Salesforce code generates pre-signed urls for S3. These are consumed by the front end to upload and download files.
This is working perfectly. Now we need to specify SSE (server side encryption).
Based on the documentation, SSE-S3 does not work with pre signed urls.
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
So I have to use SSE with customer generated keys.
http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
Here the text says this:
When creating a presigned URL, you must specify the algorithm using the x-amz-server-side-encryption-customer-algorithm in the signature calculation.
This is how I am calculating my signature and it is working fine with upload and download of files into the bucket sans the encryption.
public string getStringToSign() {
// ==== CONSTRUCT THE STRING TO SIGN ====
string stringToSign = S3Connection.AWS_HEADER_ENCRYPTION_SCHEME + '\n'
+ this.dateStampISO + '\n'
+ this.dateStampYYYYMMDD + '/' + this.awsRegion + '/s3/aws4_request' + '\n'
+ EncodingUtil.convertToHex(Crypto.generateDigest(S3Connection.AWS_ENCRYPTION_SCHEME, blob.valueOf(this.getRequestString())));
return stringToSign;
}
public blob getSigningKey() {
// ==== GENERATE THE SIGNING KEY ====
Blob dateKey = Crypto.generateMac('hmacSHA256', Blob.valueOf(this.dateStampYYYYMMDD), Blob.valueOf('AWS4' + this.accessSecret));
Blob dateRegionKey = Crypto.generateMac('hmacSHA256', Blob.valueOf(this.awsRegion), dateKey);
Blob dateRegionServiceKey = Crypto.generateMac('hmacSHA256', blob.valueOf(this.awsServiceName), dateRegionKey);
Blob signingKey = Crypto.generateMac('hmacSHA256', blob.valueOf('aws4_request'), dateRegionServiceKey);
//Blob signingKey2 = Crypto.generateMac('hmacSHA256', blob.valueOf('x-amz-server-side-encryption-customer-algorithm'), signingKey);
return signingKey;
}
public string getSignature() {
// ==== GENERATE THE SIGNATURE ====
return this.generateSignature(this.getStringToSign(), this.getSigningKey());
}
public string generateSignature(string stringToSign, blob signingKey) {
blob signatureBlob = Crypto.generateMac('hmacSHA256', blob.valueOf(stringToSign), signingKey);
return EncodingUtil.convertToHex(signatureBlob);
}
So my question is how do I add "x-amz-server-side-encryption-customer-algorithm" in this signature calculation.
Thanks in advance!
Related
I am use aws Lamda to decompress and traverse tar.gz files then uploading them back to s3 deflated retaining the original directory structure.
I am running into an issue streaming a TarArchiveEntry to a S3 bucket via a PutObjectRequest. While first entry is successfully streamed, upon trying to getNextTarEntry() on the TarArchiveInputStream a null pointer is thrown due to the underlying GunzipCompress inflator being null, which had an appropriate value prior to the s3.putObject(new PutObjectRequest(...)) call.
I have not been able to find documentation on how / why the gz input stream inflator attribute is being set to null after partially being sent to s3.
EDIT Further investigation has revealed that the AWS call appears to be closing the input stream after completing the upload of specified content length... haven't not been able to find how to prevent this behavior.
Below is essentially what my code looks like. Thank in advance for your help, comments, and suggestions.
public String handleRequest(S3Event s3Event, Context context) {
try {
S3Event.S3EventNotificationRecord s3EventRecord = s3Event.getRecords().get(0);
String s3Bucket = s3EventRecord.getS3().getBucket().getName();
// Object key may have spaces or unicode non-ASCII characters.
String srcKey = s3EventRecord.getS3().getObject().getKey();
System.out.println("Received valid request from bucket: " + bucketName + " with srckey: " + srcKeyInput);
String bucketFolder = srcKeyInput.substring(0, srcKeyInput.lastIndexOf('/') + 1);
System.out.println("File parent directory: " + bucketFolder);
final AmazonS3 s3Client = AmazonS3ClientBuilder.defaultClient();
TarArchiveInputStream tarInput = new TarArchiveInputStream(new GzipCompressorInputStream(getObjectContent(s3Client, bucketName, srcKeyInput)));
TarArchiveEntry currentEntry = tarInput.getNextTarEntry();
while (currentEntry != null) {
String fileName = currentEntry.getName();
System.out.println("For path = " + fileName);
// checking if looking at a file (vs a directory)
if (currentEntry.isFile()) {
System.out.println("Copying " + fileName + " to " + bucketFolder + fileName + " in bucket " + bucketName);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(currentEntry.getSize());
s3Client.putObject(new PutObjectRequest(bucketName, bucketFolder + fileName, tarInput, metadata)); // contents are properly and successfully sent to s3
System.out.println("Done!");
}
currentEntry = tarInput.getNextTarEntry(); // NPE here due underlying gz inflator is null;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
IOUtils.closeQuietly(tarInput);
}
}
That's true, AWS closes an InputStream provided to PutObjectRequest, and I don't know of a way to instruct AWS not to do so.
However, you can wrap the TarArchiveInputStream with a CloseShieldInputStream from Commons IO, like that:
InputStream shieldedInput = new CloseShieldInputStream(tarInput);
s3Client.putObject(new PutObjectRequest(bucketName, bucketFolder + fileName, shieldedInput, metadata));
When AWS closes the provided CloseShieldInputStream, the underlying TarArchiveInputStream will remain open.
PS. I don't know what ByteArrayInputStream(tarInput.getCurrentEntry()) does but it looks very strange. I ignored it for the purpose of this answer.
I'm having trouble with Azure Blobs and Shared Access Signatures when they expire. I need to grant access to a blob for longer than 1 hour (1 year), so I'm using a named container policy, but unfortunately . Its expiring after 1 hr
SharedAccessPolicy sharedAccessPolicy = new SharedAccessPolicy();
sharedAccessPolicy.Permissions = SharedAccessPermissions.Read;
sharedAccessPolicy.SharedAccessStartTime = DateTime.UtcNow;
//sharedAccessPolicy.SharedAccessExpiryTime = DateTime.UtcNow.AddYear(1); No need to define expiry time here.
BlobContainerPermissions blobContainerPermissions = new BlobContainerPermissions();
blobContainerPermissions.SharedAccessPolicies.Add("default", sharedAccessPolicy);
container.SetPermissions(blobContainerPermissions);
Console.WriteLine("Press any key to continue....");
Console.ReadLine();
CloudBlob blob = container.GetBlobReference(path);
string sas = blob.GetSharedAccessSignature(new SharedAccessPolicy()
{
SharedAccessExpiryTime = DateTime.UtcNow.AddDays(7),//add expiry date only when you're creating the signed URL
}
, "default");
Console.WriteLine(blob.Uri.AbsoluteUri + sas);
Process.Start(new ProcessStartInfo(blob.Uri.AbsoluteUri + sas));
Console.WriteLine("Press any key to continue....");
Console.ReadLine();
I need help constructing the Authorization header to PUT a block blob.
PUT\n\n\n11\n\n\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob\nx-ms-date:Sat, 25 Feb 2017 22:20:13 GMT\nx-ms-version:2015-02-21\n/myaccountname/mycontainername/blob.txt\n
I take this, UTF 8 encode it. Then I take my access key in my Azure account and HMAC sha256 this UTF 8 encoded string with the key. Then I output that in base64. Let's call this output string.
My authorization header looks like this: SharedKey myaccountname:output string
It is not working.
The header in Postman also has x-ms-blob-type, x-ms-date, x-ms-version, Content-Length, and Authorization. The body for now says hello world.
Can anyone help me make this successful request in Postman?
<?xml version="1.0" encoding="utf-8"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:cdeb9a5e-0001-0029-5fb5-8f7995000000
Time:2017-02-25T22:22:32.0300016Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request 'jiJtirohvi1syXulqkPKESnmQEJI4GpDU5JBn7BM/xY=' is not the same as any computed signature. Server used following string to sign: 'PUT
11
text/plain;charset=UTF-8
x-ms-date:Sat, 25 Feb 2017 22:20:13 GMT
x-ms-version:2015-02-21
/myaccountname/mycontainername/blob.txt'.</AuthenticationErrorDetail>
</Error>
EDIT:
First, I want to thank you and everyone who responded. I truly truly appreciate it. I have one last question and then I think I'll be set!! I'm not using that code - I'm doing this all by hand. If I have my key: X2iiy6v47j1jZZH5555555555zzQRrIAdxxVs55555555555av8uBUNGcBMotmS7tDqas14gU5O/w== changed slightly for anonymity - do I decode it: using an online base64decoder. Then, when I have my string which now looks like this: PUT\n\n\n11\n\ntext/plain;charset=UTF-8\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob\nx-ms-date:Mon, 27 Feb 2017 21:53:13 GMT\nx-ms-version:2015-02-21\n/myaccount/mycontainer/blob.txt\n so I run this in https://mothereff.in/utf-8 and then use this in HMAC with my decoded key: https://www.liavaag.org/English/SHA-Generator/HMAC/ - using sha256 and base64 at the end.
Is that how I get the correct string to put here?: SharedKey myaccount:<string here>
I believe there's an issue with how you're specifying StringToSign here:
PUT\n\n\n11\n\n\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob\nx-ms-date:Sat,
25 Feb 2017 22:20:13
GMT\nx-ms-version:2015-02-21\n/myaccountname/mycontainername/blob.txt\n
If you notice the error message returned from the server, string to sign by server is different than yours and the difference is that the server is using Content-Type (text/plain;charset=UTF-8) in signature calculation while you're not. Please include this content type in your code and things should work just fine.
Here's the sample code (partial only) I used:
var requestMethod = "PUT";
var urlPath = "test" + "/" + "myblob.txt";
var storageServiceVersion = "2015-12-11";
var date = DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture);
var blobType = "BlockBlob";
var contentBytes = Encoding.UTF8.GetBytes("Hello World");
var canonicalizedResource = "/" + accountName + "/" + urlPath;
var canonicalizedHeaders = "x-ms-blob-type:" + blobType + "\nx-ms-date:" + date + "\nx-ms-version:" + storageServiceVersion + "\n";
var stringToSign = requestMethod + "\n" +
"\n" + //Content Encoding
"\n" + //Content Language
"11\n" + //Content Length
"\n" + //Content MD5
"text/plain;charset=UTF-8" + "\n" + //Content Type
"\n" + //Date
"\n" + //If - Modified - Since
"\n" + //If - Match
"\n" + //If - None - Match
"\n" + //If - Unmodified - Since
"\n" + //Range +
canonicalizedHeaders +
canonicalizedResource;
string authorizationHeader = GenerateSharedKey(stringToSign, accountKey, accountName);
private static string GenerateSharedKey(string stringToSign, string key, string account)
{
string signature;
var unicodeKey = Convert.FromBase64String(key);
using (var hmacSha256 = new HMACSHA256(unicodeKey))
{
var dataToHmac = Encoding.UTF8.GetBytes(stringToSign);
signature = Convert.ToBase64String(hmacSha256.ComputeHash(dataToHmac));
}
return string.Format(CultureInfo.InvariantCulture, "{0} {1}:{2}", "SharedKey", account, signature);
}
According to your error message, it indicates that authorization signature is incorrect.
If the Content-Type "text/plain; charset=UTF-8" is not included in the header, please add it in the stringTosign and postman.
When we try to get the signature, we need to make sure the length of the blob.txt matches the Content length in the stringTosign. That means request body length should match the content length in the stringTosign.
I test it with Postman, it works correctly. We can get the signature with the code in another SO Thread. The following is my detail steps
Add the following header
Add the request body (example: Hello World)
Send the put blob request.
Update :
Please have a try to use the online tool to generate signature for test.
I'm working on a site hosted in Azure that has a download functionality. To reduce the load on our servers, the download is done use Shared Access Signatures. However, in Safari when downloading the file, the filename is wrapped in single quotes, as in myFile.txt downloads as 'myFile.txt'. This has made it so zips being downloaded have to be renamed by the client so the contents can be extracted.
Code for generated the Shared Access Signature is as follows:
CloudBlockBlob blob = container.GetBlockBlobReference(Helpers.StringHelper.TrimIfNotNull(blobName));
if (!blob.Exists())
{
return string.Empty;
}
var sasConstraints = new SharedAccessBlobPolicy();
sasConstraints.SharedAccessStartTime = DateTime.UtcNow.AddSeconds(-5);
sasConstraints.SharedAccessExpiryTime = DateTime.UtcNow.Add(duration);
sasConstraints.Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write;
var headers = new SharedAccessBlobHeaders();
string filename = blobName;
if (filename.Contains("/"))
{
filename = blobName.Substring(blobName.LastIndexOf("/") + 1, blobName.Length - blobName.LastIndexOf("/") - 1);
}
headers.ContentDisposition = "attachment; filename='" + filename + "'";
//Generate the shared access signature on the blob, setting the constraints directly on the signature.
string sasBlobToken = blob.GetSharedAccessSignature(sasConstraints, headers);
//Return the URI string for the container, including the SAS token.
return blob.Uri + sasBlobToken;
This code has worked fine in Chrome, Firefox, and IE. Is there something I'm missing with the headers? The only one I'm modifying is content-disposition.
You should use double quotes for quoted strings in HTTP headers, as outlined in RFC2616.
So replace
headers.ContentDisposition = "attachment; filename='" + filename + "'";
with
headers.ContentDisposition = "attachment; filename=\"" + filename + "\"";
So I'm trying to create a private/public key from 64 characters that I already know using bitcoinjs with the code below:
key = Bitcoin.ECKey.makeRandom();
// Print your private key (in WIF format)
document.write(key.toWIF());
// => Kxr9tQED9H44gCmp6HAdmemAzU3n84H3dGkuWTKvE23JgHMW8gct
// Print your public key (toString defaults to a Bitcoin address)
document.write(key.pub.getAddress().toString());
// => 14bZ7YWde4KdRb5YN7GYkToz3EHVCvRxkF
If I try to set "key" to my 64 characters instead of "Bitcoin.ECKey.makeRandom();" it fails. Is there a method or library that I overlooked that would allow me to use the known 64 characters in order to generate the private key in wif format and the public address?
Thanks in advance to anyone that may be able to offer some help.
You should use fromWIF method to pass your own data.
from source code of eckey.js
// Static constructors
ECKey.fromWIF = function(string) {
var payload = base58check.decode(string)
var compressed = false
// Ignore the version byte
payload = payload.slice(1)
if (payload.length === 33) {
assert.strictEqual(payload[32], 0x01, 'Invalid compression flag')
// Truncate the compression flag
payload = payload.slice(0, -1)
compressed = true
}
To create WIF from your key please follow https://en.bitcoin.it/wiki/Wallet_import_format
Here is interactive tool http://gobittest.appspot.com/PrivateKey
The solution to generate private and public key:
//public-key
var address = eckey.getBitcoinAddress().toString();
var privateKeyBytesCompressed = privateKeyBytes.slice(0);
privateKeyBytesCompressed.push(0x01);
var privateKeyWIFCompressed = new Bitcoin.Address(privateKeyBytesCompressed);
privateKeyWIFCompressed.version = 0x80;
//private-key
privateKeyWIFCompressed = privateKeyWIFCompressed.toString();
Take a look at moneyart.info for beautifully designed paperwallets.