Receiving 400 Bad request when making S3 GET request - amazon-s3

Question

I am trying to make an AWS version 4 authorization signed GET request to S3, and receive a bad request error 400 Code:InvalidRequest Message:Missing required header for this request: x-amz-content-sha256
If I prefix the header with "Authorization: ", I get error Code:InvalidArgument Message:Unsupported Authorization Type <ArgumentName>Authorization</ArgumentName> <ArgumentValue>Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXX/20200408/eu-west-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=vdchzint97uwyt3g%2fjehszrc8zpkbjsx4tfqacsqfow%3d</ArgumentValue>
I'm using Delphi XE5 with Indy's TIdHTTP component. Can anyone tell me what I am doing wrong? I have included my code below.
begin
bucket := 'mybucket.ata-test';
obj := 'test.xml';
region := 'eu-west-3';
service := 's3';
aws := 'amazonaws.com';
YYYYMMDD := FormatDateTime('yyyymmdd', now);
amzDate := FormatDateTime('yyyymmdd"T"hhnnss"Z"', TTimeZone.Local.ToUniversalTime(Now), TFormatSettings.Create('en-US'));
emptyHash := lowercase(SHA256HashAsHex(''));
host := Format('%s.%s.%s.%s', [bucket, service, region, aws]);
url := Format('%s://%s.%s.%s.%s/%s', ['https', bucket, service, region, aws, obj]);
// *** 1. Build the Canonical Request for Signature Version 4 ***
// HTTPRequestMethod
CanonicalRequest := URLEncodeValue('GET') +#10;
// CanonicalURI
CanonicalRequest := CanonicalRequest + '/' + URLEncodeValue(obj) +#10;
// CanonicalQueryString (empty just a newline)
CanonicalRequest := CanonicalRequest +#10;
// CanonicalHeaders
CanonicalRequest := CanonicalRequest + 'host:' + Trim(host) +#10
+ 'x-amz-content-sha256:' + emptyHash +#10
+ 'x-amz-date:' + Trim(amzDate) +#10;
// SignedHeaders
CanonicalRequest := CanonicalRequest + 'host;x-amz-content-sha256;x-amz-date' +#10;
// HexEncode(Hash(RequestPayload)) - (hash of an empty string)
CanonicalRequest := CanonicalRequest + emptyHash;
// *** 2. Create a String to Sign for Signature Version 4 ***
StringToSign := 'AWS4-HMAC-SHA256' +#10
+ amzDate +#10
+ UTF8String(YYYYMMDD) +'/'+ UTF8String(region) +'/'+ UTF8String(service) +UTF8String('/aws4_request') +#10
+ lowercase(SHA256HashAsHex(CanonicalRequest));
// *** 3. Calculate the Signature for AWS Signature Version 4 ***
DateKey := CalculateHMACSHA256(YYYYMMDD, 'AWS4' + SecretAccessKey);
DateRegionKey := CalculateHMACSHA256(region, DateKey);
DateRegionServiceKey := CalculateHMACSHA256(service, DateRegionKey);
SigningKey := CalculateHMACSHA256('aws4_request', DateRegionServiceKey);
Signature := lowercase(UrlEncodeValue(CalculateHMACSHA256(StringToSign, SigningKey)));
// *** 4. Create Authorisation Header and Add the Signature to the HTTP Request ***
AuthorisationHeader := 'AWS4-HMAC-SHA256 Credential='+AccessIdKey+'/'+YYYYMMDD+'/'+region+'/'+service+'/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature='+signature;
// (Gives <Code>InvalidRequest</Code> <Message>Missing required header for this request: x-amz-content-sha256</Message>)
// Have also tried
// AuthorisationHeader := 'Authorization: AWS4-HMAC-SHA256 Credential='+AccessIdKey+'/'+YYYYMMDD+'/'+region+'/'+service+'/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature='+signature;
// (Gives <Code>InvalidArgument</Code> <Message>Unsupported Authorization Type</Message>)
// *** 5. Add Header and Make Request ***
stm := TMemoryStream.Create;
try
try
Idhttp.Request.CustomHeaders.FoldLines := False;
Idhttp.Request.CustomHeaders.AddValue('Authorization', AuthorisationHeader);
Idhttp.Get(URL, stm);
except
on PE: EIdHTTPProtocolException do begin
s := PE.ErrorMessage;
Raise;
end;
on E: Exception do begin
s := E.Message;
Raise;
end;
end;
stm.Position := 0;
Memo1.Lines.LoadFromStream(stm);
finally
FreeAndNil(stm);
end;
end;
function SHA256HashAsHex(const value: string): String;
/// used for stringtosign
var
sha: TIdHashSHA256;
begin
LoadOpenSSLLibrary;
if not TIdHashSHA256.IsAvailable then
raise Exception.Create('SHA256 hashing is not available!');
sha := TIdHashSHA256.Create;
try
result := sha.HashStringAsHex(value, nil);
finally
sha.Free;
end;
end;
function CalculateHMACSHA256(const value, salt: String): String;
/// used for signingkey
var
hmac: TIdHMACSHA256;
hash: TIdBytes;
begin
LoadOpenSSLLibrary;
if not TIdHashSHA256.IsAvailable then
raise Exception.Create('SHA256 hashing is not available!');
hmac := TIdHMACSHA256.Create;
try
hmac.Key := IndyTextEncoding_UTF8.GetBytes(salt);
hash := hmac.HashValue(IndyTextEncoding_UTF8.GetBytes(value));
Result := EncodeBytes64(TArray<Byte>(hash));
finally
hmac.Free;
end;
end;

Answer 1

A few things I notice in your code:
when creating the YYYYMMDD and amzDate values, you are calling Now() twice, which creates a race condition that has the potential of causing those variables to represent different dates. Unlikely, but possible. To avoid that, you should call Now() only 1 time and save the result to a local TDateTime variable, and then use that variable in all of your FormatDateTime() calls.
dtNow := Now();
YYYYMMDD := FormatDateTime('yyyymmdd', dtNow);
amzDate := FormatDateTime('yyyymmdd"T"hhnnss"Z"', TTimeZone.Local.ToUniversalTime(dtNow), TFormatSettings.Create('en-US'));
When using TIdHTTP's Request.CustomHeaders property to set a custom Authorization header, make sure that you also set the Request.BasicAuthentication property to False as well, otherwise TIdHTTP may create its own Authorization: Basic ... header using its Request.Username and Request.Password properties. You don't want two Authorization headers in your GET request.
Idhttp.Request.BasicAuthentication := False;
You are using x-amz-content-sha256 and x-amz-date headers in your authorization calculations, but you are not adding those headers to the actual HTTP request. TIdHTTP will add the Host header for you, but you need to add the other headers yourself.
Idhttp.Request.CustomHeaders.AddValue('x-amz-content-sha256', emptyHash);
Idhttp.Request.CustomHeaders.AddValue('x-amz-date', amzDate);
Your SHA256HashAsHex() function is not specifying a byte encoding when calling Indy's TIdHashSHA256.HashStringAsHex() method (in fact, it is going out of its way to explicitly set the encoding to nil). As such, Indy's default byte encoding will be used, which is US-ASCII (unless you set Indy's GIdDefaultTextEncoding variable in the IdGlobal unit to something else). However, your CalculateHMACSHA256() function is explicitly using UTF-8 instead. Your SHA256HashAsHex() function should use IndyTextEncoding_UTF8 to match:
result := sha.HashStringAsHex(value, IndyTextEncoding_UTF8);
the input salt and output value for CalculateHMACSHA256() needs to be binary bytes, not strings, and certainly not base64-encoded or hex-encoded strings. Nothing in the Calculate the Signature for AWS Signature Version 4 documentation mentions the use of base64 at all.
var
DateKey, RegionKey, ServiceKey, SigningKey: TArray<Byte>;
...
// *** 3. Calculate the Signature for AWS Signature Version 4 ***
DateKey := CalculateHMACSHA256(YYYYMMDD, TEncoding.UTF8.GetBytes('AWS4' + SecretAccessKey));
RegionKey := CalculateHMACSHA256(region, DateKey);
ServiceKey := CalculateHMACSHA256(service, RegionKey);
SigningKey := CalculateHMACSHA256('aws4_request', ServiceKey);
Signature := CalculateHMACSHA256Hex(StringToSign, SigningKey);
...
function CalculateHMACSHA256(const value: string; const salt: TArray<Byte>): TArray<Byte>;
/// used for signingkey
var
hmac: TIdHMACSHA256;
hash: TIdBytes;
begin
LoadOpenSSLLibrary;
if not TIdHashSHA256.IsAvailable then
raise Exception.Create('SHA256 hashing is not available!');
hmac := TIdHMACSHA256.Create;
try
hmac.Key := TIdBytes(salt);
hash := hmac.HashValue(IndyTextEncoding_UTF8.GetBytes(value));
Result := TArray<Byte>(hash);
finally
hmac.Free;
end;
end;
function CalculateHMACSHA256Hex(const value: string; const salt: TArray<Byte>): string;
var
hash: TArray<Byte>;
begin
hash := CalculateHMACSHA256(value, salt)
Result := lowercase(ToHex(TIdBytes(hash)));
end;

Answer 2

I just went through a similar problem, I will leave my contribution here, because this post helped me to arrive at the solution.
In my case I needed to generate a signed url with a certain expiration time.
unit Data.Cloud.AmazonAPI.Utils;
interface
//See https://docs.aws.amazon.com/pt_br/AmazonS3/latest/API/sigv4-query-string-auth.html#query-string-auth-v4-signing-example
//Use example:
// MyUlrSigned := GetUrlPreSigned('mybucketname','/MyFolder/MyFile.zip','sa-east-1',3600);
function GetUrlPreSigned(ABucket:string;AObjectName:string;ARegion:string;AExpiresIn:Int64=3600):string;
implementation
uses
System.Classes, System.SysUtils, System.Hash, System.DateUtils;
const
AWS_ACCOUNTNAME = '<AWSAccessKeyId>';
AWS_ACCOUNTKEY = '<AWSSecretAccessKey>';
function SignString(const Signkey: TBytes; const StringToSign: string): TBytes;
begin
Result := THashSHA2.GetHMACAsBytes(StringToSign, Signkey);
end;
function BuildSignature(const StringToSign, DateISO, Region:string; AService: string; LSecretAccessKey:string): string;
function GetSignatureKey(const datestamp, region, serviceName: string): TBytes;
begin
Result := SignString(TEncoding.Default.GetBytes('AWS4'+LSecretAccessKey) ,datestamp);
Result := SignString(Result, region);
Result := SignString(Result, serviceName);
Result := SignString(Result, 'aws4_request');
end;
var
Signature:string;
SigningKey : TBytes;
begin
SigningKey := GetSignatureKey(DateISO, Region, AService);
Result := THash.DigestAsString(SignString(SigningKey, StringToSign));
end;
function GetHashSHA256Hex( HashString: string): string;
var
LBytes: TArray<Byte>;
begin
LBytes := THashSHA2.GetHashBytes(HashString);
Result := THash.DigestAsString(LBytes);
end;
function GetUrlPreSigned(ABucket:string;AObjectName:string;ARegion:string;AExpiresIn:Int64=3600):string;
var
LNow : TDateTime;
LData : string;
LTimeStamp : string;
LAccessKey : string;
LSecretAccessKey : string;
LService : string;
LAws : string;
LHost : string;
LUrl : string;
LQueryParams : string;
LCanonicalRequest : string;
LStringToSign : string;
LSignature : string;
begin
LNow := Now();
LData := FormatDateTime('yyyymmdd', LNow);
LTimeStamp := FormatDateTime('yyyymmdd"T"hhnnss"Z"', TTimeZone.Local.ToUniversalTime(LNow), TFormatSettings.Create('en-US'));
LAccessKey := AWS_ACCOUNTNAME;
LSecretAccessKey := AWS_ACCOUNTKEY;
if AObjectName.StartsWith('/') then
Delete(AObjectName,1,1);
LService := 's3';
LAws := 'amazonaws.com';
LHost := Format('%s-%s.%s', [LService, ARegion, LAws]);
LUrl := Format('%s://%s-%s.%s/%s/%s', ['https', LService, ARegion, LAws, ABucket, AObjectName]);
LQueryParams := 'X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential='+LAccessKey
+'%2F'+LData
+'%2F'+ARegion
+'%2F'+LService+'%2F'
+'aws4_request'
+'&X-Amz-Date='+LTimeStamp
+'&X-Amz-Expires='+AExpiresIn.ToString
+'&X-Amz-SignedHeaders=host';
//1 - CanonicalRequest
LCanonicalRequest := 'GET' +#10+
'/'+ABucket+'/'+AObjectName +#10
+LQueryParams +#10
+'host:'+LHost+#10
+#10
+'host'+#10
+'UNSIGNED-PAYLOAD';
//2 - StringToSign
LStringToSign := 'AWS4-HMAC-SHA256' +#10
+ LTimeStamp +#10
+ UTF8String(LData)+'/'+ UTF8String(ARegion) +'/'+ UTF8String(LService)+UTF8String('/aws4_request') +#10
+ lowercase(GetHashSHA256Hex(LCanonicalRequest));
//3 - Signature
LSignature := BuildSignature(LStringToSign,LData,ARegion,LService,LSecretAccessKey);
//4 - Signed URL
Result := LUrl+'?'+LQueryParams+'&X-Amz-Signature='+LSignature;
end;
end.