Terraform s3 event notification error - amazon-s3

I am having trouble trying to create s3 event notifications. Does anyone know the resolutions to this?
Error is:
*Error applying plan:
1 error(s) occurred:
* module.Test-S3-Bucket.aws_s3_bucket_notification.s3-notification: 1 error(s) occurred:
* aws_s3_bucket_notification.s3-notification: Error putting S3 notification configuration: InvalidArgument: Unable to validate the following destination configurations
status code: 400, request id: AD9B5BF2FF84A6CB, host id: ShUVJ+TdkpqAZfpeDM3grkF9Vue3Q/AF0LydchperKTF6XdQyDM6BisZi/38pGAh/ZqS+gNyrSM=*
Below is the code that gives me the error:
resource "aws_s3_bucket" "s3-bucket" {
bucket = "${var.bucket_name}"
acl = ""
lifecycle_rule {
enabled = true
prefix = ""
expiration {
days = 45
}
}
tags {
CostC = "${var.tag}"
}
}
resource "aws_s3_bucket_notification" "s3-notification" {
bucket = "${var.bucket_name}"
topic {
topic_arn = "arn:aws:sns:us-east-1:1223445555:Test"
events = [ "s3:ObjectCreated:*", "s3:ObjectRemoved:*" ]
filter_prefix = "test1/"
}
}

If you haven't done it already, you need to specify a policy on the topic that grants the SNS:Publish permission to S3 (only from the bucket specified in the Condition attribute) - if you are also provisioning the topic via Terraform then something like this should do it (we know, as it caught us out just a few days ago too!):
resource "aws_sns_topic" "my-sns-topic" {
name = "Test"
policy = <<POLICY
{
"Version":"2012-10-17",
"Statement":[{
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:us-east-1:1223445555:Test",
"Condition":{
"ArnLike":{"aws:SourceArn":"${aws_s3_bucket.s3-bucket.arn}"}
}
}]
}
POLICY
}
Hope that helps.

Well, I know that this is not your exact case, but I had the same error and I didn't manage to find an answer here, and because this post is the first that Google gave me, I will leave the answer to my case here in the hope that it will help someone else.
So, I notice that after Terraform apply I had this error and I went to the UI to see what happened and found this message:
The Lambda console can't validate one or more event sources for this trigger. The most common cause is when a source ARN includes a wildcard (*) character. You can manage unvalidated triggers using the AWS CLI or AWS SDK.
And guess what? I really had a wildcard (*) character in ARN like this:
source_arn = "{aws_s3_bucket.bucket.arn}/*"
So I changed it to:
source_arn = aws_s3_bucket.bucket.arn
And it worked. So, if you read this - there might be the same mistake in your case.

Related

AWS credentials missing when running userdata in a new EC2

Using terraform scripts, I create a new EC2, add policy to access an S3 bucket, and supply a userdata script that runs aws s3 cp s3://bucket-name/file-name . to copy a file from that S3 bucket, among other commands.
In /var/log/cloud-init-output.log I see fatal error: Unable to locate credentials, presumably caused by executing aws s3 cp ... line. When I execute the same command manually on the EC2 after it's been created, it works fine (which means the EC2 policy for bucket access is correct).
Any ideas why the aws s3 cp command doesn't work during userdata execution but works when the EC2 is already created? Could it be that the S3 access policy is only applied to the EC2 after the EC2 has been fully created (and after userdata has been run)? What should be the correct workaround?
data "aws_iam_policy_document" "ec2_assume_role" {
statement {
effect = "Allow"
actions = [
"sts:AssumeRole",
]
principals {
type = "Service"
identifiers = [
"ec2.amazonaws.com",
]
}
}
}
resource "aws_iam_role" "broker" {
name = "${var.env}-broker-role"
assume_role_policy = data.aws_iam_policy_document.ec2_assume_role.json
force_detach_policies = true
}
resource "aws_iam_instance_profile" "broker_instance_profile" {
name = "${var.env}-broker-instance-profile"
role = aws_iam_role.broker.name
}
resource "aws_iam_role_policy" "rabbitmq_ec2_access_to_s3_distro" {
name = "${env}-rabbitmq_ec2_access_to_s3_distro"
role = aws_iam_role.broker.id
policy = data.aws_iam_policy_document.rabbitmq_ec2_access_to_s3_distro.json
}
data "aws_iam_policy_document" "rabbitmq_ec2_access_to_s3_distro" {
statement {
effect = "Allow"
actions = [
"s3:GetObject",
"s3:GetObjectVersion"
]
resources = ["arn:aws:s3:::${var.distro_bucket}", "arn:aws:s3:::${var.distro_bucket}/*"]
}
}
resource "aws_instance" "rabbitmq_instance" {
iam_instance_profile = ${aws_iam_instance_profile.broker_instance_profile.name}
....
}
This sounds like a timing issue where cloud-init is executed before the EC2 profile is set/ready to use. In your cloud-init script, I would make a loop to run a particular AWS cli command or even use the metadata server to retrieve information about the IAM credentials of the EC2 instance.
As the documentation states, you receive the following response when querying the endpoint http://169.254.169.254/latest/meta-data/iam/security-credentials/iam_role_name:
{
"Code" : "Success",
"LastUpdated" : "2012-04-26T16:39:16Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAIOSFODNN7EXAMPLE",
"SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"Token" : "token",
"Expiration" : "2017-05-17T15:09:54Z"
}
So your cloud-init/user-data script could wait until the Code attribute equals to Success and then proceed with the other operations.

Unable to read file from encrypted s3 bucket

I'm unable to read a file from encrypted s3 bucket in a lambda.
Below is my policy document where i'm giving access to s3 as well as kms. I've attached this policy to lambda.
When i try to read a file from the bucket, I get Access Denied error.
I'm adding kms:RequestAlias condition to kms statement so that the lambda will only have access to keys which has mytoken in the alias.
I suspect this is where i'm making mistake because if i remove the condition, the lambda gets access to all keys and read the encrypted file without any issues.
Can someone help me restrict access to only keys which has mytoken in the alias
data "aws_iam_policy_document" "lambda_s3_policy_doc" {
statement {
sid = ""
effect = "Allow"
resources = [
"arn:aws:s3:::mybucket*",
"arn:aws:s3:::mybucket*/*"
]
actions = [
"s3:AbortMultipartUpload",
"s3:CreateBucket",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:PutObject"
]
}
statement {
effect = "Allow"
actions = [
"kms:Decrypt",
"kms:DescribeKey",
"kms:Encrypt",
"kms:GenerateDataKey"
]
resources = ["*"]
condition {
test = "StringLike"
variable = "kms:RequestAlias"
values = [
"alias/*mytoken*"
]
}
}
}
What worked for me (I was trying to download the files from a couple of encrypted buckets directly from the AWS console and your case is in fact slightly different) was replacing kms:RequestAlias with kms:ResourceAliases.
statement {
sid = "AllowKMSAccessUsingAliases"
effect = "Allow"
actions = [
"kms:Decrypt",
]
resources = [
"arn:aws:kms:eu-central-1:111111111111:key/*",
]
condition {
test = "ForAnyValue:StringEquals"
variable = "kms:ResourceAliases"
values = [
"alias/alias-bucket-1",
"alias/alias-bucket-2",
]
}
}
According to what the AWS documentation says, it makes sense at least for me: it seems that you should use kms:RequestAlias when you are including the alias as part of your KMS request like in the image below:
When you use kms:ResourceAliases what gets checked is the alias associated to the KMS resource involved in the operation regardless of whether the alias was explicitly included in the request or not
So probably, your lambda function, when asking for the un-encryption of a file in a bucket, is using the KMS id in the request instead of the KMS alias and if that is the case kms:RequestAlias won't work because there is no alias in the request to be checked.

Enabling load balancer logs for aws in terraform

Im using terraform 0.12.4 to attempt tor write some code to enable the ‘access logs’ for my load balancer to write logs to an s3 bucket.
So far the buckets been created and the load balancers have been created by someone else but the bit where the ‘access_logs’ were supposeed to be configured was commented out and a TODO comment was placed there also. Hmmm methinks.
Theres too much code to place here but i keep receving access denied errors when setting them up. Ive found a couple of resources detailing what to do but none work. Has anyone managed to do this in TF?
According to the documentation on the Access Logs, you need to add permissions on your bucket for the ALB to write to S3.
data "aws_elb_service_account" "main" {}
data "aws_caller_identity" "current" {}
data "aws_iam_policy_document" "allow_load_balancer_write" {
statement {
principals {
type = "AWS"
identifiers = ["${data.aws_elb_service_account.main.arn}"]
}
actions = [
"s3:PutObject"
]
resources = [
"${aws_s3_bucket.access_logs.arn}/<YOUR_PREFIX_HERE>/AWSLogs/${data.aws_caller_identity.current.account_id}/*",
]
}
}
resource "aws_s3_bucket_policy" "access_logs" {
bucket = "${aws_s3_bucket.<YOUR_BUCKET>.id}"
policy = data.aws_iam_policy_document.allow_load_balancer_write.json
}
Also, it seems server side encryption needs to be enabled on the bucket.
resource "aws_s3_bucket" "access_logs" {
bucket_prefix = "<YOUR_BUCKET_NAME>-"
}
resource "aws_s3_bucket_server_side_encryption_configuration" "access_logs_encryption" {
bucket = "${aws_s3_bucket.access_logs.bucket}"
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}

How do I access s3 bucket from IAM account using Java

I have an s3 account with AmazonS3FullAccess put when I try to use it to run s3.listObjects("name") I get a 403 error...
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>59C510407179770D</RequestId><HostId>aLPzqYkTKx6nkUWVtZWYS+2fYexzniKWkn2D9+aG6pdxBAjtxAcC85uvGC4HqDnQIifLaf+oy1E=</HostId></Error>
s3.doesBucketExistV2("name") returns true.
My policy looks like this...
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
Do I need to add the user somewhere?
Update
Looks like it could be a problem with not getting the AWS creds (which is weird becuase of this line)...
Deprecated. By doesBucketExistV2(String) which will correctly throw an exception when credentials are invalid instead of returning true. See Issue #1256.
If I run ((AmazonS3Client) s3).awsCredentialsProvider.getCredentials() it returns null. My creds are in a amazon.properties file like this...
#PropertySources({
#PropertySource("classpath:amazon.properties")
})
// amazon.properties
amazon.accessKey=${AMZN_ACCESS_KEY}
amazon.secretKey=${AMZN_SECRET_KEY}
aws.accessKeyId=${AMZN_ACCESS_KEY}
aws.secretKey=${AMZN_SECRET_KEY}
and echo $AMZN_ACCESS_KEY returns the value I would expect.
Update 2
It appears to be something with the properties not getting read properly if I am explicit like this...
BasicAWSCredentials awsCreds = new BasicAWSCredentials(key, secret);
final AmazonS3 s3 = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.withRegion(service.getRegion())
.build();
It worked, so 2 questions 1.) Why does the doesBucketExistV2 return true even when I am not logged in properly 2.) Why are the system properties not working?
You need to pass bucket name as below. The sample code should be as below
Refer here.
/* The following example list two objects in a bucket. */
var params = {
Bucket: "name",
MaxKeys: 2
};
s3.listObjects(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response

When I try to login using AWS Cognito I get an AccessDeniedException about my custom Lambda trigger

I am calling adminInitiateAuth and getting back a strange AccessDeniedException for my own lambdas.
Here is the code I'm calling:
var params = {
AuthFlow: "ADMIN_NO_SRP_AUTH",
ClientId: "#cognito_client_id#",
UserPoolId: "#cognito_pool_id#",
AuthParameters: {
USERNAME : username,
PASSWORD : tempPassword
},
};
cognitoIdentityServiceProvider.adminInitiateAuth(params, function(error, data) {
if (error) {
console.log("ERROR! Login failed: " + JSON.stringify(error), error.stack);
} else {
console.log("Login sent back: " + JSON.stringify(data));
}
});
The error message I'm getting is:
ERROR! Login failed: {"message":"arn:aws:lambda:us-east-1:201473124518:function:main-devryan-users_onCognitoLogin failed with error AccessDeniedException.","code":"UnexpectedLambdaException","time":"2017-02-25T18:54:15.109Z","requestId":"ce42833f-fb8b-11e6-929b-2f78b63faa12","statusCode":400,"retryable":false,"retryDelay":1.0853444458916783} UnexpectedLambdaException: arn:aws:lambda:us-east-1:201473124518:function:main-devryan-users_onCognitoLogin failed with error AccessDeniedException.
Does anybody know why I might be getting this error?
This was happening because I recreated my API Gateway & Lambdas (using serverless) and it turns out that the Cognito console sneakily adds permissions to contact a given Lambda function when added as a trigger through the console.
To fix this in your CloudFormation / serverless.yml file:
resources:
Resources:
OnCognitoSignupPermission:
Type: 'AWS::Lambda::Permission'
Properties:
Action: "lambda:InvokeFunction"
FunctionName:
Fn::GetAtt: [ "UsersUnderscoreonCognitoSignupLambdaFunction", "Arn"]
Principal: "cognito-idp.amazonaws.com"
SourceArn:
Fn::Join: [ "", [ "arn:aws:cognito-idp", ":", Ref: "AWS::Region", ":", Ref: "AWS::AccountId", ":", "userpool/", "#cognito_pool_id#" ] ]
To fix this in the AWS console:
Go to the Cognito Console
Choose your user pool
Go to "Triggers"
Remove your custom trigger (set it to None) and click "Save"
Now reset it back and click "Save" again
Here's an interesting Amazon forum post that led me down the right track.
I had a problem similar to yours except I was trying to configure the Lambda with my Cognito User Pool through CloudFormation.
In the link that Ryan had posted there was a code sample someone posted. Namely Cognito needed the proper permissions to invoke the lambda function.
MyLambdaInvocationPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt MyLambdaFunctionName.Arn
Principal: cognito-idp.amazonaws.com
SourceArn: !GetAtt MyCognitoUserPoolName.Arn
For someone ending up here, trying to add cognito triggers via terraform, all you need to do is to add an aws_lambda_permission resource:
resource "aws_lambda_permission" "allow_execution_from_user_pool" {
statement_id = "AllowExecutionFromUserPool"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.<lambda>.function_name
principal = "cognito-idp.amazonaws.com"
source_arn = aws_cognito_user_pool.<pool>.arn
}
Found in this great post: https://www.integralist.co.uk/posts/cognito/
you can add the permission from the lambda Role (create a policy for cognito and add to to the lamda role ) . this solve my problem when i stuck into it
while creation of cloudformation stack - I got error like
User:arn:aws::12345678:user/xyz is not authorized to perform:
cognito-idp:CreateUserPool on resource:*(Service:AWSCognitoIdentityProviderService;
Status Code: 400; Error Code: AccessDeniedException;Request ID: xxxxx)
workaround :
went on to the Stack which is in Rollback state -> checked events and could see , (creation-failed) some Roles I don't have access ,
So , I checked IAM policy assigned to me - I was not having the access.
I created a new policy and assigned to myself as an Inline Policy by Importing it from AWS.
aws-cognito-idp
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Cognito-IDP",
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction",
"cognito-idp:*"
],
"Resource": "*"
}
]
}
note: you can restrict the access on resource and cognito-idp user.
with this - I am successfully able to create and deploy cloudformation Stack for the module.