I'm getting an error in my Terraform scripts when attempting to add logging to two buckets. These are part of one of my modules, and I've successfully used them before. I've come back to deploy a new environment...and now it's not working.
I'm getting the following error:
module.dev2_environment.module.portal.aws_s3_bucket.portal_bucket: 1 error occurred:
* aws_s3_bucket.portal_bucket: Error putting S3 logging: InvalidTargetBucketForLogging: You must give the log-delivery group
WRITE and READ_ACP permissions to the target bucket
status code: 400, request id: 51AB42EFCACC9924, host id: nYCUxjHZE+xTisA1xG5syLTKVN/Rtwu8z3xF+O9GAPMdC2yGcafP4uwDURUGKd9Lx1SD8aHTcEI=
I'm executing via CLI, with Admin credentials. No code changes were made between the working state and the error. Any ideas on what could have changed? Syntax? AWS config someplace?
Terraform 11.14 and aws provider 2.16
Log bucket:
resource "aws_s3_bucket" "logs_bucket" {
bucket = "XYZ-${var.env}-cdnlogs"
acl = "log-delivery-write"
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
tags {
Finance = "dev_env"
Environment = "${var.env}"
}
}
Target Bucket:
resource "aws_s3_bucket" "portal_bucket" {
bucket = "XYZ-${var.env}-portal"
acl = "private"
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
logging {
target_bucket = "${aws_s3_bucket.logs_bucket.id}"
target_prefix = "logs/portal/"
}
website {
index_document = "index.html"
error_document = "index.html"
}
// Needed to allow logos to be uploaded the "Portal"
cors_rule {
allowed_headers = ["*"]
allowed_methods = ["GET", "HEAD", "PUT", "POST"]
allowed_origins = ["*"]
max_age_seconds = 3000
}
tags {
Finance = "dev_env"
Environment = "${var.env}"
}
}
I think the error is here
logging {
target_bucket = "${aws_s3_bucket.**logs_bucket**.id}"
target_prefix = "logs/portal/"
}
It should be
logging {
**target_bucket = "${aws_s3_bucket.portal_bucket.id}"**
target_prefix = "logs/portal/"
}
Set the values of your log-delivery-write ACL to allow Logging -> Read and Logging Write. As well as Read bucket permissions.
Related
Everything seems to work fine using Terraform, but for some reason after each apply it keeps removing and then adding back the configuration for server side encryption on all s3 buckets. If I apply the removal, it will just add it back next time I run apply.
Here is what happens after running terraform plan on my main branch with no changes made/deployed. Next time I run plan/apply it will add it back.
# aws_s3_bucket.terraform-state will be updated in-place
~ resource "aws_s3_bucket" "terraform-state" {
id = "company-terraform-state"
tags = {}
# (11 unchanged attributes hidden)
- server_side_encryption_configuration {
- rule {
- bucket_key_enabled = false -> null
- apply_server_side_encryption_by_default {
- kms_master_key_id = "arn:aws:kms:us-east-1:123456789012:key/Random-GUID-ABCD-1234" -> null
- sse_algorithm = "aws:kms" -> null
}
}
}
# (1 unchanged block hidden)
}
Possibly contributing: I setup a S3 state bucket to keep track of what I have deployed in AWS: https://technology.doximity.com/articles/terraform-s3-backend-best-practices
My state.tf file:
// This file is based on the writtings here: https://technology.doximity.com/articles/terraform-s3-backend-best-practices
terraform {
backend "s3" {
bucket = "company-terraform-state"
key = "state/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "alias/terraform-bucket-key"
dynamodb_table = "terraform-state"
}
}
// The backend configuration above is added after the state s3 bucket is created with the rest of the file below
resource "aws_kms_key" "terraform-bucket-key" {
description = "This key is used to encrypt bucket objects for terraform state"
deletion_window_in_days = 10
enable_key_rotation = true
}
resource "aws_kms_alias" "key-alias" {
name = "alias/terraform-bucket-key"
target_key_id = aws_kms_key.terraform-bucket-key.key_id
}
resource "aws_s3_bucket" "terraform-state" {
bucket = "company-terraform-state"
}
resource "aws_s3_bucket_server_side_encryption_configuration" "encryption-config" {
bucket = aws_s3_bucket.terraform-state.id
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.terraform-bucket-key.arn
sse_algorithm = "aws:kms"
}
}
}
resource "aws_s3_bucket_versioning" "versioning" {
bucket = aws_s3_bucket.terraform-state.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_acl" "acl" {
bucket = aws_s3_bucket.terraform-state.id
acl = "private"
}
resource "aws_s3_bucket_public_access_block" "block" {
bucket = aws_s3_bucket.terraform-state.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
// This table exists to prevent multiple team members from modifying the state file at the same time
resource "aws_dynamodb_table" "terraform-state" {
name = "terraform-state"
read_capacity = 20
write_capacity = 20
hash_key = "LockID"
attribute {
name = "LockID"
type = "S"
}
}
Figured it out, I was using an older provider in my main.tf. I had 3.0 set instead of 4.0 and I’m using the newer aws_s3_bucket_server_side_encryption_configuration instead of configuring the encryption in aws_s3_bucket (which would have been more suiting to the older provider).
I’m actually surprised it worked at all! Must be some features in 3.0 that were released yet.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0" # This was "~> 3.0"
}
}
}
I am trying to setup a cloudfront dist and s3 bucket with terraform. When I run terraform apply it is returning the following error:
aws_s3_bucket.app: Error putting S3 logging: InvalidTargetBucketForLogging: You must give the log-delivery group WRITE and READ_ACP permissions to the target bucket
my S3.tf file:
data "aws_iam_policy_document" "s3_policy" {
policy_id = "PolicyForCloudFrontPrivateContent"
statement {
sid = "1"
actions = ["s3:GetObject"]
resources = ["arn:aws:s3:::${local.name_env}/*"]
principals {
type = "AWS"
identifiers = ["${aws_cloudfront_origin_access_identity.origin_access_identity.iam_arn}"]
}
}
}
resource "aws_s3_bucket" "app" {
bucket = "${local.name_env}"
policy = "${data.aws_iam_policy_document.s3_policy.json}"
logging {
target_bucket = "${local.logs_bucket}"
target_prefix = "app-${var.environment}"
}
versioning {
enabled = true
}
tags = "${local.tags}"
}
You need to add an acl attribute to your aws_s3_bucket with a value of "log-delivery-write".
resource "aws_s3_bucket" "app" {
bucket = "${local.name_env}"
acl = "log-delivery-write"
...
}
UPDATE: terraform now supports custom bucket acls natively. What follows is a workaround for older versions where the predefined acls would not suffice.
Here's how to achieve this via terraform using a null resource and the AWS CLI.
resource "aws_s3_bucket" "files_bucket" {
# ...
logging {
target_bucket = "${aws_s3_bucket.logs_bucket.bucket}"
}
depends_on = [
"null_resource.logs_bucket_acl_workaround"
]
}
resource "aws_s3_bucket" "logs_bucket" {
# ...
acl = "private"
}
locals {
put_bucket_acl_cmd = "s3api put-bucket-acl --bucket ${aws_s3_bucket.logs_bucket.bucket} --grant-write 'uri=\"http://acs.amazonaws.com/groups/s3/LogDelivery\"' --grant-read-acp 'uri=\"http://acs.amazonaws.com/groups/s3/LogDelivery\"'"
}
resource "null_resource" "logs_bucket_acl_workaround" {
# cannot set bucket ACLs via terraform yet
# https://github.com/terraform-providers/terraform-provider-aws/issues/989
depends_on = [
"aws_s3_bucket.logs_bucket",
]
triggers = {
bucket = "${aws_s3_bucket.logs_bucket.bucket}"
command = "${local.put_bucket_acl_cmd}"
}
provisioner "local-exec" {
command = "aws ${local.put_bucket_acl_cmd}"
}
}
Note that ACLs in this way are only added but never removed.
I am trying to create S3 bucket using terraform from examples in the link
https://www.terraform.io/docs/providers/aws/r/s3_bucket.html
I have created a S3 module.
The issue i am facing is, for certain bucket i do not want logging enabled.
How can this be accomplished in terraform.
logging {
target_bucket = "${aws_s3_bucket.log_bucket.id}"
target_prefix = "log/"
}
Using empty string for target_bucket and target_prefix causes terraform to make an attempt to create target_bucket.
Also, i am trying to use a module.
Using the newer dynamic block support in terraform 0.12+ we pass a single-item array containing the logging settings if we want logging like so:
variable "logging" {
type = list
default = []
description = "to enable logging set this to [{target_bucket = 'xxx' target_prefix = 'logs/'}]"
}
resource "aws_s3_bucket" "s3bucket" {
dynamic "logging" {
for_each = [for l in var.logging : {
target_bucket = l.target_bucket
target_prefix = l.target_prefix
}]
content {
target_bucket = logging.value.target_bucket
target_prefix = logging.value.target_prefix
}
}
}
Can Fly.
If you want to make the values of logging optional, first make your module aws_s3_bucket.tf:
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging = "${var.logging}"
}
variable "logging" {
type = "list"
default = []
}
then in a sub-folder example add your template module.tf:
module "s3" {
source = "../"
logging = [
{
target_bucket = "loggingbucketname"
target_prefix = "log/"
},
]
}
provider "aws" {
region = "eu-west-1"
version = "2.4.0"
}
This is your version that has logging.
Next modify your module.tf to look like
module "s3" {
source = "../"
}
provider "aws" {
region = "eu-west-1"
version = "2.4.0"
}
That's your version without. This worked with:
Terraform v0.11.11
+ provider.aws v2.4.0
Updated
This is answer for v0.12.5.
module is now:
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging {
target_bucket = var.logging["target_bucket"]
target_prefix = var.logging["target_prefix"]
}
}
variable "logging" {
type=map
default={
target_bucket = ""
target_prefix = ""
}
}
Use module with logging becomes (your path to modules might differ):
module "s3" {
source = "../"
logging={
target_bucket = aws_s3_bucket.log_bucket.id
target_prefix = "log/"
}
}
provider "aws" {
region = "eu-west-1"
version = "2.34.0"
}
resource "aws_s3_bucket" "log_bucket" {
bucket = "my-tf-log-bucket"
acl = "private"
}
and without:
module "s3" {
source = "../"
}
provider "aws" {
region = "eu-west-1"
version = "2.34.0"
}
I'm working in Terraform, and am creating an S3 object/folder-with-content. I would like to exclude that object from my lifecycle policy. But I'm not sure to exclude the object (folder-object/sample) from the lifecycle policy (Terraform Code Below):
resource "aws_s3_bucket" "s3_test" {
bucket = "test-bucket-upload"
acl = "private"
key = "folder-object/sample"
tags {
Name = "test-bucket"
Environment = "lab"
}
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
lifecycle_rule {
id = "glacier-transfer"
enabled = true
transition {
days = 360
storage_class = "GLACIER"
}
}
}
Instead of excluding, use prefix to identify the objects your lifecycle rule should apply to. For example, the rule below would only apply to objects in the new_objects folder in your bucket:
...
lifecycle_rule {
id = "glacier-transfer"
enabled = true
prefix = "new_objects/"
transition {
days = 360
storage_class = "GLACIER"
}
}
...
I am new to Terraform and I am trying to create a Terraform module S3 bucket for Lambda function which is accessible cross account. Also the Cross account access should be limited to listing and getting objects.
Below is what I get to create lambda:
resource "aws_lambda_function" "test_lambda" {
filename = "lambda_function_payload.zip"
s3_bucket = "my_bucket_name"
function_name = "lambda_function_name"
role = "${aws_iam_role.iam_for_lambda.arn}"
handler = "exports.test"
source_code_hash = "${base64sha256(file("lambda_function_payload.zip"))}"
runtime = "nodejs4.3"
}
This is how my s3 bucket code looks like:
resource "aws_s3_bucket" "bucket" {
bucket = "my-bucket"
acl = "private"
lifecycle_rule {
id = "log"
enabled = true
prefix = "log/"
tags {
"rule" = "log"
"autoclean" = "true"
}
transition {
days = 30
storage_class = "STANDARD_IA"
}
transition {
days = 60
storage_class = "GLACIER"
}
expiration {
days = 90
}
}
lifecycle_rule {
id = "tmp"
prefix = "tmp/"
enabled = true
expiration {
date = "2016-01-12"
}
}
#Using this section for cors_rule
cors_rule {
allowed_headers = ["*"]
allowed_methods = ["GET"]
allowed_origins = ["https://s3-website-test.hashicorp.com"]
}
}
How I can turn this into a module ?
The necessary policies are described in the official documentation. In the following I will describe how to setup this example with a single terraform module - you have to adjust the permissions to your needs. You can also extend it, to support more than 2 AWS accounts. I haven't run the following code myself, so there might be some typos etc., but it should help you to get the general idea behind it.
Lets assume AWS account A is the account your s3 bucket is in and AWS account B is the account your lambda function is running in.
In order to place everything into a single module, you must define 2 Terraform AWS providers; one for each AWS account involved.
// Your s3 bucket's provider - used by default
provider "aws" {
.. // your setup here
}
// Your lambda function's provider - used if specified via alias
provider "aws" {
.. // your setup here
alias = "lambda"
}
First, attach a policy to your s3 bucket to allow access from account B:
data "aws_caller_identity" "lambda" {
provider = "aws.lambda"
}
data "aws_iam_policy_document" "s3_cross_account" {
policy_id = "Cross account access for s3"
statement {
sid= "1"
principals {
type = "AWS"
identifiers = ["arn:aws:iam::${data.aws_caller_identity.lambda.account_id}:root"]
}
actions = [""s3:GetBucketLocation", "s3:ListBucket"]
resources = ["arn:aws:s3:::${aws_s3_bucket.bucket.bucket}"]
}
}
resource "aws_s3_bucket_policy" "s3_cross_account" {
bucket = "${aws_s3_bucket.bucket.id}"
policy = "${data.aws_iam_policy_document.s3_cross_account.json}"
}
Next, in AWS account B create a role which can be used by your lambda function:
data "aws_iam_policy_document" "access_cross_account_s3" {
provider = "aws.lambda"
policy_id = "Cross account access for s3"
statement {
sid= "1"
actions = ["s3:ListBucket"]
resources = ["arn:aws:s3:::${aws_s3_bucket.bucket.bucket}"]
}
}
resource "aws_iam_role" "s3_cross_account" {
provider = "aws.lambda"
name = "s3_cross_account"
assume_role_policy = "${data.aws_iam_policy_document.access_cross_account_s3.json}"
}
Your lambda function code block must also have the provider = "aws.lambda" config set, so it is created in AWS account B.