Terraform AWS optional logging for S3 bucket - amazon-s3

I am trying to create S3 bucket using terraform from examples in the link
https://www.terraform.io/docs/providers/aws/r/s3_bucket.html
I have created a S3 module.
The issue i am facing is, for certain bucket i do not want logging enabled.
How can this be accomplished in terraform.
logging {
target_bucket = "${aws_s3_bucket.log_bucket.id}"
target_prefix = "log/"
}
Using empty string for target_bucket and target_prefix causes terraform to make an attempt to create target_bucket.
Also, i am trying to use a module.

Using the newer dynamic block support in terraform 0.12+ we pass a single-item array containing the logging settings if we want logging like so:
variable "logging" {
type = list
default = []
description = "to enable logging set this to [{target_bucket = 'xxx' target_prefix = 'logs/'}]"
}
resource "aws_s3_bucket" "s3bucket" {
dynamic "logging" {
for_each = [for l in var.logging : {
target_bucket = l.target_bucket
target_prefix = l.target_prefix
}]
content {
target_bucket = logging.value.target_bucket
target_prefix = logging.value.target_prefix
}
}
}

Can Fly.
If you want to make the values of logging optional, first make your module aws_s3_bucket.tf:
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging = "${var.logging}"
}
variable "logging" {
type = "list"
default = []
}
then in a sub-folder example add your template module.tf:
module "s3" {
source = "../"
logging = [
{
target_bucket = "loggingbucketname"
target_prefix = "log/"
},
]
}
provider "aws" {
region = "eu-west-1"
version = "2.4.0"
}
This is your version that has logging.
Next modify your module.tf to look like
module "s3" {
source = "../"
}
provider "aws" {
region = "eu-west-1"
version = "2.4.0"
}
That's your version without. This worked with:
Terraform v0.11.11
+ provider.aws v2.4.0
Updated
This is answer for v0.12.5.
module is now:
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging {
target_bucket = var.logging["target_bucket"]
target_prefix = var.logging["target_prefix"]
}
}
variable "logging" {
type=map
default={
target_bucket = ""
target_prefix = ""
}
}
Use module with logging becomes (your path to modules might differ):
module "s3" {
source = "../"
logging={
target_bucket = aws_s3_bucket.log_bucket.id
target_prefix = "log/"
}
}
provider "aws" {
region = "eu-west-1"
version = "2.34.0"
}
resource "aws_s3_bucket" "log_bucket" {
bucket = "my-tf-log-bucket"
acl = "private"
}
and without:
module "s3" {
source = "../"
}
provider "aws" {
region = "eu-west-1"
version = "2.34.0"
}

Related

Terraform Object Lock Configuration: AccessDenied

I have this terraform script that works perfectly fine for the whole s3 module but it cannot create the Object lock configuration resource and returns the message :
error creating S3 bucket (bucket-name) Object lock configuration: AccessDenied: AccessDenied
Status code 403, request id: ..., host id: ...
Desite the message, the S3 bucket is actually created, but I still get this error, maybe there is something missing in the policy ?
Here is my code.
module "s3_bucket" {
source = "terraform-aws-modules/s3-bucket/aws"
version = "3.4.0"
bucket = local.bucket_name
...
object_lock_enabled = true
attach_policy = true
policy = data.aws_iam_policy_document.voucher_s3_bucket.json
versioning = {
status = var.status
mfa_delete = var.mfa_delete
}
server_side_encryption_configuration = {
rule = {
apply_server_side_encryption_by_default = {
kms_master_key_id = aws_kms_key.voucher_s3_bucket.arn
sse_algorithm = "aws:kms"
}
}
}
}
data "aws_iam_policy_document" "s3_bucket_kms_key" {
statement {
sid = "AllowPutRoles"
effect = "Allow"
actions = ["kms:GenerateDataKey"]
principals {
identifiers = local.put_object_roles #we can use event_gateway iam_role for now
type = "AWS"
}
resources = ["*"]
}
statement {
sid = "AllowAdmin"
effect = "Allow"
actions = [
"kms:*",
]
principals {
identifiers = [data.aws_iam_role.admin_role.arn, data.aws_iam_role.default_role.arn, data.aws_iam_role.automation_role.arn]
type = "AWS"
}
resources = ["*"]
}
}
resource "aws_kms_key" "s3_bucket" {
tags = {
"s3_bucket" = local.bucket_name
}
enable_key_rotation = true
policy = data.aws_iam_policy_document.voucher_s3_bucket_kms_key.json
}
resource "aws_s3_bucket_object_lock_configuration" "s3_bucket_object_lock_configuration" {
bucket = local.bucket_name
rule {
default_retention {
mode = "GOVERNANCE"
years = 10
}
}
}
data "aws_iam_policy_document" "voucher_s3_bucket" {
statement {
sid = "DenyNoKMSEncryption"
effect = "Deny"
actions = ["s3:PutObject"]
principals {
identifiers = ["*"]
type = "*"
}
resources = ["${module.voucher_s3_bucket.s3_bucket_arn}/*"]
condition {
test = "StringNotEqualsIfExists"
values = ["aws:kms"]
variable = "s3:x-amz-server-side-encryption"
}
condition {
test = "Null"
values = ["false"]
variable = "s3:x-amz-server-side-encryption"
}
}
statement {
sid = "DenyWrongKMSKey"
effect = "Deny"
actions = ["s3:PutObject"]
principals {
identifiers = ["*"]
type = "*"
}
resources = ["${module.s3_bucket.s3_bucket_arn}/*"]
condition {
test = "StringNotEquals"
values = [aws_kms_key.voucher_s3_bucket.arn]
variable = "s3:x-amz-server-side-encryption-aws-kms-key-id"
}
}
statement {
sid = "AllowAdminDefault"
effect = "Allow"
actions = ["s3:*"]
principals {
identifiers = [data.aws_iam_role.admin_role.arn, data.aws_iam_role.default_role.arn]
type = "AWS"
}
resources = [
"${module.voucher_s3_bucket.s3_bucket_arn}/*",
module.voucher_s3_bucket.s3_bucket_arn,
]
}
statement {
sid = "DenyDeleteActions"
effect = "Deny"
actions = ["s3:DeleteBucket", "s3:DeleteObject", "s3:DeleteObjectVersion", "s3:PutBucketObjectLockConfiguration"]
principals {
identifiers = ["*"]
type = "AWS"
}
resources = [
"${module.s3_bucket.s3_bucket_arn}/*",
module.s3_bucket.s3_bucket_arn,
]
}
}

Terraform "ping-pong" encryption config for S3 buckets

Everything seems to work fine using Terraform, but for some reason after each apply it keeps removing and then adding back the configuration for server side encryption on all s3 buckets. If I apply the removal, it will just add it back next time I run apply.
Here is what happens after running terraform plan on my main branch with no changes made/deployed. Next time I run plan/apply it will add it back.
# aws_s3_bucket.terraform-state will be updated in-place
~ resource "aws_s3_bucket" "terraform-state" {
id = "company-terraform-state"
tags = {}
# (11 unchanged attributes hidden)
- server_side_encryption_configuration {
- rule {
- bucket_key_enabled = false -> null
- apply_server_side_encryption_by_default {
- kms_master_key_id = "arn:aws:kms:us-east-1:123456789012:key/Random-GUID-ABCD-1234" -> null
- sse_algorithm = "aws:kms" -> null
}
}
}
# (1 unchanged block hidden)
}
Possibly contributing: I setup a S3 state bucket to keep track of what I have deployed in AWS: https://technology.doximity.com/articles/terraform-s3-backend-best-practices
My state.tf file:
// This file is based on the writtings here: https://technology.doximity.com/articles/terraform-s3-backend-best-practices
terraform {
backend "s3" {
bucket = "company-terraform-state"
key = "state/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "alias/terraform-bucket-key"
dynamodb_table = "terraform-state"
}
}
// The backend configuration above is added after the state s3 bucket is created with the rest of the file below
resource "aws_kms_key" "terraform-bucket-key" {
description = "This key is used to encrypt bucket objects for terraform state"
deletion_window_in_days = 10
enable_key_rotation = true
}
resource "aws_kms_alias" "key-alias" {
name = "alias/terraform-bucket-key"
target_key_id = aws_kms_key.terraform-bucket-key.key_id
}
resource "aws_s3_bucket" "terraform-state" {
bucket = "company-terraform-state"
}
resource "aws_s3_bucket_server_side_encryption_configuration" "encryption-config" {
bucket = aws_s3_bucket.terraform-state.id
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.terraform-bucket-key.arn
sse_algorithm = "aws:kms"
}
}
}
resource "aws_s3_bucket_versioning" "versioning" {
bucket = aws_s3_bucket.terraform-state.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_acl" "acl" {
bucket = aws_s3_bucket.terraform-state.id
acl = "private"
}
resource "aws_s3_bucket_public_access_block" "block" {
bucket = aws_s3_bucket.terraform-state.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
// This table exists to prevent multiple team members from modifying the state file at the same time
resource "aws_dynamodb_table" "terraform-state" {
name = "terraform-state"
read_capacity = 20
write_capacity = 20
hash_key = "LockID"
attribute {
name = "LockID"
type = "S"
}
}
Figured it out, I was using an older provider in my main.tf. I had 3.0 set instead of 4.0 and I’m using the newer aws_s3_bucket_server_side_encryption_configuration instead of configuring the encryption in aws_s3_bucket (which would have been more suiting to the older provider).
I’m actually surprised it worked at all! Must be some features in 3.0 that were released yet.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0" # This was "~> 3.0"
}
}
}

Terraform: Adding server logging to S3 bucket

I'm getting an error in my Terraform scripts when attempting to add logging to two buckets. These are part of one of my modules, and I've successfully used them before. I've come back to deploy a new environment...and now it's not working.
I'm getting the following error:
module.dev2_environment.module.portal.aws_s3_bucket.portal_bucket: 1 error occurred:
* aws_s3_bucket.portal_bucket: Error putting S3 logging: InvalidTargetBucketForLogging: You must give the log-delivery group
WRITE and READ_ACP permissions to the target bucket
status code: 400, request id: 51AB42EFCACC9924, host id: nYCUxjHZE+xTisA1xG5syLTKVN/Rtwu8z3xF+O9GAPMdC2yGcafP4uwDURUGKd9Lx1SD8aHTcEI=
I'm executing via CLI, with Admin credentials. No code changes were made between the working state and the error. Any ideas on what could have changed? Syntax? AWS config someplace?
Terraform 11.14 and aws provider 2.16
Log bucket:
resource "aws_s3_bucket" "logs_bucket" {
bucket = "XYZ-${var.env}-cdnlogs"
acl = "log-delivery-write"
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
tags {
Finance = "dev_env"
Environment = "${var.env}"
}
}
Target Bucket:
resource "aws_s3_bucket" "portal_bucket" {
bucket = "XYZ-${var.env}-portal"
acl = "private"
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
logging {
target_bucket = "${aws_s3_bucket.logs_bucket.id}"
target_prefix = "logs/portal/"
}
website {
index_document = "index.html"
error_document = "index.html"
}
// Needed to allow logos to be uploaded the "Portal"
cors_rule {
allowed_headers = ["*"]
allowed_methods = ["GET", "HEAD", "PUT", "POST"]
allowed_origins = ["*"]
max_age_seconds = 3000
}
tags {
Finance = "dev_env"
Environment = "${var.env}"
}
}
I think the error is here
logging {
target_bucket = "${aws_s3_bucket.**logs_bucket**.id}"
target_prefix = "logs/portal/"
}
It should be
logging {
**target_bucket = "${aws_s3_bucket.portal_bucket.id}"**
target_prefix = "logs/portal/"
}
Set the values of your log-delivery-write ACL to allow Logging -> Read and Logging Write. As well as Read bucket permissions.

How to give the target bucket log-delivery group WRITE and READ_ACP permissions?

I am trying to setup a cloudfront dist and s3 bucket with terraform. When I run terraform apply it is returning the following error:
aws_s3_bucket.app: Error putting S3 logging: InvalidTargetBucketForLogging: You must give the log-delivery group WRITE and READ_ACP permissions to the target bucket
my S3.tf file:
data "aws_iam_policy_document" "s3_policy" {
policy_id = "PolicyForCloudFrontPrivateContent"
statement {
sid = "1"
actions = ["s3:GetObject"]
resources = ["arn:aws:s3:::${local.name_env}/*"]
principals {
type = "AWS"
identifiers = ["${aws_cloudfront_origin_access_identity.origin_access_identity.iam_arn}"]
}
}
}
resource "aws_s3_bucket" "app" {
bucket = "${local.name_env}"
policy = "${data.aws_iam_policy_document.s3_policy.json}"
logging {
target_bucket = "${local.logs_bucket}"
target_prefix = "app-${var.environment}"
}
versioning {
enabled = true
}
tags = "${local.tags}"
}
You need to add an acl attribute to your aws_s3_bucket with a value of "log-delivery-write".
resource "aws_s3_bucket" "app" {
bucket = "${local.name_env}"
acl = "log-delivery-write"
...
}
UPDATE: terraform now supports custom bucket acls natively. What follows is a workaround for older versions where the predefined acls would not suffice.
Here's how to achieve this via terraform using a null resource and the AWS CLI.
resource "aws_s3_bucket" "files_bucket" {
# ...
logging {
target_bucket = "${aws_s3_bucket.logs_bucket.bucket}"
}
depends_on = [
"null_resource.logs_bucket_acl_workaround"
]
}
resource "aws_s3_bucket" "logs_bucket" {
# ...
acl = "private"
}
locals {
put_bucket_acl_cmd = "s3api put-bucket-acl --bucket ${aws_s3_bucket.logs_bucket.bucket} --grant-write 'uri=\"http://acs.amazonaws.com/groups/s3/LogDelivery\"' --grant-read-acp 'uri=\"http://acs.amazonaws.com/groups/s3/LogDelivery\"'"
}
resource "null_resource" "logs_bucket_acl_workaround" {
# cannot set bucket ACLs via terraform yet
# https://github.com/terraform-providers/terraform-provider-aws/issues/989
depends_on = [
"aws_s3_bucket.logs_bucket",
]
triggers = {
bucket = "${aws_s3_bucket.logs_bucket.bucket}"
command = "${local.put_bucket_acl_cmd}"
}
provisioner "local-exec" {
command = "aws ${local.put_bucket_acl_cmd}"
}
}
Note that ACLs in this way are only added but never removed.

AWS S3 Object Lifecycle Exclution

I'm working in Terraform, and am creating an S3 object/folder-with-content. I would like to exclude that object from my lifecycle policy. But I'm not sure to exclude the object (folder-object/sample) from the lifecycle policy (Terraform Code Below):
resource "aws_s3_bucket" "s3_test" {
bucket = "test-bucket-upload"
acl = "private"
key = "folder-object/sample"
tags {
Name = "test-bucket"
Environment = "lab"
}
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
lifecycle_rule {
id = "glacier-transfer"
enabled = true
transition {
days = 360
storage_class = "GLACIER"
}
}
}
Instead of excluding, use prefix to identify the objects your lifecycle rule should apply to. For example, the rule below would only apply to objects in the new_objects folder in your bucket:
...
lifecycle_rule {
id = "glacier-transfer"
enabled = true
prefix = "new_objects/"
transition {
days = 360
storage_class = "GLACIER"
}
}
...