I'm working in Terraform, and am creating an S3 object/folder-with-content. I would like to exclude that object from my lifecycle policy. But I'm not sure to exclude the object (folder-object/sample) from the lifecycle policy (Terraform Code Below):
resource "aws_s3_bucket" "s3_test" {
bucket = "test-bucket-upload"
acl = "private"
key = "folder-object/sample"
tags {
Name = "test-bucket"
Environment = "lab"
}
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
lifecycle_rule {
id = "glacier-transfer"
enabled = true
transition {
days = 360
storage_class = "GLACIER"
}
}
}
Instead of excluding, use prefix to identify the objects your lifecycle rule should apply to. For example, the rule below would only apply to objects in the new_objects folder in your bucket:
...
lifecycle_rule {
id = "glacier-transfer"
enabled = true
prefix = "new_objects/"
transition {
days = 360
storage_class = "GLACIER"
}
}
...
Related
I have this terraform script that works perfectly fine for the whole s3 module but it cannot create the Object lock configuration resource and returns the message :
error creating S3 bucket (bucket-name) Object lock configuration: AccessDenied: AccessDenied
Status code 403, request id: ..., host id: ...
Desite the message, the S3 bucket is actually created, but I still get this error, maybe there is something missing in the policy ?
Here is my code.
module "s3_bucket" {
source = "terraform-aws-modules/s3-bucket/aws"
version = "3.4.0"
bucket = local.bucket_name
...
object_lock_enabled = true
attach_policy = true
policy = data.aws_iam_policy_document.voucher_s3_bucket.json
versioning = {
status = var.status
mfa_delete = var.mfa_delete
}
server_side_encryption_configuration = {
rule = {
apply_server_side_encryption_by_default = {
kms_master_key_id = aws_kms_key.voucher_s3_bucket.arn
sse_algorithm = "aws:kms"
}
}
}
}
data "aws_iam_policy_document" "s3_bucket_kms_key" {
statement {
sid = "AllowPutRoles"
effect = "Allow"
actions = ["kms:GenerateDataKey"]
principals {
identifiers = local.put_object_roles #we can use event_gateway iam_role for now
type = "AWS"
}
resources = ["*"]
}
statement {
sid = "AllowAdmin"
effect = "Allow"
actions = [
"kms:*",
]
principals {
identifiers = [data.aws_iam_role.admin_role.arn, data.aws_iam_role.default_role.arn, data.aws_iam_role.automation_role.arn]
type = "AWS"
}
resources = ["*"]
}
}
resource "aws_kms_key" "s3_bucket" {
tags = {
"s3_bucket" = local.bucket_name
}
enable_key_rotation = true
policy = data.aws_iam_policy_document.voucher_s3_bucket_kms_key.json
}
resource "aws_s3_bucket_object_lock_configuration" "s3_bucket_object_lock_configuration" {
bucket = local.bucket_name
rule {
default_retention {
mode = "GOVERNANCE"
years = 10
}
}
}
data "aws_iam_policy_document" "voucher_s3_bucket" {
statement {
sid = "DenyNoKMSEncryption"
effect = "Deny"
actions = ["s3:PutObject"]
principals {
identifiers = ["*"]
type = "*"
}
resources = ["${module.voucher_s3_bucket.s3_bucket_arn}/*"]
condition {
test = "StringNotEqualsIfExists"
values = ["aws:kms"]
variable = "s3:x-amz-server-side-encryption"
}
condition {
test = "Null"
values = ["false"]
variable = "s3:x-amz-server-side-encryption"
}
}
statement {
sid = "DenyWrongKMSKey"
effect = "Deny"
actions = ["s3:PutObject"]
principals {
identifiers = ["*"]
type = "*"
}
resources = ["${module.s3_bucket.s3_bucket_arn}/*"]
condition {
test = "StringNotEquals"
values = [aws_kms_key.voucher_s3_bucket.arn]
variable = "s3:x-amz-server-side-encryption-aws-kms-key-id"
}
}
statement {
sid = "AllowAdminDefault"
effect = "Allow"
actions = ["s3:*"]
principals {
identifiers = [data.aws_iam_role.admin_role.arn, data.aws_iam_role.default_role.arn]
type = "AWS"
}
resources = [
"${module.voucher_s3_bucket.s3_bucket_arn}/*",
module.voucher_s3_bucket.s3_bucket_arn,
]
}
statement {
sid = "DenyDeleteActions"
effect = "Deny"
actions = ["s3:DeleteBucket", "s3:DeleteObject", "s3:DeleteObjectVersion", "s3:PutBucketObjectLockConfiguration"]
principals {
identifiers = ["*"]
type = "AWS"
}
resources = [
"${module.s3_bucket.s3_bucket_arn}/*",
module.s3_bucket.s3_bucket_arn,
]
}
}
I have created a Terraform project in order to create s3-buckets, cross region replication, versioning and a livecycle rule to delete old versions.
The problem is, that whenever I run terraform, it want's to delete the cross region replication and livecycle rule, if it exists in AWS, and whenever it does not exist anymore, it wants to readd it. This happens without any code change.
It seems that the state is not accurate.
I already deleted everything from scratch in AWS and started from the beginning, but it didn't help. I always run into the flappy situation.
All the details:
$ terraform init && terraform plan -var-file xyz.tfvars
...
Terraform will perform the following actions:
# module.test_s3_bucket["bnpl-docs"].aws_s3_bucket.s3_bucket will be updated in-place
~ resource "aws_s3_bucket" "s3_bucket" {
id = "bnpl-docs"
tags = {}
# (11 unchanged attributes hidden)
- replication_configuration {
- role = "arn:aws:iam::....:role/bnpl-docs-s3-bucket-replication" -> null
- rules {
- id = "version-replication" -> null
- priority = 0 -> null
- status = "Enabled" -> null
- destination {
- bucket = "arn:aws:s3:::bnpl-docs-crr" -> null
- storage_class = "STANDARD" -> null
}
}
}
# (1 unchanged block hidden)
}
# module.test_s3_bucket["bnpl-docs"].aws_s3_bucket_lifecycle_configuration.s3_bucket[0] will be created
+ resource "aws_s3_bucket_lifecycle_configuration" "s3_bucket" {
+ bucket = "bnpl-docs"
+ id = (known after apply)
+ rule {
+ id = "version-retention"
+ status = "Enabled"
+ expiration {
+ days = 0
+ expired_object_delete_marker = true
}
+ noncurrent_version_expiration {
+ noncurrent_days = 30
}
}
}
Plan: 1 to add, 1 to change, 0 to destroy.
But result is both is missing in AWS after running apply.
When I rerun, i get the same output:
Terraform will perform the following actions:
# module.test_s3_bucket["bnpl-docs"].aws_s3_bucket.s3_bucket will be updated in-place
~ resource "aws_s3_bucket" "s3_bucket" {
id = "bnpl-docs"
tags = {}
# (11 unchanged attributes hidden)
- replication_configuration {
- role = "arn:aws:iam::......:role/bnpl-docs-s3-bucket-replication" -> null
- rules {
- id = "version-replication" -> null
- priority = 0 -> null
- status = "Enabled" -> null
- destination {
- bucket = "arn:aws:s3:::bnpl-docs-crr" -> null
- storage_class = "STANDARD" -> null
}
}
}
# (1 unchanged block hidden)
}
# module.test_s3_bucket["bnpl-docs"].aws_s3_bucket_lifecycle_configuration.s3_bucket[0] will be created
+ resource "aws_s3_bucket_lifecycle_configuration" "s3_bucket" {
+ bucket = "bnpl-docs"
+ id = (known after apply)
+ rule {
+ id = "version-retention"
+ status = "Enabled"
+ expiration {
+ days = 0
+ expired_object_delete_marker = true
}
+ noncurrent_version_expiration {
+ noncurrent_days = 30
}
}
}
Plan: 1 to add, 1 to change, 0 to destroy.
But result is both is created in AWS after running apply.
I have created several modules to realize what I want. The involved code:
module "test_s3_bucket" {
source = "./modules/test-s3-bucket"
for_each = local.aws_s3_bucket_map
bucket_name = each.key
versioning = each.value.version_config
}
module "test_s3_bucket_repli" {
source = "./modules/test-s3-bucket"
providers = {
aws = aws.repli
}
for_each = local.aws_s3_bucket_map_repli
bucket_name = each.key
versioning = each.value.version_config
}
module "test_s3_bucket_repli_config" {
source = "./modules/test-s3-bucket-replication"
for_each = local.aws_s3_bucket_map_repli
src_bucket = {
name = module.test_s3_bucket[each.value.src_bucket_name].name
arn = module.test_s3_bucket[each.value.src_bucket_name].arn
}
dest_bucket = {
name = module.test_s3_bucket_repli[each.key].name
arn = module.test_s3_bucket_repli[each.key].arn
}
}
Contents of test_s3_bucket-Module:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
##
# Bucket with configuration
##
resource "aws_s3_bucket" "s3_bucket" {
bucket = var.bucket_name
}
resource "aws_s3_bucket_acl" "s3_bucket" {
bucket = aws_s3_bucket.s3_bucket.id
acl = "private"
}
resource "aws_s3_bucket_versioning" "s3_bucket" {
bucket = aws_s3_bucket.s3_bucket.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_lifecycle_configuration" "s3_bucket" {
count = var.versioning == null ? 0 : 1
bucket = aws_s3_bucket.s3_bucket.id
rule {
id = var.versioning.rule_id
expiration {
expired_object_delete_marker = true
}
noncurrent_version_expiration {
noncurrent_days = var.versioning.expiration_days
}
status = "Enabled"
}
}
resource "aws_s3_bucket_public_access_block" "s3_bucket" {
bucket = aws_s3_bucket.s3_bucket.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
The contents of the test-s3-bucket-replication module:
erraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
resource "aws_s3_bucket_replication_configuration" "bucket_main" {
bucket = var.src_bucket.name
role = aws_iam_role.s3_bucket_main_replication.arn
rule {
id = "version-replication"
status = "Enabled"
destination {
bucket = var.dest_bucket.arn
storage_class = "STANDARD"
}
}
}
resource "aws_iam_role" "s3_bucket_main_replication" {
name = "${var.src_bucket.name}-s3-bucket-replication"
assume_role_policy = file("${path.module}/files/policies/sts-s3-assume.json")
}
resource "aws_iam_policy" "s3_bucket_main_replication" {
name = "${var.src_bucket.name}-s3-bucket-replication"
policy = templatefile("${path.module}/files/policies/s3-bucket-replication.json", {
source_bucket_arn = var.src_bucket.arn,
destination_bucket_arn = var.dest_bucket.arn,
})
}
resource "aws_iam_role_policy_attachment" "s3_bucket_main_replication" {
role = aws_iam_role.s3_bucket_main_replication.name
policy_arn = aws_iam_policy.s3_bucket_main_replication.arn
}
You see the two configuration parts which kind of seem to be in conflict are spread over two modules. I'm unsure if this is a problem. I'm kinda new to terraform :)
After increasing the Loglevel via TF_LOG="DEBUG", I found out that those two ressources conflicted with the bucket configuration - the aws_s3_bucket-configuration did not include any configuration for lifecycle or replication (as suggested by the documentation), but terraform expected this legacy declaration somehow.
I was using AWS Provider 3.75 and upgraded to 4.X. After this upgrade, everything works as expected.
Everything seems to work fine using Terraform, but for some reason after each apply it keeps removing and then adding back the configuration for server side encryption on all s3 buckets. If I apply the removal, it will just add it back next time I run apply.
Here is what happens after running terraform plan on my main branch with no changes made/deployed. Next time I run plan/apply it will add it back.
# aws_s3_bucket.terraform-state will be updated in-place
~ resource "aws_s3_bucket" "terraform-state" {
id = "company-terraform-state"
tags = {}
# (11 unchanged attributes hidden)
- server_side_encryption_configuration {
- rule {
- bucket_key_enabled = false -> null
- apply_server_side_encryption_by_default {
- kms_master_key_id = "arn:aws:kms:us-east-1:123456789012:key/Random-GUID-ABCD-1234" -> null
- sse_algorithm = "aws:kms" -> null
}
}
}
# (1 unchanged block hidden)
}
Possibly contributing: I setup a S3 state bucket to keep track of what I have deployed in AWS: https://technology.doximity.com/articles/terraform-s3-backend-best-practices
My state.tf file:
// This file is based on the writtings here: https://technology.doximity.com/articles/terraform-s3-backend-best-practices
terraform {
backend "s3" {
bucket = "company-terraform-state"
key = "state/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "alias/terraform-bucket-key"
dynamodb_table = "terraform-state"
}
}
// The backend configuration above is added after the state s3 bucket is created with the rest of the file below
resource "aws_kms_key" "terraform-bucket-key" {
description = "This key is used to encrypt bucket objects for terraform state"
deletion_window_in_days = 10
enable_key_rotation = true
}
resource "aws_kms_alias" "key-alias" {
name = "alias/terraform-bucket-key"
target_key_id = aws_kms_key.terraform-bucket-key.key_id
}
resource "aws_s3_bucket" "terraform-state" {
bucket = "company-terraform-state"
}
resource "aws_s3_bucket_server_side_encryption_configuration" "encryption-config" {
bucket = aws_s3_bucket.terraform-state.id
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.terraform-bucket-key.arn
sse_algorithm = "aws:kms"
}
}
}
resource "aws_s3_bucket_versioning" "versioning" {
bucket = aws_s3_bucket.terraform-state.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_acl" "acl" {
bucket = aws_s3_bucket.terraform-state.id
acl = "private"
}
resource "aws_s3_bucket_public_access_block" "block" {
bucket = aws_s3_bucket.terraform-state.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
// This table exists to prevent multiple team members from modifying the state file at the same time
resource "aws_dynamodb_table" "terraform-state" {
name = "terraform-state"
read_capacity = 20
write_capacity = 20
hash_key = "LockID"
attribute {
name = "LockID"
type = "S"
}
}
Figured it out, I was using an older provider in my main.tf. I had 3.0 set instead of 4.0 and I’m using the newer aws_s3_bucket_server_side_encryption_configuration instead of configuring the encryption in aws_s3_bucket (which would have been more suiting to the older provider).
I’m actually surprised it worked at all! Must be some features in 3.0 that were released yet.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0" # This was "~> 3.0"
}
}
}
I'm getting an error in my Terraform scripts when attempting to add logging to two buckets. These are part of one of my modules, and I've successfully used them before. I've come back to deploy a new environment...and now it's not working.
I'm getting the following error:
module.dev2_environment.module.portal.aws_s3_bucket.portal_bucket: 1 error occurred:
* aws_s3_bucket.portal_bucket: Error putting S3 logging: InvalidTargetBucketForLogging: You must give the log-delivery group
WRITE and READ_ACP permissions to the target bucket
status code: 400, request id: 51AB42EFCACC9924, host id: nYCUxjHZE+xTisA1xG5syLTKVN/Rtwu8z3xF+O9GAPMdC2yGcafP4uwDURUGKd9Lx1SD8aHTcEI=
I'm executing via CLI, with Admin credentials. No code changes were made between the working state and the error. Any ideas on what could have changed? Syntax? AWS config someplace?
Terraform 11.14 and aws provider 2.16
Log bucket:
resource "aws_s3_bucket" "logs_bucket" {
bucket = "XYZ-${var.env}-cdnlogs"
acl = "log-delivery-write"
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
tags {
Finance = "dev_env"
Environment = "${var.env}"
}
}
Target Bucket:
resource "aws_s3_bucket" "portal_bucket" {
bucket = "XYZ-${var.env}-portal"
acl = "private"
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
logging {
target_bucket = "${aws_s3_bucket.logs_bucket.id}"
target_prefix = "logs/portal/"
}
website {
index_document = "index.html"
error_document = "index.html"
}
// Needed to allow logos to be uploaded the "Portal"
cors_rule {
allowed_headers = ["*"]
allowed_methods = ["GET", "HEAD", "PUT", "POST"]
allowed_origins = ["*"]
max_age_seconds = 3000
}
tags {
Finance = "dev_env"
Environment = "${var.env}"
}
}
I think the error is here
logging {
target_bucket = "${aws_s3_bucket.**logs_bucket**.id}"
target_prefix = "logs/portal/"
}
It should be
logging {
**target_bucket = "${aws_s3_bucket.portal_bucket.id}"**
target_prefix = "logs/portal/"
}
Set the values of your log-delivery-write ACL to allow Logging -> Read and Logging Write. As well as Read bucket permissions.
I am trying to create S3 bucket using terraform from examples in the link
https://www.terraform.io/docs/providers/aws/r/s3_bucket.html
I have created a S3 module.
The issue i am facing is, for certain bucket i do not want logging enabled.
How can this be accomplished in terraform.
logging {
target_bucket = "${aws_s3_bucket.log_bucket.id}"
target_prefix = "log/"
}
Using empty string for target_bucket and target_prefix causes terraform to make an attempt to create target_bucket.
Also, i am trying to use a module.
Using the newer dynamic block support in terraform 0.12+ we pass a single-item array containing the logging settings if we want logging like so:
variable "logging" {
type = list
default = []
description = "to enable logging set this to [{target_bucket = 'xxx' target_prefix = 'logs/'}]"
}
resource "aws_s3_bucket" "s3bucket" {
dynamic "logging" {
for_each = [for l in var.logging : {
target_bucket = l.target_bucket
target_prefix = l.target_prefix
}]
content {
target_bucket = logging.value.target_bucket
target_prefix = logging.value.target_prefix
}
}
}
Can Fly.
If you want to make the values of logging optional, first make your module aws_s3_bucket.tf:
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging = "${var.logging}"
}
variable "logging" {
type = "list"
default = []
}
then in a sub-folder example add your template module.tf:
module "s3" {
source = "../"
logging = [
{
target_bucket = "loggingbucketname"
target_prefix = "log/"
},
]
}
provider "aws" {
region = "eu-west-1"
version = "2.4.0"
}
This is your version that has logging.
Next modify your module.tf to look like
module "s3" {
source = "../"
}
provider "aws" {
region = "eu-west-1"
version = "2.4.0"
}
That's your version without. This worked with:
Terraform v0.11.11
+ provider.aws v2.4.0
Updated
This is answer for v0.12.5.
module is now:
resource "aws_s3_bucket" "b" {
bucket = "my-tf-test-bucket"
acl = "private"
logging {
target_bucket = var.logging["target_bucket"]
target_prefix = var.logging["target_prefix"]
}
}
variable "logging" {
type=map
default={
target_bucket = ""
target_prefix = ""
}
}
Use module with logging becomes (your path to modules might differ):
module "s3" {
source = "../"
logging={
target_bucket = aws_s3_bucket.log_bucket.id
target_prefix = "log/"
}
}
provider "aws" {
region = "eu-west-1"
version = "2.34.0"
}
resource "aws_s3_bucket" "log_bucket" {
bucket = "my-tf-log-bucket"
acl = "private"
}
and without:
module "s3" {
source = "../"
}
provider "aws" {
region = "eu-west-1"
version = "2.34.0"
}