The terraform for a bucket creation is like below
resource "aws_s3_bucket" "a" {
bucket = "test-logs"
acl = "private"
versioning {
enabled = false
}
lifecycle_rule {
id = "log"
enabled = true
expiration {
days = 1
}
}
}
But the objects are not getting deleted after 1 day, what did I miss?
Related
I have created a Terraform project in order to create s3-buckets, cross region replication, versioning and a livecycle rule to delete old versions.
The problem is, that whenever I run terraform, it want's to delete the cross region replication and livecycle rule, if it exists in AWS, and whenever it does not exist anymore, it wants to readd it. This happens without any code change.
It seems that the state is not accurate.
I already deleted everything from scratch in AWS and started from the beginning, but it didn't help. I always run into the flappy situation.
All the details:
$ terraform init && terraform plan -var-file xyz.tfvars
...
Terraform will perform the following actions:
# module.test_s3_bucket["bnpl-docs"].aws_s3_bucket.s3_bucket will be updated in-place
~ resource "aws_s3_bucket" "s3_bucket" {
id = "bnpl-docs"
tags = {}
# (11 unchanged attributes hidden)
- replication_configuration {
- role = "arn:aws:iam::....:role/bnpl-docs-s3-bucket-replication" -> null
- rules {
- id = "version-replication" -> null
- priority = 0 -> null
- status = "Enabled" -> null
- destination {
- bucket = "arn:aws:s3:::bnpl-docs-crr" -> null
- storage_class = "STANDARD" -> null
}
}
}
# (1 unchanged block hidden)
}
# module.test_s3_bucket["bnpl-docs"].aws_s3_bucket_lifecycle_configuration.s3_bucket[0] will be created
+ resource "aws_s3_bucket_lifecycle_configuration" "s3_bucket" {
+ bucket = "bnpl-docs"
+ id = (known after apply)
+ rule {
+ id = "version-retention"
+ status = "Enabled"
+ expiration {
+ days = 0
+ expired_object_delete_marker = true
}
+ noncurrent_version_expiration {
+ noncurrent_days = 30
}
}
}
Plan: 1 to add, 1 to change, 0 to destroy.
But result is both is missing in AWS after running apply.
When I rerun, i get the same output:
Terraform will perform the following actions:
# module.test_s3_bucket["bnpl-docs"].aws_s3_bucket.s3_bucket will be updated in-place
~ resource "aws_s3_bucket" "s3_bucket" {
id = "bnpl-docs"
tags = {}
# (11 unchanged attributes hidden)
- replication_configuration {
- role = "arn:aws:iam::......:role/bnpl-docs-s3-bucket-replication" -> null
- rules {
- id = "version-replication" -> null
- priority = 0 -> null
- status = "Enabled" -> null
- destination {
- bucket = "arn:aws:s3:::bnpl-docs-crr" -> null
- storage_class = "STANDARD" -> null
}
}
}
# (1 unchanged block hidden)
}
# module.test_s3_bucket["bnpl-docs"].aws_s3_bucket_lifecycle_configuration.s3_bucket[0] will be created
+ resource "aws_s3_bucket_lifecycle_configuration" "s3_bucket" {
+ bucket = "bnpl-docs"
+ id = (known after apply)
+ rule {
+ id = "version-retention"
+ status = "Enabled"
+ expiration {
+ days = 0
+ expired_object_delete_marker = true
}
+ noncurrent_version_expiration {
+ noncurrent_days = 30
}
}
}
Plan: 1 to add, 1 to change, 0 to destroy.
But result is both is created in AWS after running apply.
I have created several modules to realize what I want. The involved code:
module "test_s3_bucket" {
source = "./modules/test-s3-bucket"
for_each = local.aws_s3_bucket_map
bucket_name = each.key
versioning = each.value.version_config
}
module "test_s3_bucket_repli" {
source = "./modules/test-s3-bucket"
providers = {
aws = aws.repli
}
for_each = local.aws_s3_bucket_map_repli
bucket_name = each.key
versioning = each.value.version_config
}
module "test_s3_bucket_repli_config" {
source = "./modules/test-s3-bucket-replication"
for_each = local.aws_s3_bucket_map_repli
src_bucket = {
name = module.test_s3_bucket[each.value.src_bucket_name].name
arn = module.test_s3_bucket[each.value.src_bucket_name].arn
}
dest_bucket = {
name = module.test_s3_bucket_repli[each.key].name
arn = module.test_s3_bucket_repli[each.key].arn
}
}
Contents of test_s3_bucket-Module:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
##
# Bucket with configuration
##
resource "aws_s3_bucket" "s3_bucket" {
bucket = var.bucket_name
}
resource "aws_s3_bucket_acl" "s3_bucket" {
bucket = aws_s3_bucket.s3_bucket.id
acl = "private"
}
resource "aws_s3_bucket_versioning" "s3_bucket" {
bucket = aws_s3_bucket.s3_bucket.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_lifecycle_configuration" "s3_bucket" {
count = var.versioning == null ? 0 : 1
bucket = aws_s3_bucket.s3_bucket.id
rule {
id = var.versioning.rule_id
expiration {
expired_object_delete_marker = true
}
noncurrent_version_expiration {
noncurrent_days = var.versioning.expiration_days
}
status = "Enabled"
}
}
resource "aws_s3_bucket_public_access_block" "s3_bucket" {
bucket = aws_s3_bucket.s3_bucket.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
The contents of the test-s3-bucket-replication module:
erraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
resource "aws_s3_bucket_replication_configuration" "bucket_main" {
bucket = var.src_bucket.name
role = aws_iam_role.s3_bucket_main_replication.arn
rule {
id = "version-replication"
status = "Enabled"
destination {
bucket = var.dest_bucket.arn
storage_class = "STANDARD"
}
}
}
resource "aws_iam_role" "s3_bucket_main_replication" {
name = "${var.src_bucket.name}-s3-bucket-replication"
assume_role_policy = file("${path.module}/files/policies/sts-s3-assume.json")
}
resource "aws_iam_policy" "s3_bucket_main_replication" {
name = "${var.src_bucket.name}-s3-bucket-replication"
policy = templatefile("${path.module}/files/policies/s3-bucket-replication.json", {
source_bucket_arn = var.src_bucket.arn,
destination_bucket_arn = var.dest_bucket.arn,
})
}
resource "aws_iam_role_policy_attachment" "s3_bucket_main_replication" {
role = aws_iam_role.s3_bucket_main_replication.name
policy_arn = aws_iam_policy.s3_bucket_main_replication.arn
}
You see the two configuration parts which kind of seem to be in conflict are spread over two modules. I'm unsure if this is a problem. I'm kinda new to terraform :)
After increasing the Loglevel via TF_LOG="DEBUG", I found out that those two ressources conflicted with the bucket configuration - the aws_s3_bucket-configuration did not include any configuration for lifecycle or replication (as suggested by the documentation), but terraform expected this legacy declaration somehow.
I was using AWS Provider 3.75 and upgraded to 4.X. After this upgrade, everything works as expected.
Everything seems to work fine using Terraform, but for some reason after each apply it keeps removing and then adding back the configuration for server side encryption on all s3 buckets. If I apply the removal, it will just add it back next time I run apply.
Here is what happens after running terraform plan on my main branch with no changes made/deployed. Next time I run plan/apply it will add it back.
# aws_s3_bucket.terraform-state will be updated in-place
~ resource "aws_s3_bucket" "terraform-state" {
id = "company-terraform-state"
tags = {}
# (11 unchanged attributes hidden)
- server_side_encryption_configuration {
- rule {
- bucket_key_enabled = false -> null
- apply_server_side_encryption_by_default {
- kms_master_key_id = "arn:aws:kms:us-east-1:123456789012:key/Random-GUID-ABCD-1234" -> null
- sse_algorithm = "aws:kms" -> null
}
}
}
# (1 unchanged block hidden)
}
Possibly contributing: I setup a S3 state bucket to keep track of what I have deployed in AWS: https://technology.doximity.com/articles/terraform-s3-backend-best-practices
My state.tf file:
// This file is based on the writtings here: https://technology.doximity.com/articles/terraform-s3-backend-best-practices
terraform {
backend "s3" {
bucket = "company-terraform-state"
key = "state/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "alias/terraform-bucket-key"
dynamodb_table = "terraform-state"
}
}
// The backend configuration above is added after the state s3 bucket is created with the rest of the file below
resource "aws_kms_key" "terraform-bucket-key" {
description = "This key is used to encrypt bucket objects for terraform state"
deletion_window_in_days = 10
enable_key_rotation = true
}
resource "aws_kms_alias" "key-alias" {
name = "alias/terraform-bucket-key"
target_key_id = aws_kms_key.terraform-bucket-key.key_id
}
resource "aws_s3_bucket" "terraform-state" {
bucket = "company-terraform-state"
}
resource "aws_s3_bucket_server_side_encryption_configuration" "encryption-config" {
bucket = aws_s3_bucket.terraform-state.id
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.terraform-bucket-key.arn
sse_algorithm = "aws:kms"
}
}
}
resource "aws_s3_bucket_versioning" "versioning" {
bucket = aws_s3_bucket.terraform-state.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_acl" "acl" {
bucket = aws_s3_bucket.terraform-state.id
acl = "private"
}
resource "aws_s3_bucket_public_access_block" "block" {
bucket = aws_s3_bucket.terraform-state.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
// This table exists to prevent multiple team members from modifying the state file at the same time
resource "aws_dynamodb_table" "terraform-state" {
name = "terraform-state"
read_capacity = 20
write_capacity = 20
hash_key = "LockID"
attribute {
name = "LockID"
type = "S"
}
}
Figured it out, I was using an older provider in my main.tf. I had 3.0 set instead of 4.0 and I’m using the newer aws_s3_bucket_server_side_encryption_configuration instead of configuring the encryption in aws_s3_bucket (which would have been more suiting to the older provider).
I’m actually surprised it worked at all! Must be some features in 3.0 that were released yet.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0" # This was "~> 3.0"
}
}
}
I am trying to setup a cloudfront dist and s3 bucket with terraform. When I run terraform apply it is returning the following error:
aws_s3_bucket.app: Error putting S3 logging: InvalidTargetBucketForLogging: You must give the log-delivery group WRITE and READ_ACP permissions to the target bucket
my S3.tf file:
data "aws_iam_policy_document" "s3_policy" {
policy_id = "PolicyForCloudFrontPrivateContent"
statement {
sid = "1"
actions = ["s3:GetObject"]
resources = ["arn:aws:s3:::${local.name_env}/*"]
principals {
type = "AWS"
identifiers = ["${aws_cloudfront_origin_access_identity.origin_access_identity.iam_arn}"]
}
}
}
resource "aws_s3_bucket" "app" {
bucket = "${local.name_env}"
policy = "${data.aws_iam_policy_document.s3_policy.json}"
logging {
target_bucket = "${local.logs_bucket}"
target_prefix = "app-${var.environment}"
}
versioning {
enabled = true
}
tags = "${local.tags}"
}
You need to add an acl attribute to your aws_s3_bucket with a value of "log-delivery-write".
resource "aws_s3_bucket" "app" {
bucket = "${local.name_env}"
acl = "log-delivery-write"
...
}
UPDATE: terraform now supports custom bucket acls natively. What follows is a workaround for older versions where the predefined acls would not suffice.
Here's how to achieve this via terraform using a null resource and the AWS CLI.
resource "aws_s3_bucket" "files_bucket" {
# ...
logging {
target_bucket = "${aws_s3_bucket.logs_bucket.bucket}"
}
depends_on = [
"null_resource.logs_bucket_acl_workaround"
]
}
resource "aws_s3_bucket" "logs_bucket" {
# ...
acl = "private"
}
locals {
put_bucket_acl_cmd = "s3api put-bucket-acl --bucket ${aws_s3_bucket.logs_bucket.bucket} --grant-write 'uri=\"http://acs.amazonaws.com/groups/s3/LogDelivery\"' --grant-read-acp 'uri=\"http://acs.amazonaws.com/groups/s3/LogDelivery\"'"
}
resource "null_resource" "logs_bucket_acl_workaround" {
# cannot set bucket ACLs via terraform yet
# https://github.com/terraform-providers/terraform-provider-aws/issues/989
depends_on = [
"aws_s3_bucket.logs_bucket",
]
triggers = {
bucket = "${aws_s3_bucket.logs_bucket.bucket}"
command = "${local.put_bucket_acl_cmd}"
}
provisioner "local-exec" {
command = "aws ${local.put_bucket_acl_cmd}"
}
}
Note that ACLs in this way are only added but never removed.
I'm working in Terraform, and am creating an S3 object/folder-with-content. I would like to exclude that object from my lifecycle policy. But I'm not sure to exclude the object (folder-object/sample) from the lifecycle policy (Terraform Code Below):
resource "aws_s3_bucket" "s3_test" {
bucket = "test-bucket-upload"
acl = "private"
key = "folder-object/sample"
tags {
Name = "test-bucket"
Environment = "lab"
}
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
lifecycle_rule {
id = "glacier-transfer"
enabled = true
transition {
days = 360
storage_class = "GLACIER"
}
}
}
Instead of excluding, use prefix to identify the objects your lifecycle rule should apply to. For example, the rule below would only apply to objects in the new_objects folder in your bucket:
...
lifecycle_rule {
id = "glacier-transfer"
enabled = true
prefix = "new_objects/"
transition {
days = 360
storage_class = "GLACIER"
}
}
...
I am new to Terraform and I am trying to create a Terraform module S3 bucket for Lambda function which is accessible cross account. Also the Cross account access should be limited to listing and getting objects.
Below is what I get to create lambda:
resource "aws_lambda_function" "test_lambda" {
filename = "lambda_function_payload.zip"
s3_bucket = "my_bucket_name"
function_name = "lambda_function_name"
role = "${aws_iam_role.iam_for_lambda.arn}"
handler = "exports.test"
source_code_hash = "${base64sha256(file("lambda_function_payload.zip"))}"
runtime = "nodejs4.3"
}
This is how my s3 bucket code looks like:
resource "aws_s3_bucket" "bucket" {
bucket = "my-bucket"
acl = "private"
lifecycle_rule {
id = "log"
enabled = true
prefix = "log/"
tags {
"rule" = "log"
"autoclean" = "true"
}
transition {
days = 30
storage_class = "STANDARD_IA"
}
transition {
days = 60
storage_class = "GLACIER"
}
expiration {
days = 90
}
}
lifecycle_rule {
id = "tmp"
prefix = "tmp/"
enabled = true
expiration {
date = "2016-01-12"
}
}
#Using this section for cors_rule
cors_rule {
allowed_headers = ["*"]
allowed_methods = ["GET"]
allowed_origins = ["https://s3-website-test.hashicorp.com"]
}
}
How I can turn this into a module ?
The necessary policies are described in the official documentation. In the following I will describe how to setup this example with a single terraform module - you have to adjust the permissions to your needs. You can also extend it, to support more than 2 AWS accounts. I haven't run the following code myself, so there might be some typos etc., but it should help you to get the general idea behind it.
Lets assume AWS account A is the account your s3 bucket is in and AWS account B is the account your lambda function is running in.
In order to place everything into a single module, you must define 2 Terraform AWS providers; one for each AWS account involved.
// Your s3 bucket's provider - used by default
provider "aws" {
.. // your setup here
}
// Your lambda function's provider - used if specified via alias
provider "aws" {
.. // your setup here
alias = "lambda"
}
First, attach a policy to your s3 bucket to allow access from account B:
data "aws_caller_identity" "lambda" {
provider = "aws.lambda"
}
data "aws_iam_policy_document" "s3_cross_account" {
policy_id = "Cross account access for s3"
statement {
sid= "1"
principals {
type = "AWS"
identifiers = ["arn:aws:iam::${data.aws_caller_identity.lambda.account_id}:root"]
}
actions = [""s3:GetBucketLocation", "s3:ListBucket"]
resources = ["arn:aws:s3:::${aws_s3_bucket.bucket.bucket}"]
}
}
resource "aws_s3_bucket_policy" "s3_cross_account" {
bucket = "${aws_s3_bucket.bucket.id}"
policy = "${data.aws_iam_policy_document.s3_cross_account.json}"
}
Next, in AWS account B create a role which can be used by your lambda function:
data "aws_iam_policy_document" "access_cross_account_s3" {
provider = "aws.lambda"
policy_id = "Cross account access for s3"
statement {
sid= "1"
actions = ["s3:ListBucket"]
resources = ["arn:aws:s3:::${aws_s3_bucket.bucket.bucket}"]
}
}
resource "aws_iam_role" "s3_cross_account" {
provider = "aws.lambda"
name = "s3_cross_account"
assume_role_policy = "${data.aws_iam_policy_document.access_cross_account_s3.json}"
}
Your lambda function code block must also have the provider = "aws.lambda" config set, so it is created in AWS account B.