I'm writing a module to create multiple S3 Buckets with all the related resources. Currently, I'm a little bit stuck on the server side encryption as I need to parametrize the KMS key id for a key that is not still created.
The variables passed to the module are:
A list of S3 buckets
A map with the KMS created
The structure of the S3 buckets is
type = list(object({
bucket = string
acl = string
versioning = string
kms_description = string
logging = bool
loggingBucket = optional(string)
logPath = optional(string)
}))
}
The structure of the KMS map is similar to
kms_resources = {
0 = {
kms_arn = (known after apply)
kms_description = "my-kms"
kms_id = (known after apply)
}
}
This variable is an output from a previous module that creates all the KMS. The output is created this way
output "kms_resources" {
value = {
for kms, details in aws_kms_key.my-kms :
kms => ({
"kms_description" = details.description
"kms_id" = details.key_id
"kms_arn" = details.arn
})
}
}
As you can see the idea is that, on the S3 variable, the user can select his own KMS key, but I'm struggling to retrieve the value. At this moment, resource looks like this
resource "aws_s3_bucket_server_side_encryption_configuration" "my-s3-buckets" {
count = length(var.s3_buckets)
bucket = var.s3_buckets[count.index].bucket
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = [
for k in var.kms_keys : k.kms_id
if k.kms_description == var.s3_buckets[count.index].kms_description
]
sse_algorithm = "aws:kms"
}
}
}
I thought it was gonna work but once terraform is giving me Inappropriate value for attribute "kms_master_key_id": string required.. I would also like that if the value does not exist the kms_master_key_id is set by default to aws/s3
The problem seems to be that you try to give a list as kms_master_key_id instead of just a string. The fact that you alternatively want to use "aws/s3" actually makes the fix quite easy:
kms_master_key_id = concat([
for k in var.kms_keys : k.kms_id
if k.kms_description == var.s3_buckets[count.index].kms_description
], ["aws/s3"])[0]
That way you first get a list with keys / that key that matches the description and then append the default s3 key. Afterwards you simply pick the first element of the list, either the first actual key or if none matched you pick the default key.
Related
I am facing a problem with for_each looping in terraform.
I have a azure resource for managed_keys as follow:
resource "azurerm_storage_account_customer_managed_key" "storage-managed-key" {
for_each = toset(var.key-name)
key_name = "Key-Client-${each.value}"
key_vault_id = azurerm_key_vault.tenantsnbshared.id
key_version = azurerm_key_vault_key.client-key[each.value].version
storage_account_id = azurerm_storage_account.storage-foreach[each.value].identity.0.principal_id
depends_on = [azurerm_key_vault_access_policy.storage]
}
I have a variable named key-name and a storage-account storage-foreach, both of them have a list(string) with some values.
My aim is to be able to loop through those 2 variables and encrypt the storage account with the respective key.
but if I run my code, I get this error:
Error: Invalid index
on main.tf line 173, in resource "azurerm_storage_account_customer_managed_key" "storage-managed-key":
173: storage_account_id = azurerm_storage_account.storage-foreach[each.value].identity.0.principal_id
|----------------
| azurerm_storage_account.storage-foreach is object with 4 attributes
| each.value is "key-name"
The given key does not identify an element in this collection value.
EDIT:
resource "azurerm_storage_account" "storage-foreach" {
for_each = toset(var.storage-foreach)
access_tier = "Hot"
account_kind = "StorageV2"
account_replication_type = "LRS"
account_tier = "Standard"
location = var.location
name = each.value
resource_group_name = azurerm_resource_group.tenant-testing-hamza.name
identity {
type = "SystemAssigned"
}
lifecycle {
prevent_destroy = false
}
}
Key vault access policies:
resource "azurerm_key_vault_access_policy" "storage" {
for_each = var.storage-foreach
key_vault_id = azurerm_key_vault.tenantsnbshared.id
tenant_id = "<tenant-id>"
object_id = azurerm_storage_account.storage-foreach[each.value].identity.0.principal_id
key_permissions = ["get", "Create", "List", "Restore", "Recover", "Unwrapkey", "Wrapkey", "Purge", "Encrypt", "Decrypt", "Sign", "Verify", "Delete"]
secret_permissions = ["get", "set", "list", "delete", "recover"]
depends_on = [azurerm_key_vault.tenantsnbshared]
}
variable "storage-foreach" {
type = map(string)
default = { "<name1>" = "storage1", "<name2>" = "storage2", "<name3>" = "storage3", "<name4>" = "storage4"}
}
variable "key-name" {
type = map(string)
default = {"<name1>" = "key1", "<name2>" = "<key2>", "name3" = "<key3>", "<name4>" = "key4"}
}
this error get repeated for each element I have in my key-name variable.
I tried the some thing but using a count instead of a for_each and it works just fine, but the problem I had with that, was if I wanted to delete the first storage account and the first key, it automatically destroy all the element coming after to then recreate them, and is not something I wanted to do.
Is there anyone who can help me to understand this error and how to fix it please?
I'm assuming the lists with keys and storage accounts are of the same length, and that you want to encrypt storage account number i with key number i. You can either use "old style", index-based loops or zip the two lists into a single list of tuples, and then iterate over the zipped list.
Solution 1: using index-based iteration
This solution does not use for_each but the meta-argument count. This way of iterating over resources in Terraform predates the newer for_each style.
resource "azurerm_storage_account_customer_managed_key" "storage-managed-key" {
count = length(var.key-name)
key_name = azurerm_key_vault_key.client-key[var.key-name[count.index]].name
key_vault_id = azurerm_key_vault_key.client-key[var.key-name[count.index]].key_vault_id
key_version = azurerm_key_vault_key.client-key[var.key-name[count.index]].version
storage_account_id = azurerm_storage_account.storage-foreach[var.storage-foreach[count.index]].identity.0.principal_id
depends_on = [azurerm_key_vault_access_policy.storage]
}
I've taken the liberty to replace the explicit key name and key vault ID by references to the attributes of your key resource.
Solution 2: using combined structure
Here, the idea is to create a structure that you can iterate over, and of which the elements combine key name and the storage account names. There's multiple ways of doing this. The easiest way is probably to "misuse" a map and treat it as a list of tuples, as you can then simply use zipmap.
resource "azurerm_storage_account_customer_managed_key" "storage-managed-key" {
for_each = zipmap(var.storage-foreach, var.key-name)
key_name = azurerm_key_vault_key.client-key[each.value].name
key_vault_id = azurerm_key_vault_key.client-key[each.value].key_vault_id
key_version = azurerm_key_vault_key.client-key[each.value].version
storage_account_id = azurerm_storage_account.storage-foreach[each.key].identity.0.principal_id
depends_on = [azurerm_key_vault_access_policy.storage]
}
Note that I, perhaps confusingly, chose var.storage-foreach to be the keys of the object. Picking the keys as the map keys would make it impossible to use the same key to encrypt multiple storage accounts. Since storage-foreach is already used to index Terraform resources, I also already know these adhere to the Terraform naming restrictions.
What's the difference?
In solution 1, your azurerm_storage_account_customer_managed_key resource names are integer-based. Re-ordering elements in the lists may cause Terraform to destroyed and re-create resources. This is not the case for solution 2, which is why I usually prefer solution 2. However, in this case solution 1 may have the advantage of being more straight-forward.
A suggestion...
If possible, I would suggest to re-evaluate how you define your variables. It likely makes more sense to ask for list of objects combining key name and storage accounts in the first place; then you can basically use solution 2 without the call to zipmap. It's almost never a good idea to have an implicit dependency between two variables like this - these lists have to have the same length, and implicitly, the contents of the lists are connected by virtue of having the same index.
I am trying to create some simple logic when calling applicationg gateway module.
When creating WAF v2 application gateway I want to specify more attributes that simple application gateway can't handle and they won't be described.
resource "azurerm_application_gateway" {
name = var.appgatewayname
resource_group_name = data.azurerm_resource_group.rg.name
location = data.azurerm_resource_group.rg.location
......................
waf_configuration {
enabled = "${length(var.waf_configuration) > 0 ? lookup(var.waf_configuration, "enabled", "") : null }"
firewall_mode = "${length(var.waf_configuration) > 0 ? lookup(var.waf_configuration, "firewall_mode", "") : null }"
............
Calling module:
module "GWdemo" {
source = "./...."
sku-name = "WAF_v2"
sku-tier = "WAF_v2"
sku-capacity = 1
waf-configuration = [
{
enabled = true
firewall_mode = "Detection"
}
Am I thinking right that if waf-configuration map is specified it should specify following settings is applied and if not than null?
When working with Terraform we often want to reframe problems involving a conditional test into problems involving a collection that may or may not contain elements, because the Terraform language features are oriented around transforming collections into configuration on an element-by-element basis.
In your case, you have a variable that is already a list, so it could work to just ensure that its default value is an empty list rather than null, and thus that you can just generate one waf_configuration block per element:
variable "waf_configuration" {
type = list(object({
enabled = bool
firewall_mode = string
}))
default = []
}
Then you can use a dynamic block to generate one waf_configuration block per element of that list:
dynamic "waf_configuration" {
for_each = var.waf_configuration
content {
enabled = waf_configuration.value.enabled
firewall_mode = waf_configuration.value.firewall_mode
}
}
Although it doesn't seem to apply to this particular example, another common pattern is a variable that can be set to enable something or left unset to disable it. For example, if your module was designed to take only a single optional WAF configuration, you might define the variable like this:
variable "waf_configuration" {
type = object({
enabled = bool
firewall_mode = string
})
default = null
}
As noted above, the best way to work with something like that in Terraform is to recast it as a list that might be empty. Because it's a common situation, there is a shorthand for it via splat expressions:
dynamic "waf_configuration" {
for_each = var.waf_configuration[*]
content {
enabled = waf_configuration.value.enabled
firewall_mode = waf_configuration.value.firewall_mode
}
}
When we apply the [*] operator to a value of a non-list/non-set type, Terraform will test to see if the value is null. If it is null then the result will be an empty list, while if it is not null then the result will be a single-element list containing that one value.
After converting to a list we can then use it in the for_each argument to dynamic in the usual way, accessing the attributes from that possible single element inside the content block. We don't need to repeat the conditionals for each argument because the content block contents are evaluated only when the list is non-empty.
I would encourage you to upgrade to Terraform v0.12.x and this should get a much easier to do. I would leverage the new dynamic block syntax to make the block optional based on whatever condition you need to use.
Here is a rough example, but should get you going in the correct direction.
dynamic "waf-configuration " {
for_each = length(var.waf_configuration) > 0 ? [] : [1]
content {
enabled = "${length(var.waf_configuration) > 0 ? lookup(var.waf_configuration, "enabled", "") : null }"
firewall_mode = "${length(var.waf_configuration) > 0 ? lookup(var.waf_configuration, "firewall_mode", "") : null }"
}
}
I am building a kafka streams topology in which I need to materialize the contents of a Global KTable in a persistent RocksDB store. This store will be further used on every instance of my application for querying.
The store consinsts of key values pairs, where the key is serialized using Avro and the payload is just a byte array.
My problem is when I try to query the store by a specific key, as shown in the code below:
fun topology() {
val streamsBuilder = StreamsBuilder()
streamsBuilder.globalTable<Key, ByteArray>(
SOURCE_TOPIC_FOR_GLOBAL_KTABLE,
Consumed.with(null /*Uses default serde provided in config which is SpecificAvroSerde*/,
Serdes.ByteArray()),
Materialized.`as`(STORE_NAME))
val kafkaStreams = KafkaStreams(streamsBuilder.build(), getProperties(), DefaultKafkaClientSupplier())
kafkaStreams.start()
}
private fun getProperties(): Properties {
val properties = Properties()
properties[StreamsConfig.BOOTSTRAP_SERVERS_CONFIG] = "kafka-brokers-ip"
properties[AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG] = "schema-registry-ip"
properties[StreamsConfig.APPLICATION_ID_CONFIG] = "app-id"
properties[ConsumerConfig.CLIENT_ID_CONFIG] = "client-id"
properties[ConsumerConfig.AUTO_OFFSET_RESET_CONFIG] = "earliest"
properties[StreamsConfig.STATE_DIR_CONFIG] = "C:/dev/kafka-streams"
properties[StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG] = SpecificAvroSerde::class.java
properties[StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG] = SpecificAvroSerde::class.java
return properties
}
And this is the function that I use to query the store:
val store = kafkaStreams
.store(STORE_NAME, QueryableStoreTypes.keyValueStore<Key, ByteArray>())
store.all().forEach { keyValue ->
if (keyValue.key == Key("25-OCT-19 USD")) {
//we get here, so the key is in the store
println("Keys are equal.")
println(keyValue) // this gets printed: KeyValue({"value": "25-OCT-19 USD"}, [B#3ca343b5)
println(store.get(Key("25-OCT-19 USD"))) //this is null
println(store.get(keyValue.key)) //this is also null
}
}
As you can see in the comments of code, there are values in the store, but when I try to query by a specific key, I get nothing back.
I would like to retrieve separately the key and value from Vault using Terraform.
For the following try, I am receiving that the value doesn't exists.
data "vault_generic_secret" "kv" {
path = "kv/test"
}
output "kv" {
value = "${data.vault_generic_secret.kv.data["Value"]}"
}
For an output as follows, I am getting the Key + Value in the following format: "key/value".
output "kv" {
value = "${data.vault_generic_secret.kv.data}"
}
It's any way to retrieve just the key and just the value separately?
Thanks
I want to automate deployments of Vmware VM's in an landscape with lots of portgroups. To be able to select the correct portgroup it would be best to enter 2 variables tenant and environment. These 2 variables are used for CMDB registration and deployment purposes.
For the deployment the variables need to be combined in to 1 new variable to pick the correct portgroup. Due to interpolation syntax it seems to be impossible to use 2 combined variables in the lookup.
How can I combine 2 variables to 1 in Terraform?
I also tried to make a local file with the correct string, but that file needs to exist before the script starts, terraform plan gives a error message that the file doesn't exist.
variable "tenant" {
description = "tenant: T1 or T2"
}
variable "environment" {
description = "environment: PROD or TEST"
}
variable "vm_network" {
description = "network the VM will be provisioned with"
type = "map"
default = {
T1_PROD = "T1-PROD-network"
T2_PROD = "T2-PROD-network"
T1_TEST = "T1-TEST-network"
T2_TEST = "T2-TEST-network"
}
}
data "vsphere_network" "network" {
name = "${lookup(var.vm_network, tenant_environment)}"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
Off the top of my head I can think of three different ways to merge the variables to use as a lookup key:
variable "tenant" {}
variable "environment" {}
variable "vm_network" {
default = {
T1_PROD = "T1-PROD-network"
T2_PROD = "T2-PROD-network"
T1_TEST = "T1-TEST-network"
T2_TEST = "T2-TEST-network"
}
}
locals {
tenant_environment = "${var.tenant}_${var.environment}"
}
output "local_network" {
value = "${lookup(var.vm_network, local.tenant_environment)}"
}
output "format_network" {
value = "${lookup(var.vm_network, format("%s_%s", var.tenant, var.environment))}"
}
output "lookup_network" {
value = "${lookup(var.vm_network, "${var.tenant}_${var.environment}")}"
}
The first option uses locals to create a variable that is interpolated already and can be easily reused in multiple places which can't be done directly with variables in Terraform/HCL. This is generally the best way to do variable combination/interpolation in later versions of Terraform (they were introduced in Terraform 0.10.3).
The second option uses the format function to create a string containing the tenant and environment variables.
The last one is a little funny looking but is valid HCL. I'd probably shy away from using that syntax if possible.