I have a problem with terraform configuration which I really don't know how to resolve. I have written module for policy assigments, module as parameters taking object with 5 attributes. The question is if it is possible to split into folder structure tfvars file. I mean eg.
I have main folder subscriptions -> folder_subscription_name -> some number of files tfvars with configuration for each of the policy assigment
Example of each of the file
testmap = {
var1 = "test1"
var2 = "test2"
var3 = "test3"
var4 = "test4"
var5 = "test5"
}
In the module I would like to iterate over all of the maps combine into list of maps. Is it good approach? How to achive that or maybe I should use something other to do it like terragrunt ?
Please give me some tips what is the best way to achive that, basically the goal is that I don't want to have one insanely big tvars file with list of 100 maps but splitted into 100 configuration file for each of the assignment.
Based on the comment on the original question:
The question is preety simple how to keep input variables for each of
resource in seperate file instead of keeping all of them in one very
big file
I would say you aim at having different .tfvars files. For example, you could have a dev.tfvars and a prod.tfvars file.
To plan your deployment, you can pass those file with:
terraform plan --var-file=dev.tfvars -out planoutput
To apply those changes
terraform apply planoutput
Related
I have got requirement saying, blob storage has multiple files with names file_1.csv,file_2.csv,file_3.csv,file_4.csv,file_5.csv,file_6.csv,file_7.csv. From these i have to read only filenames from 5 to 7.
how we can achieve this in ADF/Synapse pipeline.
I have repro’d in my lab, please see the below repro steps.
ADF:
Using the Get Metadata activity, get a list of all files.
(Parameterize the source file name in the source dataset to pass ‘*’ in the dataset parameters to get all files.)
Get Metadata output:
Pass the Get Metadata output child items to ForEach activity.
#activity('Get Metadata1').output.childItems
Add If Condition activity inside ForEach and add the true case expression to copy only required files to sink.
#and(greater(int(substring(item().name,4,1)),4),lessOrEquals(int(substring(item().name,4,1)),7))
When the If Condition is True, add copy data activity to copy the current item (file) to sink.
Source:
Sink:
Output:
I took a slightly different approaching using a Filter activity and the endsWith function:
The filter expression is:
#or(or(endsWith(item().name, '_5.csv'),endsWith(item().name, '_6.csv')),endsWith(item().name, '_7.csv'))
Slightly different approaches, similar results, it depends what you need.
You can always do what #NiharikaMoola-MT suggested . But since you already know the range of the files ( 5-7) , I suggest
Declare two paramter as an upper and lower range
Create a Foreach loop and pass the parameter and to create a range[lowerlimit,upperlimit]
Create a paramterized dataset for source .
Use the fileNumber from the FE loop to create a dynamic expression like
#concat('file',item(),'.csv')
So here is what I want as a module in Pseudo Code:
IF UseCustom, Create AWS Launch Config With One Custom EBS Device and One Generic EBS Device
ELSE Create AWS Launch Config With One Generic EBS Device
I am aware that I can use the 'count' function within a resource to decide whether it is created or not... So I currently have:
resource aws_launch_configuration "basic_launch_config" {
count = var.boolean ? 0 : 1
blah
}
resource aws_launch_configuration "custom_launch_config" {
count = var.boolean ? 1 : 0
blah
blah
}
Which is great, now it creates the right Launch configuration based on my 'boolean' variable... But in order to then create the AutoScalingGroup using that Launch Configuration, I need the Launch Configuration Name. I know what you're thinking, just output it and grab it, you moron! Well of course I'm outputting it:
output "name" {
description = "The Name of the Default Launch Configuration"
value = aws_launch_configuration.basic_launch_config.*.name
}
output "name" {
description = "The Name of the Custom Launch Configuration"
value = aws_launch_configuration.custom_launch_config.*.name
}
But how the heck do I know from the higher area that I'm calling the module that creates the Launch Configuration and Then the Auto Scaling Group which output to use for passing into the ASG???
Is there a different way to grab the value I want that I'm overlooking? I'm new to Terraform and the whole no real conditional thing is really throwing me for a loop.
Terraform: How to conditionally assign an EBS volume to an ECS Cluster
This seemed to be the cleanest way I could find, using a ternary operator:
output "name {
description = "The Name of the Launch Configuration"
value = "${(var.booleanVar) == 0 ? aws_launch_configuration.default_launch_config.*.name : aws_launch_configuration.custom_launch_config.*.name}
}
Let me know if there is a better way!
You can use the same variable you used to decide which resource to enable to select the appropriate result:
output "name" {
value = var.boolean ? aws_launch_configuration.custom_launch_config[0].name : aws_launch_configuration.basic_launch_config[0].name
}
Another option, which is a little more terse but arguably also a little less clear to a future reader, is to exploit the fact that you will always have one list of zero elements and one list with one elements, like this:
output "name" {
value = concat(
aws_launch_configuration.basic_launch_config[*].name,
aws_launch_configuration.custom_launch_config[*].name,
)[0]
}
Concatenating these two lists will always produce a single-item list due to how the count expressions are written, and so we can use [0] to take that single item and return it.
I want to know where does SimObject names like mem_ctrls, membus, replacement_policy are set in gem5. After looking at the code, I understood that, these name are used in stats.txt.
I have looked into SimObject code files(py,cc,hh files). I printed all Simobject names by stepping through root descendants in Simulation.py and then searched some of the names like mem_ctrls using vscode, but could not find a place where these names are set.
for obj in root.descendants():
print("object name:%s\n"% obj.get_name())
These names are the Python variable names from the configuration/run script.
For instance, from the Learning gem5 simple.py script...
from m5.objects import *
# create the system we are going to simulate
system = System()
# Set the clock fequency of the system (and all of its children)
system.clk_domain = SrcClockDomain()
system.clk_domain.clock = '1GHz'
system.clk_domain.voltage_domain = VoltageDomain()
# Set up the system
system.mem_mode = 'timing' # Use timing accesses
system.mem_ranges = [AddrRange('512MB')] # Create an address range
The names will be system, clk_domain, mem_ranges.
Note that only the SimObjects will have a name. The other parameters (e.g., integers, etc.) will not have a name.
You can see where this is set here: https://gem5.googlesource.com/public/gem5/+/master/src/python/m5/SimObject.py#1352
Is there a good way to iterate over model/dataset pairs in AMPL in a .run file?
Say you have two different models for the same optimization problems, and four datasets. What I would do up to now was to make a .run file for each model/dataset pair, and run each individually, or keep one script for each model and manually solve for each dataset by modifying the data (file); command. But this is obviously tedious and inviable for larger projects.
So is there any good way to do this? What I've tried is something like the following (just a skeleton for clarity):
# Some global options
for {model_ in {'1.mod', '2.mod'}} {
reset;
model (model_);
for {dat in {'1.dat', '2.dat', '3.dat', '4.dat'}} {
update data;
data (dat);
solve;
# Do some post-processing
}
}
But AMPL whines about using the for loop's dummy variable in the model and data command. I've tried declaring symbolic parameters to store the name of the model and data file, but no good either.
Of course this would only be sensible if the models were similar enough, insofar as they could be post-processed in the same way. But there should at least be a way to iterate through the data files within one model without having to go like
update data;
data 1.dat;
solve;
update data;
data 2.dat;
solve;
update data;
data 3.dat;
solve;
[...]
update data;
data 427598.dat;
solve;
right?
Your example code has a reset inside a loop. This will reset the loop parameter, which is going to cause problems. For instance, if you run this:
for {modelname in {"itertest1.mod","itertest2.mod"}}{
display (modelname);
for {dataname in {"itertest_a.dat","itertest_b.dat"}}{
display (dataname);
}
}
it will print out the file names as we might hope:
modelname = itertest1.mod
dataname = itertest_a.dat
dataname = itertest_b.dat
modelname = itertest2.mod
dataname = itertest_a.dat
dataname = itertest_b.dat
But if we add a reset statement:
for {modelname in {"itertest1.mod","itertest2.mod"}}{
reset; ### this is the only line I changed ###
display (modelname);
for {dataname in {"itertest_a.dat","itertest_b.dat"}}{
display (dataname);
}
}
then we get an error: modelname is not defined (because we just reset it).
However, even without that issue, we'll still get a complaint about using the loop variable.
Solution 1: commands
Depending on exactly what you ran, you may have seen an error message advising use of the commands statement, like so:
reset;
for {scriptname in {"script1.run", "script2.run"}}{
commands (scriptname);
}
This will then run whatever commands are in the listed .run files, and you should be able to nest that to call separate scripts to define the models and the data. Since you can't use a blanket reset without killing your loop parameter, you will need to use other options for updating the model files. AMPL offers the option to define multiple "problems" and switch between them, which may or may not be helpful here; I haven't explored it.
The commands statement is documented here, although TBH I find the documentation a bit difficult to follow.
Solution 2: code generation
If all else fails, you can write an iterated AMPL script that generates (and optionally runs) a second script containing all the model/data combinations you want to run, like so:
param outfile symbolic := "runme_temp.run";
for{i in 1..3}{
for{j in 1..4}{
printf "\nreset;" > (outfile);
printf "\nmodel model%s;",i > (outfile);
printf "\ndata data%s;",j > (outfile);
printf "\nsolve;" > (outfile);
# add post-processing here as desired
}
}
close (outfile);
include runme_temp.run;
This will create and then call "runme_temp.run" which looks like this:
reset;
model model1;
data data1;
solve;
reset;
model model1;
data data2;
solve;
etc. etc. Since this does allow you to use a blanket reset; it might simplify the process of cleaning up previous models. Not the most dignified solution, but it works and can be adapted to a wide range of things.
This could be improved by de-nesting the loops:
param outfile symbolic := "runme_temp.run";
for{i in 1..3, j in 1..4}{
printf "\nreset;" > (outfile);
printf "\nmodel model%s;",i > (outfile);
printf "\ndata data%s;",j > (outfile);
printf "\nsolve;" > (outfile);
# add post-processing here as desired
}
close (outfile);
include runme_temp.run;
In this particular example it doesn't make much difference, but multiply-nested loops can run slow in AMPL; using a single multi-index for can make a big difference to performance, so it might be better to get into that habit.
Apologies if this is a straightforward question, I couldn't find anything in the docs.
currently my workflow looks something like this. I'm taking a number of input files created as part of this workflow, and summarizing them.
Is there a way to avoid this manual regex step to parse the wildcards in the filenames?
I thought about an "expand" of cross_ids and config["chromosomes"], but unsure to guarantee conistent order.
rule report:
output:
table="output/mendel_errors.txt"
input:
files=expand("output/{chrom}/{cross}.in", chrom=config["chromosomes"], cross=cross_ids)
params:
req="h_vmem=4G",
run:
df = pd.DataFrame(index=range(len(input.files), columns=["stat", "chrom", "cross"])
for i, fn in enumerate(input.files):
# open fn / make calculations etc // stat =
# manual regex of filename to get chrom cross // chrom, cross =
df.loc[i] = stat, chrom, choss
This seems a bit awkward when this information must be in the environment somewhere.
(via Johannes Köster on the google group)
To answer your question:
Expand uses functools.product from the standard library. Hence, you could write
from functools import product
product(config["chromosomes"], cross_ids)