I have the need to create and manage multiple customer environments in AWS and I'm wanting to leverage Terraform to deploy all of the necessary resources. Each customer environment is basically the same with the exception of the URL they use to access one of the servers.
I have put together a Terraform configuration that deploys all of the resources for a given customer. BUT... How do I take that same configuration and apply it to the next customer without copying the entire Terraform directory and duplicating that for every customer. (I could have 100's of these)
I've heard workspaces and modules or both. Anyone seen a best-practice article out there on this?
Thx
You should modulerize your code, then you can easily reuse that module(from a git repository) with different variables to be used for that customer. In this case for each customer, you will end up with only a file that configures the main module.
Have one directory for each customer, with a terraform file that loads up the module(s) and configures it. If you use terraform apply in that directory then the state will also be in that directory. To make sure your team can also deploy and make changes it is suggested to use a backend such as S3, so the state will be written there. Note that you have to configure a backend for each customer in their respective directory. Make sure the backend for each customer don't clash(For example use a different path in S3).
Nicki Watt gave a good presentation on this. You can view the video here and slides at here.
Related
I ran into this situation during my training the other day.
Say im working with another guy on a terraform infrastructure, and we have shared state (ofcourse).
He creates a resource and updates the state but doesnt deploy that said resource.
After which I code an important resource and want to deploy it, but i dont want the changes the other guy made to apply along with my resource.
What would be the ideal solution for this?
Seperate workspace?
Taint his resource?
Remove his resource from the state file?
This is not possible:
"He creates a resource and updates the state but doesn't deploy that
said resource."
The state file tracks what is deployed. How would the state be updated with that resource, if the resource wasn't deployed?
What would be the ideal solution for this?
Seperate workspace?
If you are both working on resources that belong in the same environment, then that's probably not the right solution.
Taint his resource?
Tainting a resource causes an already deployed resource to be deleted and redeployed. How would that help the situation you describe?
Remove his resource from the state file?
You should never modify the state file directly.
Are you confusing the Terraform template files (.tf files) with the Terraform state file?
I'm guessing you have some terminology wrong, and what you mean to say is the resource is defined in the Terraform template, and now you need to deploy something new without including that other resource. In that case you would need to use the -target argument to deploy only certain resources.
I think in this case because there is one state file and more than one developer contributing, the use of workspaces would be best. That way both will get working sessions of the resources being deployed. Note, if you decide to go the workspace route, doing a migration of the default workspace would be needed into the new workspace. This guide can help with that.
Setup:
Private master repo and every developer has their own private fork.
Currently using CircleCI, but we'd be happy to switch to satisfy requirements
Branches on master repo are protected with merge restrictions
Requirements:
Build + test on forked pull requests
Deploy to different environments based on master repo branch updates
Not all developers can be fully trusted with production credentials
Partial Solution:
Enable building and passing secrets on forked pull requests (Reference)
Use CircleCI contexts to set environment variables per branch. This allows different deploy targets.
Problems:
All repo specific secrets as well as all global contexts are now accessible by anyone who can open a PR.
Even if we disable building on forked pull requests, anyone with write access to at least one repo can access all global contexts.
Question:
This would seems to be a very common use case. How do other companies solve it?
Is CircleCI not the right tool for this? - No, it is not (see below).
Should we build a custom solution?
Edit1:
CircleCI got back to me and surprisingly this is not a use case they support. Looking into other providers now. Above questions are still unanswered.
Edit2:
I've also contacted TravisCi and SemaphoreCi and it appears that only TravisCi supports building forked PRs and not leaking secrets into them (Reference).
SempahoreCi is missing (1) building forked PRs and (2) hiding secrets from the deployment phase in non-master workflows
CircleCi has restricted contexts, but they would require manually changing workflows. Definitely not easy to set up and I don't fully understand how they would work.
I'm familiar with Terraform and its terraform.tfstate file where it keeps track of which local resource identifiers map to which remote resources. I've noticed that there is a .serverless directory on my machine which seems to contain files such as CloudFormation templates and ZIP files containing Lambda code.
Suppose I create and deploy a project from my laptop, and Serverless spins up fooxyz.cloudfront.net which points to a Lambda function arn:aws:lambda:us-east-1:123456789012:function:handleRequest456. If I naively try to run Serverless again from another machine (or if I git clean my working directory), it'll spin up a new CloudFront endpoint since it doesn't know that fooxyz.cloudfront.net already represents the same application. I'm looking to back up the state it keeps internally, so that it modifies an existing resource rather than creates a new one. (The equivalent in Terraform would be to back up the terraform.tfstate file.)
If I wished to back up or restore a Serverless deployment state, which files would I back up? In the case of AWS, it seems like I should be backing up the CloudFormation templates; I don't want to back up the Lambda code since it's directly generated from the source. However, I'm likely going to use more than just AWS in the future, and so don't want to "special-case" the CloudFormation templates if at all possible.
How can I back up only the files I cannot regenerate?
I think what you are asking is If I or a colleague checks out the serverless code from git on a different machine, will we still be able to deploy and update the same lambda functions and the same API gateway endpoints?
And the answer to that is yes! Serverless keeps track of all of that for you within their files. Unless you run serverless destroy - no operation will create a new lambda or api endpoint.
My team and I are using this method: we commit all code to a git repo and one of us checks it out and deploys a function or the entire thing and it updates the existing set of functions properly. If you setup an environment file - that's all you need to worry about really. And I recommend leaving it outside of git entirely.
For AWS; Serverless Framework keeps track of your deployment via Cloudformation (CF) parameters/identifiers which are specific to an account/region. The CF stack templates are uploaded to an (auto-generated) S3 bucket so it's already backed up for you.
So all you really need to have is the original deployment code in a git repo and have access to your keys. Everything else is already backed up for you.
Im currently working on a big project with more then one team.
Lets say in the project there are some modules that each team working on.
In addition we are using gerrit for sometime now and there is something i couldn't find out.
My question is the following:
Is there a way to tell Gerrit that only specific people/group(on Gerrit) will have permission to review code (+2) on specific path/module on the project?
This is possible, and can be achieved by using the Gerrit OWNERS Plugin. I haven't configured this plugin myself, but we use this in our codebase to protect certain areas of code.
Every folder that needs protection contains a file named OWNERS that has the following structure.
inherited: true
owners:
- user-a#example.com
- user-b#example.com
Here is the link to a readme for the plugin. Hope you can figure out how to configure it.
https://gerrit.googlesource.com/plugins/owners/+/refs/heads/master/README.md
I think you can do this by making two separate commits. You can later add the group that you want to review the code on that specific path using gerrit interface.
I'm writing my magento extension and came up with a question. The main extension files/directory structure is quite clear. We have dirs for extension configuration files, models, helpers, database resources, frontend and backend scripts and stylesheets etc.
But what if my extension uses some files that aren't classes or resources to be included to frontend or backend?
For instance: image files that will only be attached to emails and will never be retrieved by a browser directly.
Should/could I just create a directory /app/code/community/MyNamespace/MyExtensionName/images?
The same dir tree for better readability:
app
code
community
MyNamespace
MyExtensionName
images
Or is there any other correct/recommended way to achieve that?
There's never been clear guidance on how to do this from Magento Inc. itself, and Magento's module structure doesn't offer clear guidance. The approach I've always taken is
Pretend I'm on the Magento core team
Pretend my fellow team members are sociopaths who don't care if anything I've done breaks
If you're adding frontend files for public consumption (to js, skin, etc), I always create a folder that's a lowercase version of my full module name, and drop all files in there
/js/namespace_modulename/file.js
In the case of files that aren't going to be served publicly (i.e. you only need access to them via PHP), creating a folder in the root of your module (as you've done above) is appropriate. I'd suggest something like
app
code
community
MyNamespace
MyExtensionName
assets
images
You never know when there'll be something else you want to add, and having everything under one folder will help keep the module structure clean.
There's even sort of a precedent for this in Magento's core code. Take a look at the
app/code/core/Mage/Sales/doc
folder.
Create a folder into media directory and place your files/images into that folder
media
MyExtensionName
images
And access them like
echo Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_MEDIA).'/MyExtensionName/images/pic.jpg';