I am a QA automation engineer and I am investigating docker as a potential way to run our tests.
Traditionally we have followed the git flow method where essentially where you have a dev and a master branch. Dev are constantly merging their new changes to the dev branch. When we wish to release, we will have a code cut off, where everything currently on the dev branch is deemed to be part of the next release. Script is then run to create the release candidate and this is deployed to staging. Any fixes that need to be done are made to the release branch and once ready to go to prod, new code is merged to master and deployed. Master is back merged to all branches so that everything is up to date. (described in more detail here: http://nvie.com/posts/a-successful-git-branching-model/).
So my question is with docker do you need to have this workflow? Im thinking of maybe having a workflow like describe below:
Dev start working on a new feature.
Dev pulls master, creates feature branch - does his dev work - unit tests pass, dev is happy for work to go to QA
Dev runs script to create release candidate (which would involve pulling master again in case new code has been merged to master by another dev),
Docker then spins up a container with multiple containers inside that (front end app, DB instance etc)
Tests (unit, api, selenium integration etc) are then run against this release candidate and if good deploy to production.
So do I need a staging env in the traditional sense where it is constantly available?
I think you're conflating two things: a continuous integration environment and a staging environment. Docker does make it easy to bring up a fresh instance of your entire stack for continuous integration (see drone for a good example), but generally you still need a staging environment that is always available to test against before deploying to prod. This staging environment should be running the same docker images that eventually get deployed to prod.
Related
I'm writing an API that consists of several microservices. I have the code in a private Gitlab repo. I have a custom CI/CD pipeline configured to run a couple of different steps automatically on every commit to master (e.g. build, test, deploy to a dev environment). Deploying to prod is manual.
I have written some unit tests around this code, which naturally test only small units of the code. These, of course, are run with every commit, because if they fail, that means something in the code has broken.
I also have regression tests which we run after deploying. One of these is actually a bash script that uses curl to hit my production endpoint with certain parameters and checks to make sure that I'm getting 200 responses. I have parameterized this script so I can easily point it at my dev environment (instead of prod).
I use this regression test (and others like it) to check that my already-deployed service is functioning properly. And I run it right after deploying as a final, double-check to confirm that everything is working. But I want to automate that.
My question is where does this fit in a CI/CD workflow? It wouldn't make sense to run this kind of regression test on a commit, because that commit is not necessarily coupled with a deploy. And because there are any number of reasons why the service might be down that are unrelated to whatever code changes went into the most recent commit. In other words, the pipeline should not fail because of external circumstances.
Are there any best practices for running and automating regressions tests?
Great question. There are a couple of interesting points here.
When to run the regression tests (as they exist today) in your CI / CD environment.
The obvious answer to this is to run as a post deploy step. Using the same approach you are currently using to limit the deploy step to the master branch only you can limit this post deploy step to the master branch only.
If you add more details about your environment. For example the CI / CD system that you are using and your current configuration I would be very happy to provide more concrete details on how to achieve this.
It wouldn't make sense to run this kind of regression test on a commit
An interesting approach that I have seen a couple of times. Is using a cloud service (AWS / GCloud etc.) to spin up an environment on each CI run. This means that the full pipeline can be run for every commit. While it takes more resources, it means that you can find issues prior to merging to master. Of course up to you whether the ROI adds up in your environment.
I have a following use case:
We have one solution that contains 5-10 different services (.NET Framework Web Apps of various versions) within. We have to setup CI/CD in Azure DevOps to be able to automate the deployments of each services separately (or all services at once). There will be around 5 different environments for each service.
Challenges:
We are trying to avoid having (# of services X # of environments) seperate builds and releases (~50 build/ ~50 release).
We do have to be able to deploy one service alone without others being affected.
We do have to be able to deploy ALL services all at once for mass deployments.
P.S. We are currently using trunk based development but I am thinking about moving to giflow to have branch based triggers as I feel it would be easier to manage in this case.
CI - handled by your build server (e.g. teamcity). Responsibility: Build, Test, Obfuscate, Create Packages and lastly push Packages to nuget server (.net specific). Traditionally besides the app code you also need at least 2 other packages: db migrations, infra migrations.
You build packages once and deploy the exact version everywhere else you want it to go.
https://gist.github.com/leblancmeneses/1d352bb79447cd7a486598c4dc796ef1
This script works in conjunction with https://github.com/leblancmeneses/RobustHaven.DevOps
CD - handled by something like octopus deploy. Responsibly: orchestrate deployment process across your cluster. Octopus pulls packages from nuget server and moves them to what ever environment you want and to whatever machines encompasses that environment.
https://www.robusthaven.com/presentations/DevOps
you dont really need 50 builds, you can use a single build per service (assuming builds for different environments are identical) and build from different branches. technically you can get away with a single release for 50 environments if you create your triggers\phases properly, but that would be a mess, just create a single one for each environment. I cant see how managing 50 environments on a single release is manageable.
when yaml release pipelines arrive, this becomes trivial, right now its not, unfortunatelly.
As part of my job I evaluate many software and applications.
I need to have an environment that is easy to clean (so the previous apps are not bloating my system) and always light.
One idea is to create isolated environments (either by Docker or Virtual machines) and fire up a new environment every time I need to start over with new software to evaluate.
Questions:
1.Does Docker support this? Can I use it to create new environment every few days and test software in it?
2. If not, which VM system would be suitable for this particular need?
Thanks
This is exactly what all the Continuous Integration systems do: get fresh code, build your project and run tests inside the freshly created container. This is how clean testing is done nowadays. So Docker fits perfectly your needs.
Each fresh container is a clean environment where you can run your tests in. Then you can parse the result and remove the container, for example docker run --rm -it my-image ./tests.sh
I have read what the Docker is but having hard time finding of what are the real scenarios of using Docker?
It would be great to see here your usages.
I'm replicating production environment with it, on commit on project with jenkins after building binaries i deploy there, launch the required daemons and run integration tests, all in a very short time (a few seconds over the time that takes the integration tests). Having no need to boot, and little overhead on memory/cpu/disk is great for that kind of things.
I could extend that use for development (just adding a volume where the code resides to my git repository, at least for scripting languages) to have the production environment with the code im actually editing, at a fraction of what virtualbox would require.
Also needed to test how to integrate some 3rd party code into a production system that modified DB. Cloned the DB in a container, installed the production system in another, launched both and iterated the integration until i did it well, going back to zero to try again in seconds, and faster, cheaper and more scriptable than doing it with VMs+snapshots.
Also run several desktop browser instances on containers, with their own plugins, cookies, data storage and so on separated. The docker repository example for desktop integration is a good start for it, but planning to test subuser to extend this kind of usage.
I've used Docker to implement a virtualized build server which any user could ask to run a build off their personal git branch in our canonical environment.
Each SSH connection made to the server was connected to a new container, ensuring that all builds were isolated from each other (a major pain point in the past), ensuring that the container's state couldn't be corrupted (since changes were all isolated to that single instance), and ensuring that even developers on platforms such as Windows where Docker (and other tools in our canonical build environment) couldn't be run locally would be able to run builds.
We use it for the following uses:
We have a Jenkins Container which we can use to bring up our Jenkins server. We mount the workspace using volumes so we can migrate the server easily just by copying the files and launching the container somewhere else.
We use a Jetty container to easily deploy our war files in our production and development environment.
We use a whole host of other monitoring tools such as Uptime which we have containers for so that we can bring them up and down on various hosts with a single command.
I use docker to build and test our software on several different Linux distributions (RHEL 4/5/6/7, Ubuntu 12.04, 14.04).
Docker makes it easy and fast to create minimalistic and consistent build environments.
Docker gives you the benefits that other virtualization solutions give you to a fraction of the recourse needed.
While I only have a github repository that I'm pushing to (alone), I often forget to run tests, or forget to commit all relevant files, or rely on objects residing on my local machine. These result in build breaks, but they are only detected by Travis-CI after the erroneous commit. I know TeamCity has a pre-commit testing facility (which relies on the IDE in use), but my question is with regards to the current use of continuous integration as opposed to any one implementation. My question is
Why aren't changes tested on a clean build machine - such as those which Travis-CI uses for post-commit tesing - before those changes are committed?
Such a process would mean that there would never be build breaks, meaning that a fresh environment could pull any commit from the repository and be sure of its success; as such, I don't understand why CI isn't implemented using post-commit testing.
I preface my answer with the details that I am running on GitHub and Jenkins.
Why should a developer have to run all tests locally before committing. Especially in the Git paradigm that is not a requirement. What if, for instance, it takes 15-30 minutes to run all of the tests. Do you really want your developers or you personally sitting around waiting for the tests to run locally before your commit and push your changes?
Our process usually goes like this:
Make changes in local branch.
Run any new tests that you have created.
Commit changes to local branch.
Push local changes remotely to GitHub and create pull request.
Have build process pick up changes and run unit tests.
If tests fail, then fix them in local branch and push them locally.
Get changes code reviewed in pull request.
After approval and all checks have passed, push to master.
Rerun all unit tests.
Push artifact to repository.
Push changes to an environment (ie DEV, QA) and run any integration/functional tests that rely on a full environment.
If you have a cloud then you can push your changes to a new node and only after all environment tests pass reroute the VIP to the new node(s)
Repeat 11 until you have pushed through all pre-prod environments.
If you are practicing continuous deployment then push your changes all the way to PROD if all testing, checks, etc pass.
My point is that it is not a good use of a developers time to run tests locally impeding their progress when you can off-load that work onto a Continuous Integration server and be notified of issues that you need to fix later. Also, some tests simply can't be run until you commit them and deploy the artifact to an environment. If an environment is broken because you don't have a cloud and maybe you only have one server, then fix it locally and push the changes quickly to stabilize the environment.
You can run tests locally if you have to, but this should not be the norm.
As to the multiple developer issue, open source projects have been dealing with that for a long time now. They use forks in GitHub to allow contributors the chance to suggest new fixes and functionality, but this is not really that different from a developer on the team creating a local branch, pushing it remotely, and getting team buy-in via code review before pushing. If someone pushes changes that break your changes then you try to fix them yourself first and then ask for their help. You should be following the principle of "merging early and often" as well as merging in updates from master to your branch periodically.
The assumption that if you write code and it compiles and tests are passed locally, no builds could be broken is wrong. It is only so, if you are the only developer working on that code.
But let's say I change the interface you are using, my code will compile and pass tests
as long as I don't get your updated code That uses my interface.
Your code will compile and pass tests as long as you don't get my update in the interface.
And when we both check in our code, the build machine explodes...
So CI is a process which basically say: put your changes in as soon as possible
and test them in the CI server (it should be of course compiled and tested locally first).
If all developers follow those rules,
the build will still break, but we will know about it sooner rather than later.
The CI server is not the same as the version control system. The CI server, too, checks the code out of the repository. And therefore the code has already been committed when it gets tested on the CI server.
More extensive tests may be run periodically, rather than at time of checking in, on whatever is the current version of the code at the time of testing. Think of multi-platform tests or load tests.
Generally, of course, you'll unit test your code on your development machine before checking it in.