My company has recently started using tSQLt to test our codebase. We've built a few good test suites and now we're trying to figure out the best way to commit them.
We're using an SQL server with Redgate to commit our live code to a github.com repository.
One option we thought of would be to commit the tSQLt scripts alongside our live code in the same repository, but we feel this isn't the best choice. It would mean our test code would/could be uploaded to the live servers.
Another option would be to commit the tSQLt scripts to a second repository. This solution keeps the code separate but has more overhead. When our developers want to run test cases they would have to first pull the live code onto their dev databases, then pull the tSQLt code onto their databases. Also, when developer create new live code and make the corresponding tSQLt tests, they have remember to push the live code and test code to the appropriate repository. Seems like a lot of extra work.
Has anyone run into this issue? How did you resolve it? Are there best practices when committing test code?
Thanks!!
The best practice is to keep tests and code in the same repository. Have the CI pipeline create an artifact for the code (without the tests) and a separate artifact for the tests. Then deploy both to the CI environment together with tSQLt itself and run the tests.
If the tests are passing you can send the code-artifact on to be used in down steam environments.
Related
I'm writing an API that consists of several microservices. I have the code in a private Gitlab repo. I have a custom CI/CD pipeline configured to run a couple of different steps automatically on every commit to master (e.g. build, test, deploy to a dev environment). Deploying to prod is manual.
I have written some unit tests around this code, which naturally test only small units of the code. These, of course, are run with every commit, because if they fail, that means something in the code has broken.
I also have regression tests which we run after deploying. One of these is actually a bash script that uses curl to hit my production endpoint with certain parameters and checks to make sure that I'm getting 200 responses. I have parameterized this script so I can easily point it at my dev environment (instead of prod).
I use this regression test (and others like it) to check that my already-deployed service is functioning properly. And I run it right after deploying as a final, double-check to confirm that everything is working. But I want to automate that.
My question is where does this fit in a CI/CD workflow? It wouldn't make sense to run this kind of regression test on a commit, because that commit is not necessarily coupled with a deploy. And because there are any number of reasons why the service might be down that are unrelated to whatever code changes went into the most recent commit. In other words, the pipeline should not fail because of external circumstances.
Are there any best practices for running and automating regressions tests?
Great question. There are a couple of interesting points here.
When to run the regression tests (as they exist today) in your CI / CD environment.
The obvious answer to this is to run as a post deploy step. Using the same approach you are currently using to limit the deploy step to the master branch only you can limit this post deploy step to the master branch only.
If you add more details about your environment. For example the CI / CD system that you are using and your current configuration I would be very happy to provide more concrete details on how to achieve this.
It wouldn't make sense to run this kind of regression test on a commit
An interesting approach that I have seen a couple of times. Is using a cloud service (AWS / GCloud etc.) to spin up an environment on each CI run. This means that the full pipeline can be run for every commit. While it takes more resources, it means that you can find issues prior to merging to master. Of course up to you whether the ROI adds up in your environment.
Our developers use Bitbucket as the code repository.
Dev Repository is: AbcdProject
We, from QA team, write selenium automation scripts. What is the right approach -
Should the automated scripts go under tests folder under the same repo as the Dev. Like:
AbcdProject/
-src
-tests
--unit
--functional
---AbcdAutomationScripts
----src
----pom.xml
----testng.xml
or we should have our own repo and our scripts should go under that repo? Like:
Dev Repo:
AbcdProject/
-src
-tests
--unit
QA Repo:
AbcdAutomationScripts/
-src
--pom.xml
---testng.xml
I would prefer having a separate repo for QA but I would like to know the industry standard/best practice.
Considering, we go with a separate repo for QA:
Right now, when the developer pushes the code in bitbucket, his jenkinsfile triggers the build and deploys the code in dev-server. But the question is how do I set the dependency in Jenkins Pipeline such that when developer's trigger of the build has completed and the code is deployed in dev-server, my selenium scripts in another repo should get executed.
Standard is to have the tests in the same project. Consequences:
Developers see the expectations of QA. They know what's in focus and what isn't.
They see if stuff is in focus that should't be, or vice versa, so this can help improve the test suite quality.
The downside is that devs get the option to specifically program for passing the tests instead of for improving quality. However, if this is a thing, developers aiming for the wrong goals is a symptom of deeper problems, such as developers generally being incentivized towards the wrong goals.
Developers see what DOM access paths are being used in QA. This helps them understand what paths are expected to be stable. You get a chance to fix any miscommunications about access path stability before you run into a nightmare of "every small dev change requires adapting all Selenium scripts".
Liability: Dev and QA need to coordinate their directory structure. Usually not a big issue and if you have a useful SCM (such as git, even svn should work) this isn't really a big problem, but the conventions need to be in place and understood by everybody.
QA will notice if dev starts a new development branch.
Currently i have merged automation testing code base with development code base and merged both the pom's and able to run my automation cases effectively on CI/CD pipeline y adding the stage for run tests in jenkinsfile and do mvn clean install.
Having said that, i am still looking for some better solution where i don't need to merge both pom's and handle both dev and testing code loosely coupled.
While I only have a github repository that I'm pushing to (alone), I often forget to run tests, or forget to commit all relevant files, or rely on objects residing on my local machine. These result in build breaks, but they are only detected by Travis-CI after the erroneous commit. I know TeamCity has a pre-commit testing facility (which relies on the IDE in use), but my question is with regards to the current use of continuous integration as opposed to any one implementation. My question is
Why aren't changes tested on a clean build machine - such as those which Travis-CI uses for post-commit tesing - before those changes are committed?
Such a process would mean that there would never be build breaks, meaning that a fresh environment could pull any commit from the repository and be sure of its success; as such, I don't understand why CI isn't implemented using post-commit testing.
I preface my answer with the details that I am running on GitHub and Jenkins.
Why should a developer have to run all tests locally before committing. Especially in the Git paradigm that is not a requirement. What if, for instance, it takes 15-30 minutes to run all of the tests. Do you really want your developers or you personally sitting around waiting for the tests to run locally before your commit and push your changes?
Our process usually goes like this:
Make changes in local branch.
Run any new tests that you have created.
Commit changes to local branch.
Push local changes remotely to GitHub and create pull request.
Have build process pick up changes and run unit tests.
If tests fail, then fix them in local branch and push them locally.
Get changes code reviewed in pull request.
After approval and all checks have passed, push to master.
Rerun all unit tests.
Push artifact to repository.
Push changes to an environment (ie DEV, QA) and run any integration/functional tests that rely on a full environment.
If you have a cloud then you can push your changes to a new node and only after all environment tests pass reroute the VIP to the new node(s)
Repeat 11 until you have pushed through all pre-prod environments.
If you are practicing continuous deployment then push your changes all the way to PROD if all testing, checks, etc pass.
My point is that it is not a good use of a developers time to run tests locally impeding their progress when you can off-load that work onto a Continuous Integration server and be notified of issues that you need to fix later. Also, some tests simply can't be run until you commit them and deploy the artifact to an environment. If an environment is broken because you don't have a cloud and maybe you only have one server, then fix it locally and push the changes quickly to stabilize the environment.
You can run tests locally if you have to, but this should not be the norm.
As to the multiple developer issue, open source projects have been dealing with that for a long time now. They use forks in GitHub to allow contributors the chance to suggest new fixes and functionality, but this is not really that different from a developer on the team creating a local branch, pushing it remotely, and getting team buy-in via code review before pushing. If someone pushes changes that break your changes then you try to fix them yourself first and then ask for their help. You should be following the principle of "merging early and often" as well as merging in updates from master to your branch periodically.
The assumption that if you write code and it compiles and tests are passed locally, no builds could be broken is wrong. It is only so, if you are the only developer working on that code.
But let's say I change the interface you are using, my code will compile and pass tests
as long as I don't get your updated code That uses my interface.
Your code will compile and pass tests as long as you don't get my update in the interface.
And when we both check in our code, the build machine explodes...
So CI is a process which basically say: put your changes in as soon as possible
and test them in the CI server (it should be of course compiled and tested locally first).
If all developers follow those rules,
the build will still break, but we will know about it sooner rather than later.
The CI server is not the same as the version control system. The CI server, too, checks the code out of the repository. And therefore the code has already been committed when it gets tested on the CI server.
More extensive tests may be run periodically, rather than at time of checking in, on whatever is the current version of the code at the time of testing. Think of multi-platform tests or load tests.
Generally, of course, you'll unit test your code on your development machine before checking it in.
I've never worked on tremendously huge projects and the workflow we use at work is check-out/code/compile locally to test/commit. I was wondering how a build server would change this process. How do developer test their code when the application is too huge to compile locally? They just code, commit and pray?
Absolutely not.
The developer usually has a build file which can build the project for him or her, which has some "targets" defined which do the testing. If you have a really big project, you may have certain portions of it precompiled for you, so you don't have to build the whole thing in one big chunk. You usually do your testing locally before you commit to your repository. Breaking the build in big projects can mark you as an object of ridicule and scorn. Breaking the build in really important, really big projects can be career limiting... ;-)
The build SERVER itself doesn't change this. The build server only runs your build file and the targets you tell it to.
There are also build components (I've just started using TeamCity - no affiliation) that allow "personal builds".
I haven't used it yet as we haven't got it set up properly but my understanding is that TeamCity allows running a build (and tests if they are any run on the server) with your changes before committing (and optionally the server will commit your changes if the build is succesful). in TeamCity this is called a Pre-Tested Commit.
I'm looking at versioning databases and came across the usual articles regarding how to do this (coding horror, ode to code, etc). This all make perfect sense to me, however I'm trying to find a script runner that will run the sql scripts for me. All these articles mention having something to run them automatically, but none of them make any recommendations.
Does anybody know of any utilities for running these scripts? Ideally something that works in the following way:
Runs everything in a transaction so if any single update fails, the whole thing fails
I have control over the name of the scheme version database table
Ability to have a series of scripts that are always run if an upgrade takes place
Can be run as part of an automated task
EDIT
Open Source
We Use DbUp as Script Runner in our Web Project. Its simple and nice open source tools that help you to write you own script runner with Console Application fashion.
DbUp is a .NET library that helps you to deploy changes to SQL Server
databases. It tracks which SQL scripts have been run already, and runs
the change scripts that are needed to get your database up to date.
we can run scripts from folder in filesystem or you can embed them to your assembly and run them as embedded scripts.
you can find more information and sample on their code repository on github.
http://dbup.github.com
Check out SSW SQL Deploy - it would appear to do just about all you're asking for. It keeps track of already executed scripts, it'll run a whole batch of scripts at once and on multiple servers (if required), and so forth.
It's a pretty simple, but nifty tool - highly recommended!