I am looking for a command line tool that will take googletest executables as an argument and then only runs the fastest tests from that executable. Which those are should be defineable by a given time threshold.
At the first run this tool should run all tests and on further runs it could use the runtimes from the first runs to determine which tests are too slow to be run.
The Visual Studio GoogleTest adapter has such a feature, but I need it as a command line tool, so I can implement a "runFastTests" CMake target that can be used in a fast test-driven development loop.
Thank you for your time.
Related
I'm trying to improve the testing process where I work, but without adjusting the structure.
What we have: VSTS, Selenium IDE, Testers who write test cases, but not code.
What I'd like to do is manage a way to marry our TFS continuous integration with the Selenium tests we write. These are NOT the code-driven selenium tests, but rather the IDE version where users click through, and set assertions using the IDE (All are just UI tests). I know we can export those tests plans as a .SIDE file, but what I can't figure out, is how to have our TFS server execute those as part of a deployment or build pipeline.
Ideally, developers/devops would setup projects in TFS from the onset with whatever solution makes sense to execute these Selenium .SIDE files, but afterwards, the testers would manage adding/modifying those tests cases elsewhere.
The real goal here is to not have testers writing code, or checking in code. Only writing these UI Selenium tests, but having TFS execute those as part of CI.
Researching this on the internet drives me basically always to something that requires testers to write code.
I don't think it can automate testing without code, at lease, you need a test project containing your automated tests.
Generally, in Azure DevOps, we use Visual Studio Test task to run tests. This task supports using the following tests:
Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a
filter criteria to select only specific tests.
Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. To learn more about
how to associate tests with a test case work item, see Associate
automated tests with test cases.
Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when
running tests in a continuous integration/continuous deployment
(CI/CD) pipeline.
This was a question that I had as well, and I think I found an imperfect but better solution.
I wasn't able to get my Selenium IDE tests running with Jenkins, but I was able to get them to run with TeamCity, another CI.
I created a build step like the following :
Runner type: Command Line
Working Directory: where the selenium IDE .side file is located
Run: Custom Script
With the build script content that I usually use to run my Selenium IDE Tests, such as selenium-side-runner sidefile.side
I also added the following so I could output the results in Junitor another form: --output-directory=results --output-format=junit
You can also add the following so the tests are run headlessly, this only works in Chrome : -c "goog:chromeOptions.args=[--headless,--nogpu] browserName=chrome"
Finally, I also use --filter to run one test suite at a time, but that is optional too.
I then used another build step to export the results to our test manger, xray, but I think that is beyond the scope of this question.
The problem with this solution is that it runs directly from a users individual machine still, but this can be work around.
I have a C# project that I have used OpenCover and Gallio to run the MbUnit UnitTests from Nant script. I'm trying to set this up in Bamboo. I see Bamboo has NUnit runner, but doesn't seem to have anything for MbUnit.
Any idea how one would run the MbUnit UnitTests from Bamboo ?
Most CI systems have the ability to execute generic tasks (for this type of scenario)
A quick peruse of the site shows that the Console (Builder) Task would be the one; or you could also get it working in a nant script and then use that.
You would then use OpenCover to run NUnit or MbUnit runners as required.
I have an Xcode project with a framework target and a separate target for unit tests. Since there is no executable target I can't run the profiling tools without creating a new target, linking against the framework and profiling it that way. However, this means I have to move a lot of code around which unreliable and annoying.
Is there a way I can run the profiling tools (mainly Allocations) on my testing target?
I wrote a few nunit tests that I've run successfully in nunit on win7, and via MonoDevelop on my mac.
What I'd like to do now is run these tests non-stop (or until I kill it). I can't see any way to tell MonoDevelop to keep running over and over, so I'm trying to do this via the commandline.
Is there a way to run the MonoDevelop testing tool from the command line? I don't need reporting (my tests already report internally), just re-triggering every few minutes.
If there was a commandline tool, I could just wrap it in fabric or a shell script and just run it over and over...
Why don't you just use the normal nunit command line? It is contained with the standard Mono installation, available as nunit-console command.
I'm currently trying to reduce the time taken to compile and unit test projects in TeamCity.
Currently my project takes about between 5 to 8 minutes to build.
What it does is:
Clean any existing files
Compile each project
Create the installer
Once that is done the unit tests are kicked off and it takes about 2 minutes to run.
What it does is:
Clean any existing files
Compile each project
Run all the unit tests
Now, running the unit tests only takes 5 seconds....so the clean and compile step takes about 2 minutes....and the create installer step about 3 to 6 minutes.
My first question is: Is there any way to configure team city so that it doesn't have to do the clean and compile steps again when it runs the unit tests. I believe the main reason why we have it that way is because the project build and unit tests can be run by different build agents.
My second question is: Is 5 to 8 minutes a reasonable time for a project build to take? Are there ways to optimize the compile of the projects in the solution as well as the installer creation?
Please let me know if there are any additional details I can supply that might help you to point me in the right direction of either optimising the build or just leaving things as they are.
Update to answer some questions from Nate:
When you run the build outside TeamCity, does it do a clean/compile again before the unit tests? No, because when you run it on your own machine you can specify which parts of the build script to run. We have sections for clean, compile, unit-test etc which get specified differently in team city for project build vs unit test running.
If so, can you make the build not do that before the tests? The reason we don't do this at the moment is because when different build agents are used the files needed for the unit test would not be available and thus would fail.
What are you using to build your project? we use Nant to build our project, wix for the installer, and NUnit and NCover for the unit tests and coverage reports.
What source control system are you using? Subversion
Regarding #1: When you run the build outside TeamCity, does it do a clean/compile again before the unit tests? If so, can you make the build not do that before the tests?
What are you using to build your project? What source control system are you using?
Another option consider is having multiple build configurations. At my work, we have several configurations. One that runs after each commit. It does a clean/compile and some quick tests. We then have a nightly config that does a clean/compile and all the tests. Can you do something like that?
As a rule of thumb you should keep the build that is triggered on commit as short as posible, so you should try to refactor the build configuration into a slim on commit build, and run the current configuration once every hour or so.