Is there a way to specify code to be run before all the tests in the current test run? Even when running tests across deeply nested directories? e.g.
a/
a/a_test.go
b/c/
b/c/c_test.go
d_test.go
I want to write some code that runs once before and once after all the tests in files a_test.go, c_test.go, d_test.go have run.
I know about TestMain, which sorta does what I want if I needed to do this at the package level, but this doesn't run before/after all the tests in subdirectories/subpackages. I want something that's one level above TestMain.
I'm not limited to go test, so if there's a third-party test runner that would do this for go, that would be alright as well.
I'm looking for something akin to nosetests's SetUpPackage or pytest's session scoped fixtures.
Related
I'm using GitLab pipelines to run e2e tests on various physical machines (these machines are connected to the test hardware in a 1 to 1 relation). On each machine, a GitLab runner is installed. The pipeline consists of three major parts:
prepare the test hardware (deploy, configure)
execute the e2e tests (on the test hardware)
clean up the test hardware
Currently I'm doing all of this in one job, by using the before_script, script and after_script keywords. But I would like to use multiple jobs (or even stages) for this.
The problem I'm facing is, that I can't be sure that all jobs/stages are executed on the same runner. So it might happen, that the prepare step is executed on runner1 and the execute step is executed on runner2 (even in parallel), which obviously is not what I want. The preparation is more than just creating artifacts, therefore I can't simply give it to the next job.
Tags also seems not to solve this, because a tag can only be specified for one job, not for multiple, or the complete stage.
I understand that this is not the way how runners are used normally, but I still wonder if there is a way to achieve this.
Or can someone point out another approach to solve this?
I'm using GitLab Community Edition 14.3.2.
I think you have two options here for how you can split this up -
As sytech mentioned, you can tag each machine with machine-1, machine-2, etc, which will allow you to make your jobs sticky to each runner. Since you can use variables in runner tags, you could have a job at the start that checks which runner is not running tests, and sets RUNNER_TAG or something similar to that runner, so you don't have to hardcode your runner to a single box
You could not have the test boxes run the jobs directly (presumably you're using a shell runner to do this today), and use SSH or winRM to access the box directly, and modify it from there. Then the state of your runner doesn't matter at all. This is likely the "cleaner" way to do it, so your test boxes don't have to share resources or state with the runner
As far as I know TestCafe default behaviour is to run tests in parallel.
Indeed the browsers function accepts an array of browser (which is cool).
What I would like to do however is quite different. I have fixtures based on area of my portal (search, payment etc...) and so I'd like to know if it's possible to run these tests in CLI in parallel as they are orthogonal.
The scope is of course to improve the execution time as the number test
cases will grow.
On the other hand I'd like also to catch the failures meaning that if a test ran in parallel on a specific metadata filter fails possibly we would like to stop the others too.
I am not using TestCafe's docker but our custom one with just Firefox, Chrome installed and we launch of tests in headless mode.
As a last point a great thing would be if we could run these scenario/metadata in parallel but somehow at the end of the test suite gather the reports together.
I understand the question is not easy especially because it involves either TestCafe or GitlabCi but probably someone else faced this problem too.
Thank you
If I understand you correctly, the behavior you described can be achieved by dividing the test execution among multiple CI jobs. For example, each CI job can test a particular area of your portal. For that, run TestCafe with specified metadata of your fixture/test. Also, most of the CI systems allow you to cancel all other jobs in a pipeline if one of the jobs fails (unfortunately, Gitlab hasn't released this feature yet).
On the other hand, you can use TestCafe's programmatic API: create multiple TestCafe runners, each running the desired subset of tests. However, at the end of the test execution, you'll need to merge generated reports into one report manually. Check this answer to get an idea of how to create multiple runners.
I have a small collection of integration tests that utilize selenium in a class. The idea is that these tests run every time there is a merge to the codebase, with the merge proceeding through the pipeline and having a series of tests running against the new code.
The thing is, these selenium tests have to run one at a time. They're using the browser to log into a website, and the account will just log out if more than one person tries to log into the account at once, it'll just log out, and the test will obviously fail, so I need these tests to run one at a time. I've tried using the #NotThreadSafe annotation, doesn't seem to have changed anything, and I've searched through for some sort of switch or parameter that defines how many tests run at once with no luck. These tests are using junit 4.12.
I'm using Selenium IDE 2.3.0 to record actions in my web application and create tests.
Before every test I have to clear all cookies, load the main page, log in with a specific user and submit the login form. These ~10 commands are fix and every test case needs them, but I don't want to record or copy them from other tests every time.
Is there a way to configure how "empty" test cases are created?
I know I could create a prepare.html file or something and prepend it to a test suite. But I need to be able to run either a single test or all tests at once, so every test case must include the commands.
Ok I finally came up with a solution that suits me. I wrote custom commands setUpTest and tearDownTest, so I only have to add those two manually to each test.
I used this post to get started:
Adding custom commands to Selenium IDE
Selenium supports object-oriented design. You should create a class that takes those commands that you are referring to and always executes those, in each of the tests that you are executing you could then make a call to that class and the supporting method and then execute it.
A great resource for doing this is here.
I'm trying to integrate running Fitnesse tests from MSBuild im my nightly build on TFS.
In an attempt to make it self contained I would like to start the seleniumRC server only when it's needed from fitness.
I've seen that there is a "Command Line Fixture" but it's written in java can I use that?
I think you might be able to. You can call any process easily in MSBuild using the task. However, the problem with doing this is that the exec task will wait for the Selinium process to finish before continuing, which is not the bahaviour you want. You want to run the process, keep it running during your build and then tear it down as your build finishes.
Therefore, I think you are probably going to need to create a custom MSBuild task to do this. See the following post for an example of a tasks that someone has created that will run asynchronously returning control back to the build script:
http://blog.eleutian.com/2007/03/01/AsyncExecMsBuildTask.aspx
And for an example of calling a Java program from MSBuild (but in this case synchronously) take a look at my task that calls Ant from MSBuild here
http://teamprise.com/products/build/
As part of your MSBuild task, you will want to output the process id that you created to an output property so that at the end of your build script you can call another custom MSBuild task that kills the process. It can do this by looking for the process id passed in as a variable in MSBuild and then call Process.Kill method i.e.
Process process = Process.GetProcessById(ProcessId);
process.Kill();
That said, you would need to be careful to ensure that your kill task was always executed in MSBuild by making sure it was included during error paths etc in the build. You could probably make things a bit more resilient by making the selenium RC starter task look for other seleniumRC processes and killing them before starting a new one - that way if a process didn't get closed properly for some reason, it would only run until the next build.
Anyway - my answer sounds like a lot of work so hopefully someone else will come up with an easier way. You might be able to create the seleniumRC process in the test suite start up of the FitNesse tests and kill it in the suite tear down, or you might be able to write a custom task that extends your FitNesse runner tasks and fires up seleiniumRC asynronously before running the test process and then kills it afterwards.
Good luck,
Martin.
Thanks for your replies!
This is how I've done so far.
I made a fit fixture (very simple) that starts a process with the supplied command line, in my case startSelenium.bat. The fixture returns the ProcessID so I can store that in my fitnesse context and close that session later.
I can now make a SuiteSetUp page in my fitnesse test that looks like this.
|RunCommandFixture|
|Commandline|RunCommand?|
|C:\Projects...\startSeleniumRC.bat|>>seleniumprocess|
and a SuiteTearDown like this
|RunCommandFixture|
|ProcessID|StopCommand?|
|<
That works for me. No selenium RC starts by request from my fitnesse test.
What about writing a simple .NET app that does a Process.Start("selenumRC commandline") which gets run by your build script?
If you aren't too far down the Selenium route; might I suggest that you look at similar .NET browser automation tools; specifically WatiN or ArtOfTest. The "stacks" in these are completely .NET, so getting them running on different machines is much easier.