I'm writing unit tests using Selenium and xunit and on their own they run great but if I select all of the tests (multiple classes) in Test Explorer (they appear to be grouped by class - this is not intentional) they run in parallel. Run Tests in Parallel is not selected. Each one of my tests creates and then deletes test data so they obviously can't run in parallel. One test might delete data right after another test created that data and so the test would fail. So how can I run all of my tests and not have them run in parallel? I guess I could make them all use one partial class that spans multiple files but that's not my first choice.
I found a solution (although not an explanation). Just put [Collection("Sequential")] as the first line under each namespace. This forces everything to run sequentially.
Related
I am writing new tests in Nunit. I would like the tests to get their TestCaseSource values from an excel sheet (Data-driven tests).
However, I noticed that the [SetUp] method is actually accessed AFTER the [Test] method is entered, therefore I cannot initialize the data I read from my excel sheet in the TestCaseSource.
How do I init my TestCaseSource from an excel file BEFORE each test is running?
Thanks
I have tried using a separate class like MyFactoryClass and then used
[Test, TestCaseSource(typeof(MyFactoryClass), "TestCases")]
However, this is reached Before the [Setup] method and does not recognize the name of the excel file that is named after each tests' name.
It's important, when using NUnit, to understand the stages that a test goes through as it is loaded and then run. Because I don't know what you are doing at each stage, I'll start by outlining those stages. I'll add to this answer after you post some code that shows what your factory class, your [SetUp] method and your actual tests are doing.
In brief, NUnit loads tests before it runs them. It may actually run tests multiple tiems for each load - this depends on the type of runner being used. Examples:
NUnit-console loads tests once and runs them once, then exits.
TestCentric GUI loads tests once and then runs them each time you select tests and click run. It can reload them using a menu option as well.
TestExplorer, using the NUnit 3 Test Adapter, loads tests and then runs them each time you click run.
Ideally, you should write your tests so that they will work under any runner. To do that, you should assume that they will be run multiple times for each load. Don't write code at load time, which you want to see repeated for each run. If you follow this rule, you'll have more robust tests.
So... what does NUnit do at each stage? Here it is...
Loading...
All the code in your [TestCaseSource] executes.
Running...
For each TestFixture (I'll ignore SetUpFixtures for simplicity)
Run any [OneTimeSetUp] method
For each Test or TestCase
Run any [SetUp] method
Run the test itself
Run any [TearDown] method
Run any [OneTimeTearDown] method
As you noticed, the code you write for any step can only depend on steps that have already executed. In particular, the action taken when loading the test can't depend on actions that are part of running it. This makes sense if you consider that "loading" really means creating the test that will be run.
In your [TestCaseSource] you should only call a factory that creates objects if you know in advance what objects to create. Usually, the best approach is to initialize those parameters that will be used to create objects. Those are then used to actually create the objects in the [OneTimeSetUp] or [SetUp] depending on the object lifetime you are aiming for.
That's enough (maybe too much) generalization! If you post some code, I'll add more specific suggestions to this answer.
I’m trying to run a custom test suite which includes several test cases. For example, I’ve wrote 4 test scripts(test_login_success,test_login_fail,test_register_xxx,test_register_yyy), and I just want to run test_login_* module, how to set the defaultTestSuite and add testcases to it?
The test cases you create belongs to its class. If you want to customise test runs you shall consider updating to the new Xcode 11. The new version of Xcode has test plans feature allowing you to control tests executions better.
Introduction video:
https://developer.apple.com/videos/play/wwdc2019/413/
If you prefer to stay on previous Xcode, you should add schemes for your scenarios.
Also, you can pass tests names in xcodebuild shell command.
I have to execute large number of test cases in parallel using TestNG and Selenium. Each test case will be executed in different data set using Data driven testing. How to run these test cases in parallel in different machines? We can use Parallel attribute in TestNG but that is restricted to a single machine.
Can Selenium Grid tweaked and use in this purpose? If yes how to use this or any other suggestion?
I want examples of (https://www.seleniumhq.org/docs/07_selenium_grid.jsp#when-to-use-it)
To reduce the time it takes for the test suite to complete a test
pass.
Basically it's quite complicated it needs lot of understanding i haven't done it but i know that you need to create one root machine and rest machines will be childs of the parent machine then you can run the test scripts parallel but you need to make sure that those script shouldn't be dependent otherwise their will be lot of issue
I have shared the link with you so you can check how you set up?
https://medium.com/#appening/how-to-run-your-test-on-multiple-machines-using-selenium-grid-3aa37d5d2b63
I have two tests that are dependent on same test(Using DependsON). Are these dependent test cases will be executed one after other?
I am using testNG version-6.10
Would the dependent testcases run one after the other or together depends on what your parallel execution strategy is.
If you have set parallel='methods' in your suite file, then both the dependent methods will run together.
If you have disabled parallel execution (by setting parallel=false in your suite file), then the dependent methods will run one after the other. Which runs first and which runs next is not determined (since TestNG relies on reflection to query methods from a class)
But in either cases, they will be executed only after the master test (the one on which both your tests depend on) runs successfully.
I have implemented several packages for a web API, each with their own test cases. When each package is tested using go test ./api/pkgname the tests pass. If I want to run all tests at once with go test ./api/... test cases always fail.
In each test case, I recreate the entire schema using DROP SCHEMA public CASCADE followed by CREATE SCHEMA public and apply all migrations. The test suite reports errors back at random, saying a relation/table does not exist, so I guess each test suite (per package) is run in parallel somehow, thus messing up the DB state.
I tried to pass along some test flags like go test -cpu 1 -parallel 0 ./src/api/... with no success.
Could the problem here be tests running in parallel, and if yes, how can I force serial execution?
Update:
Currently I use this workaround to run the tests, but I still wonder if there's a better solution
find <dir> -type d -exec go test {} \;
As others have pointed out, -parallel doesn't do the job (it only works within packages). However, you can use the flag -p=1 to run through the package tests in series. This is documented here:
http://golang.org/src/cmd/go/testflag.go
but (afaict) not on the command line, go help, etc. I'm not sure it is meant to stick around (although I'd argue that if it is removed, -parallel should be fixed.)
The go tool is provided to make running unit tests easier using the convention that *_test.go files contain unittests in them. Because it assumes they are unittests it also assumes they are hermetic. It sounds like your tests either aren't unittests or they are but violate the assumptions that a unittest should fulfill.
In the case that you mean for these tests to be unittests then you probably need a mock database for your unittests. A mock, preferrably in memory, of your database will ensure that the unittest is hermetic and can't be interfered with by other unittests.
In the case that you mean for these tests to be integration tests you are probably better off not using the go tool for these tests. What you probably want is to create a seperate test binary whose running you can control and write you integration test scripts in there.
The good news is that creating a mock in Go is insanely easy. Change your code to take an interface with the methods you care about for the databases and then write an in memory implementation of that interface for testing purposes and pass it into your application code that you want to test.
Just to clarify, #Jeremy's answer is still the accepted one:
Since my integration tests were only run on one package (api), I removed the separate test binary in the end and created a pattern to separate test types by:
Unit tests use the normal TestX name
Integration tests use Test_X
I created shell scripts (utest.sh/itest.sh) to run either of those.
For unit tests go test -run="^(Test|Benchmark)[^_](.*)"
For integration tests go test -run"^(Test|Benchmark)_(.*)"
Run both using the normal go test