I'm working on developing some Behavior Driven Development i.e style tests using pytest-bdd. We want to re-use the same features and more or less the same step definitions to having both stubbed and live calls to a third party API i.e. we want to reuse test code for integration and end to end testing.
I'm wondering about whether there was a convention on how to handle alternating between mocked and real calls in pytest_bdd or pytest
This question is similar: Running pytest tests against multiple backends? with an answer to add a parser option with a pytest_addoption hook placed in the top level conftest.py.
It looks like a good approach to select a stubbed or live api call in api is to add a parser option with a pytest_addoption hook. Conditional logic will need to look for those option in the relevant tests.
This answer to a similar question is the source for this approach and has more detail: https://stackoverflow.com/a/50686439/961659
Related
In Rust, is there any way to execute a teardown function after all tests have been run (i.e. at the end of cargo test) using the standard testing library?
I'm not looking to run a teardown function after each test, as they've been discussed in these related posts:
How to run setup code before any tests run in Rust?
How to initialize the logger for integration tests?
These discuss ideas to run:
setup before each test
teardown before each test (using std::panic::catch_unwind)
setup before all tests (using std::sync::Once)
One workaround is a shell script that wraps around the cargo test call, but I'm still curious if the above is possible.
I'm not sure there's a way to have a global ("session") teardown with Rust's built-in testing features, previous inquiries seem to have yielded little, aside from "maybe a build script". Third-party testing systems (e.g. shiny or stainless) might have that option though, might be worth looking into their exact capabilities
Alternatively, if nightly is suitable there's a custom test frameworks feature being implemented, which you might be able to use for that purpose.
That aside, you may want to look at macro_rules! to cleanup some boilerplate, that's what folks like burntsushi do e.g. in the regex package.
I am designing a BDD automation framework, where we are thinking of using cucumber #Before and #after hooks, can anyone suggest the best use of these?
There are many different hooks for before and after,
BeforeFeature:- Automation logic that has to run before each feature. Say in the feature file you have 10 scenarios, then for each 10 scenarios, BeforeFeature would be common. One use case is, say you want to get a global variable from config which is using by all scenarios in a feature. In this case, you don't need to go and get it during each scenario. Create a BeforeFeature hook
BeforeScenario:- Run before each scenario. Say you have 10 scenarios and you want to run one step which is common to all 10 scenarios. In this case, you can create a BeforeScenario hook. One example is, say you want to create a REST client and that REST client is common for all 10 scenarios. Go for BeforeScenario hook then.
AfterFeature:- Say if you want to clean up something, which was common to all feature then clean it in AfterFeature. This will run only after the completion of all scenarios in a feature.
AfterScenario:- Run after each scenario. If you want to clean up something after each scenario. say if you want to dispose of the REST client you have created during BeforeScenario then you can do that here.
Please find full hooks,
Can any one please tell me is there any kind of tools or eclipse base plugins available for generate relevant test cases for SalesForce platform related Apex classes. It seems with code coverage they are not expecting out come like we expect with JUnit, they want to cover whether, test cases are going through the flows of the source classes (like code go through).
Please don't get this post in wrong, I don't want anyone is going to write test cases for my codes :). I have post this question due to nature of SalesForce expecting that code coverage should be. Thanks.
Although Salesforce requires a certain percentage of code coverage for your test cases, you really need to be writing cases that check the results to ensure that the code behaves as designed.
So, even if there was a tool that could generate code to get 100% coverage of your test class, it wouldn't be able to test the results of those method calls, leaving you with a false sense of having "tested code".
I've found that breaking up long methods into separate, sometimes static, methods makes it easier to do unit testing. You can test each individual method, and not worry so much about tweaking parameters to a single method so that it covers all execution paths.
it's now possible to generate test classes automatically for your class/trigger/batch. You can install "Test Class Generator" app from AppExchange and see it working.
This would really help you generating test class and saves lot of your development time.
I am facing some difficulties in understanding test harness and related common terms like test case, test scripts in automation testing.
So this is what I got so far:
Automation testing is the use of a special software (other than the software being tested) to control the execution of tests and compare the actual results with the expected results. It also involves the setting up of test pre-conditions. This kind of testing is most suitable for tests that are frequently carried out.
Now, I am having some problems with test harness. I read that it consists of a test suite of test cases, input files, output files, and test scripts.
Now my question is what is the difference between test case and test script?
How do you use the software to test the different functions of the Acceptance Unit Testing (AUT)? I also came across some terms like suite master and case agents.
Several broad questions there, will try to answer based on my experience.
Think of a Test Harness as an 'enabler' that actually does all the work of (1)executing tests using a (2)test library and (3)generating reports. It would require that your test scripts are designed to handle different (4)test data and (5)test scenarios. Essentially, when the test harness is in place and prerequisite data is prepared (aka data prep) someone should be able to click a button or run one command to execute all your tests and generate reports.
A test harness is most likely a collection of different things that make all of the above happen. If you wrote unit tests while developing your application, that would be part of a test harness. You would also have other tests for the functionality of your app, like: user logs in to site, sees favourites pane, recent messages and notifications. Then you add in a 'runner' of sorts that goes through all of your "test scripts" and runs them (instead of you having to execute tests one at a time). If it feels like a test harness is more of a conceptual collection rather than a single piece of software, then you're understanding this correctly :-)
Now my question is what is the difference between test case and test script?
Simple but not entirely correct answer: A Test Case defines test objectives, description, pre-conditions, steps (descriptive or specific), expected results. A Test Script would then be the actual automated script that you execute to do that test. That's in an Automation context. And it changes. A lot.
What certifications like ISTQB define as test scenarios is usually referred to as test cases in some companies and countries. In others, test cases are flipped with test scripts when referring to manual testing (when the steps are given in detail but not part of an automation harness). Others say that test scripts exclusively mean automated tests. On the other hand, one can also argue that several test cases can be combined in a test script and vice-versa. So that begs the question, how does a test procedure fit in?
A test development stage can have: "Test procedures, test scenarios, test cases, test datasets, test scripts to use in testing software."
If you assume a > (is larger than/collection of) relation, how would you relate those? Rhetorical question - that differs based on where you work, who your client is, etc. Best thing is to define it with your colleagues/clients and agree on the understanding of the terms rather than the definition. I currently go with test script = automated script, based on a pre-existing manual test case or a test scenario.
Also, how do you use the software to test the different functions of the AUT?
You write different tests to test different things. Each test does certain actions and checks if the AUT's output matches what you expected - If displayed_value == expected_value. An input file could be used to provide data for the test- list of test usernames and passwords, for instance. Or run the same test with different data - login as a different user with different messages, etc.
Take a look at RobotFramework and the Selenium. A robot framework test (written in text or html files) combined with the Selenium library would allow you to write an automated test which tests something specific...like a home page validation. You would write a separate test to ensure that a user can see all his/her messages. Another to test clearing notifications. And so on.
test harness: A test environment comprised of stubs and drivers needed to execute a test.
Test harnesses and stubs will be used to replicate the missing items (components not yet included in the tests or external systems).
Often, when small-scale Integration Testing of several modules or components is performed, it is necessary to devise or improvise methods and tools to get the test data to the components under test. This is often called a test harness. Because of the need to understand the technicalities required to build a test harness this testing is almost always done by the development team.
A test harness may facilitate the testing of components or part of a system by simulating the environment in which that test object will run. This may be done either because other components of that environment are not yet available and are replaced by stubs and/or drivers, or simply to provide a predictable and controllable environment in which any faults can be localized to the object under test. These are usually bespoke programs generated by developers to help in the testing process. If they are used in a mature organisation it is quite possible that these harnesses will be considered as ‘Test Assets’ and subject to Version Control & Configuration Management.
Test harnesses contains all the information required to compile and run a test. This includes, test cases, source files under test, stubs, and Target Deployment Port (TDP) configuration settings.
A Test Harness is the collection of all the items needed to test software at the unit, module, application or system level and provides the mechanism to execute the test. Every item such as input data, test parameters, test case, test script, expected output data, test tool, and test result report is part of the test harness.
We use TestNG and Selenium WebDriver to test our web application.
Now our problem is that we often have several tests that need to run in a certain order, e.g.:
login to application
enter some data
edit the data
check that it's displayed correctly
Now obviously these tests need to run in that precise order.
At the same time, we have many other tests which are totally independent from the list of tests above.
So we'd like to be able to somehow put tests into "groups" (not necessarily groups in the TestNG sense), and then run them such that:
tests inside one "group" always run together and in the same order
but different test "groups" as a whole can run in any order
The second point is important, because we want to avoid dependencies between tests in different groups (so different test "groups" can be used and developed independently).
Is there a way to achieve this using TestNG?
Solutions we tried
At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".
We considered writing a method interceptor. However, this has the disadvantage that running tests from inside an IDE becomes more difficult (because directly invoking a test on a class would not use the interceptor). Also, tests using dependsOnMethods cannot be ordered by the interceptor, so we'd have to stop using that. We'd probably have to create our own annotation to specify ordering, and we'd like to use standard TestNG features as far as possible.
The TestNG docs propose using preserve-order to order tests. That looks promising, but only works if you list every test method separately, which seems redundant and hard to maintain.
Is there a better way to achieve this?
I am also open for any other suggestions on how to handle tests that build on each other, without having to impose a total order on all tests.
PS
alanning's answer points out that we could simply keep all tests independent by doing the necessary setup inside each test. That is in principle a good idea (and some tests do this), however sometimes we need to test a complete workflow, with each step depending on all previous steps (as in my example). To do that with "independent" tests would mean running the same multi-step setup over and over, and that would make our already slow tests even slower. Instead of three tests doing:
Test 1: login to application
Test 2: enter some data
Test 3: edit the data
we would get
Test 1: login to application
Test 2: login to application, enter some data
Test 3: login to application, enter some data, edit the data
etc.
In addition to needlessly increasing testing time, this also feels unnatural - it should be possible to model a workflow as a series of tests.
If there's no other way, this is probably how we'll do it, but we are looking for a better solution, without repeating the same setup calls.
You are mixing "functionality" and "test". Separating them will solve your problem.
For example, create a helper class/method that executes the steps to log in, then call that class/method in your Login test and all other tests that require the user to be logged in.
Your other tests do not actually need to rely on your Login "Test", just the login class/method.
If later back-end modifications introduce a bug in the login process, all of the tests which rely on the Login helper class/method will still fail as expected.
Update:
Turns out this already has a name, the Page Object pattern. Here is a page with Java examples of using this pattern:
http://code.google.com/p/selenium/wiki/PageObjects
Try with depends on group along with depends on method. Add all methods in same class in one group.
For example
#Test(groups={"cls1","other"})
public void cls1test1(){
}
#Test(groups={"cls1","other"}, dependsOnMethods="cls1test1", alwaysrun=true)
public void cls1test2(){
}
In class 2
#Test(groups={"cls2","other"}, dependsOnGroups="cls1", alwaysrun=true)
public void cls2test1(){
}
#Test(groups={"cls2","other"}, dependsOnMethods="cls2test1", dependsOnGroups="cls1", alwaysrun=true)
public void cls2test2(){
}
There is an easy (whilst hacky) workaround for this if you are comfortable with your first approach:
At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".
We had a similar problem: we need our tests to be run class-wise because we couldn't guarantee the test classes not interfering with each other.
This is what we did:
Put a
#Test( dependsOnGroups= { "dummyGroupToMakeTestNGTreatThisAsDependentClass" } )
Annotation on an Abstract Test Class or Interface that all your Tests inherit from.
This will put all your methods in the "first group" (group as described in this paragraph, not TestNG-groups). Inside the groups the ordering is class-wise.
Thanks to Cedric Beust, he provided a very quick answer for this.
Edit:
The group dummyGroupToMakeTestNGTreatThisAsDependentClass actually has to exist, but you can just add a dummy test case for that purpose..