I'm working in a company as a QA intern, we do end to end testing with browser automation.
We have lots of tests scenarios that looks like that : (we use cucumber)
Given I am logged in
When I go to my address book
Then I can see my contacts
but recently we had few bugs depending on what kind of account we are logged in so in order to tests this edges case I'm thinking of doing something like that:
Given I am logged in as with in as a project manager
When I go to my address book
Then I can see my contacts
And do that for every kind of account (project manager, commercial, etc ...)
But I'm asking about the interest of doing that .. For every user the outcome should be the same, they should all see their contacts. (even if due to a bug it was not the case)
if I start doing that way, I would be legit to have test like
Given I am logged in with a german account
and same with french, english etc...
but it would lead to an explosion of the number of tests
So do you guys test those edge case ?
If yes, how to do it efficiently ?
You can use cucumber sceanrio examples like below .
Scenario Outline:
Given I am logged in as "<userAccount>" .
When I go to my address book .
Then I can see my contacts
Examples:
|userAccount| .
|ABC| .
|XYZ| .
Usually you don't test for this in an integration test. Exactly because it leads to an explosion of test cases. Instead you re-engineer the system in such a way that you can test this at a lower level and still have confidence that the edge case is covered.
One way to do this would be to create "fake" countries. Each with either a single interesting attribute or a combination of interesting attributes.
Related
Let’s assume the following simple test case which is testing the functionality of a Banking system maintaining the balance of bank accounts:
Check Account #1234 Balance, which become the reference point (Ex: 1000 $)
Perform Deposit of 600 $
Perform Withdraw of 400 $
Check Account #1234 Balance, expecting the balance to be 200 $ over the reference point (Ex: 1200 $)
Given project pressures you and your colleague are ask to run the test suite in a concurrent fashion (could be using different browser version), given that both of you are manipulating the same account your test is sporadically failing.
In the IP sprint you are task to come up with a solution to bring consistency to the test results regardless of the number of members executing the suite in a concurrent fashion, what are the options you would consider.
There are different ways to approach your case, I would like to list some:
1 - If the concurrency is a must and if your Check Account changes something in a Database, then would be necessary to use different accounts, one per thread of execution, this way each test can run with no concerns on what the other tests are doing.
2 - If you can push for a non-concurrent solution, then you only need to run your tests serialized and at the end of each test revert back the check account to the reference point.
3 - Another way to solve this problem is to use mock data. This solution could be a little bit more complex, and it could requiere more work. But if still you want to know more about it contact your development team and let them know about your problem so that you can find a solution together.
You can read more about mocking data here:
Cypress Interceptor
Mockserver
Wiremock
Mockoon
Hope it helps!
I have a feature with a number of scenarios that need to be tested where the user is logged in as a number of possible roles (and the outcome should be the same).
Is it possible to put an Examples table into the background? For example:
Feature: My general set of tests
Background:
Given I am logged in
And I am logged in as <role>
Examples:
| role |
| Sales |
| Support |
| Admin |
| Auditor |
Scenario: tests1 blah...
Scenario Outline: tests 2 blah...
So what I want to do is iterate through the who feature file with a different Background context rather than having to maintain a separate feature file for each role. Is this possible?
No, this is not possible. The closest you could get would be a step definition "I am logged in in one of the usual roles (Sales, Support, Admin, Auditor)" where the actual role is chosen at random every time. This would of course make your tests non-deterministic, which might be acceptable if you are 100% positive that this role really has absolutely no effect on the outcome. Of course in that case you should rather consider not mentioning the role at all in your scenarios.
Remember that BDD isn't actually about testing. We're using examples to illustrate the behaviour we want. All you need is one scenario which provides an example of the behaviour.
If one of the roles provides different behaviour, then have an example of that behaviour too.
A better place to test that the outcomes are the same for all the roles might be at a unit-testing level, in whichever class links the roles to different outcomes. And of course, you can manually test it. The chances are very good that if you get it right and it works manually, it isn't going to break without breaking the one example you've got. It's enough.
If you try and cover every permutation of every role like this, you'll end up with a very slow build and far too many scenarios, none of which are interesting any more. Think in terms of examples of what you're trying to illustrate, rather than tests, and you'll have an easier time.
I've been using Gherkin (with Cucumber) for many years now. I noticed early on that Cucumber.io homepage examples refer to 'I' in the given and when statements, but not in the then statements.
I always assumed this was because given and when statements are subjective actions by a user, where as then statements should be objective measures about the state of the application under test.
However, I noticed in the official Cucumber book, their examples refer to 'I' all the way through the steps (including when using then statements).
Anyone know which method is correct?
In my experience regarding the BDD and Gherkinri, we've always considered that the User story has two actors: User and System. Not all actors are end users. For example, a role could be another system or someone who wants certain functionality in order to buy your product but will never actually use the product. It may be useful to create aggregate roles (such as consumer) and specialized roles (such as browser or frequent shopper). So having I in the Then steps is actually expected result from end-user's perspective. Example:
Then I should see that cash amount 'storedTotalAmount' and cash amount 'currentAmount' are equal 'true'
There is not a definitive rule as to whether "Then" statements should include "I". As #ekostadinov it depends on the Actors which are taking part in the test scenario.
For instance in a messaging scenario:
Given Alice is on a messaging page
When Alice posts a message to Bob
Then Bob should receive the message
In such a scenario it would make no sense to use "I" in the "Then". However if you had a slightly different scenario:
Given I am on the messaging page
When I post a message to Bob
Then I should receive confirmation that Bob received the message
Then using "I" does make sense.
Hence it depends on how you write your tests. To conclude, I prefer to use personas (i.e. particular types of users in your system) throughout my tests e.g. Alice, Bob etc. This generally removes any possible ambiguity with the use of "I".
This is an interesting topic I came across and my co-workers and I have different opinions on the matter. Should your Gherkin describe exactly what the test is doing, or ONLY show the business logic you tried to achieve in the test.
The biggest example that I run into all the time at work is that if you have access to item A, then you should be able to access A. We can have 20 different types of users with access to A, so we only choose 1 (to keep our test suite from taking 40 hours to run). So which is "better"?
A
Scenario: A user with access to item A can access A
Given I am a type 4 user with access to item A
When I try to access A
Then I am granted access to A
or B
Scenario: A user with access to item A can access A
Given I am a user with access to item A
When I try to access A
Then I am granted access to A
Notice the difference in the given statements (type 4 user)
Granted in the step definition we are going to use a type 4 user for our test, but the test is not specific to a type 4 user. Any user with item A will work for this test, we're just using a type 4 user because we need a user type to login with.
So A describes what the test is doing (Logging in with a type 4 user with access to item A)
And B describes the functionality needed to access item A (just a user with access to item A)
Before you ask, how we determine who has access to item A is a SQL call to the database looking for a specific item linked to a user.
For a cucumber test you are testing the business logic - as an acceptance test - not hte specific implementation details. So you SHOULD do the second not the first. Your Request specs or Integration tests can be more tied to specifics if you want to run tests for type X, type Y and edge cases.
I think one can think of this - and it's not a hard fast rule - as something like:
Unit test to isolate methods and test one thing at a time. Mock & stub everything else to isolate what is being tested.
Integration tests to test how things interact together to test a larger part of your stack, including the interaction of multiple objects. Some would argue that you test everything soup to nuts here, but I think there's a place in a large complex app to test lots of pieces integrated while still not testing the full request cycle.
Request specs - sometimes in simple apps these are pretty much the same as integration tests, in other cases I will do integration tests for everything except the request stack and specifically separate out my request specs. Opinions will vary.
Acceptance tests - this is where you're sitting with your question - where the tests are written in plain business language and avoid technical implementation details within the feature definitions.
Anyway, even if you ignore the thoughts on the rest of the test stack, in the specific question you're asking go for B.
I would say option B is better. The "type 4 user" sounds like an implementation detail.
However, if it is a requirement that all user types have access, then that should form part of the specification too. In that case the test should specify and test all user types.
I would say B is better. For the "Type 4 user" you could make it be part of a Background:
Backgound : User is logged in
Given "Type 4 user" is logged in
Use a place holder for type 4 user by placing it in "" so you can reuse the logged in step definition for other user that have access to item A
I have set up SpecFlow to execute a number of Selenium tests to test a website. This all works fine!
I am just considering if there might be room for some optimization on the structuring of the SpecFlow features.
All the scenarios in a specific feature should use the same login. This is something I have hardcoded in the StepDefinition using the [BeforeScenario()] hook at the moment, since I don't really wan't pollute the scenarios with login info. It isn't relevant for the test.
But at the same time, I would like to remove the hardcoded part, and move this into my feature.
My question is two part.
Can I specify login credentials in my feature description. Sort of like a Given:
Feature: As a user I want to be able to see my ongoing orders, and interact with them.
Given I am logged in with username abc and password xyz
Scenario: See list of ongoing order
Given I place an order
When I navigate to MyOrders page
Then I can see at least one order in the list
Is this good practice?
Does it make sense to do it like this, on the feature level I mean. The scenarios are not dependent on a specific order, and they would execute faster if I didn't need to log in for each scenario.
Thanks for your input.
For steps that are common to all scenarios in a feature you can use Backgrounds:
Feature: As a user I want to be able to see my ongoing orders, and interact with them.
Background:
Given I am logged in with username abc and password xyz
Scenario: See list of ongoing order
Given I place an order
When I navigate to MyOrders page
Then I can see at least one order in the list
Too much abuse of background steps can be a bad practice because it introduces coupling between your scenarios.
Another solution is to put the login part directly into your "I Place an order" step.
This will remove all the noise about login stuff since it's implicit that you need to log in to place an order.
I'd also suggest to call it "I've placed an order" instead of "I place an order".
Given steps are usually pre-condition that describes what happened prior using the functionnality (when steps)
When I first read your post, I thought of Dan North: Who's domain is it anyway? and that influences me to think that we should be trying to write our tests so that they stick to a single knowledge domain. As you talk about specific users and navigation and orders in a list, well its very like the example he gives in that your specification is crossing domains. This kind of leads me towards almost making your specification less specific, i.e. its not about navigating and checking a list, do you have an order or not!
Scenario: See list of ongoing order
Given I am logged in with username abc and password xyz
And I place an order
Then should have at least one order
I wanted to give you another example link, to a post I cant find, which talks about the benefits that a team has got by actually defining some example users and what value that gives their testing. So in your business domain you might have a user called Jack who will have placed a large order in the system, and another prospective client Jill with no orders. In this way tests such as
Given I am Jack
When I search for my orders
Then I will find 1
Given I am Jill
When I search for my orders
Then I will find 0
can guarantee that you are only testing your Search functionality, and less about getting the setup in place.
I'd also suggest that you have a look at Liz Keogh: Acceptance Criteria vs Scenarios who asserts that more general broad definitions are less valuable than very specific examples. (It is Specification By Example after all :-) )
So to answer your question, I think having the specified user is a good thing, but doing it how you are doing it is heading a very complicated route.
The scenario examples that are given are completely not in line with the Feature.
Feature: As a user I want to be able to see my ongoing orders, and interact with them.
Background:
Given I am logged in as John Doe
Scenario: This is taking to long
And I have placed an order more than 29 days ago
When I navigate to MyOrders page
Then I can see my order in the list
and I am able to cancel it
Scenario: Track package
And I have been notified my order has been send
When I navigate to MyOrders page
Then I can see my order in the list
and I am able to navigate to the postman's site and see its status
Scenario: Review item
And I have received my package
When I navigate to MyOrders page
Then I can see my order in the list
and I am able to review the item
Scenario: Contact Seller
And I have placed an order two weeks ago
When I navigate to MyOrders page
Then I can see my order in the list
and I am able to ask the seller what is going on
As you can see the When and the Then is starting to duplicate itself and you might want to consider tucking those away.