I'm writing some acceptance tests for a donation form. I'm using Codeception. For the sake of this example, lets say that the donation form has 3 parts:
Enter your personal information
Enter either Credit Card and Direct Transfer
Submit and receive e-mail confirmation
For the acceptance test I'd like to test the whole process--for both credit card AND direct transfer. Steps 1 and 3 are essentially the same between the two donation processes, but--obviously--you can't run the second step by itself (the donation form wouldn't submit without step 1).
So I'm wondering, would it be "normal" in this case to write two tests (e.g. canDonateWithCreditCard() and canDonateWithDirectTransfer()) that both test all three parts of the process? Even though that's partly testing the same thing twice?
If not, what would be the preferred way to do it?
This is perfectly acceptable at my work we have a sizable automation suite where the same pages get executed multiple times, because of scenarios similar to what you outlined above.
The only caveat I would mention is when building your tests (I don't know how codeception works) but look to build your tests using something along the lines of the page object model (http://martinfowler.com/bliki/PageObject.html) this will mean even though you have multiple tests that may implement the same scenarios each test doesn't have its own implementation of those steps.
This depends on your approach.
1. You can create two different test cases performing the action.
2. You can have a logic in your test to pass the mode of transfer as an argument to the method and perform activities accordingly.
It's always ideal to use Page object model to encapsulate all actions in each page class and also to avoid redundancy.
If both Credit card and Direct transfer actions navigate to a new page, create a new object of the page according to the argument passed, and call the method to do the transfer action.
A simple page object class can be created like this:
http://testautomationlove.blogspot.in/2016/02/page-object-design-pattern.html
Related
Scenario1
When a new user clicks on sign up page
And provides login ID
Then user is signed up and can view profile page.
Scenario2
When user clicks on the edit profile page
And updates his address
Then updated profile should be visible to user
The scenarios are written in a feature file in the same sequence. When writing the cucumber file for the same, I am creating a user in the scenario 1. And in scenario 2 , the same user is being updated. In a way, scenario2 is dependent on 1 as it is updating the same user that was created in scenario1.
My question is should the scenarios be created so that they are dependent on other scenarios.
Or they should be independent of each other execution.. in which case I should create a new user in scenario2 and then perform and update on it and assert it.
Cucumber explicitly recommends you do not make your Scenarios depend on each other. From the FAQ's:
"Each scenario should be independent; you should be able to run them in any order or in parallel without one scenario interfering with another.
Each scenario should test exactly one thing so that when it fails, it fails for a clear reason. This means you wouldn’t reuse one scenario inside another scenario.
If your scenarios use the same or similar steps, or perform similar actions on your system, you can extract helper methods to do those things."
(Sidenote: From personal experience, I can tell you that tests that depend on each other / the state of the system, will very quickly become very hard to maintain. I'd highly recommend you to make your tests independent!)
Let's say I have a Main.story file which contains-
Login Scenario
Search Scenario
AddToCart Scenario
UpdateQuantity Scenario
Checkout Scenario
But now what if I only want to run-
Login Scenario → Search Scenario → AddToCart Scenario → Checkout Scenario and skipping UpdateQuantity Scenario
How could I possibly achieve this without removing/deleting anything from the story file.
BDD style scenarios are meant to be completely independent. In this case, you would have 2 scenarios:
Scenario: I can check out with an updated order quantity
Given I login
And I search
And I add to the cart
And I update the quantity
When I checkout
Then I get a confirmation email (or whatever)
Scenario: I can purchase items
Given I login
And I search
And I add to the cart
And I update the quantity
When I checkout
Then I get a confirmation email (or whatever)
If there's a concern about each of those steps actually being several steps and that looking really ugly in the scenario with 20 givens, so long as each of those actions is tested separately, you can use a compound step. That's a step that, in its definition, calls other steps (through code, not Gherkin). You take the same actions, but the Gherkin has far fewer entries.
The important part is that no scenario should require action from a previous scenario in BDD.
To further clarify, there is a way to re-order things if you really must, but it's very bad practice. If you choose to go that road, see this question:
How do i execute story files in specific order in serenity BDD Jbehave
I want to perform rigrous testing on Payment Gateway(2checkout) and Pay Pal. For testing, I need to simulate a large number of successful, failed and halted transactions (transaction stopped due to system crash/reboot). But I don't want to make actual payments.
1. Is there any way I can make a test transaction on payment gateway, using fake card numbers or something else.
2. What are the possible advance testing scenarios for Payment Gateway testing?
For example:
Changing the amount, unmask CVV or card from Inspect
element.
List item
There are two options :
Using the PayPal Sandbox (Application Testing), or
Using Dependancy Injection (Unit Testing).
Both would work but I would suggest a Dependancy Injection approach. Assuming you have a separate object that only interacts with PayPal and then other objects that do your actual application logic (and error handling, etc) then you can just create a dummy version of the PayPal interaction object (that always returns true, or conditionally returns false, whatever) and then test your various application classes in detail.
I would suggest you only one solution, look at this Git PayPal-Android SDK and go through the README.md file. Last link tells you how to create a sandbox PayPal account to create dummy transactions across your sandboxed account developer account.
If you have doubts, you can refer Part 1 and Part 2 of AndroidHive tutorial for this.
This is an interesting topic I came across and my co-workers and I have different opinions on the matter. Should your Gherkin describe exactly what the test is doing, or ONLY show the business logic you tried to achieve in the test.
The biggest example that I run into all the time at work is that if you have access to item A, then you should be able to access A. We can have 20 different types of users with access to A, so we only choose 1 (to keep our test suite from taking 40 hours to run). So which is "better"?
A
Scenario: A user with access to item A can access A
Given I am a type 4 user with access to item A
When I try to access A
Then I am granted access to A
or B
Scenario: A user with access to item A can access A
Given I am a user with access to item A
When I try to access A
Then I am granted access to A
Notice the difference in the given statements (type 4 user)
Granted in the step definition we are going to use a type 4 user for our test, but the test is not specific to a type 4 user. Any user with item A will work for this test, we're just using a type 4 user because we need a user type to login with.
So A describes what the test is doing (Logging in with a type 4 user with access to item A)
And B describes the functionality needed to access item A (just a user with access to item A)
Before you ask, how we determine who has access to item A is a SQL call to the database looking for a specific item linked to a user.
For a cucumber test you are testing the business logic - as an acceptance test - not hte specific implementation details. So you SHOULD do the second not the first. Your Request specs or Integration tests can be more tied to specifics if you want to run tests for type X, type Y and edge cases.
I think one can think of this - and it's not a hard fast rule - as something like:
Unit test to isolate methods and test one thing at a time. Mock & stub everything else to isolate what is being tested.
Integration tests to test how things interact together to test a larger part of your stack, including the interaction of multiple objects. Some would argue that you test everything soup to nuts here, but I think there's a place in a large complex app to test lots of pieces integrated while still not testing the full request cycle.
Request specs - sometimes in simple apps these are pretty much the same as integration tests, in other cases I will do integration tests for everything except the request stack and specifically separate out my request specs. Opinions will vary.
Acceptance tests - this is where you're sitting with your question - where the tests are written in plain business language and avoid technical implementation details within the feature definitions.
Anyway, even if you ignore the thoughts on the rest of the test stack, in the specific question you're asking go for B.
I would say option B is better. The "type 4 user" sounds like an implementation detail.
However, if it is a requirement that all user types have access, then that should form part of the specification too. In that case the test should specify and test all user types.
I would say B is better. For the "Type 4 user" you could make it be part of a Background:
Backgound : User is logged in
Given "Type 4 user" is logged in
Use a place holder for type 4 user by placing it in "" so you can reuse the logged in step definition for other user that have access to item A
Is it acceptable to write a "Given When Then When Then" test in Gherkin?
A real-life example is as follows all AllPlayers.com
Scenario: Successfully register a user
Given I am on homepage
And I am not logged into an account
When I follow "create a new account"
And I fill in "First Name" with "Bobby"
And I fill in "Last Name" with "Bricks"
And I fill in "E-mail" with "bbricks#example.com"
And I select "Jun" from "Birthday Month"
And I select "22" from "Birthday Day"
And I select "1985" form "Birthday Year"
And I select "Male" from "Gender"
And I fill in "Password" with "123testing"
And I fill in "Confirm Password" with "123testing"
And I solve the captcha math problem
And I click "Create new account"
Then I should see "the user dashboard"
And I should see the Registration Wizard
When I push "Proceed to next step"
Then the "First Name" field should contain "Bobby"
And the "Last Name" field should contain "Bricks".
I know it works using behat, so parsing it isn't a problem. I'm just trying to write better tests. I could write in the first then And the Registration Wizard should be filled out with data but that doesn't seem specific enough...
Suggestions?
It depends on the target audience of the feature as written. It seems highly likely that the gherkin you've got there was not written with a stakeholder (i.e. somebody not-techie but has a vested interest in the business and the website). BDD is really about the conversation about requirements and expectations - and Gherkin is a tool which gives a standard/recognised way that everyone should be able to read that you can write the requirements and expectations; in a way that serves as automated tests for a developer and perhaps test scripts for a tester.
Trying to take my developer hat off now - I would say that a business stakeholder would rather read, and understand easily...
Scenario: Should be able to successfully register on website
Given I am new to the website
And I want to register for a user account
When I go to the registration form
And I complete all the required registration details correctly
Then I will be registered on the website
And I will be automatically logged in
You can still build the same test behind the scenes of this specification - but this specification has larger readership, it is a more easily understood requirement that anyone should understand. I'm not saying what you have got has no value - far from it. It will be a very valid test. But it is quite developer specific, and highly coupled to the UI implementation (if you refactor/redesign the UI, you now need to refactor your Requirements...).
I started off having plenty of gherkin specifications much like yours - and I still use them on occasion. Once your testing framework has built up a little gherkin is a really great way of kind of writing data-driven/configurable unit tests; and they still have great value to my development process. But I do try to separate the more "pure" specifications from my "developer" ones - but folder and tags/categories.
Edit: I guess in summary what I'm getting at is... what you have is a great "test", but a fairly bad "requirement". Stick with it though!
Yes, more than one When/Then cycle is appropriate in a Gherkin scenario when the real-world scenario calls for it.
SaxonMatt's answer makes the excellent point that scenarios are best written in stakeholder language rather than in the language of UI manipulation, and that doing so often reduces the length of a scenario, but that misses the exact point of the question. Let's take the bull by the horns.
Gherkin was designed for acceptance tests: tests which test that stakeholder-level requirements have been completely implemented, i.e. that the software actually provides value to stakeholders. Sometimes providing value takes more than one action-response cycle. Consider the following scenario:
Scenario: Guest buys a product
# This scenario starts with the user not logged in, which doesn't require a step
Given there is a product named "Elliptical Juicer"
When I go to the product page for "Elliptical Juicer"
And I add the product to my shopping cart
Then I should see 1 product in my shopping cart
When I request to check out
Then I should see the account creation form
When I create an account
Then I should see the checkout form with 1 product, "Elliptical Juicer"
When I check out
Then I should see the checkout success page with 1 product, "Elliptical Juicer"
And I should receive a checkout confirmation email with 1 product, "Elliptical Juicer"
(Note that when I have more than one When/Then cycle in a scenario I like to separate them with blank lines so they stand out.)
There are several reasons why this scenario is best written with multiple When/Then cycles:
Before the user checks out, they should see one product in their shopping cart (only as a digit in the site header, so the step doesn't mention the product name). There is no way to test this requirement at the end of the scenario. (Well, the test could collect the information immediately after the user adds the product to their cart and assert the expected count at the end of the scenario, but that would be pointlessly sneaky and confusing.) Instead, assert the correct count at the natural place in the scenario, as soon as it is visible to the user.
Similarly, Then I should see the account creation form and Then I should see the checkout form with 1 product, "Elliptical Juicer" can test important requirements at the points in the scenario at which it is natural to test them.
Suppose we didn't care about what the user sees during the process, only whether they get to the end of the scenario with their product on the way. We might then omit the intermediate Then steps:
Given there is a product named "Elliptical Juicer"
When I go to the product page for "Elliptical Juicer"
And I add the product to my shopping cart
And I request to check out
And I create an account
And I check out
Then I should see the checkout success page with 1 product, "Elliptical Juicer"
And I should receive a checkout confirmation email with 1 product, "Elliptical Juicer"
And I create an account comes as a surprise, doesn't it? It requires the reader to infer that a guest user is asked to create an account during checkout. It's clearer to say so explicitly, as in the first version of the scenario that I gave.
Suppose none of the above concerns convinced us and we wrote a separate Gherkin scenario for each point in the overall scenario where we needed to assert that requirements have been met:
Scenario: Guest adds a product to their shopping cart
Given there is a product named "Elliptical Juicer"
When I go to the product page for "Elliptical Juicer"
And I add the product to my shopping cart
Then I should see 1 product in my shopping cart
Scenario: Guest with a product in their shopping cart attempts to check out
Given I have a product in my shopping cart
When I request to check out
Then I should see the account creation form
Scenario: Guest creates an account
Given I have a product named "Elliptical Juicer" in my shopping cart
And I am on the account creation form
When I create an account
Then I should see the checkout form with 1 product, "Elliptical Juicer"
Scenario: Newly registered user checks out
Given I am a user
And I have a product named "Elliptical Juicer" in my shopping cart
And I am on the checkout form
When I check out
Then I should see the checkout success page with 1 product, "Elliptical Juicer"
And I should receive a checkout confirmation email with 1 product, "Elliptical Juicer"
That's awful! First, none of the scenarios is what a stakeholder would think of as a scenario. Second, when one of the intermediate states changes, two steps will have to change: the step which asserts the intermediate state and the Given step which sets up the intermediate state for the next scenario. Each of those Given steps is an opportunity to set up the wrong state, i.e. make an integration error. This set of scenarios has much less value as an integration test suite than did the single scenario. You might almost have written a series of unit tests.
It's true that writing every scenario end-to-end is likely to lead to some duplication. Just as you tolerate duplication more in unit tests than you would in regular code, tolerate duplication even more in Gherkin scenarios than you would in unit tests. Don't compromise on understandability. Break up scenarios and use Givens only at crucial points (such as creation of a product in the example above), and do so knowing that you're diluting your scenarios' integration-testing power.
Also, keep in mind that acceptance tests should be only part of your automated test suite. Write only enough acceptance tests to cover critical scenarios, and cover the details with unit tests. Often enough, the solution to duplication among acceptance tests is to replace one with a unit test.
I would also say No.
In a separate post of mine Daniel F found this fantastic article. Here is the relevant section:
Given-When-Then steps must appear in order and cannot repeat. A Given may not follow a When or Then, and a When may not follow a Then. The reason is simple: any single When-Then pair denotes an individual behavior. This makes it easy to see how, in the test above, there are actually two behaviors covered: (1) searching from the search bar, and (2) performing an image search. In Gherkin, one scenario covers one behavior. Thus, there should be two scenarios instead of one. Any time you want to write more than one When-Then pair, write separate scenarios instead. (Note: Some BDD frameworks may allow disordered steps, but it would nevertheless be anti-behavioral.)
https://automationpanda.com/2017/01/30/bdd-101-writing-good-gherkin/
I would say no.
When a test fails it should tell you where in your system the failure has occurred.
Long tests like in your example tend to be brittle and require a higher level of maintenance.
You need to define what your test is testing (which should be one thing) reading your test
it could be a form validation test.
it could be a registration test.
it could be a user dashboard test.
It would require an amount of time to investigate where the failure is and where that relates to in the code.
I would also say No.
The Given is a precondition for setup.
The When is an action (which can be a do nothing)
The Then form asserts.
If you need more actions then break the test down.
This will become far more useful once the first Then's fail for localising the problems.