I've noticed a problem that seems to crop up with several of my otherwise cleanly organized Selenium test classes (though the problem could equally apply to unit or other feature/acceptance tests). I'll have a bunch of tests that are all logically grouped together:
login with valid password
login with valid password and go to page Foo
login with valid password and go to page Bar
login with valid password and then log out
etc.
All of these tests have "login" as part of them, so I add that to the setUp method:
self.login("fake#example.com", "testuser", "testpassword")
But then I need to add one more test to the suite:
fail to login with invalid password
And then I'm not sure what to do, because that test doesn't do the same login action as the others.
At this point, I'm left with several different options ... none of which I really like:
Start a whole other suite just for that last test (even though it logically belongs with the others)
Leave the setUp logic, but make my last test logout as its first action (which results in the test doing a pointless login/logout before even starting)
Leave the setUp logic, but add a line to the setup to skip the login if we're on the last test (ie. if self._testMethodName == "last test" ..., but this just feels hacky)
Take the login out of the setUp (but then I have login lines in every other test in the suite, which don't really add anything to the test)
My question is: which of the above approaches is the "best practice" (or is there a fifth option that I'm missing which is the real best practice)?
If you're testing the login page of the application, then why you're placing the tests inside the setup() method.
I believe setup method is used to set the environment for your tests. eg. Opening the browser, Loading the website/ application etc.,
You could use different Test method for testing the login page, and you can pass various inputs to test it.
Related
I'm writing some end-to-end tests for my application using Cucumber and Selenium. I'm keeping every scenario totally independent of one another, as advised on the Cucumber website. However, my application has session based authentication, so every time a new scenario is run, it will required a login process in order to first access the site. Right now my approach is to put the login scenario as a background scenario for all other scenarios, like so:
Background: User is Logged In
Given I am on the login screen
When I enter my login details
And I click submit
Then I should be logged in
However, this feels like a lot of duplicated 'code'. Moreover, having each scenario run independently requires a new WebDriver instance being created and a browser being run for each scenario, which feels a bit inefficient?
Can anyone advise firstly on how I can avoid duplicating the background scenario in every other scenario (if possible) and secondly if having a separate WebDriver instance for each scenario is the correct approach?
First of all each of your scenarios is going to have to login and create a new session. Thats the price you pay to do end to end testing. The cost of this in runtime should be relatively small with a standard login process as most login screens are simple, and have short rendering times and very little db access. It really is a very bad idea to tray and save runtime here by trying to share sessions between scenarios.
To start tidying your cukes you could follow Daniel's answer, but instead of nesting the steps I would recommend extracting the code to a helper method and calling that instead.
To do this elegantly with power, i.e. deal with different users with roles and extra attributes you need to do a bit more. You can see a detailed example of this here (https://github.com/diabolo/cuke_up).
To use this effectively follow the commit history and focus mostly on the features folder.
This has some framework code that allows you to register/create users and then use them in lots of different configurations. There is some underlying code that is a little complex, which gives you the power to create users who know their own passwords, as well as add other attributes like roles.
The end result is that you can write a step definitions like
Given I am registered
Given I am an admin
Given I am logged in
Given I am logged in as an admin
which are implemented as
Given 'I am registered' do
#i = create_user
end
Given 'I am an admin' do
#i = create_user role: 'admin'
end
Given 'I am logged in' do
#i = create_user
login as: #i
end
Given 'I am logged in as an admin' do
#i = create_user role: 'admin'
login as: #i
end
note: the variable #i is used to pass the user from one step to the next
You don't have to worry about repetition here as all the steps make calls to the same helper methods. You can use the patterns shown here in a much wider context to simplify your features/
This example is in Ruby. You can group up the steps used for Login in login_steps.rb file.
In the .feature file you'll need to write a step like "Given the user is logged in". You can pass in login data in this step as well, if you want. Then in the login_steps.rb file, you create:
Given(/^the user is logged in$/) do
step('I am on the login screen')
step('I enter my login details')
step('I click submit')
step('I should be logged in')
end
I'm sure you can find the equivalent in any other language. Now you can write a the background like:
Background: Given the user is logged in
and it will be used before each scenario of that specific .feature file
As for the Webdriver, as far as I'm aware, you create a session when the test starts and you quit when it ends.
Hope it helps!
I'm new to Gherking and trying to write my first scenarios as best as I can, but I regularly find myself in situations where I'm really tempted to add an "Else" to my scenario. "Given, When, Then" becomes "Given, When, Then, Else". I know that the "Else" keyword is not defined and so not implemented in Gherkin tools but it doesn't care to me because I don't use these tools.
Do you think it is correct to write this :
Example :
Scenario : Application starts
Given I start the application
When I already have an open session
Then I see the home screen
Else I see the login screen
Or is it better to write two different scenarios :
Scenario : Application started by authenticated user
Given I have an open session
When I start the application
Then I see the home screen
Scenario : Application started by unauthenticated user
Given I don't have an open session
When I start the application
Then I see the login screen
In short no, but here are options to handle multiple variants of a scenario:
If it was only tailing elements of the scenario steps that differed you could of moved early steps in to a common 'Background' section, making repeated differing scenarios shorter and clearer.
But from your example it is all steps differing slightly so you can:-
accept the repitition of multiple scenarios
Or
parametise the differences and use data tables in the 'given' and 'then' sections to give before and after values.
Or (my prefernece)
Use the 'scenario outline' syntax that uses an examples table to provide sets of data fixtures with their expected results. These replace in the scenario steps as runtime. The scenario is then 'played out' once for each row in the examples table.
So:
Scenario : Application started by authenticated user
Given I have an open session
When I start the application
Then I see the home screen
Scenario : Application started by unauthenticated user
Given I don't have an open session
When I start the application
Then I see the login screen
Becomes:
Scenario Outline: Application Start and login
Given Application started by <AuthenticationStatus> user
And I have <SessionState> session
When I start the application
Then I see the <FirstScreen> screen
Examples:
|AuthenticationStatus |SessionState |FirstScreen|
|Authenticated |open |home |
|Un-Authenticated |not open |login |
IMHO for 2 scenarios it might not be worth the loss of readabiltiy but for more than that I think it's definitely worth it.
Using Chimp.js, Cucumberjs and WebdriverIO, I'm trying to run login step definitions in a browser instance, before other step definitions that depend on a user to be logged in. And possibly without adding them into the Background over and over again for each feature file.
Is this possible? I'm quite new to Wedbdriver.io and Cucumber and any advice would be a great help. Please let me know if more info is needed.
Personally I don't think this is a good idea. To log someone in you have to specify 'who' the user is. Later when your application becomes more complex you might have interactions between different users. Hiding any of this from the scenario is not good.
What you can do is combine user specification and login in single steps e.g.
Given I am logged in as an admin
Given Fred is logged in as a sales executive
etc.
If you are clever about how you implement these steps you can keep things fairly dry by extracting helper methods from the step definitions and using global variables to store people e.g.
Given 'I am logged in as an admin' do
#i = create_user role: admin
login as: admin, user: #i
and reuse these methods in other login steps.
If you organise your features well, you can background alot of these calls e.g.
Feature: Basic admin ops
Background:
Given I am logged in as an admin
Scenario: I can foo
When I foo
Scenario: I can bar
When I bar
Some final thoughts ...
Each scenario is there to drive a particular piece of development. Compared to the work of doing the development writing "Given I am logged in" is trivial.
When something goes wrong knowing that you were supposed to be logged in is an essential piece of information.
Suppose I have a web page with several links on it. Also it has few buttons which execute some JavaScript.
So should I create one Java class for testing each of these links and elements or should I test all the links in just one test method and other elements in another one(so ending up with two Scripts).
Is there a another way of gouping these tests.
Thank you.
I have found that writing test cases based on actions is much more useful than writing based on pages.
Obviously, we would love to have everything automated. But realistically, this isn't possible. So we test what is most important...which happens to be: 1. The primary purposes of the product you are testing, and 2. The security of the product.
To make this easier to understand, lets say I have a Checkout page.
If I were to test based on a page, I would make sure every link on the page would work. I would verify that I can type in a new number in the quantity field, and make sure that the page verifies that the credit card number I type in is correct.
This method would somewhat test Security, but beyond that, it would test the UI. What would happen if I clicked on the Checkout button, and I was sent to the right page, but the item I was trying to checkout disappeared? That is a huge problem, but the test based on the page would work.
However, if I were to test based on actions (go to this page, add to cart, type in personal information, and then checkout), now I have made sure that the most important part of your program works, checked security, and even a little UI testing.
All in all, write your testing to do what the average user would do. No normal person is going to sit on the same page, testing out every little feature on that page.
It depends on whether you like to see 1/1 tests passed or 2/2 tests passed.
I'm currently writing some functional tests, and I started wondering what's the best philosophy between these two.
Situation
My application has some secured page, that need the user's group to have the right credentials to have access. These user are split into 2 groups : the 'collaborator group', and the 'accountable group'. Credentials are given to the groups.
Possible philosophies
Solution 1: Tests the credentials a.k.a. Test the features.
For each secured page, I test the
access with 2 users : one with the
right credential, and only this one,
and one without the right credential.
Pros: Tests only the fact that the page is secured against a specific credential
Cons: Doesn't test the "final" application behavior, as wanted (and user) by the client.
Solution 2: Test the groups a.k.a. test the requirements
For each secured page, I test the
access with a user of each group, and
check that only the allowed groups
gain access to the secured page.
Pros: Tests the "final" application behavior, as wanted (and user) by the client.
Cons:
Tests are tieds with the tests fixtures
Tests will have to change if the business rules changes or if more groups are created.
Thank you.
I think the second solution is the good one. The credentials will be tested as far as they are associated with a group.
Pros: Tests the "final" application behavior, as wanted (and user) by the client.
This is the most important part. Functional tests aims to test the final application in every possible cases. If you want to test the fact that your credentials have the same behavior with a user or a group, you'd better use unit tests.
Cons: Tests will have to change if the business rules changes or if more groups are created.
Your tests cases will always have to be updated if the business of your application changes. As you do with your unit tests. If your modify the code of a function, you check if your unit tests are still able to control each case. It's the same way with functional tests.
Maintaining your tests (and the fixtures they need to run) is a very tedious task, but it's the only way to ensure you're code is strong.
Hope it helped.
I would do both tests. The first one like you pointed out, does not need updating, but is testing the crucially important fact that users without entitlements do not have access. The second is the more comprehensive test and like #TimotheeMartin pointed out, tests will always need to be updated when the code changes.