Can we write " Given As a user I can't (cannot) login into the system" in Gherkin as a negative sentence?
You can definitely write negative phrases, but the example sentence is too broad. One alternative may be this (somewhat more realistic) scenario: "As a non-admin user, I cannot log into the admin page."
But that gives us no information about how to deal with this situation. Ideally when this happens, you want to state what should happen instead of letting the user log in.
This can be written as:
Given user type is not admin
When user tries to log in
Then throw error/redirect to a different page/break the whole internet and set the world aflame etc.
Related
Goal:
Determine if a functional test was successful.
Scenario:
We have a functional requirement: "A user should be able to signup with username and password. The username has to be a valid email-adress. The password has to be at least 8 characters long".
We have a method "SignupResult UserManager.Signup(string username, string password)".
We want a happy-test with valid intputs, and a sad-test with invalid inputs.
Sub-Systems of the UserManager (e.g. Database) can be either mocked or real systems.
Question:
What would be the best way to determine if the user was successfully signed up. I can imagine the following options:
If any of the sub-system was mocked, one could check if a specific function like "DB.SaveUser(...)" was called. This would destroy the idea of a functional test being a blackbox test and requires that the test-writer has knowledge of the implementation.
If we use real sub-systems, one could for example check if the row in the DB exists. That would be not adequate like the attempt above.
One could use another function like "UserManager.CheckUser(...)" to check if the user was created. This would introduce another method that is tested, also there may be operations that would have no "test-counterpart", or one would have to implement them, just for testing - that seems not ideal.
We could check the result "SignupResult" and/or check for exceptions thrown. This would require defining the interface of the method. This also would require all methods to return a sensible value - I guess this will be a good approach anyway.
To me the last methods seems to be the way to go. Am I correct? Are there other approaches? How would we check side-effects like "an email was sent to the new user" ?
You may want to acquaint yourself with the concept of the Test Pyramid.
There's no single correct way to design and implement automated tests - only trade-offs.
If you absolutely must avoid any sort of knowledge of implementation details, there's really only way to go about it: test the actual system.
The problem with that is that automated tests tend to leave behind a trail of persistent state changes. For example, I once did something like what you're asking about and wrote a series of automated tests that used the actual system (a REST API) to sign up new users.
The operations people soon asked me to turn that system off, even though it only generated a small fraction of actual users.
You might think that the next-best thing would be a full systems test against some staging or test environment. Yes, but then you have to take it on faith that this environment sufficiently mirrors the actual production environment. How can you know that? By knowing something about implementation details. I don't see how you can avoid that.
If you accept that it's okay to know a little about implementation details, then it quickly becomes a question of how much knowledge is acceptable.
The experience behind the test pyramid is that unit tests are much easier to write and maintain than integration tests, which are again easier to write and maintain than systems tests.
I usually find that the sweet spot for these kinds of tests are self-hosted state-based tests where only the actual system dependencies such as databases or email servers are replaced with Fakes (not Mocks).
Perhaps it is the requirement that needs further refinement.
For instance, what precisely would your user do to verify if she has signed up correctly? How would she know? I imagine she'd look at the response from the system: "account successfully created". Then she'd only know that the system posts a message in response to that valid creation attempt.
Testing for the posted message is actionable, just having a created account is not. This is acceptable as a more specific test, at a lower test level.
So think about why exactly users should register? Just to see response? How about the requirement:
When a user signs up with a valid username and a valid password, then she should be able to successfully log into the system using the combination of that username and password.
Then one can add a definition of a successful login, just like the definitions of validity of the username and password.
This is actionable, without knowing specifics about internals. It should be acceptable as far as system integration tests go.
I'm writing some end-to-end tests for my application using Cucumber and Selenium. I'm keeping every scenario totally independent of one another, as advised on the Cucumber website. However, my application has session based authentication, so every time a new scenario is run, it will required a login process in order to first access the site. Right now my approach is to put the login scenario as a background scenario for all other scenarios, like so:
Background: User is Logged In
Given I am on the login screen
When I enter my login details
And I click submit
Then I should be logged in
However, this feels like a lot of duplicated 'code'. Moreover, having each scenario run independently requires a new WebDriver instance being created and a browser being run for each scenario, which feels a bit inefficient?
Can anyone advise firstly on how I can avoid duplicating the background scenario in every other scenario (if possible) and secondly if having a separate WebDriver instance for each scenario is the correct approach?
First of all each of your scenarios is going to have to login and create a new session. Thats the price you pay to do end to end testing. The cost of this in runtime should be relatively small with a standard login process as most login screens are simple, and have short rendering times and very little db access. It really is a very bad idea to tray and save runtime here by trying to share sessions between scenarios.
To start tidying your cukes you could follow Daniel's answer, but instead of nesting the steps I would recommend extracting the code to a helper method and calling that instead.
To do this elegantly with power, i.e. deal with different users with roles and extra attributes you need to do a bit more. You can see a detailed example of this here (https://github.com/diabolo/cuke_up).
To use this effectively follow the commit history and focus mostly on the features folder.
This has some framework code that allows you to register/create users and then use them in lots of different configurations. There is some underlying code that is a little complex, which gives you the power to create users who know their own passwords, as well as add other attributes like roles.
The end result is that you can write a step definitions like
Given I am registered
Given I am an admin
Given I am logged in
Given I am logged in as an admin
which are implemented as
Given 'I am registered' do
#i = create_user
end
Given 'I am an admin' do
#i = create_user role: 'admin'
end
Given 'I am logged in' do
#i = create_user
login as: #i
end
Given 'I am logged in as an admin' do
#i = create_user role: 'admin'
login as: #i
end
note: the variable #i is used to pass the user from one step to the next
You don't have to worry about repetition here as all the steps make calls to the same helper methods. You can use the patterns shown here in a much wider context to simplify your features/
This example is in Ruby. You can group up the steps used for Login in login_steps.rb file.
In the .feature file you'll need to write a step like "Given the user is logged in". You can pass in login data in this step as well, if you want. Then in the login_steps.rb file, you create:
Given(/^the user is logged in$/) do
step('I am on the login screen')
step('I enter my login details')
step('I click submit')
step('I should be logged in')
end
I'm sure you can find the equivalent in any other language. Now you can write a the background like:
Background: Given the user is logged in
and it will be used before each scenario of that specific .feature file
As for the Webdriver, as far as I'm aware, you create a session when the test starts and you quit when it ends.
Hope it helps!
For something that REALLY needs to be set up before every scenario then BeforeScenario is used.
Sometimes there are things that only need to be set up for some scenarios, but for a significant proportion of them. For example, if the scenario needs a "regular user account" to exist, then it goes on to login as that user and do some stuff.
You can make an #BeforeScenario #RegularUser method that will run for each Scenario tagged as #RegularUser. So scenarios end up looking like:
#RegularUser
Scenario login as a regular user
Given I am on the login page
When I login to a regular user account
Then the welcome screen is displayed
The alternative is to
Scenario login as a regular user
Given a regular user exists
And I am on the login page
When I login to a regular user account
Then the welcome screen is displayed
"a regular user exists" would be associated with the method that creates the regular user.
With the first approach, I can make #AfterScenario #RegularUser that will delete the user after the scenario ends. So that seems "a good thing".
But the 2nd approach kind of looks nicer to me from a BDD viewpoint. But its limitation is that then there is nothing happening at the end of the scenario to delete the user.
Which approach is the better practice?
In my opinion, i would go with first option.
The problem is that, both options are not so good, because normally you have context of this scenario added:
In order to ..
As a Regular User
I need to ..
That should show what was you intention of running this scenario. The process of creating RegularUser don't adding value to scenario itself.
Then, if you need to confirmation in scenario of having regular user - then is OK. But in here in my opinion, it is not the point - then second option is not so good - we have context to add informations like that.
That is why, I think first option (not ideal, but better that second) is good solution.
I don't know in Behat functionality to set up user by context, but tag option should be more elastic.
Using Chimp.js, Cucumberjs and WebdriverIO, I'm trying to run login step definitions in a browser instance, before other step definitions that depend on a user to be logged in. And possibly without adding them into the Background over and over again for each feature file.
Is this possible? I'm quite new to Wedbdriver.io and Cucumber and any advice would be a great help. Please let me know if more info is needed.
Personally I don't think this is a good idea. To log someone in you have to specify 'who' the user is. Later when your application becomes more complex you might have interactions between different users. Hiding any of this from the scenario is not good.
What you can do is combine user specification and login in single steps e.g.
Given I am logged in as an admin
Given Fred is logged in as a sales executive
etc.
If you are clever about how you implement these steps you can keep things fairly dry by extracting helper methods from the step definitions and using global variables to store people e.g.
Given 'I am logged in as an admin' do
#i = create_user role: admin
login as: admin, user: #i
and reuse these methods in other login steps.
If you organise your features well, you can background alot of these calls e.g.
Feature: Basic admin ops
Background:
Given I am logged in as an admin
Scenario: I can foo
When I foo
Scenario: I can bar
When I bar
Some final thoughts ...
Each scenario is there to drive a particular piece of development. Compared to the work of doing the development writing "Given I am logged in" is trivial.
When something goes wrong knowing that you were supposed to be logged in is an essential piece of information.
I'm writing acceptance tests in Gherkin where I want to test for multiple changes in the UI of a web app based on an initial action. Here's an example:
Scenario: Cancel editing a new text asset
Given the user "test_user#fakedomain.com" is logged in
When the user navigates to "/build/"
And the user clicks the "Sandbox" link
And the user inputs "Test story for canceling editing of a new text asset" for the "title" field
And the user inputs "Test User" for the "byline" field
And the user inputs "My summary, so exciting!" for the "summary" textarea
And the user clicks on "Untitled Section" in the section list
And the user clicks the "Text" icon in the "center" container
And the user inputs the following text in the rich text editor:
"""
Test text for asset. This is cool.
"""
And the user clicks the "cancel" button
Then the following text is not present:
"""
Test text for asset. This is cool.
"""
And the "Image" icon is present
And the "Text" icon is present
When the user refreshes the browser
And the user clicks on "Untitled Section" in the section list
Then the following text is not present:
"""
Test text for asset. This is cool.
"""
When the user opens the asset drawer
Then the following text is not present:
"""
Test text for asset. This is cool.
"""
Note that there are multiple groups of When/Then steps, to test for responses of the initial action. While most implementations of steps ignore the prefix keyword, and I'm pretty sure that I can get this test to run, is there a better way to test the different outcomes? Is it better practice to write multiple scenarios with the same setup but different "Then" statements?
Remember that you should test only ONE behaviour/feature at a time.
The rule of thumb is that you should use only one When step:
Given some state before action
And some other state before action
...
When only one action
Then assert correct output
And assert correct output
...
You see - only one line of When, without any Ands under When. If you use many When steps instead, you create test script, not a specification. Your tests will be hard to understand and you will notice that you add more and more steps when the underlying implementation changes.
You also need to keep underlying logic hidden, because you don't want to change it every time you change something irrelevant. Example:
And the user inputs "My summary, so exciting!" for the "summary"
textarea
What if you change the summary field from a textarea to an input type? You have to change the scenario (maintenance nightmare) or left you scenario lying (worse than not having scenario). You should write instead:
When the user describes it as "so exciting!"
But still, the structure of the whole scenario is bad. Ask yourself the question: what I want to check? If I were a person that wants to understand the business logic of the feature I would like to see something like:
Scenario: Cancel editing a new text asset
Given I edit the Sandbox details with some data
When I cancel editing
Then Sandox details should be empty
That's it!
How to achieve it? Move all irrelevant logic deeper, use the PageObject pattern etc. And read about Specification By Example
In contrast to the previous answer, there is no strict rule on how one can use define the steps. Cucumber official documentation at https://cucumber.io/docs/gherkin/reference/ recommends to use only one When due to the fact that only behavior is listed in one acceptance criteria. This is just a recommendation, but not a rule. And Cucumber documentation is mostly focused on the Acceptance criteria specification, but not the test automation.
So, it is definitely possible to follow how it best suits your requirement when it comes to automation testing.
I'd recommend using fewer "And" keywords by merging different steps. Below points I recommend (It is a recommendation, but not a rule :) ):
Use only one Given, When and Then keywords in one flow of a scenario, use "And" keyword if you need to specify extra steps at the appropriate event
If you come across using too many "And" keywords, try to merge multiple such steps
In reality, we cannot just use only one When in the test automation, because:
Number of tests in the automation increases
Total execution time for the automation goes up
We may be doing a lot of redundant actions across multiple scenarios if we use only one When. Consider that you need to perform 5 steps as the initial condition for testing 10 different actions. Here, you will perform these 5 steps for 10 times when you use only one "When" - This consumes too much time, and causes the tests to be unstable, and causes more load on your application
Due to the increased number of tests, we need to spend more time to analyze the results, more time to maintain
I also recommend testing the behavior based on the requirement. If your requirement is about verifying something in the "test area" rather than the "input area", your step should indicate that. Remember that if the requirement is changed, development is code changed, and hence the test automation code is also changed.