We have the following UI as shown in the image. These parameters are cascaded i.e they are inter-dependent. If you select continent then respective countries will come and then when you select country respective city will come.
I want to automate testing of each option. This was just a dummy UI. In my case these fields are dynamic i.e generated on the fly through shell/groovy scripts and I have more than 10 such fields.
I have seen Robot Framework and Job-DSL Plugin but I am not able to write test cases for these option selection. Also I have seen some tools which record your steps and generate a test file according to steps performed based on option selected and buttons clicked?
Can some one guide me for the optimum tool or platform so as to do Automation testing?
It's hard to say what is 'the optimal way' but here's what I would do:
I assume that all selections are based on the Jenkins jelly calling a method in your code, I suggest you put the effort in combining these calls in a normal unit test first. There you can try all possibilities in a much faster way.
Then when it comes to real UI test, record a Selenium session and translate that into the source code of your choice.
Related
I have several tests to run and all of them share a certain number x of initial actions (say login, fill form fields, click buttons, etc.), then they diverge.
Is it possible to let the browser execute the first x actions just once, save the current state and then execute all the test separately (in parallel if possible), each one with a separate browser instance?
Thanks
You should try to avoid duplicating effort in your tests. However, you must aim for consistency above all, and maintainability is probably just as important.
What that means is that using the browser in a way a real user wouldn't (I think your state-saving idea counts) is very risky for consistency, and may fail to give you the meaningful results you need.
Another alternative - a 'monolithic' test that attempts to cover multiple scenarios within one user session - is also problematic, because it's slower to run and slower to write and debug.
To be honest I think the idea of "browser state" is one that isn't a good fit for the real web.
My suggestion is to run dedicated, self-contained, clean tests - even if they do duplicate things like login/registration forms. However, if it is important to minimise the length of your test runs, try running them in parallel: ideally on multiple VMs, or via Selenium Grid.
I read Bob Martin's brilliant article on how "Given-When-Then" can actual be compared to an FSM. It got me thinking. Is it OK for a BDD test to have multiple "When"s?
For eg.
GIVEN my system is in a defined state
WHEN an event A occurs
AND an event B occurs
AND an event C occurs
THEN my system should behave in this manner
I personally think these should be 3 different tests for good separation of intent. But other than that, are there any compelling reasons for or against this approach?
When multiple steps (WHEN) are needed before you do your actual assertion (THEN), I prefer to group them in the initial condition part (GIVEN) and keep only one in the WHEN section. This kind of shows that the event that really triggers the "action" of my SUT is this one, and that the previous one are more steps to get there.
Your test would become:
GIVEN my system is in a defined state
AND an event A occurs
AND an event B occurs
WHEN an event C occurs
THEN my system should behave in this manner
but this is more of a personal preference I guess.
If you truly need to test that a system behaves in a particular manner under those specific conditions, it's a perfectly acceptable way to write a test.
I found that the other limiting factor could be in an E2E testing scenario that you would like to reuse a statement multiple times. In my case the BDD framework of my choice(pytest_bdd) is implemented in a way that a given statement can have a singular return value and it maps the then input parameters automagically by the name of the function that was mapped to the given step. Now this design prevents reusability whereas in my case I wanted that. In short I needed to create objects and add them to a sequence object provided by another given statement. The way I worked around this limitation is by using a test fixture(which I named test_context), which was a python dictionary(a hashmap) and used when statements that don't have same singular requirement so the '(when)add object to sequence' step looked up the sequence in the context and appended the object in question to it. So now I could reuse the add object to sequence action multiple times.
This requirement was tricky because BDD aims to be descriptive. So I could have used a single given statement with the pickled memory map of the sequence object that I wanted to perform test action on. BUT would it have been useful? I think not. I needed to get the sequence constructed first and that needed reusable statements. And although this is not in the BDD bible I think in the end it is a practical and pragmatic solution to a very real E2E descriptive testing problem.
I am looking to create a Selenium page object for the menu bar of our application. However, the menu items will be different based on the license that the customer has (about 70% will be the same throughout). I am looking at a couple of approaches (I'm using Java):
Create all the methods for all the possible menu items in once class, and then just depend on the test case writer to use the correct methods.
Create a different page object for each different license (about 5 as of right now).
Which one would be better in the long term?
Thanks.
I have the exact same situation for my current job. I use the option #1 as it is the most maintable on the long term. Others solutions will add extra complexity which is not justified.
I'm always testing with "full" licence (the operator can do anything). Then, I have some specific tests to test the GUI is only showing what it is supposed to show with limited licence.
The page object I use will fail if called on a menu that does not exist on the screen (something that turn my test cases red). There are methods on my framework to determine the presence (or not) of a menu.
Note: If you are testing the security of your software (to ensure the operator cannot gain extra access), you need to use other technique. For such test you have to bpass the GUI and "attack" your server directly.
Why not create an enumerator for the license types then pass that enum into your method calls on the Page Object? The method could then use if/else or a switch to return appropriate values or take appropriate action based on the given license.
I suggest to use Java Inheritance
Create a class with the basic account and extend it for complex accounts.
See example here: Java docs: An Example of Inheritance
I'm new to Selenium, and also fuzz testing. I see that Selenium IDE only allows the fixed test cases. But then fuzz testing seems to be helpful.
So what's behind a fuzz testing, what kind of tests does Selenium offer, is this a black box or white box testing.
Any help would be appreciated.
For a short answer:
Selenium is mostly about black-box testing, but you could do some whiter testing also with Selenium.
Selenium RC gives you much more freedom to do fuzz testing than Selenium IDE.
For a long answer, see below:
In this post I would try to explain the concept of randomly testing your web application using Selenium RC.
Normally speaking, a black-box testing technique like Selenium gives you a good freedom to
(1) Enter any value to a certain field
(2) Choose any field to test in a certain HTML form
(3) Choose any execution order/step to test a certain set of fields.
Basically you
use (1) to test a specific field in your HTML form (did you choose a good maximum length for a field), your JavaScript handling of that field's value (e.g. turning "t" into today's date, turning "+1" into tomorrow's date), and your back end Database's handling of that variable (VARCHAR length, conversion of numerical string into numerical value, ...).
use (2) to test ALL possible fields
use (3) to test the interaction of the fields with each other: is there a JavaScript alert popped up if the username field was not entered before the password field, is there a database (e.g. Oracle) trigger "popped up" when certain condition is not met.
Note that testing EVERYTHING (all states of your program, constructed by possible combinations of all variables) is not possible even in theory (e.g.: consider testing your small function used to parse a string, then how many possible values does a string have ?). Therefore, in reality, given a limited resource (time, money, people) you want to test only the "most crucial" execution paths of your web application. A path is called more "crucial" if it has more of the properties: (a) is executed frequently, (b) a deviation from specification causes serious loss.
Unfortunately, it is hard to know which execution cases are crucial, unless you have recorded all use cases of your application and select the most frequent ones, which is a very time consuming process. Furthermore even some bugs at the least executed use case could cause a lot of trouble if it is a security hole (e.g. someone steals all customers' password given a tiny bug in an URL handling of some PHP page).
That is why you need to randomly scan the testing space (i.e. the space of values used in those use cases), with the hope to run-something-and-scan-everything. This is called fuzz testing.
Using Selenium RC you could easily do all the phases (1), (2) and (3): testing any value in any field under any execution step by doing some programming in a supported language like Java, PHP, CSharp, Ruby, Perl, Python.
Following is the steps to do all these phases (1), (2) and (3):
Create list of your HTML fields so that you could easily iterate through them. If your HTML fields are not structurized enough (legacy reason), think of adding a new attribute that contains a specific id, e.g. selenium-id to your HTML element, to (1) simplify XPath formation, (2) speed up XPath resolution and (3) to avoid translation hassle. While choosing the value for these newly added selenium-id, you are free to help iterating while fuzzing by (a) using consecutive numbers, (b) using names that forms a consistency.
Create a random variable to control the step, say rand_step
Create a random variable to control the field, say rand_field
Eventually, create a random variable to control the value entered into a certain field, say rand_value.
Now, inside your fuzzing algorithm, iterate first through the values of rand_step, then with each such iteration, iterate through rand_field, then finally iterate through rand_value.
That said, fuzz testing helps to scan your whole application's use case values space after a limited execution time. It is said that "a plague of new vulnerabilities emerge that affected popular client-side applications including Microsoft Internet Explorer, Microsoft Word and Microsoft Excel; a large portion of these vulnerabilities were discovered through fuzzing"
But fuzz testing does not come without drawback. One if which is the ability to reproduce a test case given all those randomness. But you could easily overcome this limitation by either doing one of the following:
Generating the test cases before hand in a batch file to be used in a certain period of time, and apply this file gradually
Generating the test cases on the fly, together with logging down those cases
Logging down only the failed cases.
To answer more on if Selenium is black or white box.
Definitions about black-box and white-box
Black box: checks if one box (usually the whole app) delivers the correct outputs while being fed with inputs. Theoretically, your application is bug free if ALL possible input-output pairs are verified.
White box: checks the control flow of the source. Theoretically, your application is bug free if ALL execution paths are visited without problem.
But in real life, you cannot do ALL input-output pairs, nor ALL execution paths, because you always have limited resources in
Time
Money
People
With selenium: you mimic the user by entering a value or do a certain click on a web application, and you wait if the browser gives you the behavior you want. You don't know and don't care how the inner functionality of the web application actually work. That's why a typical Selenium testing is black-box testing
Is there a way to modularize JMeter tests.
I have recorded several use cases for our application. Each of them is in a separate thread group in the same test plan. To control the workflow I wrote some primitives (e.g. postprocessor elements) that are used in many of these thread groups.
Is there a way not to copy these elements into each thread group but to use some kind of referencing within the same test plan? What would also be helpful is a way to reference elements from a different file.
Does anybody have any solutions or workarounds. I guess I am not the only one trying to follow the DRY principle...
I think this post from Atlassian describes what you're after using Module controllers. I've not tried it myself yet, but have it on my list of things to do :)
http://blogs.atlassian.com/developer/2008/10/performance_testing_with_jmete.html
Jared
You can't do this with JMeter. The UI doesn't support it. The Workbench would be a perfect place to store those common elements but it's not saved in JMX.
However, you can parameterize just about anything so you can achieve similar effects. For example, we use the same regex post processor in several thread groups. Even though we can't share the processor, the whole expression is a parameter defined in the test plan, which is shared. We only need to change one place when the regex changes.
They are talking about saving Workbench in a future version of Jmeter. Once that's done, it's trivial to add some UI to refer to the element in Workbench.
Module controllers are useful for executing the same samples in different thread groups.
It is possible to use the same assertions in multiple thread groups very easily.
At your Test Plan level, create a set of User Defined variables with names like "Expected_Result_x". Then, in your response assertion, simply reference the variable name ${Expected_Result_x}. You would still need to add the assertion manually to every page you want a particular assertion on, but now you only have to change it one place if the assertion changes.