I have a group that has some non-technical people creating tests and test suites using the Selenium IDE. I'd like for that group to be able to work independently, yet after the fact be able to run a series of those suites with minimal button clicks. There are a lot of reasons why I'm not just writing tests using some 'native' language (groovy or java) and making this easy to use for the team will help adoption of testing.
So, I would like to be able to just instruct the members of the team to open a single 'suite' (or equivalent) and run it and it would then run each of the suites that I have designated as part of the 'master suite' (if you follow me).
I know that I could just maintain a list of the suites that are part of our automated tests, but it would be easier for me to sell if it was possible to just open up a single file and click 'go' and then walk away and see the results after coming back from getting a cup of coffee or something.
If your reason for not going with a native language is because of your non-technical people, then your automation strategy will fail.
Sorry for being blunt, but there is a reason why there is the IDE, and there is the native language support. They both serve very different purpose, and if you don't approach automation with the respect that it's a programming exercise, then your automation strategy will fail.
Selenium IDE is extremely limited. You are unable to string multiple test suites together. You only have the option of creating one huge a** list of test cases in one suite, or loading your suites in one at a time.
Go WebDriver - everything that you want to do, is extremely limited, if even possible using the IDE.
Yes, I wrote a framework that does that. You can record as many "Selenium Builder" scripts as you like and they will be ran by my framework in a multi-threaded fashion as a group. Just fork your own copy of my framework and then modify it to your uses.
Related
Why can't we just record all of our test cases in Selenium IDE, export it to Java/WebDriver and run it in WebDriver with Eclipse?
I need a clear explanation as I am very much confused in using WebDriver!
And can anyone please explain why IDE recorded scripts fail in WebDriver?
why cant we just record all of our test cases in IDE, export it to java/webdriver and run it in webdriver
Great question, and here is the answer:
Selenium IDE is a Record and Playback tool, which is very easy to use, but it's very unreliable. Record and playback is typically a frowned upon in web applications. Since web applications are frequently changed, the IDE is not an ideal solution for a production environment, because of the maintenance nightmare that may arise.
Let me give you a practical example. You record your test, and you find an element with a dynamic ID. Sure we can import it into eclipse, but what happens when that test starts failing down the road? why not simply make your test agile and independent to catch these in the first place.
It also boils down to your principles of test automation. Test automation in MY opinion (and several other professionals), believe that test automation should be approached from a programming perspective. Programmers should write the tests, and maintain the tests. Ideally, your quality assurance personnel should be trained to write and maintain their own tests.
So again, back to your question, the IDE is designed to be a quick solution to automation, NOT a solution to a full regression suite.
And can anyone please explain why IDE recorded scripts fail in Webdriver?
I haven't used the IDE in a while, but the reason they fail, is because the scripts that are exported, are simply the steps, not an entire java file. This also is because Selenium IDE exportations are supposed to be agnostic when it comes to how to run your test. Say I'm a user of jUnit.. what if Selenium IDE exported it to TestNG all the time? That wouldn't be fair.. honestly i'd rather create my own tests than changing that one line every single time i create my test file.
You may read the full text of a research conducted, called Why do Record/Replay Tests
of Web Applications Break?
Why can't we just record all of our test cases in Selenium IDE, export it to Java/WebDriver and run it in WebDriver with Eclipse.
You can actually do this with Selenium IDE quite easily. Record your test case / test suite in Selenium IDE, export to "Java / JUnit 4 / Webdriver" to a .java file. This will generate a JUnit test that you can import and run from Eclipse (with the correct version of JUnit of course).
It's not 100% reliable, and you may need to make some manual changes/corrections, but in general it works pretty well. Start with a single small testcase and work from there.
Why not just record in the IDE and play the recording with WebDriver???
The IDE Does Not Know What to Wait For
Suppose Bob is performing manual checks and recording his interaction with Selenium IDE. He performs an operation that takes a while to update the GUI, and when the operation has finished, he clicks a button. Bob knows that the GUI has finished updating because a spinner that was show when the operation started is removed when the operation is finished. How is the IDE going to capture the fact that the operation must have finished before clicking on the button? (Note here this is not a case where the button is disabled until the operation is finished.) This is not a hypothetical: when you test dynamic tables (like those managed by DataTables), the user can change anything at any time.
This can be one of the reasons a sequence of commands created with the IDE would fail with WebDriver. It just does not know what to wait for.
It is possible to add waits manually in the IDE but if you are doing this, then you are no longer "just record[ing] all of our test cases". And what you are doing becomes more like writing code for WebDriver.
Users are Inefficient
Not long ago there was a Selenium question in which the user wanted to get Selenium to click on the last record in a table that had multiple pages of records. So the question was, how can I page through the table, all the way down to the last page and then click the last record? Someone (maybe me, maybe someone else) pointed out that if the table is sortable, it could be sorted in reverse order and then the Selenium code could click on the first record. That's 2 operations rather than p+1 operations: clicking p times, where p is the number of pages, plus 1 time for the click on the record.
When we write code for WebDriver, we have the opportunity to write the tests to avoid taking the scenic route to get the results we want. See the next point for the technical details as to why it matters.
Selenium Operations Can Be Costly
If the software that runs your Selenium commands is local, and your browser is local, you may not feel that Selenium operations can be costly but if you run your commands on one machine and the browser is remote, then you will notice a significant slowdown. For instance, if you spawn your browser in a Sauce Labs VM or in a BrowserStack VM to run a test suite, network delays are going to add significant time to how long it takes the suite to complete. For a complete application test suite, this can mean many minutes more.
The IDE produces a sequence of commands that each require a round-trip between the Selenium script and the browser. Each round-trip adds up. Suppose I want to check that two elements contain the same text. I'm not going to use Selenese because I don't usually use the IDE but using WebDriver code in Python, the script could be:
a = driver.find_element_by_id("a")
b = driver.find_element_by_id("b")
assert_equal(a.text, b.text)
This code requires 4 round-trips: one round-trip per find_element... and one per access to the text field. The same test could be written:
a_text, b_text = driver.execute_script("""
var a = document.getElementById("a");
var b = document.getElementById("b");
return [a.textcontent, b.textContent];
""");
assert_equal(a_text, b_text);
This needs only one round-trip. When you use the IDE to record actions, the sequence of commands is like the earlier snippet: lots of round-trips. When you write your code for WebDriver, you have the opportunity to optimize as you code.
This does not mean that the Selenium IDE has no use but I would never ever think of just recording tests with it and then playing those recordings with WebDriver.
If you are a newbie, you can actually use 80% of your IDE script as your webdriver JAVA script, u just have to slightly improvise your exported IDE script and it will work fine.
But when you start testing complex functionalities you have to learn some of the basic java methods to work it out!
I beg to differ. I see no reason in having to maintain extra code when a simple record and playback can achieve same results in a fraction of the time.
I do agree with the impl of the recorded script has to be agile. It can't just be as simple as the build in commands and one has to make it agile and even inplement their own macros (aka commands/aliases) specific to the app under test.
Still, if implemented right and with the help of few Selenium IDE plugins like flow control it becomes a breeze to maintain such complex test suits and have them played back via the HTML Runner. I've used Selenium IDE HTML scripts to fully cover complex sites and reused the same script for all responsive modes of the app.
Maintanance is simple as the IDE will help you quickly adjust a failed cmd and AJAX waits can easily be accounted for correctly by waiting for DOM content to change on the page. I'm still not convinced of this AJAX excuse for dismissing Selenium IDE.
For a Maven integration of flow controls command/aliases/macros used by Selenium IDE and Maven's SureFire test plugin see:
https://github.com/paulbors/sideflow
Feel free to use it in your product and improve as needed.
webdriver allows you to execute your test cases captured with Selenium IDE in a Selenium server or via RC to multiple servers. This includes options to integrate with Selenium-enabled providers (see link at the bottom).
For example, if you've captured your test case in IDE, you could easily integrate with your own selenium server, or a provider like Saucelabs to handle the running of your test cases in Firefox on Ubuntu, Safari on OSX and IE10 on Windows 8. As with any test case, you can run it from your IDE (Eclipse / IDEA / etc ) your builder (maven / gradle /), and / or via your CI system.
Check out https://saucelabs.com/selenium for some examples. We've also used our selenium test cases for load testing with Soasta - WebDriver allows simple IDE tests to be used in many different contexts.
Because when the complexity of your code will increase, it will be difficult for you to manage it using IDE. Maintenance of your scripts will be difficult using IDE.
Also Exporting test cases from IDE to webdriver is not 100% reliable.
Other people have said a lot of helpful answers. One thing that is missing is that if you write code instead of using the IDE you have the power to make changes to the database. It depends on the website you are testing how useful this is, but here are some useful examples I've experienced. In one case the website had multiple sections of a class. One of our tests was merging these sections into one. The IDE couldn't do this. But with webdriver we can add all of our test data into the database, perform the merge, and then remove the data from the database.
Another advantage that webdriver has is Selenium Grid. With Grid you can run your tests in parallel and on multiple browsers. This makes your testing run faster and you can more thoroughly test the code.
I am new to Selenium and have spent about a week using IDE. So far, so good. However, some other parts of our application involve heavy drag-n-drop capabilities, and I'm not so sure about that...which brings me to my question...what type of specific things can WebDriver do that the IDE cannot do? I read the Selenium front page, and know that WebDriver can make more "robust" tests, but I'm not entirely clear what robust means in this context? Longer? More capabilities?
What I'm looking for is what type of particular tests could be run with WebDriver that couldn't be run in the IDE? And are any of them related to drag-n-drop? This will inform my decision about whether to keep on using the IDE or to switch over. I think it's better to use something from the beginning than to create a lot of tests and then switch over.
Thanks much in advance.
You can use the full features of your chosen language. For example, using the IDE you are stuck to using the IDE's API, however if you choose to use the WebDriver explicitly you can combine it with C# and use C#'s full features (such as LINQ).
A few of the IDE commands are not directly converted with the WebDriver.
You can integrate a pure WebDriver test into a CI (Continuous Integration) server.
It is not limited to just Firefox, whereas the IDE is (since the IDE is only available in Firefox).
You can improve and extend Selenium more easily. For example using Extension methods and inheritance. (For instance, extending Selenium so that whenever it cannot find an element, it takes a screenshot of the page)
It will be more robust, as the IDE will not always give you the most reliable way of finding an element. For instance, if the element you require is nested within a table, the XPath that the IDE will give you will infer directly to specific table rows and cells.
For instance something like this:
//table[1]/tr[1]/td[2]/table[1]/tr[1]/td[3]/a[1]/b
Is not the most reliable XPath in the world. Those without any knowledge of XPath or automated testing or how Selenium works, will stick with that and then waste hours figuring out why this XPath query fails. Others will change it, and just use the XPath as a base query.
The IDE should usually be used for those who don't have much programming knowledge (e.g some testers), if you want your developers and/or the test developer's to use Selenium, go directly with the WebDriver. It will feel more like programming with a fully-fledged language instead of scripting (after all, the scripts that the IDE generates are purely HTML files). The idea is generally for those who don't have much programming knowledge to simply hit 'play' and let Selenium do the work.
Drag-and-drop are applicable within the IDE and WebDriver.
Most of the development in Selenium (it seems) goes into extending the WebDriver code, the IDE may be left behind in some new features.
Using WebDriver directly, you can also use the full features of a unit testing framework (usually NUnit with C# or Junit/TestNG for Java). This allows things like repeating tests for each browser (i.e repeating tests in Chrome, Firefox and IE) or defining certain logic before and after tests (i.e adding user memberships, creating users, setting up other test data).
Fellow developers might be more inclined to help out if it is using a programming language as opposed to a scripting language (ie WebDriver is programmed in a programming language, the IDE scripts are just HTML files). Putting it in your solution and having it as part of the nightly build process helps to ensure developers help out in your tests progression.
Thinking about it, another example is a bug is fixed but introduces a new bug which causes your test to fail. Who's responsibility is it to fix the test? You or them? Having it use WebDriver directly, they can just look up the source code to Selenium and figure out the API usage themselves. With the IDE, it is literally a HTML file that calls some javascipt. Not much documentation on it.
To sum up: if you don't have much programming experience, either learn a language and use WebDriver with it directly, or just stick with the IDE. Either will do the job.
IDE cannot be used for serious automation. I would sussgest learn the API and use it with Java/C#/Python. It will help you churn out some good automation and will payback really well.
You can always use the IDE to learn how the code is generated.
Our codebase contains code in multiple languages, ranging from Python over C# to MATLAB and LaTeX. Currently we have unit tests in each individual language (using language-specific frameworks). This makes test automation cumbersome, especially collecting and checking all the different reports.
I am therefore looking for a test automation tool that
is portable (at least Windows + Linux)
is language-independent
can be extended (custom reporting, additional languages, ...)
Ideally the tool would connect via plugins to the existing tool-specific frameworks like Python's unittest, C#'s NUnit, etc.
Is there such a tool? If not, how do you handle test scenarios like this?
Perhaps you're looking for something like the Test Anything Protocol?
The Test Anything Protocol (TAP) is a protocol to allow communication between unit tests and a test harness. It allows individual tests (TAP producers) to communicate test results to the testing harness in a language-agnostic way.
It doesn't avoid the need for language-specific tests, but it does make sure that your test results are all in a uniform format, so that you can build cross-langauge reports. It's also fairly simple, so if your language of choice doesn't have a TAP producer, it is not very difficult to write one (and maybe share it back to the world).
If you go with TAP as already suggested, then a complete test infrastructure around that is Tapper.
The central server assumes some external Linux tools when you need its automation layer but besides that everything else is language and OS independent. It can receive TAP reports regardless how they were started: manual tests, crontab-driven, or the built-in automation which can set up machines from scratch or start tests via ssh.
See above link for more info, especially have a look at the linked presentations.
Summary:
TAP as protocol for writing tests
Tapper for the infrastructure
I have been working on Test Automation from last few months and have been using the tool named "Testcomplete". But I have noticed that the tools do not matter a lot in the field of automation. Only thing you expect from an automation tool is the ability of the tool to spit out the recognition strings for the different controls used in the test application.
Apart from this, you will always have to build a automation framework which will serve your needs writing code.
So my question is, Is my thinking that automation tools do not matter a lot in the field of automation correct? In the sense, you can use any tools to get your automation running. Or Do the tools really matter? (Please ignore the costs factor of the tools). Also if I need to learn a new automation tool, then what do I concentrate on? Or how do I go about learing the tool? In short, what exactly does "learning a tool" mean?
My 3 best reasons for choosing which tool to use:
it works. This is important, not all tools work in all scenarios i.e. flash, silver light, adobe air, legacy apps with no automation support, etc.
whole team skills. This includes not only testers, but also developers. Test automation shouldn't be an isolated effort, developers should also collaborate on it. This is far easier when dev and test are using the same language/platform.
price. Doesn't have to be free (but it could), but of course its an important factor.
Personally we use the same test runner as the one for the unit tests. That along with extra third party automation pieces that do the plumbing for you.
Some additional thoughts on why the tool is important:
Community - What's the user community like? Are there a lot of user-generated resources out there to help?
Support - (if vendor) What's customer support like? Do they fix problems quickly? Is it easy to find solutions to common problems?
Extensibility - Often in test automation, you'll need to roll your own or code work-arounds, if the tool does not support a particular type of object in your application. How easy is it to extend the product? What programming language does the tool use? What kind of support do you get from the IDE?
An other piece of advice: sometimes you'll need wrapper classes around certain frameworks. We were using WatiN, which was really good at its time, but it lacked Chrome (it had a small percentage that time) support. The thing that killed WatiN for us was the lack of coping with new FireFox releases: FireFox 8 was out, and we had to run on our tests on FireFox 3.6...
Selenium was the solution, but it had a totally different logic and we already had more than a 1000 tests.
So we had to create a wrapper class around Selenium to "fake" it was WatiN. We had some issues, but we had to rewrite only some special cases... And not all tests.
The point is, sometimes, support for frameworks just cease to exist. But with an own framework focusing on what the test actually does instead of how it works would save you in this situation.
Variety of Test Automation Frameworks and Test Automation Tools are already available in the market. Thus, I would not recommend the built your own Test Automation Framework at all.
As far as selection of automation tools is a concern, I would say it does matter on the following basis:
Support: How much level of support you have when you are choosing an automation tool for your project.
Community: How big community is using that tool and how responsive that community is about sharing knowledge?
Pricing: (Proprietary or Open Source) Last but not the least is the pricing of the automation tool that you are planning to introduce in your project.
QA teams' expertise also matters sometimes. For example, in case your QA team does not have a developer or semi-developer skills vs Not-Technical QA Team, etc.
Regarding the Automation framework, there are many automation frameworks also available in the market already, therefore no need to reinvent the wheel. and selection of automation framework mostly depends on your selection of scripting language.
For example, if you choose python as your scripting language, then you have option to choose UnitTest, PyTest etc. as an automation framework.
In case of Java, you have option of JUnit and TestNG as an automation framework.
and so on, the base on your selection of scripting language.
Finally, when it comes to structuring your automation framework, it solely depends on many things as the following:
Your nature of the project
Single product vs multiple products
and many more...
Check an example of multiple product automation project directory structure. https://github.com/pancht/python-selenium-framework
I hope, in some way, I would have helped you out in giving an answer to your question.
Thanks,
Panchdev Singh Chauhan
My goal is:
Our customers could generate new web-tests.
Our continuous integration server makes a test-environment deployment; it should execute the tests against it
The test could also be run against some other environment.
(Final acceptance tests should be made by the customer, to test fonts etc, but this would be a great pre-acceptance check for our test-environment. Customers could focus on other things than now.)
Usually some property (like text field id) has changed or something and the tests will break in a few weeks. It seems that recorded tests broke often, so it's better to easily record a new one than trying to maintenance and modify an old test.
Now, I found a whole new approach. Maybe recording is not the right way.
How about, if our customers could make use cases in a human readable own language which the machine would understand and compile to web-recording (with Domain Specific Language, DSL).
This is not sci-fi, it has been already made, so read on. :-)
I have tried to use these automatic web testing frameworks:
Visual Studio web test (Customers can't execute)
Selenium (Works only with Firefox, our customers have IE)
WatiN (.NET version of Watir, recorder seems to be a bit buggy)
HP Quick Test Pro (Not easy enough to make new tests)
None of these have provided actually what I need... but Selenium is the closest one.
Our customers speak Finnish, so in the beginning of a software project, in specification phase, user writes a use case like:
Avaa "OmaLomake"
Syötä "Tuomas" kohtaan "nimi"
Paina "Seuraava"
Translation:
Open "MyForm"
Insert "Tuomas" into field "name"
Press "Next"
Now... This is a human-readable use case, but also it can be compiled to automatic web acceptance test. Open, Insert, into field and Press are keywords, others are values.
What kind of DSL tool would be good for this?
Microsoft is making a new DSL-making-tool in their Oslo-project called MGrammar. It means that you can make a custom language to make it easy for non-technical people to work with machines. (The same basic idea that was (and failed) with Cobol and Visual Basic.)
I found that someone has already made this kind of DSL with MGrammar, but it is for Watin, not Selenium:
http://www.codinginstinct.com/2008/11/creating-watin-dsl-using-mgrammar.html
So the continuous integration server process will be:
Fetch a new version from source control (as usual).
Build, run unit tests and analyze the code (as usual).
Make an installation package and tag version in version control (as usual).
Compile use cases to web tests
Run web tests
Accept/Reject the software :-)
Running a web-test in the continuous integration server usually means a lot of configuration work. So, before I try this, I'm curious, what do you think?
Have you used same kind of setup, and what are your experiences? (What exact environment?)
How about DSL, will it have enough power for use cases or will it be another endless development task? Will the customers ever generate the tests?
First of all, Selenium does work with IE and other browsers as well as Firefox; cross browser support is one of its strengths. Here's the list of supported browsers.
However, if you want a human language-based DSL for writing your tests, take a look at Cucumber - the syntax is almost exactly like your example above. Cucumber already has Finnish language support - see the examples at this link.
Fitnesse and Selenium Integration tools such as Selenesse(http://github.com/marisaseal/selenesse) or Fitnium(http://www.magneticreason.com/tools/fitnium/fitnium.html) can also serve your purpose. However, you need to find answer for who will put the element locators in the test cases written by customers. If customers put the locators using the recorders, it may not be possible to maintain. If customers write the steps and a automation tester/developer can put those locators using regex, custom location strategy, this approach may work.
The TestPlan software uses a specialize language for writing tests. It is highly domain specific and works very well in web environments. It supports the Selenium backend so you gain that compatibility, plus it can run without a browser, for even faster tests. I have used it on some fairly large web projects in the type of setup you are looking for.
Your example script might look like this:
GotoURL /SomePage
Click MyForm
SubmitForm with
%Params:name% Tuomos
%Submit% value:Next
end
That's it. It nicely describes what the user wants to do and is a functioning test. You can combine scripts into units and have custom function as well. So if you really wanted you could write the Finish equivalents to the names.