How automated test works and how can I try one? How tools for automated testing works and what they do?
If possible please post with examples to clarify the ideas.
Any help on this topic is very welcome! Thanks.
Automated testing means writing a script for the tasks that we test manually.
Tools include softwares where we write a few lines of code in a sequence as we wish to perform a partiular test. Then running that script to perform the tests and generate results.
Automated testing saves the hours that we manually spend in repeating a series of test cases for a particular test.
Probably the best place to start is the xUnit libraries (JUnit, PHPUnit, jsUnit, etc.). If you're interested in testing of Web interfaces, there's something called Selenium. They provide lots of code examples to look at. These tools allow you to set up some input values, run some code, and then it allows you to verify the final output of that code matches your expectations (aka assertions).
In more sophisticated development teams, these automated tests are run every time new code gets submitted to the project to make sure no new bugs are introduced. As Priyanka mentioned, they save lots of time and eliminate the possibility of human error because the tests are run automatically where they would otherwise be done manually.
I'm sorry for not being more specific. This is a very broad topic of discussion.
Related
I have a group that has some non-technical people creating tests and test suites using the Selenium IDE. I'd like for that group to be able to work independently, yet after the fact be able to run a series of those suites with minimal button clicks. There are a lot of reasons why I'm not just writing tests using some 'native' language (groovy or java) and making this easy to use for the team will help adoption of testing.
So, I would like to be able to just instruct the members of the team to open a single 'suite' (or equivalent) and run it and it would then run each of the suites that I have designated as part of the 'master suite' (if you follow me).
I know that I could just maintain a list of the suites that are part of our automated tests, but it would be easier for me to sell if it was possible to just open up a single file and click 'go' and then walk away and see the results after coming back from getting a cup of coffee or something.
If your reason for not going with a native language is because of your non-technical people, then your automation strategy will fail.
Sorry for being blunt, but there is a reason why there is the IDE, and there is the native language support. They both serve very different purpose, and if you don't approach automation with the respect that it's a programming exercise, then your automation strategy will fail.
Selenium IDE is extremely limited. You are unable to string multiple test suites together. You only have the option of creating one huge a** list of test cases in one suite, or loading your suites in one at a time.
Go WebDriver - everything that you want to do, is extremely limited, if even possible using the IDE.
Yes, I wrote a framework that does that. You can record as many "Selenium Builder" scripts as you like and they will be ran by my framework in a multi-threaded fashion as a group. Just fork your own copy of my framework and then modify it to your uses.
Could someone explain what an automation test is and why I would use it. I read from the wiki page that a tester would create a automation script? What kind of scripting language can be used to do this?
Automation tests are carried out to check the behavior of an application against expected behavior. Normally used in regression testing where you validate that a newer version of the application doesn't hinder any of the previous version's features. These might also be carried along with manual testing.
Coming to the scripting language part, this might help you:- https://softwareengineering.stackexchange.com/questions/19292/best-language-or-tool-for-automating-tedious-manual-tasks
In simple words, If you are doing regression test or testing same piece of code over and over you can automate that manual process. That's called automation testing.
You can use several different scripting languages to achieve this and it's depends on which tool you are using. Some popular automation tools are Selenium, QTP, Loadrunner, Jmeter, SOAP UI etc.
You want to check your login with more than 1000 of users how much time you will spend to run this test case ?
In the same way you want to test you mobile API's before it used by developer how will you test?
There are lots of thing for that you have to go for automation In small application, sites you can work as an tester after that when those app's sites will grow will large data than those product owner will move for automated test cases
I have been writing tests for my Ruby code for a while, but as a frontend developer I am obviously interested in bring this into the code I write for my frontend code. There is quite a few different options which I have been playing around with:
CasperJS
Capybara & Rspec
Jasmine
Cucumber or just Rspec
What are people using for testing? And further than that what do people test? Just JavaScript? Links? Forms? Hardcoded content?
Any thoughts would be greatly appreciated.
I had the same questions a few months ago and, after talking to many developers and doing a lot of research, this is what I found out. You should unit test your JavaScript, write a small set of UI integration tests and avoid record and playback testing tools. Let me explain that in more detail.
First, consider the test pyramid. This is a interesting analogy created by Mike Cohn that will help you decide which kind of testing you should be doing. At the bottom of the pyramid are the unit tests, which are solid and provide fast feedback. These should be the foundation of your test strategy and thus occupy the largest part of the pyramid. At the top, you have the UI tests. Those are the tests that interact with your UI directly, like Selenium does for example. Although these tests might help you find bugs, they are more expensive and provide very slow feedback. Also, depending on the tool you use, they become very brittle and you will end up spending more time maintaining these tests than writing actual production code. The service layer, in the middle, includes integration tests that do not require an UI. In Rails, for instance, you would test your REST interface directly instead of interacting with the DOM elements.
Now, back to your question. I found out that I could greatly reduce the number of bugs in my project, which is a web application written in Spring Roo (Java) with tons of JavaScript, simply by writing enough unit tests for JS. In my application, there is a lot of logic written in JS and that is the kind of thing that I am testing here. I am not concerned about how the page will actually look or if the animations plays as they should. I test if the modules I write in JS will execute the expected logic, if element classes are correctly assigned and if error conditions are well handled. For these tests, I've been using Jasmine. This is a great tool. It is very easy to learn and has nice mocking capabilities, which are called spies. Jasmine-jQuery adds more great functionality if you are using jQuery. In particular, it allows you to specify fixtures, which are snippets of the HTML code, so you don't have to manually mock the DOM. I have integrated this tool with maven and these tests are part of my CI strategy.
You have to be careful with UI tests, specially if you rely on record/playback tools like Selenium. Since the UI changes often, these tests keep breaking and you will spend a lot of time finding out if the tests really failed or if they are just outdated. Also, they don't add as much value as unit tests. Since they need an integrated environment to run, you will mostly like run them only after you finished developing, when the cost of fixing things is higher.
For smoke/regression tests, however, UI tests are very useful. If you need to automate these, then you should watch out for some dangers. Write your tests, don't record them. Recorded tests usually rely on automatically generated xpaths that break for every little change you do on your code. I believe Cucumber is a good framework for writing these tests and you can use it along with WebDriver to automate the browser interaction. Code thinking about tests. In UI tests, you will have to make elements easier to find so you don't have to rely on complex xpaths. Adding class and id elements where you usually wouldn't will be frequent. Don't write tests for every small corner case. These tests are expensive to write and take too long to run. You should focus on the cases that explore most of your functionality. If you write too many tests at this level you will probably test the same functionality that you have previously tested on your unit tests (supposing you have written them).
In my current project I am using Spock and Geb to write the UI tests. I find these tools amazing. They are written in Groovy, which suits better my Java project.
There are lots of options and tools for that. But their choice depends on whether you have a web UI or it's a desktop app?
Supposing from the tools you've mentioned it's Web UI. I would suggest Selenium (aka WebDriver): http://seleniumhq.org/docs/
There is a variety of languages it supports (Ruby is in the list). It can be run against a variety of browsers, ad it's quite easy to use with lots of tutorials and tips available.
Oh, and it's free, of course :)
I though as this post gets a lot of likes, I would post my answer to my question as I do write lots of tests now and how you test front end has moved on a lot now.
So in terms of FE testing I spent lot of time using karma with Jasmine, although karma will work nicely with other test suites like mocha & qunit. While these are great and karma allows you to interface directly with browsers to run your tests. The downside is as your test suite gets large it can become quite slow.
So recently I have moved to Jest which is much faster and if your writing react app, using enzyme with snap shot testing give you really good coverage. Talking of coverage Jest has Istanbul coverage built in and set up and mocking is really easy simple to use. The downside it doesn't test in browser and it using something called jsdom which is fast, but does have a few nuisances. Personally I don't find this a big deal particularly when I compile my code through webpack/babel which means the cross browser bugs are fairly few and far between, so generally isn't an issue if you manually test anyway (and imo you should).
In terms of working within the rails stack, this much easy now that the webpacker gem is now available and using npm and node is generally much more excepted. I would recommend using nvm to manage your node versions
While this isn't strictly testing, I would also recommend using linting as this also picks up a lot of issues in your code. For JS I use eslint with prettier and scss/css I use stylelint
In terms on what to test, I think as Carlos talks about the test pyramid is still relevant, after all the theory doesn't change, just the tools. I would also add to be practical about tests, I would always test, but to what level and coverage will depend on the project. It is important to manage your time and spending hours/days testing a short lifecycle project. Larger/longer term projects the benefits of a larger test suite is obviously greater.
Anyway I hope that helps people that look at the question.
My goal is:
Our customers could generate new web-tests.
Our continuous integration server makes a test-environment deployment; it should execute the tests against it
The test could also be run against some other environment.
(Final acceptance tests should be made by the customer, to test fonts etc, but this would be a great pre-acceptance check for our test-environment. Customers could focus on other things than now.)
Usually some property (like text field id) has changed or something and the tests will break in a few weeks. It seems that recorded tests broke often, so it's better to easily record a new one than trying to maintenance and modify an old test.
Now, I found a whole new approach. Maybe recording is not the right way.
How about, if our customers could make use cases in a human readable own language which the machine would understand and compile to web-recording (with Domain Specific Language, DSL).
This is not sci-fi, it has been already made, so read on. :-)
I have tried to use these automatic web testing frameworks:
Visual Studio web test (Customers can't execute)
Selenium (Works only with Firefox, our customers have IE)
WatiN (.NET version of Watir, recorder seems to be a bit buggy)
HP Quick Test Pro (Not easy enough to make new tests)
None of these have provided actually what I need... but Selenium is the closest one.
Our customers speak Finnish, so in the beginning of a software project, in specification phase, user writes a use case like:
Avaa "OmaLomake"
Syötä "Tuomas" kohtaan "nimi"
Paina "Seuraava"
Translation:
Open "MyForm"
Insert "Tuomas" into field "name"
Press "Next"
Now... This is a human-readable use case, but also it can be compiled to automatic web acceptance test. Open, Insert, into field and Press are keywords, others are values.
What kind of DSL tool would be good for this?
Microsoft is making a new DSL-making-tool in their Oslo-project called MGrammar. It means that you can make a custom language to make it easy for non-technical people to work with machines. (The same basic idea that was (and failed) with Cobol and Visual Basic.)
I found that someone has already made this kind of DSL with MGrammar, but it is for Watin, not Selenium:
http://www.codinginstinct.com/2008/11/creating-watin-dsl-using-mgrammar.html
So the continuous integration server process will be:
Fetch a new version from source control (as usual).
Build, run unit tests and analyze the code (as usual).
Make an installation package and tag version in version control (as usual).
Compile use cases to web tests
Run web tests
Accept/Reject the software :-)
Running a web-test in the continuous integration server usually means a lot of configuration work. So, before I try this, I'm curious, what do you think?
Have you used same kind of setup, and what are your experiences? (What exact environment?)
How about DSL, will it have enough power for use cases or will it be another endless development task? Will the customers ever generate the tests?
First of all, Selenium does work with IE and other browsers as well as Firefox; cross browser support is one of its strengths. Here's the list of supported browsers.
However, if you want a human language-based DSL for writing your tests, take a look at Cucumber - the syntax is almost exactly like your example above. Cucumber already has Finnish language support - see the examples at this link.
Fitnesse and Selenium Integration tools such as Selenesse(http://github.com/marisaseal/selenesse) or Fitnium(http://www.magneticreason.com/tools/fitnium/fitnium.html) can also serve your purpose. However, you need to find answer for who will put the element locators in the test cases written by customers. If customers put the locators using the recorders, it may not be possible to maintain. If customers write the steps and a automation tester/developer can put those locators using regex, custom location strategy, this approach may work.
The TestPlan software uses a specialize language for writing tests. It is highly domain specific and works very well in web environments. It supports the Selenium backend so you gain that compatibility, plus it can run without a browser, for even faster tests. I have used it on some fairly large web projects in the type of setup you are looking for.
Your example script might look like this:
GotoURL /SomePage
Click MyForm
SubmitForm with
%Params:name% Tuomos
%Submit% value:Next
end
That's it. It nicely describes what the user wants to do and is a functioning test. You can combine scripts into units and have custom function as well. So if you really wanted you could write the Finish equivalents to the names.
I look for a tool/framework to make automatic acceptance-testing. The interface to create new tests should be so easy, that a non-programmer (customer, boss) will be able to add specifications for which will be tested automatically.
It should be some way to execute the tests from command-line, to include a run of the tests in automatic builds.
I prefer Java and Open-Source, but my question isn't restricted in that way.
What do you can recommend and please explain why your tool/framework is the best in the world.
http://fitnesse.org/ appears to meet all of the qualifications you want. It is one I have used with success.
I think that several of the options are very good and you should test them to see which fits your team :
Cucumber (Ruby)
Fitnesse
Robot framework (Python/Java)
Behave for Java
SpecFlow (.net)
Another framework you may want to look at is Robot Framework. To see how test cases look like, take a look at the Quick Start Guide.
I've found a framework named Concordion that may fulfill my needs.
What you ask for appears to be for a very well-defined system with a very specific sets of inputs and a high degree of automation built-into the system or developed for your system.
Commercial applications such as HP Quick Test Pro isn't non-technical enough and requires an additional framework such as one from Sonnet, which is a step in the right direction, but neither is open source or java-based.
Even with a framework in place, it's quite a bit of work to make this work in an automated way. I'd like you to consider the time needed to develop the framework vs the time to manually run these tests and verify that you are using your time well.
How about Cucumber:
Feature: Acceptance testing framework
Scenario: an example speaks volumes
Given a text example
When it is read
Then the simplicity will be appreciated
You would need a developer to discuss with the boss what each of those lines really means and implement the step definition to drive it:
Given /^a text example$/ do
file.open("example.txt", "w") { |file| file.write "text example" }
end
When /^it is read$/ do
SystemUnderTest.read("example.txt")
end
Then /^the simplicity will be appreciated$/ do
SystemUnderTest.simplicity.should be_appreciated
end