I am new to Cucumber testing and I am trying to understand when to use Cucumber and when to use RSpec. For my models, I know I should be testing them with RSpec, and I know that I don't need to write RSpec request specs if I write Cucumber stories instead. The thing I don't know when to use what is with views and controllers. Should I be using RSpec to test my views and controllers, or can I just skip them because I am using Cucumber for my higher level testing? Any advice would be greatly appreciated! Thanks!
I use RSpec to drive the development and testing of my models and to some extent my controllers, and Cucumber to drive the development and testing of my views (and subsequently their integration with the controllers).
I don't feel writing Cucumber tests allows me to "skip" writing RSpec tests because a lot of development takes place in the model layer before any views are created.
If you keep writing Cucumber scenarios instead of unit tests, then you will probably find that your tests become very slow to run. If the tests are slow, you probably won't run them very often, so your development pace will slow down.
You can use RSpec for integration tests operating at the same level as Cucumber. If you need to share and discuss features with non-developers, then Cucumber is a huge help. But even if you're a solo developer, Cucumber helps by acting as a 'mental guard' between the behaviour you need and the underlying implementation.
The standard practice most people follow is to write extensive unit tests for the model, minimal tests for the controller, and very rarely any unit tests for the view. Then you would use Cucumber to check everything is interacting properly (be sure that you scenarios cover any loops/conditions in the controllers and views).
I'm not sure if it's OK to recommend a book here, but The RSpec Book by David Chelimsky (ISBN 9781934356371) does provide a nice practical example of how Cucumber and RSpec can be interleaved to behaviour-drive the development of a small command-line game in Ruby.
In a nutshell: writing your Cucumber features adopting the point of view of your end-user, and your RSpec specs using your own point of view as a developer is a good balance between execution speed of the test suite, testing granularity (how precise are the error messages when your tests fail) and documentation coverage (the Cucumber features can be seen as an always up-to-date documentation - the very own RSpec documentation is a great example of that).
Just a small addition that might be helpful to you. Many people think (i also do) that cucumber is more of a hassle than a practical test framework for the programmer. As long as you do not have to cooperate by discussing scenarios with clients that do not know much about programming, i would highly recommend that you do not use cucumber. I'm with DHH on this one. But RSpec, use and even abuse :)
Related
And further, how they relate together or even if they do?
What would one do to understand the various pieces to a simple question, how to I properly build a testing facility for my (web or other) application?
Agile Development is a banner term for many things, too numerous to mention, including Scrum and TDD. It typically, but not always follows the Agile Manifesto.
SCRUM
This is a particular flavour of agile. This diagram from wikipedia shows the process:
See wikipedia for more info.
Unit Testing
This is the art of writing code that tests code. Failing tests indicate a problem in your solution.
Test Driven Development
This is the practice of writing tests before code, some of the advantages being that untested code isn't added to the solution, and that the code written is testable.
A proper testing facility, usually leverages something along the lines of xunit, junit, nunit, mstest depending on the framework used, these tests are typically ran via a Continuous Integration build on some kind of build server. That is a build that runs everytime the code changes, that executes tests. This way problems are identified quicker.
I find it not so easy as one might think to start with TDD in Rails.
I've searched for some examples but no-one really explains how to do TDD/BDD with Rspec AND cucumber. They usually use an old version of rspec, or prefer one on the other (so you find cucumber tests and rspec tests separated). And by the way the say what to do, but the real process is often not explained at all (I've read a lot on make them work togheter with Capybara, but only once on how to proceed on bdd with them).
I'd like to know how to use them togheter, how to avoid test duplication (disabling view specs and other tests that should be done with cucumber), and maybe an explanation of what not to test (for example I find it useless to test the HTML output in most cases, it can be useful to test if you're logging in to match what you're seeing, for example a flash message).
Can you please provide a step-by-step list of how do you proceed with BDD using this two tools togheter? It would be nice a fresh tutorial (using latest versions) or maybe a screencast.
Thanks.
I highly recommend you the Rspec book which is exactly what you are looking for :
It explains how to use Rspec and Cucumber together with a simple 2 loops process : The outer big loop is a Test - Code - Refactor process you do with cucumber and each step is made of multiple iteration of a Test - Code - Refactor process you do with Rspec.
That books explains at the same time how and when to use each of the two tools.
If you want a broader vision of TDD-BDD, I also suggest you the GOOS book which is more language/tool agnostic and more process oriented.
Unit testing is a practice of writing code tests. TDD is the practice of writing them "before". BDD is the practice of writing behavior/spec driven tests. Can I write BDD "after" or do I have to do it always "before"?
If you write BDD "after", and it's not BDD, then what it is called?
By definition of Behaviour Driven Development, you cannot write the behaviour tests after the code, however that does not mean that doing so isn't useful. You may get more benefit from writing the spec tests first, but they are still useful as regression system tests for your application. So while you're technically not practicing BDD, writing these tests is a good idea. One of the big perks of BDD is that it guides the development of the particular behaviour, so you are losing a lot of value by adding them later, but they still serve some use.
This is the same as writing unit tests after the code in TDD. It's technically not TDD, but having the tests is obviously still useful.
Behavior-Driven Development (BDD) is a variation of Test-Driven Development (TDD) and just like with TDD you should write your tests first.
Some people call BDD for TDD done right, or the way it was intended. Also, you could say that BDD is a mix of Domain-Driven Development (DDD) and TDD.
BDD after development is not BDD, and it is a case of validation rather than specification.
However as the other guys mentioned, it does not mean that adding in an acceptance test suite after-the-fact has no value. You will be building a suite of regression acceptance tests that validate behaviour, before proceeding with further development (large refactoring jobs or new features being added).
From experience I would say if you are to do this task, it is best that the key developers who wrote the production code stay well away from writing the acceptance tests (hopefully in the form of Gherkin scripts); and those that are writing them go back to the original requirements documentation (if any) and most definitely collaborate with some of the stakeholders in doing so. This will help make sure that the acceptance tests you write are closer to specification.
I like the observation that BDD-After is simply a case of writing validation. I also appreciate the comments that a developer doing BDD-After misses some of the other benefits of BDD-As-You-Go. One point that seems worth adding is that writing a secenario/test before the implementation and then having the test pass is also a type of validation that the test itself is sound. Writing a passing test for a feature that already works (BDD-After) may leave a developer wondering if their test will "fail appropriately" should a feature get broken.
I've got analysis paralysis looking at all the different functional testing options for a new grails (v1.3.4) application. I've looked at
WebDriver/Selenium (which I've used before)
WebTest/Canoo
Geb
Tellurium
Grails Functional Test
and there must be others. I think some of the criteria that I would use to make a decision include (in no particular order):
Likely longevity, active development
Can do ajax/javascript
Support for PageObject or similar patterns
Maturity
Headless (eg htmlunit) is ok, especially if it makes things go faster
Good reporting
Support for NTLM credential provider or similar
Compact, robust test scripts
Takes advantage of groovy language
I would be particularly interested to hear from people who have tried more than one framework. Thanks!
I maintain the plugin for WebTest but I'd recommend giving Geb a try. I haven't used it personally on a project yet but I think it will tick off most of your criteria.
It is the most actively developed (IMO) but is quite new. It is built on WebDriver/Selenium so should also be a good fit with your past experience.
WebTest doesn't see a lot of development these days and does not have built in support for page objects. But it does give you great reports. It's downside is it's historical foundation of Ant. It makes it hard to test in a dynamic fashion as your test steps are built up when the test case is executed and then the actual test is run as a second pass.
I have used GFunc as well but the lack of reporting is a real pain and it does not get as much development as Geb. It is far more "groovy" than WebTest though and is a thin wrapper over HtmlUnit allowing you to "roll your own" testing functionality quite easily.
cheers
Lee
Grails Functional Test is HtmlUnit only. I wouldn't suggest coding directly to HtmlUnit these days with the other frameworks that are available that give you page object and better abstraction out of the box.
I've used the webdriver plugin which seems to work pretty well and has page object built in - but the page object support is coupled to the JUnit hierarchy, which makes it hard for me to use because I want to also use Cucumber.
I'm very excited about Geb and am hearing a lot of success stories with it but haven't gotten to finish setting it up myself.
There are a lot of testing methods out there i.e. blackbox, graybox, unit, functional, regression etc.
Obviously, a project cannot take on all testing methods. So I asked this question to gain an idea of what test methods to use and why should I use them. You can answer in the following format:
Test Method - what you use it on
e.g.
Unit Testing - I use it for ...(blah, blah)
Regression Testing - I use it for ...(blah, blah)
I was asked to engage into TDD and of course I had to research testing methods. But there is a whole plethora of them and I don't know what to use (because they all sound useful).
1. Unit Testing is used by developers to ensure unit code he wrote is correct. This is usually white box testing as well as some level of black box testing.
2. Regression Testing is a functional testing used by testers to ensure that new changes in system has not broken any of existing functionality
3. Functional testing is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. Functionality testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic
.
This Test-driven development and Feature Driven Development wiki articles will be of great help for you.
For TDD you need to follow following process:
Document feature (or use case) that
you need to implement or enhance
in your application that
currently does not exists.
Write set of functional test
cases that can ensure above
feature (from step 1) works. You may need to
write multiple test cases for
above feature to test all different
possible work flows.
Write code to implement above feature (from step 1).
Test this code using test cases you
had written earlier (in step 2). The actual
testing can be manual but I would recommend to create automated tests
if possible.
If all test cases pass, you are good to
go. If not, you need to update code (go back to step 3)
so as to make the test case pass.
TDD is to ensure that functional test cases which were written before you coded should work and does not matter how code was implemented.
There is no "right" or "wrong" in testing. Testing is an art and what you should choose and how well it works out for you depends a lot from project to project and your experience.
But as a professional Test Expert my suggestion is that you have a healthy mix of automated and manual testing.
(Examples below are in PHP but you can easily find the correct examples for what ever langauge/framework you are using)
AUTOMATED TESTING
Unit Testing
Use PHPUnit to test your classes, functions and interaction between them.
http://phpunit.sourceforge.net/
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.webinade.com/web-development/functional-testing-in-php-using-selenium-ide
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
This answer is (almost) identical to one that I gave to another question. Check out that question since it had some other good answers that might help you.
How can we decide which testing method can be used?
I usually do the following things:
Page consistency in case of multi-page web sties.
Testing the database connections.
Testing the functionalities that can be affected by the change I just made.
I test functions with sample input to make sure they work fine (especially those that are algorithm-like).
In some cases I implement features very simply hard-coding most of the settings then implement the settings later, testing after implementing every setting.
Most of these apply to applications, too.
Well before going to the answer i would like to clear testing concept about multiple methods.
There are six main testing types which cover all most all testing methods.
Black Box Testing
White Box Testing
Grey Box Testing
Functional Testing
Integration Testing
Usability Testing
Almost all Testing methods lies under these types, you can also use some testing method in multiple types like you can use Smoke testing in black box or white box approach on the basis of resources available to test.
So for testing a web site completely you need to use at least following testing methods on the basis of resources available to test. These are at least methods which should be used to test a web site, but there may be some more imp methods on the basic of nature of website.
Requirement Testing
Smock Testing
System Testing
Integration Testing
Regression Testing
Security Testing
Performance & Load Testing
Deployment Testing
You should at least use all of above (8) testing methods to test a web site no matter what testing type you are focusing. You can automate you test in some areas and you can do this manually it all depends upon the resources availability.
There is specifically no hard and fast rule to follow any testing type or any method. As you know "Testing Is An ART" so art don't have rules or boundaries. Its totally up to you What you use to test and how.......
Hope you got the answer of question.
Selenium is very good for testing websites.
The answer depends on the Web framework used (if any). Django for example has built-in testing functions.
For PHP (or functional web testing), SimpleTest is pretty good and well... simple. It support Unit Testing (PHP only) and Web Testing. Tests can run in the IDE (Eclipse), or in the browser (meaning on your server).
The other answers posted so far focus on unit/functional/performance/etc. testing, and they are all reasonable.
However, one the key questions you should ask is, "how effective is my testing?".
This is often answered with test coverage tools, that determine which parts of your application actually get exercised by some set of tests. The ideal test coverage tool lets you test your application by any method you can imagine (including all the standard answers above) and will then report what part and what percentage of your code was exercised. Most importantly, it will tell you what code you did not exercise. You can then inspect that code and decide if more testing is warranted, or if you don't care. If the untested code has to do with "disk full error handling" and you belive that 1TB disks are common, you might decide to ignore that. If the untested code is the input validation logic leading to SQL queries, you might decide that you must test that logic to ensure that no SQL injection attacks can occur.
What test coverage tools let you do it to make a rational decision that you have tested adequately, using data about what parts of your code has been exercised. So regardless of how you test, best practices indicates you should also do test coverage analysis.
Test coverage tools can be obtained from a variety of sources. SD provides a family of test coverage tools that handle C, C++, Java, C#, PHP and COBOL, all of which are used to support web site testing in various ways.