I'm trying to figure out a good practice approach of when I should use Background over a Before hook.
I've browsed around the web looking at other peoples blog post but there's so much over lap.
Any opinions when to use which, and perhaps more importantly, when definitely not to use one of them?
My take on differentiating between the two comes down to using the tests as communication. If the functionality that you're adding is necessary for anyone reading the plain language Scenarios and features, it should go in the Background section. If the functionality is simply part of the implementation of your test framework, it should go into a Before hook.
Keeping the scenarios and features clear is an important part of building good Cucumber tests.
I guess another mental check you could use would be to ask yourself if someone was testing your scenarios manually, would they need to do the functionality that you want to put in the Background/before hook. If they would, I'd put it in the Background section in the feature file. If they wouldn't, put it in the hook back in your code.
Related
I have been using Karate for the past 6 months, I am really impressed with the features it offers.
I know karate is meant to test API(s) individually but we are also trying to use it for E2E tests that involves calling multiple scenarios step by step.
Our feature file looks like below
1.Call Feature1:Scenario1
2.Call Feature2:Scenario2
.....
Note: We are re-using a scenarios for both API Testing and E2E testing.Sometimes I find it difficult to remember all feature files.
Can we chain the scenario call like java, I doubt feature file will let us to do that. We need your valuable suggestion. pls let us know if you feel our approach is not correct
First, I'd like to quote the documentation: https://github.com/intuit/karate#script-structure
Variables set using def in the Background will be re-set before every Scenario. If you are looking for a way to do something only once per Feature, take a look at callonce. On the other hand, if you are expecting a variable in the Background to be modified by one Scenario so that later ones can see the updated value - that is not how you should think of them, and you should combine your 'flow' into one Scenario. Keep in mind that you should be able to comment-out a Scenario or skip some via tags without impacting any others. Note that the parallel runner will run Scenario-s in parallel, which means they can run in any order.
So by default, I actually recommend teams to have Scenario-s with multiple API calls within them. There is nothing wrong with that, and I really don't understand why some folks assume that you should have a Scenario for every GET or POST etc. I thought the "hello world" example would have made that clear, but evidently not.
If you have multiple Scenario-s in a Feature just run the feature, and all Scenario-s will be executed or "chained". So what's the problem ?
I think you need to change some of your assumptions. Karate is designed for integration testing. If you really need to have a separate set of tests that test one API at a time, please create separate feature files. The whole point of Karate is that there is so little code needed - that code-duplication is perfectly ok.
Let me point you to this article by Google. For test-automation, you should NOT be trying to re-use things all over the place. It does more harm than good.
For a great example of what happens when you try to apply "too much re-use" in Karate, see this: https://stackoverflow.com/a/54126724/143475
Please let me know the difference between the hand written code and recorded scripts in automation testing tools like coded ui or any other tools.
Regards,
Raj
By 'hand-written' I'll assume you mean manually coded...
I can see a few reasons. Coding experience is brilliant. It will be a worthwhile investment if you code your own tests because you can learn a lot about the testing framework you are using (CodedUI, Selenium etc) but also the language you are using (Java, C#). Manually coding these tests, using built in framework methods, will serve you well and give you much more knowledge than an automatic play back tool would.
Automatic playback tools can produce horrific code. Code that is ugly, badly named, no best practices followed, and unreliable location methods.
Playback tools will simply use the most simple way to find an element. This is not always the best. A classic example is XPath.
Most notably, XPath is a powerful tool, it can get you any element you need (or at least, I've never found a situation where XPath cannot be used), but playback tool's will produce horrific XPath queries based purely on position...let's take an example.
You've got a page that has 100 feed items. You want to verify after a particular action a feed item is shown on this page, but not only is it shown but it is the first one. You cannot use ID's etc, because the markup is badly made and so you must use XPath.
A playback tool might make a very odd XPath like: //div[1]/span[2]/table[1]/tbody[1]/tr[10]/[td[2]/a[text()='Test'].
Looks weird, right?
This will work a few times, but what happens if the app gets another tr element shoved at the top of the table? Now, tr[10] wont be the element you want, it'll be tr[11].
Through manual coding, you can account for this, you can put in logic to work around this. Playback tool's wont.
I highly recommend coding these tests yourself. You do not need a few years experience to do this, you do not need any prior programming degrees. You need time.
Playback tools will also be limited in what they can do...you want to take a screenshot when a test fails? I highly highly doubt a playback tool will do this, you'll need to put in logic yourself. However, this isn't hard to do yourself.
There might be a business reason too - playback tools can convert manual tests into automated tests faster, but they won't be reliable - you'll need to have time to dedicate to making them reliable and fast. Time that would better be spent coding them yourself in the first place.
I have been reading a lot about test-driven development and decided that I want to give it a go on a small project. For reference, I am currently reading 'Growing Object-Oriented Software, Guided by Tests'.
I understand how to unit test my application and how to unit test certain parts of the UI as well, but I am struggling to set up end-to-end tests. For example, testing that a certain path through my whole application produces the correct output (this is my basic understanding of an end-to-end test).
It's not necessary to simulate click events, but it is necessary to have some sort of connection to the UI.
Am I right in thinking that I need a combination of "Logic" tests (test without launching the app), "Application" tests (test with launching the app) and the asynchronous functionality of something like GHUnit to accomplish this?
EDIT:
After reading some of the answers below, it sounds like I'm looking for functional end-to-end testing, but I think I should give an example of a test as I imagine it.
Start the application.
Call the login function with a test users credentials. (Note: doesn't necessarily need UI automation).
Verify a label on the window says "Logging In...".
After successfully verifying the user, verify the label now says "Welcome, Adam!".
KIF sounds like it could work, as it has steps to check changes in UI elements and it looks like there is a Mac OSX branch also. I'm sure I could also write a small class which constantly polls the UI for changes I expect and times-out after a certain time, but I'm wondering if this the right way to go about it.
However, perhaps I am trying to take what I am reading in 'Growing Object-Oriented Software, Guided by Tests' and trying to apply it too literally to Cocoa.
Another UPDATE:
So I've been reading the advice so far, checked the various places linked to and started to implement something whilst still referencing the book. I think what I'm really trying to get at is the Test-Driven-Development part. What stood out most in said book, was that they described what they wanted to happen from a users perspective first with acceptance tests.
I realise that solid unit testing will be necessary as soon as I start writing methods, but I was keen to write some high-level acceptance tests first, using some of the UI. I have started to write my own application "driver" class, using some similar methods to the GHAsyncTestCase ideas to help me accomplish this. Does this sound correct/useful/necessary?
I really appreciate all the comments so far and they have definitely helped me work out in my own head what I'm trying to do and what various areas of testing there are. I will finish up this question soon, as it is getting rather large, so any final advice is welcome!
I think the key thing that I got from "Growing Object Orientated Software" was to decouple as much as possible from the UI. Without code to look at it's harder to give suggestions but with your revision I'd think that separating the "verify a label says.." bit from the UI. What is setting this message, and can you just test for that event?
The more you can decouple from the UI the more you can unit-test (quicker and easier) rather than integrating other frameworks or drivers of UI elements.
You might be interested in Square's KIF framework: http://corner.squareup.com/2011/07/ios-integration-testing.html
It looks really cool for integration/UI testing.
I believe you can use the Accessibility features to script the UI. I'd check the WWDC 2011 videos for one entitled "Design Patterns to Simplify Mac Accessibility". They did something similar in 2010.
Based on your response to #Norman, I guess you're looking for recommendations that span both functional end-to-end as well as UI-based end-to-end but perhaps a UI automation framework might change your mind? Something intrusive like FoneMonkey might be helpful:
http://www.gorillalogic.com/fonemonkey
If that doesn't work for you, I'd be interested in knowing why & what "gap" you perceive in such UI driven tests versus code-based functional testing?
So I'm trying to convert myself to a more test- and behaviour- driven approach to my development. It's good for me, and I've seen good results in the few projects I've used it for so far.
My current project is a FUSE-based filesystem - I want to add some functionality over basic filesystem access so FUSE seemed like a good fit. All I really need to do is implement a set of functions that fit the appropriate interface, wrap it up appropriately, and go.
However, test first, I remind myself. I've already written a set of cucumber features to lay out the basic expectations of how the overall app should work, so now it's time to get down to testing the innards.
Now, I could just write unit tests for each of the functions I need to write for the interface, and then get to coding the interface - but that doesn't seem overly test-driven to me. Sure the tests exist, but the interface is really what's driving things.
Am I going about this wrong? Or am I expecting too much?
Give me a "what-what" in the comments if you think this should be community wiki - I can't even decide if this has a right answer.
Step 1. What is one thing the interface must do? One thing.
Step 2. How will you prove it does that one thing?
Step 3. Write a test to prove the interface really does that one thing.
Step 4. Run the test -- it will fail. You haven't actually written the interface.
Step 5. Code the interface.
Step 6. The test will pass.
Move on to the next thing the interface must do.
This has little to do with the functions you've already designed. This is totally focused on externally visible feature the interface must have. It may turn out that your functions are the right thing. Or it may turn out that you over-engineered these functions. Or under-engineered them. The point is to drive your design from the things a component must do and the tests to prove what it must do.
Even if it's only focused on Ruby, The rspec book has a good introduction on the BDD cycle.
I want to add some functionality over basic filesystem access so FUSE seemed like a good fit
It is hard to develop fuse fs. Two main problems is VERY hard debugging and multi-threading. Also I had (and now have) problems with testing my fs. Maybe inotify will satisfy your requirements.
I've started a new role in my life. I was a front end web developer, but I've now been moved to testing web software, or more so, automating the testing of the software. I believe I am to pursue a BDD (Behavior Driven Development) methodology. I am fairly lost as to what to use, and how to piece it together.
The code that is being used/written is in Java to write a web interface for the application to test. I have documentation of the tests to run, but I've been curious how to go about automating it.
I've been directed to Cucumber as one of the "languages" to help with the automation. I have done some research and come across a web site for a synopsis of BDD Tools/Frame works,
8 Best Behavior Driven Development (BDD) Tools and Testing Frameworks. This helped a little but then I got a little confused of how to implement it. It seems that Selenium is a common denominator in a lot of the BDD frameworks for testing a GUI, but it still doesn't seem to help describe what to do.
I then came across the term Functional Testing tool, and I think that confused me even more. Do they all test a GUI?
I think the one that looked like it was all one package was SmartBear TestComplete, and then there is, what seems to be, another similar application by SmartBear called, SmartBear TestLeft, but I think I saw that they still used Cucumber for BDDing it. There a few others that looked like they might work as well, but I guess the other question is what's the cheapest route?
I guess the biggest problem I have is how to make these tests more dynamic, as the UI/browser dimensions can easily change from system to system, and how do I go about writing automation that can handle this, and tie into a BDD methodology?
Does anyone have any suggestions here? Does anybody out there do this?
Thanks in advance.
BDD Architecture
BDD automation typically consists of a few layers:
The natural language steps
The wiring that ties the steps to their definition
The step definitions, which usually access page objects
Page objects, which provide all the capabilities of a page or widget
Automation over the actual code being exercised, often through the GUI.
The wiring between natural language steps and the step definitions is normally done by the BDD tool (Cucumber).
The automation is normally done using the automation tool (Selenium). Sometimes people do skip the GUI, perhaps targeting an API or the MVC layer instead. It depends how complex the functionality in your web page is. If in doubt, give Selenium a try. I've written automation frameworks for desktop apps; the principle's the same regardless.
Keeping it maintainable
To make the steps easy to maintain and change, keep the steps at a fairly high level. This is frequently referred to as "declarative" as opposed to "imperative". For instance, this is too detailed:
When Fred provides his receipt
And his receipt is scanned
And the cashier clicks "Refund to original card"
And the card is inserted...
Think about what the user is trying to achieve:
When Fred gets a refund to his original card
Generally a scenario will have a few Givens or Thens, but typically only one When (unless you have something like users interacting or time passing, where both events are needed to illustrate the behaviour).
Your page objects in this scenario might well be a "RefundPageObject" or perhaps, if that's too large, a "RefundToCardPageObject". This pattern allows multiple scenario steps to access the same capabilities without duplication, which means that if the way the capabilities are exercised changes, you only need to change them in one place.
Different page objects could also be used for different systems.
Getting started
If you're attacking this for the first time, start by getting an empty scenario that just runs and passes without doing anything (make the steps empty). When you've done this, you'll have successfully wired up Cucumber.
Write the production code that would make the scenario run. (This is the other way round from the way you'd normally do it; normally you'd write the scenario code first. I've found this is a good way to get started though.)
When you can run your scenario manually, add the automation directly to the steps (you've only got one scenario at this point). Use your favourite assertion package (JUnit) to get the outcome you're after. You'll probably need to change your code so that you can automate over it easily, by e.g.: giving relevant test ids to elements in your webpage.
Once you've got one scenario running, try to write any subsequent scenarios first; this helps you think about your design and the testability of what you're about to do. When you start adding more scenarios, start extracting that automation out into page objects too.
Once you've got a few scenarios, have a think about how you might want to address different systems. Avoid using lots of "if" statements if you can; those are hard to maintain. Injecting different implementations of page objects is probably better (the frameworks may well support this by now; I haven't used them in a while).
Keep refactoring as you add more scenarios. If the steps are too big, split them up. If the page objects are too big, divide them into widgets. I like to organize my scenarios by user / stakeholder capabilities (normally related to the "when" but sometimes to the "then") then by different contexts.
So to summarize:
Write an empty scenario
Write the code to make that pass manually
Wire up the scenario using your automation tool; it should now run!
Write another scenario, this time writing the automation before the production code
Refactor the automation, moving it out of the steps into page objects
Keep refactoring as you add more scenarios.
Now you've got a fully wired BDD framework, and you're in a good place to keep going while making it maintainable.
A final hint
Think of this as living documentation, rather than tests. BDD scenarios hardly ever pick up bugs in good teams; anything they catch is usually a code design issue, so address it at that level. It helps people work out what the code does and doesn't do yet, and why it's valuable.
The most important part of BDD is having the conversations about how the code works. If you're automating tests for code that already exists, see if you can find someone to talk to about the complicated bits, at least, and verify your understanding with them. This will also help you to use the right language in the scenarios.
See my post on using BDD with legacy systems for more. There are lots of hints for beginners on that blog too.
Since you feel lost as to where to start, I will hint you about some blogs I have written that talks a bit about your problem.
Some categories that may help you:
http://www.thinkcode.se/blog/category/Cucumber
http://www.thinkcode.se/blog/category/Selenium
This, rather long and old post, might give you hints as well:
http://www.thinkcode.se/blog/2012/11/01/cucumberjvm-not-just-for-testing-guis
Notice that versions are dated, but hopefully it can give some ideas as what too look for.
I am not an expert on the test automation but I am currently working on this part. So let me share some idea and hope it will help you at the current stage.
We have used selenium+cucumber+intellij for testing web application. We have used testcomplete+cucumber+intellij for testing java desktop application.
As to the test of web application, we have provided a test mode in our web application, which allows us to get some useful details of the product and the environment; and also allows us to easily trigger events through clicking the button and inputting text into the test panel under test mode.
I hope these are helpful for you.