The company I work for is starting a new web application, and I have requested that front end developers make this application Automation Friendly.
The previous application was using the react framework, very few elements had unique ID's (or any unique identifier at all). This time around, I have asked the developers to include a custom data attribute, specifically for automation.
I am looking for anyone who may have experience in this kind of situation.
What have you asked your developers to do to assist in automation?
Are there any standards, or guidelines for naming elements in an application to suit Selenium automation?
are custom data attributes the best way to go? are there other options?
Any advice/guidance would be greatly appreciated!
Web Applications can be difficult to test if they aren’t made with testing in mind. This is especially true for Single Page Applications (SPAs). SPAs support heavy interaction without incurring additional page loads (e.g. Facebook, Gmail). Instead of page loads, these SPAs use AJAX requests to relay data back and forth from the server.
As per #ChrisChua from ThousandEyes these are some of the best practices to keep in mind as you develop your web application to make testing easier:
Add classes that are meaningful.
Classes should indicate the element’s functionality and state.
Use functional names in IDs and classes for action elements.
Dynamically generated classes and IDs are not helpful for testing.
Add targetable DOM feedback to indicate application state.
Never hard-code content in test code!
Conclusion
It's true some of these are not easy changes, as the developer may have to think harder about using test-friendly designs rather than "something that just works". However, it will definitely help with maintainability of the testing, which would reduce costs in the long run.
tl; dr
A couple of references:
Nicolas Gallagher’s article on HTML Semantics and Front-end Architecture
Yandex’s Block Element Modifier
W3C’s recommendation on good class names
Pamela Fox’s article on Frontend Architectures
Test Driven Development makes it easier to automate. In my opinion, testers should be developers, or ex-developers who are good at finding and preventing mistakes. Write the tests in the same project that the solution is being developed in. Developers can also put in ids, even though they don’t want to sometimes, and if the “testers” are good they can even submit pull requests (in GitHub for example) for code improvements that will allow them to test better. Think of testers as a piece of your development team, where everyone is allowed to contribute code. It helps with accountability and improves autonomy. Everyone is there to help each other and if all the code is in the same project, and all code is reviewed and approved before merging to master, everyone is a potential developer if everyone is IT. The old days of manual testing are dying. Separating Dev from Testing is a brick wall. Tear it down.
Related
We are a couple of entrepenours who have developed a cross-browser app and a backend administration system for the app. Or actually, we paid a company to develop it. Now we want it tested profesionally, but we dont want to use the same company for this purpose.
The tests may involve
Integration Testing
Functional Testing
System Testing
Stress Testing
Performance Testing
Usability Testing
For some of the tests, we think that the actual source code is required. We dont feel komfortable giving our source code away "just like that", to unknown parties, so what are the common methods for having external companies testing ones software?
you don't really need to give the source to perform the mentioned tests. you need to provide working environments or provide binaries and instructions how to deploy them. it seems sufficient for 1,2 (i don't know what does 3 means), 4 and 5. It's way too late for usability testing. it should have been done during UI design phase (how do you want to test it right now?)
but those tests are not sufficient. you forgot about penetration testing. and above tests are black box testing and they can show you how the application works.
but if you have any real plans for this application you must be sure it's maintainable. and for this you need the whitebox testing, you have to analyze the code.
you can start with automated analyzing to check the overall quality of the code. but at the end you will still need good programmers to perform the code review. but you don't really have to give them the code. you can invite them to your office and let them review the code on your workstations. unless your idea is so simple and brilliant that only one look at the code is enough to reproduce it. in such case you will need to sign NDA or give some shares to the expert who will take care of the quality
I work in a small organization that has built an enterprise SaaS solution. Up until this point our workflows have had no programmatic interface. We're moving to a model that will allow for an end user to do anything programmatically that can be done in the UI. I'm looking for suggestions in terms of the language/framework that you would use to build that programmatic layer.
From an organizational perspective I would like the current UI team to also have ownership of the API. That team is familiar with PHP, Rails, and Javascript. Our current back-end code is written in Scala. I'm leaning toward not doing the APIs in Scala because it doesn't seem like the right tool for the job and the lack of subject matter expertise around it on the UI team.
From a functionality perspective most of the APIs will be fairly simple database operations (CRUD) with perhaps some simplistic business logic applied on top (search for example).
I'm a bit intrigued by using Node.js for this as everyone on the team is really strong with Javascript. That being said I don't just want to hop on the semi-new technology bandwagon. Because it is enterprise software, unit testing frameworks, reusability, and extendability are all important considerations as well.
Any suggestions?
I realize this question was about technology options, but there's a fundamental concern that seems really important to call out:
From an organizational perspective I would like the current UI team to also have ownership of the API.
While this sounds like a logical approach, it may not work out well unless you're UI team is made up of really solid engineers. SaaS API development is arguably one of the most challenging aspects of modern software design. A great API will make everyone's lives easier, while a poor API will bring your system to its knees and leave you completely clueless as to why.
As a quick example, if you don't solve the end user's needs in the right way, you're likely to force a number of n+1 problems on them (and thus, on you.)
There is a bunch of great material out there about how to design great APIs and even more about the pitfalls of designing a bad one. Generally speaking, most of the UI devs I've worked with, particularly ones that are only familiar with scripting languages, are not people I would entrust to API design. Instead I would utilize them as customers (in a Scrum sense) who guide the design by describing end-user needs.
I faced something like this on a previous project, where we ended up going with a combo of Esper and our own DSL written using ANTLR 3.0. Our biggest concern with using a fully funcional runtime, was sandboxing the user's code.
That said, I think Node.JS would be one of the easier ones to sandbox and it fits your needs. Maybe using something like this: http://gf3.github.com/sandbox/ or looking into Cloud9's code to see how they keep things safe. I also like that with Node.js you could give your users a pretty niffy editor using Ace.
Also check out this post: How to run user-submitted scripts securely in a node.js sandbox?
I've searched for a while, but I can't find anything related on Google or here.
Me and some friends were debating starting a company, so I figure it might be good to do a quick pilot project to see how well we can work together. We have a designer who can do HTML, CSS and Flash, enjoys doing art, but doesn't like to do HTML and CSS... And 2 programmers that are willing to do anything.
My question is, from an experienced site builder's perspective, what steps do we do - in chronological order - to properly handle a website? Does the designer design the look and feel of the site, then the programmers fill in the gaps with functionality? Or do the programmers create a "mock-up" of the site with most of the functionality, then the designer spices it up? Or is it more of a back-and-forth process?
I just want to know how a professional normally handles it.
Update:
A recap taking some of the notes from each post.
Step 1: Define requirements. What will your site/application do?
Step 2: Use cases. Who will use the application, and what will they do with it? This doesn't have to be done with a bunch of crazy UML diagrams, just use whatever visual aids you think work best for you. Find a CMS vendor, or a search vendor, or both. While planning, maybe do some competitor analysis, and see how those in similar fields have done theirs.
Step 3: Visual proof-of-concept. This is done by your designer, NOT your programmers... Programmers are notoriously bad at UI. Use an image program like Photoshop, not an HTML editor. Leave it fluid and simple at first. Select the three-color theme for the site (two primaries and an accent.) Get a sense of how you want to lay things out, keeping in mind the chosen CMS and/or search functionality. Focus hard on usability, add pizzaz later. Turn the created concept into JPEG mock-ups, or create a staging site to allow the client to view the work. A staging site will allow for future releases to be tested prior to moving it to production.
Step 4: Once the site is conceptualized by your designers, have your HTML/CSS developer turn it into markup. He/she should shoot for XHTML compliance and test on as many major browsers as you can. Also a good time to set up versioning/bug tracking/management systems, to keep track of changes, bugs, and feedback.
Step 5: Have your programmers start turning your requirements into software. This can and should be done in parallel with Step 4- there's no reason they can't be coding up the major pieces and writing tests while the UI is designed and developed.
Step 6: Marry up the final UI design with the code. Test, Test, Test!!
Step 7: Display end result to client, and get client sign-off.
Step 8: Deploy the site to production.
Rinse, Repeat...
Step 1: Define requirements. What will your site/application do?
Step 2: Use cases. Who will use the application, and what will they do with it? This doesn't have to be done with a bunch of crazy UML diagrams, just use whatever visual aids you think work best for you.
Step 3: Visual proof-of-concept. This is done by your designer, NOT your programmers. Use an image program like Photoshop, not an HTML editor. Leave it fluid and simple at first. Select the three-color theme for the site (two primaries and an accent.) Get a sense of how you want to lay things out. Focus hard on usability, add pizzaz later.
Step 4: Once the site is conceptualized by your designers, have your HTML/CSS developer turn it into markup. He/she should shoot for XHTML compliance and test on as many major browsers as you can.
Step 5: Have your programmers start turning your requirements into software. This can and should be done in parallel with Step 4- there's no reason they can't be coding up the major pieces and writing tests while the UI is designed and developed.
Step 6: Marry up the final UI design with the code. Test, Test, Test!!
Rinse, Repeat...
There is no one universal way. Every shop does it differently. Hence, a warning: gross generalizations follow.
Web development typically consists of much shorter release cycles, because it's so simple to push out a release, compared to client-side software. Thus the more "agile" methods are more frequently used than the "waterfall" models encountered in developing client software.
Figure out what, exactly, you're building.
Take care of all the legal stuff (e.g. what business entity you'll be forming, how will each team member be compensated for their work, will there be health benefits, etc).
Mockups. I suggest having the designers do the mockups since programmers are notoriously bad at UI design.
Set up some sort of bug tracking / case management system so that you have a centralized place for all your feature requests and bug reports.
Start coding.
Once you have a simple version of your app, get some people to test it out to make sure you're on the right path.
???
Profit!
As a first step, I'd recommend doing a bit of up-front design using an approach such as paper prototyping, to lock down what it is you want your website to do, and roughly how you want it to look.
Next up, read up on the Agile approach to software development and see if you like the sound of what it suggests. It tends to work best with smaller, well-motivated teams.
Figure out the minimum amount of functionality you can create that you can deliver as a product so that you can get user feedback as soon as possibly. Then expect to iteratively add functionality to the product over time.
The Web Style Guide provides a pretty detailed overview of the process.
You should mix and match the lists provided here for your needs.
I just want to make sure you know one thing...
Customers are "stoopid" when it comes to web design.
You will have to claw, scrape, drag, gnash, rip, and extricate every requirement from their naive little souls. If you fail to do so? Guess who gets the blame?
The road you now look down is a hard one filled with competition, stress, and risk. It requires endurance, faith, patience, and the ability to eat ramen 5 of 7 days a week.
To add (or repeat) Dave Swersky's list.
Gather requirements from clients
Do some competitor analysis. Gather
screen shots of competitor sites.
Build a sitemap /wireframe - What is
the structure/content of the site?
Get designers to create JPG mockups.
They may use the screen shots for
"inspiration"
Get feedback from
clients based on JPEG's
Create HTML
mockups from JPEG's
Get feedback
from clients. Go back to step 4 if
necessary
Implement HTML using
technology of choice
Unit test the site
UAT and obtain sign off.
Deploy to live
client feedback is critical, they should be involved in every step to ensure a successful implementation.
Hope this helps
In addition to the steps outlined in other answers, I'd add this (to be added somewhere near the end of the "cycle"):
x. Once you have a more or less end to end solution, set up a staging site.
y. Get client sign off on staging site.
z. Deploy to production site.
Celebrate! But not too hard, there's almost always going to be a few iterations of changes, because users rarely know exactly what they really want the first time around.
So, when (not if), the client asks for changes, you can work on the changes and promote them to the staging site first! This is important because a) it gives clients a chance to preview changes before the whole world sees them b) if the integrity of the data on the production site is important, you can hopefully weed out any issues on the staging site before they impact production data.
Just to give something on the other side of the coin. Where I work, we have for the past couple of years, worked on a redesign of the company's website. Here are some highlights of the process:
Identify vendors for various functions that will be needed. In this case that meant finding a Content Management System vendor as well as a Search vendor.
Get a new design for the site that can be applied to what was selected in the first step.
Using system integrators and in-house developers, start to build some of the functionality for the site and take the flexible, customizable software in 1 and make it useful for the organization. Note that this is where a couple of years have been spent getting this working and some business decisions ironed out.
Release a preview site to verify functionality and fix bugs, add enhancements as needed.
Note that in your case you may not have the same budget but there are various CMS frameworks out there to select as well as how much integration do you want to have for the site? Does it have to talk to a half-dozen different systems? In the case I mentioned above there are CRM integrations, ESB integrations, search integrations, and translation integrations to give a few examples of where things had to be wired up correctly.
In response to the comment, be sure you and the client know what is meant by "simple" as if there is any e-commerce functionality, forums, or personalization these are examples where it can be important to know what is needed now and have an idea of what is needed down the road as there can likely be a ton of things that customers may want but you have to figure out some of the nitty-gritty details at points in the future. For example, some people may think that Google is simple, and from an end-user perspective it is though how many computers does Google have running how many different applications doing how much processing 24/7? Quite a bit, I'd imagine. Simple is good, but sometimes making something look simple can be incredibly hard to do.
Recently I've came up with the question is it worth at all to spent development time to generate automatic unit test for web based projects? I mean it seems useless at some point because at some point those projects are oriented on interactions with users/clients, so you cannot anticipate the whole possible set of user action so you be able to check the correctness of content showed. Even regression test can hardly be done. So I'm very eager to know to know the opinion of other experienced developers.
Selenium have a good web testing framework
http://seleniumhq.org/
Telerik are also in the process of developing one for web app testing.
http://www.telerik.com/products/web-ui-test-studio.aspx
You cannot anticipate the whole
possible set of user action so you be
able to check the correctness of
content showed.
You can't anticipate all the possible data your code is going to be handed, or all the possible race conditions if it's threaded, and yet you still bother unit testing. Why? Because you can narrow it down a hell of a lot. You can anticipate the sorts of pathological things that will happen. You just have to think about it a bit and get some experience.
User interaction is no different. There are certain things users are going to try and do, pathological or not, and you can anticipate them. Users are just inputting particularly imaginative data. You'll find programmers tend to miss the same sorts of conditions over and over again. I keep a checklist. For example: pump Unicode into everything; put the start date after the end date; enter gibberish data; put tags in everything; leave off the trailing newline; try to enter the same data twice; submit a form, go back and submit it again; take a text file, call it foo.jpg and try to upload it as a picture. You can even write a program to flip switches and push buttons at random, a bad monkey, that'll find all sorts of fun bugs.
Its often as simple as sitting someone down who's unfamiliar with the software and watching them use it. Fight the urge to correct them, just watch them flounder. Its very educational. Steve Krug refers to this as "Advanced Common Sense" and has an excellent book called "Don't Make Me Think" which covers cheap, simple user interaction testing. I highly recommend it. It's a very short and eye opening read.
Finally, the client themselves, if their expectations are properly prepared, can be a fantastic test suite. Be sure they understand its a work in progress, that it will have bugs, that they're helping to make their product better, and that it absolutely should not be used for production data, and let them tinker with the pre-release versions of your product. They'll do all sorts of things you never thought of! They'll be the best and most realistic testing you ever had, FOR FREE! Give them a very simple way to report bugs, preferably just a one button box right on the application which automatically submits their environment and history; the feedback box on Hiveminder is an excellent example. Respond to their bugs quickly and politely (even if its just "thanks for the info") and you'll find they'll be delighted you're so responsive to their needs!
Yes, it is. I just ran into an issue this week with a web site I am working on. I just recently switched-out the data access layer and set up unit tests for my controllers and repositories, but not the UI interactions.
I got bit by a pretty obvious bug that would have been easily caught if I had integration tests. Only through integration tests and UI functionality tests do you find issues with the way different tiers of the application interact with one another.
It really depends on the structure and architecture of your web application. If it contains an application logic layer, then that layer should be easy to unit test with automating tools such as Visual Studio. Also, using a framework that has been designed to enable unit testing, such as ASP.NET MVC, helps alot.
If you're writing a lot of Javascript, there have been a lot of JS testing frameworks that have come around the block recently for unit testing your Javascript.
Other than that, testing the web tier using something like Canoo, HtmlUnit, Selenium, etc. is more a functional or integration test than a unit test. These can be hard to maintain if you have the UI change a lot, but they can really come in handy. Recording Selenium tests is easy and something you could probably get other people (testers) to help you create and maintain. Just know that there is a cost associated with maintaining tests, and it needs to be balanced out.
There are other types of testing that are great for the web tier - fuzz testing especially, but a lot of the good options are commercial tools. One that is open source and plugs into Rails is called Tarantula. Having something like that at the web tier is a nice to have run in a continuous integration process and doesn't require much in the form of maintenance.
Unit tests make sense in TDD process. They do not have much value if you don't do test-first development. However the acceptance test are a big thing for quality of the software. I'd say that acceptance test is a holy grail of the development. Acceptance tests show whether the application satisfies the requirements. How do I know when to stop developing the feature --- only when all my acceptance test pass. Automation of acceptance testing a big thing because I do not have to do it all manualy each time I make changes to the application. After months of development there can be hundreds of test and it becomes unfeasible (sometime impossible) to run all the test manually. Then how do I know if my application still works?
Automation of acceptance tests can be implemented with use of xUnit test frameworks, which makes a confusion here. If I create an acceptance test using phpUnit or httpUnit is it a unit test? My answer is no. It does not matter what tool I use to create and run test. Acceptance test is the one that show whether the features is working IAW requirements. Unit test show whether a class (or function) satisfies the developer's implementation idea. Unit test has no value for the client (user). Acceptance test has a lot of value to the client (and thus to developer, remember Customer Affinity)
So I strongly recommend creating automated acceptance tests for the web application.
The good frameworks for the acceptance test are:
Sahi (sahi.co.in)
Silenium
Simpletest (I't a unit-test framework for php, but includes the browser object that can be used for acceptance testing).
However
You have mentioned that web-site is all about user interaction and thus test automation will not solve the whole problem of usability. For example: testing framework shows that all tests pass, however the user cannot see the form or link or other page element due to accidental style="display:none" in the div. The automated tests pass because the div is present in the document and test framework can "see" it. But the user cannot. And the manual test would fail.
Thus, all web-applications needs manual testing. The automated test can reduce the test workload drastically (80%), but manual test are as well significant for the quality of the resulting software.
As for the Unit testing and TDD -- it make the code quality. It is beneficial to the developers and for the future of the project (i.e. for projects longer that a couple of month). However TDD requires skill. If you have the skill -- use it. If you don't consider gaining the skill, but mind the time it will take to gain. It usually takes about 3 - 6 month to start creating a good Unit tests and code. If you project will last more that a year, I recommend studding TDD and investing time in proper development environment.
I've created a web test solution (docker + cucumber); it's very basic and simple, so easy to understand and modify / improve. It lies in the web directory;
my solution: https://github.com/gyulaweber/hosting_tests
I've started a new role in my life. I was a front end web developer, but I've now been moved to testing web software, or more so, automating the testing of the software. I believe I am to pursue a BDD (Behavior Driven Development) methodology. I am fairly lost as to what to use, and how to piece it together.
The code that is being used/written is in Java to write a web interface for the application to test. I have documentation of the tests to run, but I've been curious how to go about automating it.
I've been directed to Cucumber as one of the "languages" to help with the automation. I have done some research and come across a web site for a synopsis of BDD Tools/Frame works,
8 Best Behavior Driven Development (BDD) Tools and Testing Frameworks. This helped a little but then I got a little confused of how to implement it. It seems that Selenium is a common denominator in a lot of the BDD frameworks for testing a GUI, but it still doesn't seem to help describe what to do.
I then came across the term Functional Testing tool, and I think that confused me even more. Do they all test a GUI?
I think the one that looked like it was all one package was SmartBear TestComplete, and then there is, what seems to be, another similar application by SmartBear called, SmartBear TestLeft, but I think I saw that they still used Cucumber for BDDing it. There a few others that looked like they might work as well, but I guess the other question is what's the cheapest route?
I guess the biggest problem I have is how to make these tests more dynamic, as the UI/browser dimensions can easily change from system to system, and how do I go about writing automation that can handle this, and tie into a BDD methodology?
Does anyone have any suggestions here? Does anybody out there do this?
Thanks in advance.
BDD Architecture
BDD automation typically consists of a few layers:
The natural language steps
The wiring that ties the steps to their definition
The step definitions, which usually access page objects
Page objects, which provide all the capabilities of a page or widget
Automation over the actual code being exercised, often through the GUI.
The wiring between natural language steps and the step definitions is normally done by the BDD tool (Cucumber).
The automation is normally done using the automation tool (Selenium). Sometimes people do skip the GUI, perhaps targeting an API or the MVC layer instead. It depends how complex the functionality in your web page is. If in doubt, give Selenium a try. I've written automation frameworks for desktop apps; the principle's the same regardless.
Keeping it maintainable
To make the steps easy to maintain and change, keep the steps at a fairly high level. This is frequently referred to as "declarative" as opposed to "imperative". For instance, this is too detailed:
When Fred provides his receipt
And his receipt is scanned
And the cashier clicks "Refund to original card"
And the card is inserted...
Think about what the user is trying to achieve:
When Fred gets a refund to his original card
Generally a scenario will have a few Givens or Thens, but typically only one When (unless you have something like users interacting or time passing, where both events are needed to illustrate the behaviour).
Your page objects in this scenario might well be a "RefundPageObject" or perhaps, if that's too large, a "RefundToCardPageObject". This pattern allows multiple scenario steps to access the same capabilities without duplication, which means that if the way the capabilities are exercised changes, you only need to change them in one place.
Different page objects could also be used for different systems.
Getting started
If you're attacking this for the first time, start by getting an empty scenario that just runs and passes without doing anything (make the steps empty). When you've done this, you'll have successfully wired up Cucumber.
Write the production code that would make the scenario run. (This is the other way round from the way you'd normally do it; normally you'd write the scenario code first. I've found this is a good way to get started though.)
When you can run your scenario manually, add the automation directly to the steps (you've only got one scenario at this point). Use your favourite assertion package (JUnit) to get the outcome you're after. You'll probably need to change your code so that you can automate over it easily, by e.g.: giving relevant test ids to elements in your webpage.
Once you've got one scenario running, try to write any subsequent scenarios first; this helps you think about your design and the testability of what you're about to do. When you start adding more scenarios, start extracting that automation out into page objects too.
Once you've got a few scenarios, have a think about how you might want to address different systems. Avoid using lots of "if" statements if you can; those are hard to maintain. Injecting different implementations of page objects is probably better (the frameworks may well support this by now; I haven't used them in a while).
Keep refactoring as you add more scenarios. If the steps are too big, split them up. If the page objects are too big, divide them into widgets. I like to organize my scenarios by user / stakeholder capabilities (normally related to the "when" but sometimes to the "then") then by different contexts.
So to summarize:
Write an empty scenario
Write the code to make that pass manually
Wire up the scenario using your automation tool; it should now run!
Write another scenario, this time writing the automation before the production code
Refactor the automation, moving it out of the steps into page objects
Keep refactoring as you add more scenarios.
Now you've got a fully wired BDD framework, and you're in a good place to keep going while making it maintainable.
A final hint
Think of this as living documentation, rather than tests. BDD scenarios hardly ever pick up bugs in good teams; anything they catch is usually a code design issue, so address it at that level. It helps people work out what the code does and doesn't do yet, and why it's valuable.
The most important part of BDD is having the conversations about how the code works. If you're automating tests for code that already exists, see if you can find someone to talk to about the complicated bits, at least, and verify your understanding with them. This will also help you to use the right language in the scenarios.
See my post on using BDD with legacy systems for more. There are lots of hints for beginners on that blog too.
Since you feel lost as to where to start, I will hint you about some blogs I have written that talks a bit about your problem.
Some categories that may help you:
http://www.thinkcode.se/blog/category/Cucumber
http://www.thinkcode.se/blog/category/Selenium
This, rather long and old post, might give you hints as well:
http://www.thinkcode.se/blog/2012/11/01/cucumberjvm-not-just-for-testing-guis
Notice that versions are dated, but hopefully it can give some ideas as what too look for.
I am not an expert on the test automation but I am currently working on this part. So let me share some idea and hope it will help you at the current stage.
We have used selenium+cucumber+intellij for testing web application. We have used testcomplete+cucumber+intellij for testing java desktop application.
As to the test of web application, we have provided a test mode in our web application, which allows us to get some useful details of the product and the environment; and also allows us to easily trigger events through clicking the button and inputting text into the test panel under test mode.
I hope these are helpful for you.