Question
Is it worth building a web application front-end for my department's automated regression tests? I've searched quite a bit and I don't think anything like this exits. Basically the web application would allow a user to specify a URL, expected inputs, and expected outputs and an expected return URL. On the back-end a headless browser would be running on the server to test the scenario just defined by the user, most likely using calls to a headless browser... I've searched quite a bit to see if something as simple as this exists but I haven't had any luck. I've found lots of tools for allowing programmatic operation of browser commands but a web front-end for testing another web application I have not.
Background
My team has dedicated automated regression tests that the testers run on their local machines. The tests are written in Python, utilize some Selenium integration plugins, and use an excel spreadsheet as input on what to test. They are maintained by the QA department.
Problem
Nobody outside the QA team knows how extensive these regression tests
are because they exist only on individual laptops.
They have no central repository, and the dev team has no means of
actively updating these tests as we build new features. We must leave
it 100% up to the QA department.
The business analysts don't have access to the results of these
tests. Because of all this, a lot of uncertainty exists around our
automated testing increasing reluctance to change things without
instructing the QA team to perform full scale manual regression
tests...
This has led me to consider putting all of our Selenium tests in the cloud behind a user-friendly web front-end that anyone can use and access from anywhere. They could then easily create new tests using dropdown menus. Everyone, developers, testers, and business analysts, can see whats covered in a test sequence and update them as we add new features. I believe this would also make it easier to have Jenkins jobs trigger tests to run at timed intervals if web application exposed web service hooks for jenkins... But I feel like perhaps I'm re-inventing the wheel. Is what I'm proposing to build worth it?
Personally i would not spend too much time in creating a website to accept user input to create a testscript. Instead I would spend that time in creating a solid test framework and use Jenkins to trigger the tests.
You also need to consider the 'website' maintenance in future. What will happen if some new feature has to be included in the website? QA/BA team will depend on the developer to add the feature.
I think it is better to use keyword driven framework - where you can write your entire test in spreadsheet. [In my project QA people who are not familiar with programming create test scripts with this approach].
As Jenkins web based application - anyone can trigger your automated regression tests. Even the BAs (in my project, that is what i have done). No technical skill is required. We can also pass parameters through jenkins. Parameters can be anything from text to a file. So, you can upload a file which contains the steps to be executed to the jenkins job and the rest should be taken care by your test framework.
You would definitely need a central repository. It is a must have. You can take a look at VisualSVN server. It is easy and FREE.
Keyword Driven framework using Selenium:
http://www.testautomationguru.com/keyword-driven-framework-for-localization-testing-using-selenium-webdriver/
Continuous regression & results:
http://www.testautomationguru.com/continuous-regression-testing-best-practises/
Smoke Test after each build:
http://www.testautomationguru.com/automated-smoke-test-best-practises/
Related
I am trying to formalize the development workflow and here's the first draft. Welcome suggestions on the process and any tweaks for optimization. I am pretty new when it comes to setting up the processes and would be great to have feedback on it. P.S: We are working on an AWS Serverless application.
Create an issue link in JIRA - is tested by. The link 'is tested by' has no relevance apart from correcting displaying the relation while viewing the story.
Create a new issue type in JIRA - Testcase. This issue type should have some custom fields to fully describe the test case.
For every user story, there will be a set of test cases that are linked to the user story using the Jira linking function. The test cases will be defined by the QA.
The integration/e2e test cases will be written in the same branch as the developer. E2E test cases will be written in a separate branch as it's a separate repository (Open for discussion).
The Test case issue type should also be associated with a workflow that moves from states New => Under Testing => Success/Failure
Additionally, we could consider adding capability in the CI system to automatically move the Test case to Success when the test case passes in the CI. (This should be possible by using JIRA API ). This is completely optional and we will most probably be doing it manually.
When all the Test cases related to a user story to success, The user story can then be moved to Done.
A few points to note:
We will also be using https://marketplace.atlassian.com/apps/1222843/aio-tests-test-management-for-jira for test management and linking.
The QA should be working on the feature branch from day 1 for adding the test cases. Working in the same branch will enable the QA and developer to be always in Sync. This should ensure that the developer is not blocked waiting for the test cases to be completed for the branch to be merged into development.
The feature branch will be reviewed when the pull request is created by the developer. This is to ensure that the review is not pending until the test cases have been developed/passed. This should help with quick feedback.
The focus here is on the "feature-oriented QA" process to ensure the develop branch is always release-ready and that only well-tested code is merged into the develop branch.
A couple of suggestions:
For your final status consider using Closed rather than Success/Failure. Success/Failures are outcomes rather than states. You may have other outcomes like cancelled or duplicate. You can use the Resolved field for the outcomes. You could also create a custom field for Success/Failure and decouple it from both the outcome and status. You ideally do not want your issue jumping back in forth in your workflow. If Failure is a status then you set yourself up for a lot of back and forth
You may also want to consider a status after New Test Creation for the writing of the test case and a status after that such as Ready for Testing. This would allow you to more specifically where the work is in the flow and also capture the amount of time that is spent writing tests, how long test cases wait, and how much time is spent actually executing tests and defect remediation
Consider adding a verification rule to your Story workflow that prevents a story from being closed until all the linked test cases are closed
AIO Tests for Jira, unlike other test management systems, does not clutter Jira, by creating tests as issues. So you need not create an issue type at all.
With it's zero setup time, you can simply start creating tests against stories. It has a workflow from Draft to Published (essentially equaling Ready for Testing).
The AIO Tests jira panel shows the cases associated with the stories and their last execution status to get a glimpse of the testing progress of the story as shown below. It allows everyone from the Product to the Developer to get a glimpse of the testing status.
You can also create testing tasks and get a glimpse of the entire execution cycle in the AIO Tests panel.
It also has a Jenkins plugin + REST APIs to make it part of your CI/CD process.
Just wondering if there is a way to integrate TFS with TestRail to replace(get rid of entirely) the Test Hub within TFS to use TestRail to record Test Plans?
My concern with removing the Test Hub, would be if Test Rail can still reference IDs in Bug and Stories within TFS and vice versa?
You currently cannot remove the standard Hubs in Visual Studio Team Services / TFS or replace them with something else. You can enable an extension that either adds its own functionality under said hub as a separate tab or adds another Test Rail hub to the top level (if there is any) or write your own. Extensions currently cannot leave their sandbox to overwrite standard functionality.
There is nothing preventing anything from keeping track of work item numbers anywhere, so the second part of your question, whether it would have broken any form of integrations, that's unlikely.
If you are on TFS, you could try creating a custom process template that doesn't have the Test Case, Shared Step, Test Suite and Test Plan work item types, this will likely at least completely cripple the existing functionality. In the on-prem version you can also customize the files on disk, I've never tried, but it's likely that you could probably hack the test hub away. That would be a totally unsupported scenario through.
I'm looking for a some sort of management/reporting tool that collects the results of tests, stores them for reporting, then lets users generate reports based on those tests.
We have numerous test running tools that run on a variety of platforms, but all output test results in the JUnit format. The test are not specific to hardware or platform, but rather generic. What we would like to do is have an automated (or manual) test run be able to submit it to a central location along with additional information, like platform, OS, hardware configuration and maybe user defined data. The management/reporting tool would store this data.
Then, a manager would be able to go to the tool and request (or more likely, access a dashboard that developers have setup) an update on the current status. This could be a list of test results that were run in a particular configuration, or a hardware status, or just the results of specific test(s).
Any suggestions?
We built a test management tool Enterprise Tester (www.enterprisetester.com) that allows users to pull automated test results in nUnit, nUnit, XSLT, Selenium etc and report off the results.
In addition to pulling the results and reporting you are able to trace these tests back to requirements that have been created giving you the ability to see test coverage and the status of this coverage on dashboards. If you are using JIRA (or google) or anything that uses open social gadgets you can pass these gadgets to other tools also.
Feel free to contact me directly if you would like to talk further about it
Regards
Bryce
You can also try an open source project called Allure Framework. It was created specially for showing test execution results in a nice form. A set of popular test frameworks such as JUnit, PHPUnit, TestNG, py.test, RSpec, Scalatest are already supported. Other ones such as NUnit will be supported soon.
Check out Hudson. it's very useful and configurable
I'd like to run tests that simulate users modifying certain data at the same time for a grails application.
Are there any plug-ins / tools / mechanisms I can use to do this efficiently? They don't have to be grails specific. It should be possible to fire multiple actions in parallel.
I'd prefer to run the tests on functional level (so far I'm using Selenium for other tests) to see the results from the user perspective. Of course this can be done in addition to integration testing if you'd recommend to run concurrent modification tests on integration level as well.
I have used Geb (http://grails.org/plugin/geb/) for this recently. It is a layer on top of WebDriver and Selenium etc.. Its very easy to write a Grails script to act as a user in your app and then just run several instances on different consoles. Geb uses a jQuery style syntax for locating stuff in the DOM which is very cool:
import geb.Browser
import geb.Configuration
includeTargets << grailsScript("_GrailsInit")
target(main: "Do stuff as fast as possible") {
Configuration cfg = new Configuration(baseUrl: "http://localhost:8080/your_app/")
Browser.drive(cfg) {
go "user/login"
$("#login form").with {
email = "someone#somewhere.com"
password = "secret"
_action_Login().click()
}
...
}
}
setDefaultTarget(main)
Just put your script in scripts/YourScript.groovy and then you can do "Grails YourScript" to run it. I tracked down some concurrency issues by just running several of these at full speed. You do need to build a war and deploy it properly as Grails in dev mode is very slow and runs out of permgen space quite quickly.
Just an idea: it seems difficult to make client starts at the same time, but can they wait for each other just before modifying data?
Such as, a client keeps logging its process: "Client x access DATA", "Client x editing DATA" in a file. They also keep looking this log file, to see other clients' progress. Then don't permit a client complete editing a DATA until another client comes in to edit that DATA.
I've found Grinder to be an excellent tool for heavy load testing. Running multiple instances performing the same tests at one time can often uncover concurrency issues in your app that you wouldn't find with normal tests.
If you want to do this within Unit Tests or in-code Integration Tests, you could always spin up multiple threads in code and have them perform the task you're trying to test.
Are you primarily interested in load testing multiple active users, as opposed to those who just have a HttpSession? Solid load testing is predicated on really really good func. testing however. How are your functional tests organized and executed to-day? Grails has a plug-in* for that, too, and it appears to be in the Top of the Pops at the plug-in portal.
Are you attempting to test out how the optimistic locking mechanism performs under load?
If the former use case is the one that means more, it sounds like you may be looking for JUnitPerf. Here is the --> download
*functional-test <1.2.7> -- Functional Testing
WebTest is built on Ant which provides the parallel task. You might be able to use this in conjuction with the Webtest plugin to run some actions in parallel. I've never tried it though.
Have a look at MultithreadedTC. It looks like it could be used to exercise certain interleaving cases where multiple threads are executing your code in ways you consider potentially risky.
I doubt you'll find a convenient way to test specific multithreaded interleaving cases with Selenium because Selenium controls a browser which sends requests to your server. I haven't heard of a way to instrument code for multithreaded interleaving tests when the threads are started as real web requests to a running web server.
I've actually looked through Selenium questions on here and didn't find quite what I was looking for.
Basically I have about 10 "use cases" for smoke testing my site. Basic things such as, can a user log in, can they register etc..
I want to set this up on an interval such as every 10 minutes run these tests.
Is this possible with Selenium Remote Control? Does anyone have a link/tutorial they could point me towards. I'm fairly confident this is possible but I'm just not 100% sure how to get it all set up and running.
Thanks in advance.
Yes this can be done with Selenium RC. I have some Selenium Tutorials on my site. I have set up a basic user experience monitor before using Selenium RC, C#/Nunit and Windows Scheduled tasks to start the job to check the speed of our web app through the day.
Since I recommend using Selenium RC you can use any language you want to write the test.
Dending on what kind of environment you're working with you could use something like JUnit and an automated build system like hudson. This gives you all sorts of notification infrastructure when something goes wrong.
I have known people to run a script like this against both test and production systems (with a fixed user). In test environments you can discover programming mistakes, in production you can assess the up-ness of your system at a far more interesting level than pings or process watching.
Take a look at New Relic's Synthetics product, which wraps Selenium and provides periodic runs with alerting.