I have found a lot of references to UI automation for modern UI applications. But I'm not searching a way to record click/touch paths rather than a program that just simulates random input (from single to multi touch). Somehow comparable to the stress test tool that is available for Microsoft Pixelsense. Does anyone knows a tool?
Related
I am working on implementing accessibility (for visually impaired individuals) for one of our web application. It need to be ARIA compliant. Right now we are testing our changes with screen reader manually.
For example we have Tree control in our application. I open NVDA screen reader and then navigate through my Tree Nodes. NVDA screen reader speaks out
Node XYZ expanded, (When I expand XYZ node with right arrow key)
Node XYZ collapsed, (When I collapse XYZ node with left arrow key)
Along with the voice it also write down this text.
But all this is manual. Now we want to setup automated test cases for the same so that any regression bugs can be caught by are test cases. Do there exist any such tool which we can use to automate our test cases. Any direction will be helpful.
PS: Just for a sake of comparison. We have nunit to write test cases for c# application. After writing test cases we integrate them into our build process. Any breaking change is caught when we run the build. I am looking for something similar to test out our aria compliance and screen reader's behavior with our web application.
I don't know of any existing tools for testing screen readers, however, there are accessibility APIs that test websites and web applications.
axe-core from Deque Systems is widely used and well-supported.
I wrote a python package to run automated web accessibility tests that uses axe-core and selenium.
While it isn't quite what you are looking for, it does cover about 60% of accessibility guidelines, including aria roles and attributes. It should help with determining screen reader usability.
You could integrate axe into C#, similar to my python package and the Java package, also created by Deque.
I hope this helps!
It sounds like you're already performing some pretty good manual accessibility testing against your web application, which no automated testing tool is going to be able to replicate completely. That said, if you're looking to take care of any low-hanging fruit with an automated solution, like Kimberly suggested, there are several automated accessibility testing tools out there that you can relatively easily integrate into your existing web application's testing framework that might help you.
One such tool is Continuum, which doesn't have a C#-based library offering at the moment, but could be used in a separate testing framework to be run against your web application after it has already been built. This may be preferable depending on your use case, as code linters for accessibility aren't perfect and are highly language-dependent, whereas testing the HTML of your web application more closely matches the screen reader use case you say you're trying to test for. You could even integrate Continuum into your existing CI/CD process to make sure your application is tested during development as opposed to afterwards, to reduce your manual accessibility testing load.
Continuum has a few sample projects to get you started, depending on your technologies of choice. Free versions are available at webaccessibility.com if you're interested. Most of them are Java- or JavaScript-based at the moment.
Appreciate this is quite an old question, but having explored this area a lot recently thought was worth updating with the state as of 2023 as there is now some progress in this space.
Current tooling available at time of writing (that I’m aware of, may not be exhaustive):
guidepup - NodeJS automation for VoiceOver and NVDA supporting all keyboard commands and getters for spoken phrases (disclaimer: I’m the author).
auto-vo - CLI for navigating sequentially through a page with VoiceOver and reporting the spoken phrases, also exports a separate Node module for some interactions with VoiceOver.
screen-reader-reader - NodeJS automation for VoiceOver and NVDA for starting, stopping, and getting spoken phrases.
web-test-runner-voiceover - NodeJS plugins for #web/test-runner to automate VoiceOver testing.
nvda-testing-driver - .NET automation for NVDA supporting all keyboard commands and getters for spoken phrases.
assistive-webdriver - NodeJS implementation of a Webdriver server that allows remote testing of screen readers (e.g. NVDA, JAWS) running in a VM.
As stated in other answers, there are also a number of static analysis tools such as axe, as well numerous browser extensions offering similar static analysis, and companies such as Assistiv Labs offering remote environment services to interact with screen readers manually (similar to SauceLabs/BrowserStack/etc. but for screen readers, magnification etc. - no affiliation and haven’t used services so can’t vouch, simply an observation).
Worth calling out that none of these cover the full range of a11y requirements - there is more to a11y than just screen readers. A combined/layer approach including automation, manual testing, and user testing likely preferred.
Is there a possibility to do a automation testing of application developed using WxWidgets on windows platform? If so how do I do that?
wxWidgets uses native widgets so any automated testing solution for Win32 programs would work with wxWidgets applications. However IME the typical "point-and-click" tests are not that great in practice and it's better to write the tests for the GUI code in the same way as you do it for the rest of the program.
There are two problems that need to be solved when doing it though: first, how to trigger various actions in the GUI. This is addressed by wxUIActionSimulator. It is not perfect, but if your application has a reasonable keyboard interface (as it should), it should be enough.
The second problem is related to the control flow in the GUI applications: if some action results in opening of a modal dialog, the test can't continue until the dialog is closed by user, which is inappropriate for unattended tests. wxWidgets provides (still somewhat experimental and not documented yet) wxTEST_DIALOG macro for dealing with this, its use is explained in this comment.
Combination of these two approaches allows to write tests for the real-life GUI applications and, moreover, the tests are portable and not limited to Windows platform.
I am new to testing. The commonly used terms like Framework and tool confuses me a lot. Can anyone please explain me the difference between a Framework like STAF[software testing automation framework]and Tool like selenium.
Also how to select a tool for a particular framework. What are the criterias used for selection?
Brief explanations are welcomed!!
Tool:
Simply put, a tool is a software. In case of test automation, tools are software that let you automate your tests on an application. There are many test automation tools that you can choose from depending on your requirements. Some examples are Selenium, UFT, Visual Studio CUIT, Jamo Solutions Meux Test, T-Plan Robot, Telerik Test Studio etc.
Often, you'll have to write tests in the tools using a supported programming language. For instance, testers using UFT need to code in VB while those using Visual Studio can code in both VB and C#. However, some testing tools (like Telerik Test Studio) let you write script-less tests where your tests will consist of a bunch of easily understandable keywords, not code.
Framework:
The most popular test automation tools like Selenium and Visual Studio provide all the basic features you require to build your own tests. However, they do not provide ready-made features (like Reporting and Exception Handling) for testing. This requires the creation of a 'Framework' which is nothing but a collection of code written using a tool of one's choice that makes testing an application easy. Simply put, a framework is what you create with a tool (or a collection of tools) to test your application.
A typical framework consists of two parts: test scripts and function libraries. Test Scripts are the pieces of code that need to be executed to perform actions on the application under test (AUT). Function Libraries are classes consisting of important functions that are called by your test scripts. These can include timing functions, reporting/logging functions, exception handling functions, data communication functions etc.
You can also use an external database to pass test data to your test scripts during run-time instead of hard-coding it in your test scripts. Frameworks that employ external databases are called data-driven frameworks. The external database can be of your choice, be it a SQL Server, an XML file or a simple Excel spreadsheet. Data-driven frameworks make use of APIs or include custom-made classes that let you communicate with the database to transfer data.
Another type of framework is the keyword-driven framework. These frameworks are used in long-term test automation projects that require scripting of thousands of test cases. The main objective of these frameworks is to reduce the time taken to script a test case by reusing code that has already been written. They often include very strong function libraries which enable scripting of test cases using just predefined keywords. For example, common actions on an application like login and logout are performed by single line codes like:
Actions.Login();
and
Actions.Logout();
where Actions is a Function Library that consists of the Login() and Logout() functions. This massively reduces the script size and the long-term maintenance requirements of the test script, among other benefits.
Of course, you can either build a test automation framework and use it for your own application or create a generic test automation framework and make it available to the testing community for everyone to use, which is what STAF is.
Selection of testing tools:
To address your second question, there is no straight-forward answer to it. There are a number of criteria that can affect your decision. But in the end, it is all about your requirements and the requirements of your AUT.
If it is a Windows desktop app, you have to use Coded UI Tests in
Visual Studio.
If it is a Web application, you can use Selenium, UFT, Visual Studio or Test Studio.
If it is a mobile app, you can use Appium, Jamo Solutions Meux Test or T-Plan Robot.
If you want to test your mobile app over a large number of devices
and platforms, you can use cloud-based tools like Sauce Labs,
Perfecto Mobile or Device Anywhere.
If you are short on budget, you'll be better off using open source tools over
commercial tools, and so on.
Application Testing is a huge industry now and there is no dearth of testing tools available in the market. You will find the perfect tool for you if you know what you want and do some research on Google.
I will try to answer what I believe people normally use these terms to mean, lets start with the simpler term: A tool.
A tool like selenium is what actually does the automation, it has an API that will work for pretty much anything it covers (in this case websites) but it knows nothing about how the website you want to test works, this means it deals with low level constructs such as elements on a page and clicks.
A framework is normally just wrapping a tool to make it easier to make a test by imparting knowledge of your application, a standard example is login.
Say you want a test that checks when you enter a correct username and password in you get access to application. Using just selenium it would something like:
driver.findElement(By.id("username").sendKeys("MyUsername");
driver.findElement(By.id("password").sendKeys("password123");
driver.findElement(By.id("login").click();
Thats pretty simple, but as you can guess login is going to be used a lot across your tests and so it makes sense to move this into a place that makes it easier to reuse (both from a less code stand point and maintainability). This is where a framework comes into play, normally with selenium it will be page objects (see here)
Base on my understanding:
TOOLS
We "USE" tools to meet our objective (can be own self or your small groups of team).
Example: We use Selenium IDE as a tools helps us to automate some repetitive steps to do certain verification during our smoke test.
FRAMEWORK
We "DESIGN" a framework to meet the organization mission.
Things to consider when we design the framework including:
Maintainability
Reusability
Data Driven
Reporting
Schedule running through CI tools like Jenkins
Example: We design a Test Automation Framework by using WebDriver + Java + TestNG + ANT, to meet the objective to identify our current code base stability, and the test will be trigger and run by jenkins in a daily basis, SSRS report will captured in a daily basis as well each time the test finished. Stakeholders can review the daily code stability report anytime he/she needed.
Hope that can help you :D
I am trying to evaluate the capabilities of the below IBM Rational tools to handle functional testing automation and performance testing for the "Finacle - Infosys" core banking system.
IBM Rational Functional Tester (RFT)
IBM Rational Performance Tester (RPT)
Also, I would like to confirm whether any additional components are required to be added for both tools to support my testing needs (e.g. plugin to test web services or handling Oracle NCA JInitiator ... etc.).
Appreciate your kind help and support.
Note: I apologize for posting the question twice on the "Software Quality Assurance and Testing" and the main "StackOverFlow" sites.
Rational Functional Tester(RFT) is a Functional/GUI Testing/Automation tool.
I am not sure about the application you are using however RFT provides a very good support for Html /Java /.Net /Win based applications (to name a few)
I am giving a brief description on what RFT can do and then you can decide if it meets your requirement for functional testing or automation.
RFT can be installed on an Eclipse IDE or Visual Studio IDE or Both and you can use whatever you are comfortable with.
RFT provides you the capability to create your test scripts using a Recorder which simply records the action you perform on your application , capturing the objects you interacted all along and saving the object map. You can create Verification points (data verfication point , property verification point etc) or use Datapool to run some tests with different sets of data.
The script thus generated can be further fine-tuned if required post recording also.
Once you have a script ready you can playback the script and then RFT would start executing the actions the script contains, RFT would use the object map to determine the properties and the hierachy of the object to find the controls and perform actions on them. It may also use the saved verification points if any, to verify any data/property . Or may use the datapool to data-drive the script.
Another way you could use RFT is by harnessing the power of descriptive programming (you would require some basic Java/VB skills).
Using descriptive programming you would essentially create the script on your own without using the recorder. You could use the find() api provided by RFT to find objects and then perform actions / verfication test etc. When using descriptive scripting we would usually unbind ourselves from the object map as we are defining what to find and where to find etc.
Also a combination of two can be used where you can use some objects from the object map and some can be found using the find() api. This would entirely depend on your the kind of test case you want to write and type of application you are automating/testing.
You could also use Simplified scripting which records a script in plain english like format and also allows capturing of application visuals enabling a user to modify a script even if the application is offline using the captured visuals.
As checked and confirmed with IBM representatives and according to IBM Portal (Link Here), the use of both Rational Performance Tester (RPT) and Rational Functional Tester (RFT) should be supported.
We will have a project running to conduct both functional testing automation and performance testing for the "Finacle - Infosys" core banking system during the coming 12 months, after-which I can post any updates on this question of mine.
I'm looking for a code coverage tool that I can use with a BlackBerry application. I'm using J2ME-Unit for Unit Testing and I want to see how much of my code is being covered by my tests.
I've tried using Cobertura for J2ME but after days of wrestling with it I failed to get any results from it. (I believe that the instrumentation is un-done by the RAPC compilation). And despite this message, the project seems to be dead.
I've looked at JInjector but the project seems very incomplete. There is little (if any) documentation and although it claims to be able to work with BlackBerry projects, I haven't seen any places where it has been used for that purpose. I've played with the project quite a bit but to no avail.
I've also tried the "Coverage" view in the BlackBerry JDE, even though I use Eclipse for development. The view stays permanently blank, regardless of clicking "Refresh" and running the application from the JDE.
I've looked at most of the tools on this SO thread, but they won't work with J2ME/BlackBerry projects.
Has anyone had any success with any code coverage tools on the BlackBerry? If so, what tools have you used? How have you used them?
If anyone has managed to get JInjector or Cobertura for J2ME to work with a BlackBerry project, what did you have to do to get it working?
I can't speak for Coberatura or JInjector, because I don't know how they collect test coverage probe data.
What is
critical is how this data is captured (does it need Java runtime support only available in standard Java VMs?) and how it is exported to the test coverage display/report generation tools.
Our SD Java Test Coverage tool instruments your source code; at runtime this produces an array of native Java booleans representing the coverage data, without need for any special VM support. Normally, this array is exported directly to a file, used by the test coverage display mechanism, by a TCVDump method provided with the test coverage tool, as your application exits.
Java (and other programming langauges used) in embedded systems often requires custom methods to extract the test coverage data. You might need to code a special dump procedure (in Java) to write out that boolean array to an accessible place. Our experience with building such custom dump procedures is that they are generally pretty simple (a few dozen lines); the real trick is deciding how/where to put the data, so that it can be easily moved to the target file. Mostly this is just a peculiar pair of copies, the first of which copies the boolean array to some staging location, and the second which writes the staged data into the destination file. (The standard TCVdump method is provided in source form to enable this kind of customization).
While I haven't specifically looked at BlackBerry, if you can write the data anywhere, you can pretty much be assured you can achieve this. We've had success with other embedded hand-set systems, such as Symbian, doing this.
If you want a complete overview of how to generally instrument code for test coverage following this strategy, see this paper: Branch Coverage for Arbitrary Languages Made Easy
I was actively involved with JInjector while working at Google. We were able to use it to successfully obtain code coverage for Blackberry applications. The application lifecycle for Balckberry apps is less predictable than J2ME and we found we had to tweak the application code to ensure the coverage data was gathered. I didn't personally work on the blackberry apps, several other engineers did. I'd hoped we'd create an example blackberry application and make it available on the jinjector site, but events and life got in the way.
If you would be willing to provide a sample blackberry apps with some unit tests, I'd be willing to spend a few hours trying to help you get the code coverage working. I'm not actively working with either J2ME or Blackberry (I'm currently working on Android apps when I have time to experiment with mobile) so I'm quite rusty. I have a day job that doesn't involve much mobile test automation, however I continue to work on ways to improve the test automation for mobile apps e.g. http://code.google.com/p/mwta/downloads/list for Android Test Automation.
I'm julianharty at gmail.com