We are looking for an automated testing software for our web application. We need to come up with a solution or software that our non-it staffs could write test cases as well as the developers.
For example I've run through some of them such as: SmartBear, National Instrument and IBM. Most of these guys are MS Windows based or commercial Linux distros which remove them from our list since we are all Debian based.
Any recommendation or guideline would be much appreciated.
Ps. We don't have any budget limit!
You're going to have a hard time getting tooling for non-technical testers to build test cases if you limit yourselves to Debian OS for developing and running the tests on. There's no reason you couldn't have a few Windows system to manage your test suites from -- those would run against your web site just fine, regardless of what stack it's hosted on. That would open you up to the tools you mentioned (and Telerik's Test Studio, the tool I help promote).
Those Windows systems could easily be run via whatever virtualization host you prefer, so you wouldn't even need physical systems to deal with that. You could easily share the same source control repository as your devs, too, since nearly every decent SCM has Windows clients.
If you're unwilling to consider having a few Windows boxes around for your testing, then you'll need to have a look at getting all your testers proficient in APIs and frameworks like WebDriver and Robot Framework. The Pages gem from Jeff Morgan (#chzy) in Ruby would be another option, as would Adam Goucher's Saunter (in Python).
Related
I'm currently setting up a new build server and I'm interested in any suggestions the community may have about software such as Hudson or CruiseControl.NET that may simplify and add additional value to the build process.
Previously I had a build server set up using custom batch files which would run msbuild and other such tools and these were triggered by subversion hooks to allow for a continuous builds to be done per branch. The idea was that eventually we would also execute automated tests and/or static analysis although we never really got that far. This server also acted as our source code repository, a test machine for web project builds, and a web server for custom dashboard and portal for developers on the team.
At this point my thoughts are to separate some of the responsibilities of the old build server and at least a Build Server which is responsible only for creating builds, a web server which is responsible for acting as the intranet style dashboard site for developers, and perhaps an additional web server as the Subversion repository. If it turns out to be better or easier to keep the Subversion code on the same server as SvnServe then I'll probably opt to place the Subversion repository on the web server but still keep the build server separate. Having no personal experience with any of the popular build server and CI solutions out there I'm curious how CruiseControl.NET, Hudson or other solutions would fit into this type of configuration. It appears that both of CC.NET and Hudson have web interfaces for example but the documentation doesn't clearly layout how this plays out with different hardware/system configurations so I'm not sure if either requires the web portion to be on the build server itself or not.
As far as technologies I'm dealing with .NET/C# based code which is a mix of Web/WinForms/WPF and we use a few separate Subversion repositories to host these projects. Additionally it would be nice to support Visual FoxPro and Visual Source Safe for some legacy applications. I would also like to get more team members involved in monitoring builds and would like to eventual have developers create build setups for their own projects as well with as much simplicity as possible. Also I should mention that I have no experience setting up a Java based web application in IIS but I do have quite a bit of experience setting up and managing ASP.NET applications so if that may make .NET based products more favorable unless I can be convinced otherwise.
UPDATE (after researching Hudson): After all the recommendations for Hudson I started looking into what is involved to get it up and running on my two Windows 2008 servers. From what I can gather the web portion (master) would run on my webserver but it seems that IIS isn't supported so this would greatly complicate things since I want to host it on the same machine as my other web applications. On the build server, I would be installing a second copy of Hudson that would act as a slave and only perform builds that are delegated to it by the master. To get this to work I would be installing Hudson as a Windows Service and would also need to install some unix compatibility utilities. Unfortunately the UnxUtils download link appears to be broken when I checked as well so I can't really move forward until I get that resolved. All of this is really sounding just as complex if not more complex than installing CruseControl.NET. For now this unfortunately leaves me to looking into CruiseControl.NET and TeamCity.
UPDATE (about TeamCity): After looking into TeamCity a little closer I realized that at least the server portion is also written in Java and is deployed in a manner very similar to Hudson. Fortunately it appears that Tomcat can be used to host servlets inside IIS although I can't find a good straight forward guide to describe how to actually do accomplish this. So skipping that for now I looked further when I ran into what looks like what might be a major snag.
TeamCity Professional edition only
supports TeamCity Default
Authentication and does not support
changing the authentication scheme.
Since windows authentication is likely the direction we will want to go, it's now looking like it might be back to evaluating CruiseControl.NET or possibly Hudson if I can get my hands on the UnxUtils and also find out more about how I can host the dashboard portion of Hudson within my existing IIS configuration. Any pointers?
UPDATE (about Jenkins): I ended up experimenting enough with Hudson that I ended up with a reasonable build server setup that I'm happy with and that can be extended to do much more if I need. Of course I went the rout of converting to Jenkins once Oracle took over Hudson and Jenkins is what I'm using today with little bits of powershell to help tie things together. I'm very happy with this approach right now and besides being Java based, Jenkins has quite a bit of support for other development environments such as .NET and MSBuild.
I'd vote for TeamCity here. Its is very, very easy to get stood up and running, integrates with all your .NET stuff without any trouble. The builds themselves are run by agents which can be on the build server or another machine depending on requirements--they could even be on a machine running an entirely different OS on a different network in a different country.
I highly recommend using Hudson. Not only will it allow you to build .NET applications on a continual basis, but you can also run code analysis and unit tests as well. It's easy to install (just deploy a WAR file to a web server such as Tomcat) and has many configuration options. There is also a large number of plugins available that you can use, many written by other Hudson users. Best of all, it is free and actively supported.
For our decision making process we started with following overview.
http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix
Our main objective was java, easy to configure/use even after nobody created a job for 6 months. We moved away from a old version of Cruise Control, since nobody really knew how to use it. Some of the commercial products are nice if you want to go beyond just continuous integration. Have a look and decide for yourself.
Be careful, I don't know how up to date this matrix is. So some of the projects might have implemented more functions right now.
An interesting alternative could be Jira studio by Atlasian. If you use the hosted version you don't have much on support issues and it comes with subversion, bamboo, and goodies (jira+greenhopper, confluence, crucible, fisheye). http://www.atlassian.com/hosted/studio/
I agree with Wyatt Barnett. TeamCity is the best choice. It is very easy to configure and use. Moreover, TeamCity has a Free Professional Edition. Previously we used CruiseControl.NET on our project. This is also a powerful tool, but it is very complicated and hard to understand.
What s.ermakovich said: Both TeamCity and Hudson separate the web UI from build agents. You shouldn't need to install IIS on a build agent. You'd need to install a JVM and the agent software on any build node - very straightforward.
I need to provide our web developers an easy and quick way to test their code on multiple browsers. Here's my current plan:
Get a Mac
Install Windows XP and Linux over VMWare
Install all possible major browsers on these OSes, including on the Mac and the god-forsaken IE6.
This will allow developers to use the system to test their applications.
But is it possible to give them some sort of desktop sharing tool, so they can test remotely... keeping in mind that the their systems can be windows, linux(linus?) or macs.
Or am I doing it all wrong?
There are a few viable options I have used:
Get hardware. If you develop on Macs and have an old Windows box laying around, you might as well use it. You then need to figure out how you are going to connect to it. I have used:
(a) remote control tool (like VNC) to
a shared box. At one company we had a
IE6 testing box we all VNCed into
(b) Synergy on my desktop (which
allows sharing keyboard and mouse)
(c) Walking
VMs. Some developers like this because they have everything on one box, and can take it with them. You'll probably need multiple VMs for different versions of Windows. I've done this with both Parallels and VMWare.
External service. #chotchki mentioned one, but there are many others.
My current favorite is 1b, but they are all workable.
To answer your question: VMs are a reasonable solution.
There a web service that already does this Browser Shots. You can also install the software on your own systems if you want to host your own.
I seem to end up evaluating a lot of software. This requires me to constantly install all kinds of things on my system. It creates a huge clutter and I spend a lot of time during the install process, and if I don't like it, then removing everything I've done. Much of my evaluation tends away from the features of the software being evaluated and toward how difficult it is to install. I'm sure I miss good software which may have actually been a better choice, because of this startup cost.
With the advent of VM software like VMWare Player and VirtualBox, it would be much easier to sell someone like me your software, if you just provided an image that I could load into the VM and run. I'd be looking at the features almost immediately rather than fighting with which revision of whatever. The VM would take care of all of this for me.
Am I missing something, or should vendors and OSS start distributing VMs for their wares?
Most of my evaluations are for server side software installed on Linux, so OS licensing is not the issue.
VMs require that the operating system have a valid license key. For free operating systems this wouldn't be an issue, but if you're developing for something like Windows machines, each time they send out a demo version of their software, they're sending out a license key that they would have to pay for.
This would be incredibly expensive for most companies.
The only downside I would say IMHO is the size of the images, if say you have a 20 MB application, do you really want to download/transfer an entire OS just for that application.
I would say a better approach would be to have a ready to go VM and then you simply take a snapshot (on Virtual Box, I assume similar feature exist in other players)
Then simply install the applciation inside your sandbox environment, and then just Zap it when done (i.e. return to your Snapshot)
Darknight
This can be done for softwre that runs on open source platforms, and VMware have a library of images which do just this (though the images that are used for evaluating commercial software is generally for infrastructure-type things that have very, very complex installation requirements):
http://www.vmware.com/appliances/
However, if the software is for the Windows platform, you don't really have the opportunity to do this, as Microsoft's Windows licensing would prevent it. Unless, you're Microsoft, of course, in which case you can in fact do this - and MS has done this to permit easier evaluation of such software as Visual Studio, SQL, and many others:
http://technet.microsoft.com/en-us/bb738372.aspx?ppud=4
Novell has an appliance builder called Suse Studio that lets you pick the software you want, it builds out a VM with the software (and dependencies, etc) for you. You can then try out the VM, download it, etc.
Whether the software you're looking for is available or not is a different matter.
Disclaimer: I work for Novell (though not with the Suse team)
But yes, if you can deal with the OS licensing issues, or possibly host trial environments yourself, this is a very effective way for a vendor to demo their app. The problem is that all vendors don't always have the infrastructure (or lack the awareness) to do so.
Microsoft provides fully-provisioned VM's for time-limited trials of their software. So if you want to trial select Microsoft products in that manner, you can do that today.
There is no sign, though, that Microsoft will make this available to third party Windows software vendors.
In the SaaS (Software-as-a-Service) world, you can get fully-provisioned virtual servers that include Windows and your software of interest on a pay-as-you-go basis, based on both Linux and Windows. For example, see Amazon Web Services
For windows, you may be better off developing a portable application that runs from a usb key. That is how Embarcadero distribute All Access. I received a 4 gb usb key that contained multiple applications. Most could be run straight from the key without installation. I believe Embarcadero will be licensing the technology at some stage.
If you are using a programming language such as Delphi or C++ with little in the way of external dependencies, a portable application is straight forward to develop. For .net, it is much harder, but can be done with Mono, or something like Virtual Application Studio.
Is anyone out there aware of any good or even reasonable tools for automated testing on the Windows CE / mobile platforms. Potential tools that I am aware of include TestQuest, Countdown, SOTI pocket controller, and Eggplant. Are there any more that I have missed?
Alternatively, is anyone aware of a VNC or remote display tool for Windows mobile that replicates the Windows visual object hierarchy on the PC, rather than displaying the entire device as a single bitmap? If this could be done, mainstream desktop automation tools could be applied to Windows mobile.
N.B. I have already read this related question which is useful, but am looking for a viable off the shelf alternative. This post is following up on a number of related posts in the PDA/Embedded section of SQAforums.
I realize that your question is directly "are there tools to do the automated testing on CE", but have you considered perhaps aiming your automation at a version of the app which can be accessed from a standard desktop environment? In this way, you are open to all of the standard automation tools.
For example, I have worked on a few projects where we needed to perform automated testing for the device. In all cases, the RF device was really just a web browser connecting to a web based app. The same URL and simple forms could be plugged into a standard desktop browser and be automated by any of the usual automation toolsets. Automation never replaces manual testing, so what we did on those projects was automate regression testing of the same web interface that was used by the RF devices, but still do some sanity manual testing directly on the devices.
Also, with regards to the VNC/bitmap issue, I've been down that road before and agree that it is a nightmare. Using standard desktop UI automation on a VNC bitmap is not only unreliable and unmaintainable, but slow - in most tools, the CPU churns away searching the entire bitmap from top left to bottom right for the desired image. Really really slow.
Check Hopper, a test tool for Windows Mobile.
What are the key use cases for the use of virtualization -- that is, running one or more "virtual PCs" using software such as VMWare and Microsoft Virtual PC -- for software development?
Also -- are there other instances/uses of virtualization that aren't covered by my definition above (use of a tool like MS Virtual PC or VMWare), and that are useful to developers?
My impetus for asking is this StackOverflow comment by Metro Smurf asserting "You'll wonder how you ever developed without it!", regarding use of virtualization.
(Please include just one use case per response. Thanks!)
Application testing in multiple environments is one obvious use of virtualization that I'm aware of. Testing your application on other operating systems (without requiring additional physical computers to do so), as well as testing that involves software that generally only allows you to install a single version on a given machine (such as the Internet Explorer browser; running both IE6 and IE7 on the same machine is not an officially supported configuration), are good candidates for virtual machine usage.
If your build-server is running in a VM, you can make a snapshots of it for every software release in order to be 100% sure that you can recreate the build environment (in case you want to make patches to old releases, for example).
If you set up snapshots of your development environment (and back them up) it can be very easy to resume productivity if your computer breaks down. When your machine breaks down right before your release - and you can resume immediately with all your tools installed and configured, it can be a lifesaver.
The simplest case which applies to my current situation is that we have a complex client-server environment and with virtualization every developer can very quickly get a baseline set of operating systems to deploy their local build to and verify end to end functionality.
Locally you have your dev box, and N client boxes which get re-initialized as fresh OSes each time you want to try a build. Essentially it's the test environment equivalent of a 'make clean' where even the client workstation gets replaced with a new OS.
Quickly distributing environments between team members is a very nice use case to for virtualization especially if you have a lot of various components, tools, etc.. This can save you a ton of time with new hires, contractors, or other individuals who need an environment quickly.
Many presenters use a VM for presentations - it allows them to revert immediately to reset the presentation for the next day, transfer all presentation materials quickly between computers, and not have to show your attendees your messy My Documents folder.
Using virtualization for sales activities is also a great use case. You can take a snapshot at a particular time that you can save as your demo baseline. Then once you run through the demonstration and change the data, etc. you can restore back to your previous baseline for future demonstrations. You can also capture multiple baselines and pick and choose which baseline best fits the upcoming demo.
Test environments. If you have more than one setup that a system needs to be targeted for (e.g. Windows & Linux, XP & Vista) then a machine with lots of RAM and VMWare (or on of the others) is a good way to manage the environments.
Another is developing on one system and targeting another. For example, at one point I did some J2EE work on a workstation running Linux where the client was I.E. 5.5. A VM with Windows 2000 and IE 5.5 would let me test the application.
Reasons I use virtual machines for development.
Isolate different development environments.
Testing environments.
Easy recovery due to computer hardware failure/upgrade.
Ability to "roll-back" changes to your development environment if something corrupts it.
Currently, I am using VirtualBox for my VM setup. I used to use VirtualPC, but I REALLY hated not having any type of "snapshot" feature (like VMware and VirtualBox have).
We develop software for use in our SaaS application, our production environment has a large number of servers and their software environment needs to be absolutely predictable; we can't have ANYTHING installed extra, or absent from our development machines.
Moreover, our application requires a number of different server types in order to function properly (at least 7 last time I counted); mostly they can't be installed on the same (virtual) machine - at least, not without violating the "same software as production" requirement.
In order to have a consistent environment, it's necessary to use VMs. I don't know how anyone ever manages without them.
Snapshots and rollbacks are nice too, but I use them only occasionally (really useful during installation / upgrade tests).
Suppose you're developing a new version of your software, and checking that the upgrade from the previous version works correctly... how long does it take to do a test cycle without being able to rollback the box? Do you have to reinstall the OS then the old version? Can you guarantee that the uninstall really uninstalls everything?
Being able to test/retest your deployment process is a huge savings.
Developing Add-Ins for different versions of Microsoft Office (using Visual Studio Tools for Office).
My main work machine has Office 2007. When I work with Add-Ins for Office 2003 I use a virtual machine with Visual Studio and Office 2003.
I'm suprised that nobody has mentioned the VMware record/replay feature (awesome video demo) which is great for debugging.
I have a headless server running ESXi which runs various machines for building installers (so I don't have to give up processing power on my desktop), automated testing (server is faster than any desktop) and various test environments (about 20 different configurations) so that the support team can easily jump onto a configuration that closely matches a customers system.
When you have one really beefy server running VMs that can be shared between support, test and dev teams, you introduce huge cost savings. In all, we're running ~25 VMs on ESXi (dual-quad core Xeon 2.5G + 8Gb RAM) shared between 5-10 people, some of the developers use Virtual PC and then I use VMware Workstation on my desktop. All of the Mac users here use VMware Fusion as well
I am surprised that no one has mentioned the benefit of increased security by isolating, for example, the database server and web server in different VM's.
Some server applications can use VMs too. When one vm is not used much, the server can locate the resources to other vms.
Some sort of test environment: if you are debugging malware (either writing or developing a pill against it) it is not clever to use the real OS. The only possible disadvantage is that the viruses can detect that they are being run in the virtualization. :( One of the possibilities to do it is because the VM engines can emulate a finite set of hardware.