Short version: What's the best way to automate compatibility testing
against a large number of third-party programs?
The details:
I develop a program whose core feature is interacting with a variety of
different pieces of music player software via their respective RPC
interfaces. The RPC itself typically happens either via D-Bus or via some
client library specific to a particular player. Since each music player
has its own unique RPC interface, my program requires special code to
handle each.
Testing all this code is increasingly a problem for me. At last count
there are fifteen (!) different music players my program knows how to talk
to, and the interface details can vary from one version of a player to the
next. Manually testing my program against the latest version of each of
the players I'm trying to support, as well as a few older versions, is
tedious and error-prone, so I'm looking for a way to automate this as much
as possible.
The test cases themselves aren't the problem; those are just a matter of
calling a sequence of functions on a player's RPC interface and checking
the return values and/or asynchronous callbacks for the expected result.
No, the problem is having a framework to run the tests automatically.
Here are the challenges I see:
Each player maintains persistent state, usually as dotfiles under the
user's home directory. The state consists of things like the music
library, playlists, etc. These files need to be reverted to a known
initial state before each test. (Deleting it entirely isn't always an
option, since then the GUI-based players will present a setup wizard the
next time they start instead of running normally.)
Those initial states may be partially dynamic. For example, a music
library will contain full paths to the music files within it, but the
paths to the actual "music" files used for testing will vary from
machine to machine and won't be known until runtime.
The players to test against will probably be installed under
non-standard locations which will vary from system to system, in order
to have multiple versions of each installed in parallel. The framework
will probably need to know which player and version it's testing against
before the player is started, so it can initialize the player's state
files accordingly.
Since I don't have any control over development of the music players my
program interacts with, I can't modify their behavior to make it easier
for me to test against them.
What I'd like to do is set up a VM with a bunch of different players (and
a bunch of different versions of each player) installed, and then be able
to test my program against each of them in turn automatically. Ideally,
it would be possible for someone else to set up their own VM to run tests
in themselves, presumably only needing to tell the test framework which
players are installed where.
So, what's the best way to automate compatibility testing against a large
number (several dozen) of third-party programs?
In case it affects the recommendations, my program is written in Python,
and I'm using GNU autotools as the build framework.
If this only windows environment there is one way to go and that is with MS Hyper-V.
Link(http://www.microsoft.com/virtualization/en/us/solution-application-development.aspx)
They have support for having a image of for example of a Vista installation and by scripting creating a new copy of it and push an installation into the new image and have a basic installation + added software up and running within minutes.
The MS Office team use this and test engineers can order any version, language edition of windows in combination of any release of office within a few minutes..
Big problem is that this cost a good bit of money and Hyper-V is a quite complex product to set up.
As an alternative maybe could be to use Virtual Box (Open Source) and write your own script to automate installations of an image with new versions of software for test. I have done this with the standard image reading script upon start up from a network folder to install any software for test.. Its not fully automated solution but saved my team a lot of time.
Related
This might be the wrong forum, but its filled with so many smart people so someone might know a solution.
One of my customers has given me several assignments that require a quite similar yet wery different development setup (different versions of class-libraries and such) and the problem is that each time I need to switch between the projects I need to do a lot of configuring to get the compilation to work and not include wrong versions of tools used.
If I dont get it right there might be a lot of cleanup afterwards.
It takes me at least an hour to switch projects. Often several.
Now I realize that the customer has an issue with a lot of branching in their setup and they are working on that, but thats a long process.
So.. My question is. Is there some tool that allows me to take "snapshots" of working project-environments and switch between them?
I'm working on windows.
There are many options. One of them is virtualization: running an entire operating system on top of your operating system. It will emulate a fake hard drive, fake network card, etc. This will run an entire desktop 'in a box'.
In the last years containerization is very popular: running a separate environment (file system, OS configuration) but leaving the heavy stuff to the host (in your case, Windows) such as networking, hardware, etc. This is geared towards running a single application (though this is flexible) 'in a box'.
In your case, since you need to encapsulate an entire development environment, it would seem to be best to go with a virtual machine.
A no-cost solution is Virtualbox. Paid alternatives such as VMWare may offer better usability, performance or features.
If your class libraries are in a specific location, you could containerize the compiler: you work on your PC, then build inside of a 'box' with its own environment. Docker is often used.
Depending on what language you are programming in, there may be a specific solution. Python has VirtualEnv, for example.
I make a game using vb.net & wpf. But I want this run on Windows/Linux/Mac.
How can I do it?
I'm sorry my english is really bad :D I use Google translate.
Implementing a solution for multiple platforms is not an easy task and you need to be familiar with all the platforms you plan to support, starting with trivialities such as different path schemes and ending with checking every reference you require by your project settings for its compatibility to the target platform.
Please have a look at http://www.mono-project.com. When you install the mono package to your system, you can run your compiled .exe as it is from the shell under certain circumstances.
Obviously, you need to decide whether you try to create one application that runs on all target platforms OR if you want to create platform specific applications all referencing to the same game engine.
If you stick very close to the framework not using external references, chances are higher to achieve the former. If the main logic of your game can easily be compartmentalized into a dedicated project, the latter is the way to go.
In general, cross-plattform compatibility is more easily to achieve if your application backend consists of a console application to be accessed by a webbrowser installed on the system - using a web frontend though. But as long as you do not require accelerated graphics, this should be feasible.
The company I work for is building a managed force.com application as an integration with the service we provide.
We are having issues working concurrently on the same set of files due to the shoddy tooling that is provided with the force.com Eclipse plugin. If 2 developers are working on the same file, one is given a message that he can't save -- once he merges he has to manually force the plugin to push his changes to the server along with clicking 2 'Are you really sure' messages.
Basically, the tooling does a shoddy job of merging in changes and forces minutes of work every time the developer wants to save if another person has modified the file he's working on.
We're currently working around this by basically 'locking' individual files by letting co-workers know who is editing a file.
It feels like there has got to be a better way in this day and age. Does anyone know of a different toolset we could use, process we could change, or anything we can do to make this easier?
When working with the Force.com platform my current organisation has found a number of different approaches can work depending on the situation. We all use the Eclipse Force.com plugin without issues and have found the following set ups to work well.
We have a centralised version control system that we deploy from using a series of ant commands to a developer org instance. We then depending on the scope of the work either separate it off into chunks with each developer having their own development org and merging the changes and testing them regularly, or working in a single development org together (which if you have 2 developers should be no major problem) allowing you to have almost instant integration.
If you are both trying to work on the same file you should be pair programming anyway, but if working on two components of a similar system together, sharing the same org can allow you to develop in a fast and flexible manner by creating the skeleton of the system you wish to use and then individually fleshing out the detail.
I have used both methods extensively and a I say, work really well depending on the situation.
Each developer could work in separate development sandbox (if you have enterprise edition, I think 10 sandboxes with full config & limited amount of data are included in the fee?). From time to time you would merge your changes (diff tool from any version control system should be enough) and test them in integration environment. The chain development->integration->system test->Q&A-> production can be useful for other reasons too.
Separate trick to consider can be used if for example 2 guys work on the same trigger. I've learned it on the "DEV 401" course for Developers.
Move all your logic to classes. Seriously. They will be simpler to unit test too.
Add custom field (multi-select picklist) to User object. Values should be equal to each separate feature people are working on. It can hold up to 500 values so you should be safe.
For User account of developer 1 set "feature1" in the picklist. Set "feature2" for the other guy.
In the trigger write an if that tests presence of each picklist value and enters or leaves the call to relevant class. This wastes 1 query but you are sure that only the code you want will be called.
Each developer keeps on working in his own class file.
For integration test of both features simply set the multiselect to contain both features.
I found this trick especially useful when other guy's code turned out to be non-optimal and ate too many resources. I've just disabled his feature on my user account and kept on working.
This trick can be to some extend applied to Visualforce pages too (if you can divide them into components).
If you don't want to waste query - use some logic like "user's first name contains X" ;)
We had/have the exact same problem, we have a team of 10 Devs working on a force.com application that has loads of apex classes (>300) and VF pages (>300).
We started using Eclipse plugin but found it:
too slow working outside of the USA each time a save is called takes > 5 sections
to many merge issues with a team of 10 developers
Next we tried developing in our own individual sandboxes and then merging code. This is ok for a small project but when you have lots of files and need changes to be pushed between sandboxes it becomes impossible to manage as the only thing worse then force.com development tooling, is force.coms deployment/build tools. No automation its all manual. No easy way to move data between sandboxes either.
Our third approach was to just edit all our VF pages and Apex code in the browser. (not using their embedded editor that shows up in the bottom half of the page because that is buggy and slow) but just using the regular Editor under setup > develop > Apex classes. This worked ok. To supplement this we also had a scheduled job that would download all our code and save it into our SVN repository. We also built a tool that allow us to click a folder on our desktop and zip its contents and deploy it as static resources for us.
However this approach still has its short comings, i.e. it is slow and painful to develop in the cloud, their (salesforce) idea of Development As a Service is crazy. Also we have no real SCM we only have it acting as backups.
Bottom line is force.com is a CRM and not a Development platform, if you can? run, flee, get away from it as fast as you can. Using it for anything apart from a CRM is more trouble then it is worth. Even their Slogan "No Software" makes me laugh everytime
I'm not familiar with force.com, but couldn't you use source control and pull all the files down from force.com into your repository. Then you could all do your work, and merge your changes back into the mainline. Then whenever it's necessary push the mainline up to force.com?
Take a look at the "Development Lifecycle Guide: Enterprise Development on the Force.com Platform". You can find it on developer.force.com's documentation page.
You might want to consider working on separate static resources and pages and then just being careful when editing objects, classes etc.. If most of your development happens on client side code (page, staticresource, lightning component/app) you might be interested in this project: https://github.com/bvellacott/salesforce-build . In any case I strongly suggest using version control. If not on a server then at least locally on your machine, in case your peers overwrite your work.
I'm looking for a code coverage tool that I can use with a BlackBerry application. I'm using J2ME-Unit for Unit Testing and I want to see how much of my code is being covered by my tests.
I've tried using Cobertura for J2ME but after days of wrestling with it I failed to get any results from it. (I believe that the instrumentation is un-done by the RAPC compilation). And despite this message, the project seems to be dead.
I've looked at JInjector but the project seems very incomplete. There is little (if any) documentation and although it claims to be able to work with BlackBerry projects, I haven't seen any places where it has been used for that purpose. I've played with the project quite a bit but to no avail.
I've also tried the "Coverage" view in the BlackBerry JDE, even though I use Eclipse for development. The view stays permanently blank, regardless of clicking "Refresh" and running the application from the JDE.
I've looked at most of the tools on this SO thread, but they won't work with J2ME/BlackBerry projects.
Has anyone had any success with any code coverage tools on the BlackBerry? If so, what tools have you used? How have you used them?
If anyone has managed to get JInjector or Cobertura for J2ME to work with a BlackBerry project, what did you have to do to get it working?
I can't speak for Coberatura or JInjector, because I don't know how they collect test coverage probe data.
What is
critical is how this data is captured (does it need Java runtime support only available in standard Java VMs?) and how it is exported to the test coverage display/report generation tools.
Our SD Java Test Coverage tool instruments your source code; at runtime this produces an array of native Java booleans representing the coverage data, without need for any special VM support. Normally, this array is exported directly to a file, used by the test coverage display mechanism, by a TCVDump method provided with the test coverage tool, as your application exits.
Java (and other programming langauges used) in embedded systems often requires custom methods to extract the test coverage data. You might need to code a special dump procedure (in Java) to write out that boolean array to an accessible place. Our experience with building such custom dump procedures is that they are generally pretty simple (a few dozen lines); the real trick is deciding how/where to put the data, so that it can be easily moved to the target file. Mostly this is just a peculiar pair of copies, the first of which copies the boolean array to some staging location, and the second which writes the staged data into the destination file. (The standard TCVdump method is provided in source form to enable this kind of customization).
While I haven't specifically looked at BlackBerry, if you can write the data anywhere, you can pretty much be assured you can achieve this. We've had success with other embedded hand-set systems, such as Symbian, doing this.
If you want a complete overview of how to generally instrument code for test coverage following this strategy, see this paper: Branch Coverage for Arbitrary Languages Made Easy
I was actively involved with JInjector while working at Google. We were able to use it to successfully obtain code coverage for Blackberry applications. The application lifecycle for Balckberry apps is less predictable than J2ME and we found we had to tweak the application code to ensure the coverage data was gathered. I didn't personally work on the blackberry apps, several other engineers did. I'd hoped we'd create an example blackberry application and make it available on the jinjector site, but events and life got in the way.
If you would be willing to provide a sample blackberry apps with some unit tests, I'd be willing to spend a few hours trying to help you get the code coverage working. I'm not actively working with either J2ME or Blackberry (I'm currently working on Android apps when I have time to experiment with mobile) so I'm quite rusty. I have a day job that doesn't involve much mobile test automation, however I continue to work on ways to improve the test automation for mobile apps e.g. http://code.google.com/p/mwta/downloads/list for Android Test Automation.
I'm julianharty at gmail.com
I need to build a fairly simple app but it needs to work on both PC and Mac.
It also needs to be redistributable on a disc or usb drive as a standalone desktop app.
Initially I thought AIR would be perfect for this (it ticks all the API requirements), but the difficulty is making it distributable, as the app would require the AIR runtime to be installed to run.
I came across Shu Player as an option as it seems to be able to package the AIR runtime with the app and do a (silent?) install.
However this seems to break the T&C from Adobe (as outlined here) so I'm not sure about the legality.
Another option could be Zinc but I haven't tested it so I'm not sure how well it'll fit the bill.
What would you recommend or suggest I check out?
Any suggestion much appreciated
EDIT:
There's a few more discussions on mono usage (though no real conclusion):
Here and Here
EDIT2:
Titanium could also fit the bill maybe, will check it out.
Any more comments from anyone?
EDIT3 (one year on): It's actually been almost a year since I posted that question but it seems some people still come across it every now and then, and even contribute an answer, even a year later.
Thought I'd update the question a bit. I did not get around to try the tcl/tk option at the end, time constraint and the uncertainty of the compatibility to different os versions led me to discard that as an option.
I did try Titanium for a bit but though the first impressions were ok, they really are pushing the mobile platform more than anything, and imho, the desktop implementation suffers a bit from that lack of attention. There are also some report of problems with some visual studio runtime on some OSs (can't remember the details now though).. So discarded that too.
I ended up going with XULRunner. The two major appeals were:
Firefox seems to work out of the box on most OS version, so I took it as good faith that a XULRunner app would likely be compatible with most system. Saved me a lot of testing and it turned out that it did run really well on all platforms, there hasn't been a single report of not being able to start the app
It's Javascript baby! Language learning curve was minimal. The main thing to work out is what the additional xpcom interfaces are and how to query them.
On the down side:
I thought troubleshooting errors was a sometimes difficult task, the venkman debugger is kinda clunky, ended up using the console more than anything.
The sqlite interface is a great asset for a desktop app but I often struggled to find relevant error infos when something didn't work - maybe i was doing it wrong.
It took a little while to work out how to package the app as a standalone app for both PC and Mac. The final approach was to have a "shell" mac app and a shell pc app and a couple of "compile" script that would copy the shells and add the custom source code onto it in the correct location.
One last potential issue for some, due to the nature of xulrunner apps, your source code will be deployed with the app, you can use obfuscation if you want but that's something to keep in mind if you want to protect your intellectual property
All in all, great platform for a cross-platform app. I'd highly recommend it.
Tcl/Tk has one of the best packaging solutions out there. You can easily wrap a cross-platform application (implemented in a fully working virtual filesystem) with a platform-specific binary to get a single file executable for just about any modern desktop system. Search google for the terms starkit, starpack and tclkit. Such wrapped binaries are tiny in comparison to many executables these days.
Many deride Tk as being "old" or "immature" but it's one of the oldest, most stable toolkits out there. It uses native widgets when such widgets exist.
One significant drawback of Tcl/Tk, however, is that it lacks any sort of printing support. If your application needs to print you'll have to be a bit creative. There are platform-specific solutions, and the ability to generate postscript documents, and libraries to create pdfs, but it takes a little extra effort.
Java is probably your best bet, although not all Windows PCs will necessarily have Java (most should). JavaFX is new enough you can't count on it - you'll probably find a lot of machines running Java 1.5 or (shudder) 1.4. I believe recent Mac OS still ships with 1.5 (latest version may have changed to 1.6).
Consider JavaFX
It would run everywhere with a modern JRE ..!
AIR could be an option, but only if you don't mind distributing two different files (the offline runtime installer and your app), and expecting the user to run one and then the other. You do have to submit an online form at Adobe's site saying you agree to distribute the offline installer as-is, rather than digging out individual DLLs or whatever, before they give you the installer.
Unfortunately there's currently no way to get both an AIR app and the runtime to install from one file though. I'm not sure what the deal with Shu is, or whether it's doing anything that isn't kosher.
i would recommended zink. it has all the functionalities you require for desktop. however, the las time i used it it was a bit glitchy.
i was hung up by trying to write a 6M file to the disk. thought it trough and changed the code to write 512K chunks at a time (3min work, fast).
probably it still has some little annoying glitches like making you think on root lvl but the ease of use and the features are just way too sweet to ignore.