Is there a tool to run unit tests on previous versions of software that's in source control?
The idea would be a bug surfaced and I want to know when it was introduced so I write a new test and the software checks out each back version from source control, running the test on each one, until the test doesn't fail anymore or we reach the beginning.
We use subversion but I'm curious of if anything like this exists in general.
Mercurial has a built in command called bisect that essentially does what you are looking for.
It is designed to work with a user-written script but in a nutshell, it does a binary search where your script (which runs the unit tests) tells bisect if the checked out revision "passes" or "fails" and based on that it moves through the history until it finds the revision where the bug was introduced.
I'm not sure if such a tool exists for SVN, but I've found bisect with Mercurial to be very useful for this sort of thing.
Bisect in Mercurial (and Git) perform exactly this job, except of ckecking each back version - it finds source of problem faster
Just about any version control system lets you check out a specific version of an entire build. And lets you track the history/the changes of any specific file(s) of the build.
Normally, I just take a simple "divide and conquer" approach:
a) Check out a really old version into a scratch directory
b) Build and confirm it DOESN'T have the bug
c) Manually compare the old and current versions and make "educated guesses" as to "what changed".
d) Check out a version between the old and current version (based on what I found in step c).
e) Build and test.
f) If it has the bug, check out version between a) and d).
If it doesn't have the bug, check out a version between d) and the current.
g) Rinse and repeat
And yes, some or all of this can certainly be scripted.
In bash (if you're on Linux), or in the scripting language of your choice.
Related
Is there a way to run an ant build such that you get an output of what the build would do, but without actually doing it?
That is to say, it would list all of the commands that would be submitted to the system, output the expansion of all filesets, etc.
When I've searched 'ant' and 'test', I get overwhelming hits for running tests with ant. Any suggestions on actually testing ant build files?
It seems, that you are looking for a "dry run".
I googled it a bit and found no evidence that this is supoorted.
Heres a bugzilla-request for that feature, that explains things a bit:
https://issues.apache.org/bugzilla/show_bug.cgi?id=35464
This is impossible in theory and in practice. In theory, you cannot test a program meaningfully without actually running it (basically the halting problem).
In practice, since individual ant tasks very often depend on each other's output, this would be quite pointless for the vast majority of Ant scripts. Most of them compile some source code and build JARs from the class files - but what would the fileset for the JAR contain if the compiler didn't actually run?
The proper way to test an Ant script is to run it regularly, but on a test system, possibly a VM image that you can restory to the original state easily.
Here's a problem: You have target #1 that builds a bunch of stuff, then target #2 that copies it.
You run your Ant script in test mode, it pretends to do target #1. Now it comes to target #2 and there's nothing to copy. What should target #2 return? Things can get even more confusing when you have if and unless clauses in your ant targets.
I know that Make has a command line parameter that tells it to run without doing a build, but I never found it all that useful. Maybe that's why Ant doesn't have one.
Ant does have a -k parameter to tell it to keep going if something failed. You might find that useful.
As Michael already said, that's what Test Systems - VM's come in handy- are for
From my ant bookmarks => some years ago some tool called "Virtual Ant" has been announced, i never tried it. So don't regard it as a tip but as something someone heard of
From what the site says =
"With Virtual Ant you no longer have to get your hands dirty with XML to create or edit Ant build scripts. Work in a completely virtualized environment similar to Windows Explorer and run your tasks on a Virtual File System to see what they do, in real time, without affecting your real file system*. The actual Ant build script is generated in the background."
Hm, sounds to good to be true ;-)
..without affecting your real file system.. might be what you asked for !?
They provide a 30day trial license so you won't lose no money but only the time to have a look on..
Does anyone know of an existing solution to help write tests for a NSIS script?
The motivation is the benefit of knowing whether modifying an existing installation script breaks it or has undesired side effects.
Unfortunately, I think the answer to your question depends at least partially on what you need to verify.
If all you are worried about is that the installation copies the right file(s) to the right places, sets the correct registry information etc., then almost any unit testing tool would probably meet your needs. I'd probably use something like RSpec2, or Cucumber, but that's because I am somewhat familiar with Ruby and like the fact that it would be an xcopy deployment if the scripts needed to be run on another machine. I also like the idea of using a BDD-based solution because the use of a domain-specific language that is very close to readable text would mean that others could more easily understand, and if necessary modify, the test specification when necessary.
If, however you are concerned about the user experience (what progress messages are shown, etc.) then I'm not sure that the tests you would need could be as easily expressed... or at least not without a certain level of pain.
Good Luck! Don't forget to let other people here know when/if you find a solution you like.
Check out Pavonis.
With Pavonis you can compile your NSIS script and get the output of any errors and warnings.
Another solution would be AutoIT.
You can compile your install using Jenkins and the NSIS command line compiler, set up an AutoIT test script and have Jenkins run the test.
I love the maven-versions-plugin but sometimes I forget to run it for a while. Is there a way to make a maven build fail (and thus have a continuous build fail) when certain important dependencies are out of date?
I think you're approaching this incorrectly. Mail yourself the output of the maven-versions-plugin if you want, but don't fail the build due to changes outside of your control.
Even more, why would you want to needlessly update to the latest versions? I have seen many tricky problems appear due to upgrades which have brought slight changes to previous behaviour.
This, in general, is a bad practice - to update versions automatically. There is no practical reason of using the latest version of any package. If the library you're using satisfies your requirements you should stay with this version for security/stability reasons. And forever.
I think that maven-versions-plugin is an anti-pattern itself.
ps. When and if you want to do integration testing of modules developed by different teams/programmers, it is "integration testing". Even in this case I still think that on-fly version updating is the wrong approach. Root project should not do this integration testing, instead, every sub-module (or JAR, in your case), has to be responsible for integration testing of itself together with the rest of the system. When a sub-module increases its version it has to validate whether everything is still fine, and only then has to release a new version to the repository. And when the sub-module is doing the validation it has to be dependent on statically specified version numbers.
First the question, then some background.
We're using Visual Studio 2008, C# 3.0 and .NET 3.5, and TFS 2008 as our VCS.
If I execute this command against our TFS database, to show information about a merge commit:
tf changeset 13469 /noprompt
I get output like this (redacted):
Changeset: 13469
User: Lasse
Date: 12. november 2010 14:06:06
Comment:
Some text here.
Items:
merge, edit $/path/to/target/filename.txt
... more merged files
... some blurb about reviewer texts, etc. nothing important/useful here
This was merged from a different path in the same database, but this information is not available here.
For instance, if I merged from $/path/to/main/ down to $/path/to/branch/, the path to the main project is not available in the merge changeset. (note, please don't say that I'm merging the wrong way, it doesn't matter in this case so I just made it simple.)
So, the question is this: Is there any way I can find out where that changeset was merged from? Which branch it came from? ... and which changeset it originated as in that branch (like 13468? 13462? 13453? ...)
Background
We haven't used much branching and merging so far, except for simple stuff like "tagging" a release.
From now on we're looking at using branching much more active, but this creates a challenge.
Let's say I open up our bug tracker, take the topmost bug, fixes it, and checks it in. This is done in one branch, let's say this is the master branch.
Now, at some point, a tester is going to verify that the hotfix we're going to release has this bug fixed, so he opens up our product and wants to verify before he starts that the bugfix has actually gone into this build.
When we didn't use branching, we simply took the changeset number of the commit that ultimately fixed a case and typed that into the case itself. Additionally, our product was built with a build-number (4th part of version number) identical to the changeset that was the latest changeset that became part of the build.
This way, the tester could simply look at the case, the version number and easily deduce if the build had that changeset or not. If the changeset number in the version number was equal to or higher than the one in the case, the changeset was part of that build.
With branches, that doesn't work. If I commit changeset X on the master branch, but forget to merge, the tester can't simply say "If I run version X or higher, I go that fix" any more.
Note that we're not using TFS work items, so there's no easy built-in way to link commits and cases.
The reason I asked about the TFS history output was that I assume that if I can see that changeset 13469 really came from another branch, and corresponds to changeset 13462 there, and the programmer has noted 13462 on the case, I can say "13462 is now part of the build, because it was merged to the right branch, became 13469, and the build output has version 13470."
In other words, I could build a tool that as part of the build looked at the history of the database and grabbed all the necessary information and stored it in a database, so that I could take cases on our ready-to-test list and compare against the version number of the executable the tester was running, and just list all cases that is both ready to test and part of that build.
So my question is really this: Does anyone have any hints to how we can solve this? Perhaps we're boneheaded and needs to be told the right way to do this, so if you got any good ideas, let me know.
I hear and feel your lament here, as we've run into the same limitation. With TFS 2008, there's no easy way to see that history. With TFS 2010, and the branch visualizer, it gets easier.
If this is something you really need, you could potentially write it yourself using the TFS API. You would have to walk your way back through the various changesets for the files. It would be relatively straightforward to code:
Get merge changeset
Get prior merge changeset
Determine merge source from the first changeset
Get history for the file between the dates of the two changesets.
I've done this manually before, but you could either do this in C# code, or, alternatively, write a PowerShell script to do this.
I'm looking for a version numbering scheme that expresses the extent of change, especially compatiblity.
Apache APR, for example, use the well known version numbering scheme
<major>.<minor>.<patch>
example: 4.5.11
Maven suggests a similar but more detailed schema:
<major>.<minor>.<patch>-<qualifier>-<build number>
example: 4.5.11-RC1-3732
Where is the Maven versioning scheme defined? Are there conventions for qualifier and build number? Probably it is a bad idea to use maven but not to follow the Maven version scheme ...
What other version numbering schemes do you know? What scheme would you prefer and why?
I would recommend the Semantic Versioning standard, which the Maven versioning system also appears to follow. Please check out,
http://semver.org/
In short it is <major>.<minor>.<patch><anything_else>, and you can add additional rules to the anything else part as seems fit to you. eg. -<qualifier>-<build_number>.
Here is the current Maven version comparison algorithm, and a discussion of it. As long as versions only grow, and all fields except the build number are updated manually, you're good. Qualifiers work like this: if one is a prefix of the other, longer is older. Otherwise they are compared alphabetically. Use them for pre-releases.
Seconding the use of semantic versioning for expressing compatibility; major is for non-backwards compatible changes, minor for backward-compatible features, patch for backward-compatible bugfixes. Document it so your library users can express dependencies on your library correctly. Your snapshots are automated and don't have to increment these, except the first snapshot after a release because of the way prefixes are compared.
Purely for completeness, i will mention the old Apple standard for version numbers. This looks like major version. minor version. bug version. stage. non-release revision. Stage is a code drawn from the set d (development), a (alpha), b (beta), or fc (final customer ship - more or less the same as release candidate, i think).
The stage and non-release revision are only used for versions short of proper releases.
So, the first version of something might be 1.0.0. You might have released a bugfix as 1.0.1, a new version (with more features) as 1.1, and a rewrite or major upgrade as 2.0. If you then wanted to work towards 2.0.1, you might start with 2.0.1d1, 2.0.1d2, on to 2.0.1d153 or whatever it took you, then send out 2.0.1a1 to QA, and after they approved 2.0.1a37, send 2.0.1b1 to some willing punters, then after 2.0.1b9 survived a week in the field, burn 2.0.1fc1 and start getting signoffs. When 2.0.1fc17 got enough, it would become 2.0.1, and there would be much rejoicing.
This format was standardised enough that there was a packed binary format for it, and helper routines in the libraries for doing comparisons.
After reading a lot of articles/QAs/FAQs/books I become to think
that [MAJOR].[MINOR].[REV] is most useful versioning schema to
describe compatibility between project version (versioning schema
for developer, does not for marketing).
MAJOR changes is backward incompatible and require changing
project name, path to files, GUIDs, etc.
MINOR changes is backward compatible. Mark introduction of new
features.
REV for security/bug fixes. Backward and forward compatible.
This versioning schema inspired by libtool versioning semantics and by articles:
http://www106.pair.com/rhp/parallel.html
NOTE: I also recommend provide build/date/custom/quality as additional info (build
number, build date, customer name, release quality):
Hello app v2.6.34 for National bank, 2011-05-03, beta, build 23545
But this info is not versioning info!
Note that a version number scheme (like x.y.0 vs. x.y) can be constrained by external factors.
Consider that announcement for Git 1.9 (Januaury 2014):
A release candidate Git v1.9-rc2 is now available for testing at the usual places.
I've heard rumours that various third-party tools do not like the two-digit version numbers (e.g. "Git 2.0") and started barfing left and right when the users install v1.9-rc1.
While it is tempting to laugh at them for their sloppy assumption, I am also practical and
do not mind calling the upcoming release v1.9.0 to help them.
If we go that route (and I am inclined to go that route at this moment), the versioning scheme will be:
The next release candidate will be v1.9.0-rc3, not v1.9-rc3;
The first maintenance release for v1.9.0 will be v1.9.1 (and Nth one be v1.9.N); and
The feature release after v1.9.0 will be either v1.10.0 or v2.0.0, depending on how big the feature jump we are looking at.