I think it's better to release the version of the software which your developers actually tested; I therefore tend to delete the 'debug' target from the project/makefile, so that there's only one version that can be built (and tested, and debugged, and released).
For a similar reason, I don't use 'assertions' (see also Are assertions always bad? ...).
One person there argued that the reason for a 'debug' version is that it's easier to debug: but, I counter-argued that you may eventually want to support and debug whatever it is you released, and so you need to build a release which you can if necessary debug ... this may mean enabling debug symbols, and disabling some optimizations, even in the 'release' build.
Someone else said that "this is such a bad idea"; it's a policy I evolved some years ago, having been burned by:
Some developers' testing their debug but not release versions
Some developers' writing bugs which show up only in the release version
The company's releasing the release version after inadequate testing (is it ever entirely adequate?)
Being called on to debug the release version
Since then I've seen more than one other development shop follow this practice (i.e. not have separate debug and release builds).
What's your policy?
Having separate debug and release builds is a good idea, because it does make development easier.
But debug builds should be for development only, not for testing. You test release builds only. And you don't use developers to test those builds, you use testers.
It's a simple policy that gives the best of both worlds, IMO.
Edit: In response to a comment, I think it's obvious that debug and release builds (can) generate different code. Think "-DDEBUG" vs. "-DNDEBUG", and "#if defined(DEBUG)", etc.
So it's vital that you test the code that you end up shipping. If you do generate different code in debug and release builds, that means testing twice - regardless of whether or not it's tested by the same person.
Debug symbols are not that big an issue, however. Always build with debugging symbols, keep a copy of the unstripped binary, but release a stripped binary. As long as you tag each binary with a build number somehow, you should always be able to identify which unstripped binary corresponds to the stripped binary that you have to debug...
How to strip binaries and load symbols in your debugger from an external source is platform-dependent.
This might be minor, but it adds up to what others have said here. One of the advantages of having QA test release builds is that over time the built in debugging and logging capabilities of your software will advance due to the needs of developers who need to figure out why things are going wrong in QA.
The more the developers need to debug release builds, the better tools you'll have later when customers start having issues.
Of course, no reason for developers to work on release builds as part of the development cycle.
Also, I don't know any software company that has long enough cycles to afford the overhead of switching QA from debug to release builds halfway through a version's testing period. Having to do a full QA cycle is something that all too often happens pretty rarely.
Our policy is to have developers work on Debug builds, but EVERYONE else (QA, BAs, sales etc) runs the release version.
Yesterday I had to fix a bug that only showed up in the release build it, was obvious what was happening simply BECAUSE it only showed up in release
It's first one here in this shop, and I've been here 18 months or so.
Where things get hairy is when the Release build does different things to the debug build - Yes, I have been to Hell and seen this in some very old, very ropy production code.
I see no reason why not to have both if the only difference between the configurations are debug symbols and optimisations.
so you need to build a release which
you can if necessary debug ... this
may mean enabling debug symbols, and
disabling some optimizations, even in
the 'release' build.
Ummm... it sounds like you're doing a debug build to me... right?
The part where you went wrong is this statement:
I think it's better to release the
version of the software which your
developers actually tested
Developers don't test code. Tests test code.
Your unit tests should test ALL build configurations. Do not make your developers work with one hand tied behind their back - let them use all the debugging tools they have at there disposal. A Debug build is one of these.
Regarding asserts: the use of assertions greatly depends on whether or not you program by contract. If you do, then assertions merely check the contract in a debug build.
As per my answer in the linked thread, we also use the same build for debug and release for very similar reasons. The 10%-20% performance gains from the optimiser tend to be very minor when compared to manual optimisations at algorithm level. A single build removes many potential bugs. Specifically;
Uninitialised variables and small buffer overflows may end up with very different results in debug and optimised release builds.
Even with the symbolic information available, debugging an optimised release can be difficult as the object doesn't match the source, e.g. variables may have been optimised out and code may have been re-arranged. Thus bugs reported in tested release builds can be more difficult, and hence time-consuming, to track down.
Having compared unoptimised and optimised builds under automated regression tests, the performance gains provided by the optimisation don't provide enough extra value to have two builds in my case. It is may be worth noting that the software that I develop is very CPU hungry (e.g. creating and manipulating large surface models).
When developing with Java, I hate non-debug versions. When an exception is thrown, you get no line information which makes it hard or even impossible to track bugs down. Also, the runtime difference between debug and non-debug is around 5% with Java 5 or later, so this is really no issue and with todays hard disks, size doesn't matter anymore.
On the plus side using debug versions:
Stack traces contain all the information you need
Variables can be examined
If you have a problem in production, you can simply attach to the running process without having to stop the server first to install a debug version.
You won't get caught by clever optimization bugs
The build is more simple (just one artifact)
Developers work with debug builds, QA and everyone else uses the release version, which we call "production". The main advantage to this is that in the debug build, we can add lots of extra code and assertions. Some objects contain extra pieces of information that have no use except when viewing code in the debugger. Some objects validate themselves periodically to make sure that all the state information is consistent. These things make the debug version much slower, but they have helped us find no end of bugs that would have been hell to find in the production build.
As I said, all of our QA and performance testing uses production builds, and we do occasionally run into problems that show up in production but not in debug. But they're relatively rare, and as a developer, the advantages of debugging a debug build rather than a production build far outweigh that problem.
I think it depends on the project size and what type of build system and testing that you are using.
If you have an automated build system in place, and it's simple to run unit and functional tests on a given build, then you should never have any problems with multiple build types.
I've always subscribed to the "Ship what you debug, so you can debug what you ship" approach, for all the reasons you list in your question.
In my opinion this discussion missing a very important point:
It really depends upon what kind of project it is!
If you create a native (C/C++) project you will in effect be forced to create debug builds, simply because compiler optimizations can make debugging near impossible in some cases.
If you create web applications you might rather wish to simply have one build (although "build" is rather misleading for some web applications) that can enable logging features during runtime.
Although a native C++ project and a PHP web application are obviously not all kinds of project that exist, I hope my point got across.
P.S.: When developing for C#, you run into a border case since although using a debug build disables compiler optimizations, in my experience you will not run into nearly as much differences as with C++
here we develop in debug mode and do all unit testing in release mode. we are a small shop with just a few (under 12) application to support ranging from Classic ASP, ASP.Net, VB.Net, and C#.
We also have a dedicated person to handle all testing, debugged problems are thrown back to the developers.
We always build both, never even considered not doing so. Enabling debug options increases your code size and slows performance, possibly not an issue with your type of software when testing but what if the customer is running your code plus 5 other apps...
The issues with testing can be sorted out by using automated testing so you're release build can be effortlessly tested when you think you're ready to release. The failure of your developers or company to properly test release builds is not a failure in the idea of release and debug builds but in your developers and or company.
On your last point, I have never been called upon to debug a release build, just to fix it...
It's a tradeoff. Given that CPU cycles are cheap and getting cheaper while human cycles remain expensive, it makes a lot of sense to maintain only a single version of a large, complex program -- the debug(gable) version.
Always using assertions always is a safer policy than never using them. If producing separate debug and release versions, re-enable whatever #defined symbols you need to guarantee that assertions are enabled in the release version also.
I think the tradeoff is simple: yes, with only a release build, you really test what's actually being shipped. On the other hand, you do pay a price in ease of debugging for your developers and/or performance for the user, so it's up to you to check both cases.
On most medium- to large-size projects, ease of debugging will ensure a better product for your users in the end.
See this What's your most controversial programming opinion?
quote:
Opinion: Never ever have different
code between "debug" and "release"
builds
The main reason being that release
code almost never gets tested. Better
to have the same code running in test
as it is in the wild.
By removing the "debug target", you are forcing developers to debug on the release version of the software. What that probaly means in practice is two things:
1) "release builds" will have optimizations disabled (otherwised developers can't use a debugger)
2) No builds will have special PREPROCESSOR macros altering their execution.
So what you will really be doing is merging the release and debug configurations rather than eliminating just the "debug" mode.
I personally have done this with iOS development with no ill-effects. The amount of time spent in our written code is less than 1% of what is really happening, so the optimizations were not significant contributors. In this case, they really did seem to cause an increase in bugs, but even if they didn't, the idea of testing one way, then giving to QA with different code introduces just one more factor to consider with issues.
On the other hand, there are cases where the optimizations are necessary, where they are useful, and even where there is enough time for testing both. Usually, the changes between debug and release are so minor that it doesn't cause anyone any issues at all.
If you've got a real QA group who can be counted on to fully test the thing, I'd say make debug builds until you get close to the release, and then make sure a full QA cycle is done on the same build that's going out the door.
Although in at least one case we released something that still had some debug code in it. The only consequence was it ran a tiny bit slower and the log files were pretty damn big.
In my company we have both Debug and Release.
- The developers use the debug version to properly find and fix bugs.
- We are using TDD and so we have a big test suite that we run on our server that tests both debug and release build configurations as well as 64/32 builds we have as well.
So if using the "debug" configuration helps a developer to find a bug faster there is no reason not to use it - when the code goes into the server (to be further tested) or reviewed we use the "Release" one.
I learned to build the release version with .PDB files long ago so that I could debug the release version. What a lot of programmers tend to forget is that when you run the debug version, with all the optimizations turned off, you are debugging a different program altogether. It may behave like the release build (for the most part), but it is still a different program than the release build.
In addition, debugging the release build is not that difficult. And if you get a crash dump, you have to be able to do it anyway.
Related
I am a beginner in Cypress Automation Testing. I have one confusion. When we need to add our Automation scripts to run with GitHub workflows to trigger when we push a commit, for what environment should we write tests? In the local environment at localhost or for the staging site of the project?
Could anyone please get my confusion cleared on this Automation Testing and how it should be written and How can we add Cypress Automation Tests with GitHub CI/CD?
Thanks.
Ok, let me give this a shot. Of course, I do not know the exact setup of the project that you are working on, but let me give you some pointers, so you can decide for yourself what works best in your setting.
My answer is based on the assumption that you are building an automated regression test set in Cypress with the primary goal to prevent production incidents. In addition, it aims save you tons of 'manual testing' for each release to production because you want to make sure everything is still working properly.
First of all, you want your automated tests to run on a stable environment(*). If the environment is not stable, many tests will fail for many reasons, and those are usually not the right ones. You'll spend more time figuring out why your tests are failing, than actually catching issues with it. This makes a local, dev environment not really suited for the task, so I would not pick a localhost environment for this. Especially not when you have multiple developers working in your team, each with their own localhost.
A test environment is already a way more stable environment. You want your tests to only fail when you have an actual issue on your hands. As a rule of thumb, the 'higher' you go, the more stable.
Second, you want to catch the issues early in the game, so I would definitely make sure that the tests can run on the environment where all code comes together for the first time (in other words, the environment that has the master branch or whatever your team calls that branch). This is usually the test environment. In my projects, I initially build the set for this environment, and ideally, I run it daily. Your tests won't always pass here (bonus if they do), and that is OK... as long as you understand why they don't ;-)
Some things to keep in mind are integrations or connecting systems, and whether you need those for your tests to pass. In general, you don't want to be (too) dependent on (third-party) integrations for you test cases to go green. Sometimes, when those integrations are vital to the process that you need to test, it is inevitable. However, integrations are often not (fully) set up on test/lower environments. There are workarounds for this, like stubs, but let's not get into that now - that's a whole different topic.
Third, you want your tests to run on a production-like environment on the code exactly in the state that it goes to production. This is usually the acceptance, staging or pre-production environment, i.e. the last one before production. These environments often have all integrations in place and are often very similar to production. If you find an issue here, it's almost guaranteed that it is also an issue in production. This is IMO where you want to integrate your tests into your CI/CD pipeline. Ideally, your full automated set is in the pipeline, but in practice, you should only add the tests that are stable and robust, otherwise your production deployments will be blocked very often.
So, long story short, my advice: write your tests for your test environment, where you do your 'manual testing' (I hate that term BTW, all testing is manual... as if there is such a thing as 'manual coding') and run it early and often. Then put the stable ones in the pipeline of the production deployment. If you only have local, staging and production, it should be staging.
If your developers want to run the set on their local environments, they can still do that - you can share the tests with them or even better, they can take it from the repository and run it locally - but I don't think you should make it part of the deployment process always and everywhere. It will slow down your process massively.
You can work with environment variables to easily switch for the environment where you want to run your tests: https://docs.cypress.io/guides/guides/environment-variables#Setting
I hope this helps. I'm looking forward to read what others have to say about this, too.
Happy Testing!
Jackie
PS. I see that you also asked about how to add Cypress to your CI/CD pipeline. I think that should be a completely separate topic. It is also way too high level to answer. Maybe it's best to start here: https://docs.cypress.io/guides/continuous-integration/introduction#What-you-ll-learn
(*) I'm talking stable environment here, but this also includes stable code and even a stable application. If your application and code is in a very early stage, really ask yourself whether you already want to start automating your functional UI tests in Cypress - chances are that many things will change (many times) and you'll spend hours updating your tests. Maybe it is better to only think about the scenarios that you want to automate at that stage of the project.
What are the differences between the different Rebol 3 branches, especially with the new REN branch?
Is it the platforms they'll run on, the feature set, code organization, the C standard compliance?
This is an answer destined to become outdated, hence set to Community Wiki. This information is as of Sep-2015. So if updating this answer after some time has passed, please modify the date as well.
Binary download of Rebol3 from rebol.com
Last build was 5-Mar-2011 and pre-dates the open source release.
No GUI support, no HTTPS support, no serial port support, no UDP support, no smart console...
No 64-bit builds. Binaries are for Windows x86, OS/X (PPC or x86), Linux (x86 or PPC), FreeBSD x86.
While Rebol2 binaries are archived for many "esoteric" systems (BeOS, AIX, Windows DEC Alpha, QNX, Solaris...) similar binaries were not provided for Rebol3. The only "weird" build is for Amiga, and only an OS4 PowerPC Amiga. No successful builds of Rebol3 for Amiga emulators have been reported.
Open source release of Rebol3 on Github rebol/rebol
Open-sourcing was on 12-Dec-2012.
The rebol.com binary downloads were not rebuilt as part of this release. However, a community member (#earl here on SO) created a build farm at rebolsource.net that follows this GitHub master whenever it updates. Given that GitHub's rebol/rebol master hasn't been updated since March 2014, this dynamism is currently underused.
Building the source at time of release got an executable not distinguishable (?) in functionality from the builds on 5-Mar-2011. This suggests few changes to the source were made besides some cleanup and Apache-licensing edits to prepare for publication.
Minor patches and bugfixes were integrated sporadically, with most PRs sitting idle. Last PR accepted at time of writing was Mar 3, 2014, which is over a year ago.
The most noticeable "breaking" PR that did get approved was to repurpose the FUNCTION name. It was considered to be worth breaking the old arity 3 form to let the word be taken for the much more useful implementation as locals-gathering FUNCT. (This also brought Rebol in alignment with Red, whose FUNCTION is arity 2 and acts similarly.) FUNCT was kept around as-is for legacy code.
The most major non-breaking PR that was taken is probably not requiring blocks around IF, UNLESS, or EITHER bodies. This has been received well among those who know it's there, as fitting the freeform and non-boilerplate philosophy of the language. It allows some code constructs to get "prettier" and gives programmers more choice, while it doesn't seem to cause any more problems than anything else. It's certainly less of a speedbump than if [condition] [...], in fact it seems almost no one knows this feature got added, so it must not be biting anyone. (If anyone can bend ears over at Red to make sure it gets IF and IF/ONLY then that would be ideal.)
RETURN/REDO was removed. Rationale was that it permitted functions to effectively behave with variable arity, and that this was unnecessary and took terra firma away by no longer being able to predict a function's arity from its spec. Perhaps this stance warrants a second look...as Lisp users who are pressuring for the addition of Lisp-style macros seeming aren't worried about that very much. (Here in the StackExchange universe, this provoked a Programmers.SE question Would Rebol (or Red) benefit from Lisp-style Macros?, which hasn't gotten much in the way of answers yet.)
The fork by Saphirion: "Saphir"
Prior to the open-sourcing of Rebol, Saphirion AG had a special relationship with Rebol technologies. They had access to the source and were taking responsibility for most of the development work for Rebol3 GUI features. They also added several other things like HTTPS.
Saphir is available as a binary download from their website, but only provided for 32-bit Windows. There was at one time an experimental .APK for Android from Saphirion.
Some (but not all) of Saphir's source was released after the open-sourcing. Notable omissions were the android build and some Rebol3 code for encapping...a way of injecting compressed scripts and resources into binaries of the interpreter without needing to recompile it.
(Note: Under Apache2 license there is no requirement to release source code for one's derived work.)
"Community" Integration at Rebolsource on GitHub
With the GitHub rebol/rebol being held up on integrations, a fork at rebolsource/r3 was established to be a "community build" where work could be staged.
Rebolsource changes were conservative, seemingly aimed toward showing process for how GitHub's rebol/rebol might adopt changes "in the spirit in which Rebol was conceived" should that repository be delegated to the community. (For that spirit, see this.) Hence it integrated non-controversial bugfixes and tweaks, instead of large third-party cryptography libraries for implementing HTTPS. Also: no allowance for adding build dependencies besides a C compiler (no GNU autotools, for instance).
Binaries for the community build were produced on an as-needed basis for those requesting them who could not build it themselves.
Atronix Engineering's Rebol "3.0" at Github zsx/r3
Atronix is an industrial automation solutions provider that uses Rebol. How they do so is described in a video here by David den Haring, director of Engineering, and their ZOE software is built on their version of Rebol.
After the open sourcing, Atronix partnered with Saphirion to port the GUI to Linux. Atronix publishes their source publicly as it is developed, and David den Haring notes in the video above that they have only one proprietary component they developed (an industrial control driver). Other than that they are happy to share the source for all Rebol development they do.
Atronix integrated the 64-bit patches from Rebolsource, created a Windows 64-bit target, and offer up-to-date binaries of their development branch for Windows and Linux x86/x64, as well as Linux ARMv7.
Besides having the features of Saphir, the Atronix build added support for CALL with /INPUT, /OUTPUT, /ERROR. It also added a Foreign Function Interface, implementing LIBRARY!, ROUTINE! and STRUCT! for communicating with non-Rebol dynamic libraries. It brings in encapping support as well on Windows and Linux.
Rebol's "religion" was at times at odds with expedience, so the Rebol-based build process was replaced when needed by hand-edited makefiles and Visual Studio projects. The FFI library introduced a dependency on GNU autotools to build.
All Atronix builds include the GUI, so there is no "Core" build. And again, only Linux and Windows.
Ren-C
(Bias Note: This fork is the initiative #HostileFork started, knows the most about, and will speak most enthusiastically about.)
Ren-C started as an an extraction of a Core build out of Atronix's codebase. That gave it features like HTTPS, the enhanced CALL, and Foreign Function Interface to essentially all the platforms that Rebolsource was able to build for. Updates Jul/Sep-2015 Ren/C supports line continuations in the console, user infix functions, several bugfixes...
Ren-C makes large-scale changes and fixes fundamental issues in R3-Alpha, which are tracked on a Trello that provides more information. There is a new FAQ as a GitHub wiki. Critical issues like definitionally-scoped returns have been solved, with continuous work on other outstanding problems.
Though Atronix's R3/View required some additional dependencies, Ren/C pushed back to being able to be built with nothing besides a C compiler, and eliminated all handmade makefiles/projects.
Beyond Windows, Linux and Mac in both 32-bit and 64-bit variants, Ren/C has also been built for smaller players like HaikuOS and yes, even Syllable. This is interesting more for the demonstration of how broadly turnkey builds of the C89 code work (simply as make -f makefile.boot) as opposed to there being a particularly large userbase of those particular OSes!
From the point of view of language rigor, Ren/C is pushing on modern techniques. Although it can still build as C89, it can be built as C99 and C11 as well. It has also been verified to build as C++98 through C++14, and with some strategic modifications under #ifdef __cplusplus it can take advantage of modern C++ as a kind of static analysis tool over the C code. Warnings are raised, type errors all fixed up, and it's "const correct". The necessary changes were carefully considered to make Rebol's baseline C code not just more correct but cleaner and clearer source across the board.
From a point of view of C developers, Ren/C should be stable, organized, and commented enough for anyone who knows C to "modify with confidence" and try new features. That means being able to implement definitionally scoped returns (actually written, but not pushed), or try developing features like NewPath.
From a point of view of architecture, Ren/C is intended to not have an executable at all...but to be a library for embedding a Rebol interpreter into other programs. It is now the basis for Ren/C++, which was designed to anticipate working with Red as well.
From a point of view of testing, Ren/C intends to whip everything into shape for engineering rigor and zero bug tolerance. This means avoiding practices like zero-filling memory to obscure uninitialized memory accesses, using Address Sanitizer, Valgrind, and a test suite that can pass the highest settings on both.
While enabling all the extra functionality has made Ren/C's executable nearly twice the size of Rebolsource's, there's not yet been any audit to see how this can be brought down. It has been confirmed that there are duplicate copies of Zlib and PNG encoding/decoding--for instance (Saphirion included LodePNG, likely to work around a bug in the existing PNG because it was easier than fixing it...yet did not mothball the previous code). Also, being able to do a build which selectively integrates only the codecs you want to use is on the agenda.
Ren/C currently has the stakeholders from Atronix and Rebolsource participating in its development and direction, which strengthens the likelihood that it may evolve into "the" Rebol Core. It is now being linked in as the code backing Ren Garden, and using a similar approach it may be set up as the library used by Atronix's R3/View...then Rebolsource...and perhaps ultimately rebol/rebol itself.
The fork by Oldes
(Bias Note: this edit is added 28-Feb-2019 by Oldes himself)
Forked from the community branch. Main focus on keeping the code close to the original Carl's release without blindly taking everything from Atronix/Saphirion but still trying to pick-up the good things from these branches slowly.
Not like Ren-C, this version is not trying to introduce new syntax, but rather be closer to the original Rebol2 and new Red language
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I develop a library that is used by other software. Typically this library ends up packaged in Debian, Fedora, etc., and its "reverse-dependencies" also end up packaged and using it.
So, I guess this makes me an "upstream maintainer." I simply use autotools to produce a tarball, and packagers then use that to produce .deb files, etc. Now, something that has bothered me for quite some time is the disconnect between maintainers and packagers. I feel like every time I do a release, even if it is simply a bugfix release, I am potentially causing headaches for everyone down the chain.
Possible problems:
I introduced a bug that wasn't caught in testing, even though I tried extensively to test various configurations -- I don't have unlimited testing resources and it is a small library so I am mostly on my own although there are 1 or 2 other interested people who help out, but generally only test on one platform.
I forgot to bump the version number, causing confusion
I did bump the version number but forgot to bump the SO version (you know, the thing that specifies API/ABI compatibility, and is independent from the software release version)
I made a small change but accidentally caused an API incompatibility without thinking (e.g. made something "const" that should have been all along, didn't realize it would break people's code)
I made a small change but accidentally caused an ABI incompatibility -- e.g., changed a constant in a header file, wasn't thinking and forgot that this would be "baked in" in software compiled against a previous version
I have done pretty much all of these things at some time or other in the past. Due to these previous mistakes, these days I probably spend more time testing than actually developing, and still end up making mistakes. The mistakes are often not that bad, after all people understand, mistakes happen, but they sometimes cause people to drop using the library, without even talking to me or communicating on the mailing list, which sucks -- if those people were so interested, it would be cool if they had helped test before I published a release -- but anyways, you get the idea.
So, rather than just compiling and running the unit tests, my testing process now involves some pretty extensive steps. In particularly, I am now using "apt-cache rdepends" to find software that uses my library, and I install it and switch the binary out to test for ABI compatibility. Then, I uninstall it, and "apt-get source" it, and compile it against the new version to test for API compatibility.
This kind of testing involves,
understanding other peoples software and figuring out where and how it exercises my code
compiling other peoples software, including figuring out their other dependencies and how to get everything working -- for large projects this can be a nightmare.
some projects using my software are actually plugins for other projects, meaning I have to additionally get the host program working
many projects using my library are GUI-oriented, so I have to navigate and learn some software I don't even know or use, and then guess when I have got it to a place where it is actually calling out to my library
my library works on Linux, Windows, and OS X, and often I don't have enough machines and operating systems around to test on. For example, a huge problem with my last release was a bug that only showed up in Linux on x86_64. I had tested on Linux i386, and OS X 64-bit, but somehow these platforms didn't show the bug, it was particular to the Linux-64-bit combination which I had neglected testing because I didn't have the right hardware and assumed I'd covered enough ground.
As you can imagine this is not a light task, and makes for huge delays before publishing a given release, delaying the dissemination of bugfixes, etc. The worst thing is that my project is not even a large library, and is a hobby project of mine, so all of this feels like huge overhead just for something I do in my spare time. I'd rather be developing features than just defending against my own potential mistakes for every little change I make. But, it currently has 42 rdepends listed in Ubuntu, to give you an idea, and I'm proud that it is useful to other people so I want to be able to develop and improve it without worrying so much about breaking things for everyone.
My question is, how can I improve the efficiency of this testing process? Are there for example any tools that will automatically compile "rdepends" packages against a new version of my library and give me a report? Or somehow download compiled binaries of rdepends and test loading them against my ABI without actually necessarily requiring me to navigate the GUI of some unknown software?
how can I improve the efficiency of this testing process?
The main problem is communication, apart the fact that you lack scripts that automate the process. You can do pre-releases of your packages, mailing the distributions that your library supports, etc. or instead of maintaining the packages yourself insert them into some mayor Distro and let some experimented maintainer do the stuff.
You can always break people stuff, just don't do it so frequently. Remember that people need stability in some certain sense so you may document very well each change so people using your library can't say you didn't tell them.
About tools... you should find your own pace. Maybe some buildbots (AFAIK some projects lend build bots), maybe script automatizing the process you build stuff, etc. etc. etc., did I said etc.? The problem is too broad and there are effectively too many solutions that makes any suggestion non-viable. You may want to check https://softwareengineering.stackexchange.com/q/150466/104338 as "some" methods but, again, you should find your own pace.
I'm currently using OCUnit that ships with Xcode 3.2.4 for doing unit testing of my application. My workflow is often to set some break points in a failing unittestin order to quickly inspect the state. I'm using Apple's OCUnit setup:
http://developer.apple.com/library/ios/#documentation/Xcode/Conceptual/iphone_development/135-Unit_Testing_Applications/unit_testing_applications.html
but the setup from above gives me some headaches. Apple distinguish between Application tests and Logic tests. As I see it:
You cannot debug logic tests. It's as if they're invisibly run when you build your project.
You can debug application tests, but you have to run these on the device and not the simulator (what is the reason for this?)
This means that everything moves kind of slowly with my current workflow. Any hints on getting app tests to run on the simulator? Or any pin pointers to another test-framework?
Would eg. google-toolbox-for-mac work better in general or for my specific needs?
Also, general comments on using breakpoints in the unit tests are welcome! :-)
I have used the Google Toolbox testing rig in the past and it worked fine, it ran both on the Simulator and the device and I could debug my tests if I wanted to. Recently I got fed up with bundling so much code with each of my projects and tried the Apple way.
I also find the logic/app tests split weird, especially as I can’t find any way to debug the logic tests now. (And if you’re using some parts of AVFoundation that won’t build for Simulator, you are apparently out of luck with logic tests completely, which seems strange.) One of the pros is that I can run the logic tests quickly during build.
I guess this does not help you that much – the point is that you can debug the tests under GTM. And you might also want to check out this related question.
I know this isn't really a good answer, nor is it completely helpful to your cause. But I've been using NSLog to run my unit tests (if nothing outputs to the console, then success). When I comment out my tests method then the tests wouldn't run. I found this much more predictable and reliable than OCUnit. I'd much rather use a real true unit tester, but it was too frustrating to deal with the often strange errors that could occur from OCUnit and also the other shortfalls/lack of features you describe above.
If you are developing a commercial desktop application, what's your release process?
Sample process:
Develop it: Patch bugs, add features, etc.
Feature Freeze (do not fix, add anything unless it's absolutely required)
Test it
If everything is OK release it, if it's not fix it, test it, release it
I think the most crucial question is what's your approach to "feature freeze > test > release" cycle?
Or do you test it more frequently that you don't need such a cycle and your software is always ready for public release?
Depends on your QA budget. I've used both - using CI, at least on branch is always packaged and ready for release.
Feature-freeze > test > relase, the bigger QA budget, the shorter releases. Also, first stage is entirely dependand on release planning.
Also, how much developers test their work and how much coverage You have affect decision a lot.