Background
1 x Dev SQL Server
1 x UAT SQL Server
1 x Prod SQL Server
Developers use SSMS to view SQL Server objects and code and make changes directly to these objects in SQL Server itself.
Challenge
We have multiple developers potentially making changes to the same database object (let’s say a stored procedure or a view). The challenge arises from different bits of work happening on the same object where the delivery timescales for release for each of the bits of work are different. This means we end up with someone having completed their changes on the dev object, but releasing the changes into the next environment along may fail as the view (for example) contain may another developer’s changes too, and those changes themselves may require other objects. The business may not be expecting that other’s developer’s work to be released anyway, as there may be days/weeks of effort still to put into it before release. But that doesn't help the developer who's ready to go into the next environment.
How do we get round that?
How should each developer have started off, before they started making changes, to avoid dependency issues when releasing?
How can a developer “jump the queue” and release their bits of work, equally without scuppering anyone starting off their particular change too.
This is not a perfect answer, nor is it the only potential answer - but it's a good start. It's based on my experience within a relatively small shop, where tasks are re-prioritised frequently and changes required after testing etc.
Firstly - it's about process. You need to make sure you have a decent process and people follow it. Software etc can help, but it won't stop people making process errors. There are a lot of products out there to help with this, but I find making small steps is often a good start.
In our shop, we use Git source control for managing codes and releases. These script the entire database structure and views/etc, and are used to manage any changes to those scripts.
In general, we have a 'release' branch, then 'feature' branches for updates we're working on, and 'hotfix' branches for when we do changes to live on the fly (e.g., fixes etc).
When working on a specific branch, you check out that branch and work on it. Any change to the database has to go into an appropriate branch.
When ready to go live, you merge the feature/hotfix branches into that release branch when they're released. This way the 'release' branch always exactly matches what is on the production database.
For software, we use Redgate Source Control integratated with SSMS, but there are definitely others available (e.g., ApexSQL Source Control). You can also do it manually, but I wouldn't suggest it.
You don't have to, but you can also use a git GUI (e.g., SourceTree) to manage your branching and merging etc.
There are additional software products that can help to manage releases/etc (including scripting etc) but the source control aspect should be the biggest help with the main issue (being able to work on different things and helping ensure no clashes).
Regarding Git and how to use it (or SVN etc) - if you haven't used them before, they're a bit weird and take some getting used to. We had a few re-starts with a few different processes before we came up with an approach we liked. It will also take some time to run into the different issues that can arise - so you cannot expect this to just fix it out of the box.
1 source control
Any source control system GIT/TFS to manage your code and control changes
2 branching/release strategy
Git Flow! F.e. main branch with current working source code (main, develop whatever you call it), each developer works on his own feature branch, after he done his work he test it by deploying on DEV environment and running tests. After that it could be merged into release branch that will go live on PROD.
Also you need to consider merge vs rebase strategy (some link).
3 and some SCRUM
The most basic: 2 weeks for sprint, after end of the sprint you create new release branch and deploy it on UAT for testing. During next sprint release is tested on UAT, developers work on tasks from new sprint. Deploy tested release on PROD, developers have there 3rd sprint and UAT is ready for new release to be deployed. And so on.
4 more then one DEV environments
Based on the number of developers you need more DEV environments.
Related
Here is the scenario.
We are developing a product where we have a base product and regional variations for the product. We have all the common code checked into the main trunk while we have created 2 branches (branch_us, branch_uk) for the variations off of the main trunk. There is common code that is constantly being checked into the main trunk and the code that is being checked into branch_uk,branch_us is dependent on the code that is checked into the main trunk. This is being done because we expect more regions to added in future releases and as a result we want to have max reuse as well as thin regional variations layer.
Based on the current strategy, the developer will have to develop locally and then manually check-in the common files into main_trunk and regional variations into branch_uk & branch_us. Then everytime code is checked into the main_trunk, we will have to perform a merge from main_trunk->branch_uk & main_trunk->branch_us before we can perform a build for branch_uk & branch_uk (two separate deployments) because of dependency of new code in branch_uk/us branch to the new common code in main_trunk. This model seems extremely painful to think about and unproductive.
I'm by no means an expert on TFS. Here is what I am seeking opinion on:
Is there a way TFS can dynamically pull changes into branch_uk/branch_us from the main_trunk without doing a manual merge after every check-in (in the main_trunk)?
Do you guys have any other recommendations on the code management process that might be more effective/productive than the current one?
Any thoughts and feedback will be much appreciated!
This seems like a weird architecture to me, but of course I'm coming at it from a position of almost total ignorance, so there might be a compelling reason to approach it that way.
That being said: It sounds to me like you don't have a single application with two regional variations, you have two separate applications that share a common ancestor. The short answer to your question is "No". A slightly longer answer is "No, but you could write code to automate it."
A more thoughtful question-answer is "Are you sure centralized version control is the right tool for the job?" It might be more intuitive to use Git for this. What you have are, in effect, a base repository and two forks of that repository. Developers can work against whatever fork makes sense, and if something represents a change that should apply to all localizations, open a pull request to have the change merged into the base repository. This would require more discipline on the part of the developers, since they would have to ensure that their commits are isolated such that they can open a pull request that contains just commits that apply to the core platform. Git has powerful but difficult history-rewriting tools that can assist. Or, of course, they could just switch back and forth between working on the core platform, then pulling changes from the core platform back up to the separate repositories. This puts you back to where you started, but Git merges are very fast and shouldn't be a big issue.
Either way, thinking of the localizations are a single application is your mistake.
A non-source control answer might involve changing the application's architecture so that all localizations run off of the same codebase, but with locale-specific functionality expressed in a combination of configuration flags and runtime-discoverable MEF plugins, or making a "core" application platform that runs as an isolated service, and separately developed locale-specific services that express only deviations from the core application platform.
I'm currently experimenting with Bazaar, trying to find a configuration that works well for the project I am currently working on with a team of developers. Our model is currently centralised but I am open to persuasion about changing this if I can see significant benefits. Despite advocating it, I will not be able to change the version control system... some people don't know what's good for them.
At present, the project is divided into tiers. It's a web project and consists of a data access layer, a business/domain logic layer and a web layer (and a bunch of other application level projects such as services that that sit on the domain).
At present I find a bazaar repository with a number of working trees in there, one for each of the tiers I have mentioned.
Question part 1
Is there a better alternative to using working trees inside a respository for each tier?
What I have considered:
Putting everything into one fat branch/working tree (I am told this was purposely avoided, because of the necessity to check out everything). Developers are creating their own local setups by checking out subsets of the multiple repositories. For example if I am working on one of the services, I checkout the service, the business layer and the data access layer to a local directory. In the current setup though, I can checkout a top tier application which doesn't affect anything else, make a change to it and commit it back, without checking out the entire repository, which is nearly 1GB in size.
What I would like to remedy:
The problem is really that my web tier is reliant on a version of the
business layer, which in turn is reliant on the data access layer.
With a project organised like this, I have no idea which version of
the business and data access layers were current for a given commit
on the web layer. This feels very bad to me. I want to be able to
checkout versions of our application relative to some commit to one
of the working trees. At the moment we are attempting to keep track of
this across all of the working trees with tagging, but it feels
pretty wrong to me.
Question part 2
If I branch one of these tiers for a release branch, and find that a change in the root of the branch needs to be in that release, how do I push only those required changes into the branch?
So assuming that the structure of one working tree (or possibly a real branch) exists for each of these projects, we would like to have a branch for each of these, which contain a version for the a particular release. So the website tree has a website_rls1 branch, which encapsulates the state of development for that particular release. General development for a later release is going to happen concurrently. So if one file out of many happens to be relevant to this release, I would like to merge that single file into my release branch. What is the preferred method of doing this. As I understand it, Bazaar does not support merging/pulling individual files.
What I have considered:
Just putting the files I want into my local checkout of the release branch and committing
What I would like to remedy:
Following this consideration is going to kill off version information for this file. I want to keep the version information in tact, but only merge in a subset of changes. I essentially want to push individual files up to the child branch, or something to the effect of.
Any thoughts appreciated.
#1 There are two plugins for managing multiple dependent bzr repositories:
https://launchpad.net/bzr-externals
https://launchpad.net/bzr-scmproj
They have different approaches and are suited to different situations.
You may also want to consider solving this problem at the build level. Use a continuous integration system like Jenkins and a dependency resolution system like Ivy or Maven. QA should be done using automated builds from this system so that bugs that are filed can refer to a particular build number which should include in its logs the versions of the various dependencies used to produce that build.
#2 There isn't really a way to do that merge. Bazaar doesn't track cherry-pick merges (yet). A better approach would be to make the original fix in the release branch and then merge it back into its parent.
I am the QA Test Lead for a large enterprise software company with a team of over 30 developers and a small team of QA Testers. We currently use SVN to do all our code and schema check in which is then built out each night after hours.
My dilemma is this: All of developments code is promoted from their machine to the central repository on a daily basis into a single branch. This branch is our production code for our next software release. Each day when code is checked in, the stable branch is de-stabilized with this new piece of code until QA can get to testing it. It can sometimes take weeks for QA to get to a specific piece of code to test. The worst part of all of this is that we identify months ahead of time of what code is going to go into the standard release and what code will be bumped to the next branch, which has us coding all the way up until almost the actual release date.
I'm really starting to see the effects of this process (put in place by my predecessors) and I'm trying to come up with a way that won't piss off development whereby they can promote code to a QA environment, without holding up another developers piece of code. A lot of our code has shared libraries, and as I mentioned before it can sometimes take QA awhile to get to a piece of code to test. I don't want to hold up development in a certain area while that piece of code is waiting to be tested.
My question now is, what is the best methodology to adopt here? Is there software out there than can help with this? All I really want to do is ensure QA has enough time to test a release without any new code going in until it's tested. I don't want to end up on the street looking for a new job because "QA is doing a crappy job" according to a lot of people in the organization.
Any suggestions are greatly appreciated and will help with our testing and product.
It's a broad question which takes a broad answer, and I'm not sure if I know all it takes (I've been working as dev lead and architect, not as test manager). I see several problems in the process you describe, each require a solution:
Test team working on intermediate versions
This should be handled by working with the dev guys on splitting their work effort into meaningful iterations (called sprints in agile methodology) and delivering a working version every few weeks. Moreover, it should be established that feature are implemented by priority. This has the benefit that it keep the "test gap" fixed: you always test the latest version, which is a few weeks old, and devs understand that any problem you find there is more important than new features for next version.
Test team working on non stable versions
There is absolutely no reason why test team should invest time in version which are "dead on arrival". Continuous Integration is a methodology by which "breaking the code" is found as soon as possible. This require some investment in products like Hudson or home-grown solution to make sure build failure are notices as they occur and some "Smoke Testing" is applied to them.
Your test cycle is long
Invest in automated testing. This is not to say your testers need to learn to program; rather you should invest in recruiting or growing people with their knowledge and passion in writing stable automated tests.
You choose "coding all the way up until almost the actual release date"
That's right; it's a choice made by you and your management, favoring more features over stability and quality. It's a fine choice in some companies with a need to get to market ASAP or have a key customer satisfied; but it's a poor long-term investment. Once you convince your management it's a choice, you can stop taking it when it's not really needed.
Again, it's my two cents.
You need a continuous integration server that is able to automate the build and testing and deployment. I would look at a combination of Apache Hudson, JUnit (DBUnit), Selenium and code quality tools like Sonar.
To ensure that the code that the QA is testing is unique and not constantly changing, you should make the use of TAGs. A tag is like a branch except that the contents are immutable. Once a set of files have been checked in / committed you cannot change and then commit on top of those files. This way the QA has a stable version of code they are working with.
Using SVN without branching seems like a wasted resource. They should set up a stable branch and a test branch (ie. the daily build). When code is tested in the daily build it can be then pushed up to the development release branch.
Like Albert mentioned depending on what your code is you might also look into some automated tests for some of the shared libraries (which depending on where you are in development really shouldn't be changing all that much or your Dev team is doing a crappy job of organization imho).
You might also talk with your dev team leaders (or who ever manages them) and discuss where they view QA and what QA can do to help them the best. Ask: Does your dev team have a set cut off time before releases? Do you test every single line of code? Are there places that you might be spending too much detailed time testing? It shouldn't all fall on QA, QA and dev need to work together to get the product out.
I need some concrete ideas for this.
We're looking at changing our version control system, and I'd like for us to use Mercurial. It would ease a lot of pain related to some internal processes, as well as pose some challenges.
One of those challenges is that the version control system we're currently using is not a distributed one, and thus has the concept of revision numbers for each changeset, which we've used internally.
Basically, when the programmer checks in the final change that fixes a case in our case management system, Visual Studio responds with which changeset number this became, and the programmer then affixes that to the case which basically says "If you're running a version of our product with the last version number this value, or higher, then you have all the changes in that version."
However, with Mercurial that doesn't work, as revision numbers can and will change as commits come in from different branches.
So I'm wondering how I can get something similar.
Note, this is not about release management. Releases are much more controlled, but on-going tests are more fluid, so we'd like to avoid having a tester continuously trying to figure out if the test-cases on his list are really available in the version he's testing on.
Basically, I need the ability for someone testing to see if the version they're testing on has the changes related to a test-case, or not.
I was considering the following:
The programmer commits, and grabs the hash of the changeset
The programmer affixes this to the case in our case tracker
The build process will have to tag (not in the Mercurial way) the version so that it knows which changeset it was built from
I have to make it easy to take the hash of the changeset our product was built from, look it up in the changeset log of the repository that is used for our build machine, and then figure out if the product changeset is the same as, or an ancestor of, each case in the test list.
So I have two questions:
Is this a feasible approach? I'm not adverse to creating a web application that makes this easy to handle
Does anyone know of an alternate process that would help me? I've looked at tagging, but it seems that tagging ends up adding merge pressure, is that something I want? (ie. adding/moving a tag ends up as a commit, which needs to be merged with the rest of the system)
Is there anything out there that would help me out of the box, that is, have someone made, or know of, something like this already?
Any other ideas?
Is it right to say that you're looking for a lightweight tagging process linked to your build process?
I'm not keen on the idea of the programmer grabbing the last hash and sticking it somewhere else - sounds like the sort of manual process that you couldn't rely on happening. Would you be able to build a process around programmers adding the case number to their commit message so something could later link the commit to the original case? When the case was marked as "closed" you could pick up all commits against the case.
Lots of case control systems have this - Fogbugz, for example.
Both bitbucket and google code keep a branching timeline, that shows visually what has been merged, by who, and when. I suspect that this might be what you want to do: it's a very simple way to resolve issue 4.
How they do that, I don't know, but the tools are out there. BitBucket offers commercial code hosting.
I manage a small team of developers who up till recently have all been working on independent projects.
We have now all come together to work on one holistic project and its really tough. People are changing things, without consulting other programmers and its very difficult to manage.
We are also working in a pure production environment (no test/production)
I am a bit lost on where to start, I am looking at products like Team Foundation Server or their is a product by Red gate called "Source control"
Does anyone know any good books on the above subject, I cannot be the only person in this situation? :)
Source Code:
If you're looking for how to use source control (versus how to get your team to a controlled source environment) you should check out Source Control HOWTO by Eric Sink. The disclaimer here is that he has written a source control product but the HOWTO isn't specifically tailored to it.
Database Schema:
If you're looking for how to get your database (structure not data) in source control (since it was mentioned in the title) then you should check out this Coding Horror post: Get your database under version control. It covers some basic concepts and points to a few articles on how to actually implement getting your database in source control.
Source Control Software:
If you are looking for actual source control software, a list of applications can be found here: List of Source Control Systems with Visual Studio Plugins.
Don't make the mistake I did when putting the database definition under source code control by allowing branches and merges. This led developers (and managers) to assume they could have several versions of a table in the same database, leading to programmers overwriting each others changes, and having incompatible versions of source code and table definition all over the place.
As you might well imagine this lead to a great deal of wasted time and many frayed tempers.
Depending if time or money is more important, do you have the option to being someone in for a short time with the specific task of improving your processes? They would be able to set you up a test environment, install a version-control server and get your guys (and you) up to speed.
The first thing that you need to do is to give every developer their own copy of the database.
Secondly, you need each dev to be able re-create a baseline database from the master.
Once each dev is happy with their changes to the database, you will then need them to create a series of patches for their changes.
Lastly, each one of these patches needs to be able to checked in to source-control, and re-distributed to the other devs.
If you are looking for a good source control server, consider Subversion, with Tortoise SVN as your client.
Take a look at DBSourceTools. (http://dbsourcetools.codeplex.com).
It's specifically designed to help developers use source control for database changes.
This tool will allow you to baseline your database at a specific point, and create a named version (v1).
All files are scripted to disk, and very easy to add to a Subversion repository.
Have a look at the patching process for creating and using patches.
Have fun.