We have a production database with texts that shows on our web site.
We also have development servers with multiple branches (several copies of the production database).
The poblem we have is that during development we add and change texts in each branch. And we also change texts in our production environment.
If development and production has changed the same text it's hard to find our how to merge these changes.
We were thinking of that we only can make changes on the production database and only add texts on development database. But that would give us many many text with different keys that has pretty much the same data.
How do you handle text changes between environments?
Thanks!
This is a fairly common problem - Martin Fowler wrote about it a while ago (http://martinfowler.com/articles/evodb.html).
THere's no nice, simple, painless solution - but http://www.amazon.com/Recipes-Continuous-Database-Integration-ebook/dp/B000RH0EI4 is probably the best book on the topic....
It's a fairly major undertaking, and requires a lot of discipline from your development team - but it's worth it if you're running into the problems you describe.
It boils down to scripting your database creation/modification tasks, and committing those scripts to source code control. You use a naming convention to determine the order in which to run the scripts, and then have an automated process to run them when setting up an environment, or deploying a new version to that environment.
Related
Background
1 x Dev SQL Server
1 x UAT SQL Server
1 x Prod SQL Server
Developers use SSMS to view SQL Server objects and code and make changes directly to these objects in SQL Server itself.
Challenge
We have multiple developers potentially making changes to the same database object (let’s say a stored procedure or a view). The challenge arises from different bits of work happening on the same object where the delivery timescales for release for each of the bits of work are different. This means we end up with someone having completed their changes on the dev object, but releasing the changes into the next environment along may fail as the view (for example) contain may another developer’s changes too, and those changes themselves may require other objects. The business may not be expecting that other’s developer’s work to be released anyway, as there may be days/weeks of effort still to put into it before release. But that doesn't help the developer who's ready to go into the next environment.
How do we get round that?
How should each developer have started off, before they started making changes, to avoid dependency issues when releasing?
How can a developer “jump the queue” and release their bits of work, equally without scuppering anyone starting off their particular change too.
This is not a perfect answer, nor is it the only potential answer - but it's a good start. It's based on my experience within a relatively small shop, where tasks are re-prioritised frequently and changes required after testing etc.
Firstly - it's about process. You need to make sure you have a decent process and people follow it. Software etc can help, but it won't stop people making process errors. There are a lot of products out there to help with this, but I find making small steps is often a good start.
In our shop, we use Git source control for managing codes and releases. These script the entire database structure and views/etc, and are used to manage any changes to those scripts.
In general, we have a 'release' branch, then 'feature' branches for updates we're working on, and 'hotfix' branches for when we do changes to live on the fly (e.g., fixes etc).
When working on a specific branch, you check out that branch and work on it. Any change to the database has to go into an appropriate branch.
When ready to go live, you merge the feature/hotfix branches into that release branch when they're released. This way the 'release' branch always exactly matches what is on the production database.
For software, we use Redgate Source Control integratated with SSMS, but there are definitely others available (e.g., ApexSQL Source Control). You can also do it manually, but I wouldn't suggest it.
You don't have to, but you can also use a git GUI (e.g., SourceTree) to manage your branching and merging etc.
There are additional software products that can help to manage releases/etc (including scripting etc) but the source control aspect should be the biggest help with the main issue (being able to work on different things and helping ensure no clashes).
Regarding Git and how to use it (or SVN etc) - if you haven't used them before, they're a bit weird and take some getting used to. We had a few re-starts with a few different processes before we came up with an approach we liked. It will also take some time to run into the different issues that can arise - so you cannot expect this to just fix it out of the box.
1 source control
Any source control system GIT/TFS to manage your code and control changes
2 branching/release strategy
Git Flow! F.e. main branch with current working source code (main, develop whatever you call it), each developer works on his own feature branch, after he done his work he test it by deploying on DEV environment and running tests. After that it could be merged into release branch that will go live on PROD.
Also you need to consider merge vs rebase strategy (some link).
3 and some SCRUM
The most basic: 2 weeks for sprint, after end of the sprint you create new release branch and deploy it on UAT for testing. During next sprint release is tested on UAT, developers work on tasks from new sprint. Deploy tested release on PROD, developers have there 3rd sprint and UAT is ready for new release to be deployed. And so on.
4 more then one DEV environments
Based on the number of developers you need more DEV environments.
My company has decided that each developer should have its own database locally for testing the Asp.net web application. I am against the decision as I think there should be a centralised test database for testing. But the developers put an argument that if they test the same table and other developers delete the records while I add the records at the same time it will create incorrect result. So what you guys think.
Each developer should have their own database if they are going to be making any changes to the data structure or values. If one of the devs adds/removes something that prevents a portion of code from working it will only affect that dev instead of all the devs. This also allows devs to get more comfortable with the data structure and making changes to it since they can break/fix their own environment as much as they want.
There should be a testing environment that has its own database where the current revision of the project can run for tests.
No, it's not a good practice (for integration tests for example), but it's acceptable for functional tests (or smoke tests) on development environment. The thing is that, it's better to integrate database and application from the very initial steps, thus you will avoid issues, bugs, waste of time. Also, make sure the developers don't make changes on schema, you should have a unique and well identified database version.
But the developers put an argument that if they test the same table and other developers delete the records while I add the records at the same time it will create incorrect result.
Sounds like a really bad web application. A good web application should be able to support more than one simultaneous user!
For that reason alone, perhaps your team should at least have a phase where people are sharing a common database.
I am the QA Test Lead for a large enterprise software company with a team of over 30 developers and a small team of QA Testers. We currently use SVN to do all our code and schema check in which is then built out each night after hours.
My dilemma is this: All of developments code is promoted from their machine to the central repository on a daily basis into a single branch. This branch is our production code for our next software release. Each day when code is checked in, the stable branch is de-stabilized with this new piece of code until QA can get to testing it. It can sometimes take weeks for QA to get to a specific piece of code to test. The worst part of all of this is that we identify months ahead of time of what code is going to go into the standard release and what code will be bumped to the next branch, which has us coding all the way up until almost the actual release date.
I'm really starting to see the effects of this process (put in place by my predecessors) and I'm trying to come up with a way that won't piss off development whereby they can promote code to a QA environment, without holding up another developers piece of code. A lot of our code has shared libraries, and as I mentioned before it can sometimes take QA awhile to get to a piece of code to test. I don't want to hold up development in a certain area while that piece of code is waiting to be tested.
My question now is, what is the best methodology to adopt here? Is there software out there than can help with this? All I really want to do is ensure QA has enough time to test a release without any new code going in until it's tested. I don't want to end up on the street looking for a new job because "QA is doing a crappy job" according to a lot of people in the organization.
Any suggestions are greatly appreciated and will help with our testing and product.
It's a broad question which takes a broad answer, and I'm not sure if I know all it takes (I've been working as dev lead and architect, not as test manager). I see several problems in the process you describe, each require a solution:
Test team working on intermediate versions
This should be handled by working with the dev guys on splitting their work effort into meaningful iterations (called sprints in agile methodology) and delivering a working version every few weeks. Moreover, it should be established that feature are implemented by priority. This has the benefit that it keep the "test gap" fixed: you always test the latest version, which is a few weeks old, and devs understand that any problem you find there is more important than new features for next version.
Test team working on non stable versions
There is absolutely no reason why test team should invest time in version which are "dead on arrival". Continuous Integration is a methodology by which "breaking the code" is found as soon as possible. This require some investment in products like Hudson or home-grown solution to make sure build failure are notices as they occur and some "Smoke Testing" is applied to them.
Your test cycle is long
Invest in automated testing. This is not to say your testers need to learn to program; rather you should invest in recruiting or growing people with their knowledge and passion in writing stable automated tests.
You choose "coding all the way up until almost the actual release date"
That's right; it's a choice made by you and your management, favoring more features over stability and quality. It's a fine choice in some companies with a need to get to market ASAP or have a key customer satisfied; but it's a poor long-term investment. Once you convince your management it's a choice, you can stop taking it when it's not really needed.
Again, it's my two cents.
You need a continuous integration server that is able to automate the build and testing and deployment. I would look at a combination of Apache Hudson, JUnit (DBUnit), Selenium and code quality tools like Sonar.
To ensure that the code that the QA is testing is unique and not constantly changing, you should make the use of TAGs. A tag is like a branch except that the contents are immutable. Once a set of files have been checked in / committed you cannot change and then commit on top of those files. This way the QA has a stable version of code they are working with.
Using SVN without branching seems like a wasted resource. They should set up a stable branch and a test branch (ie. the daily build). When code is tested in the daily build it can be then pushed up to the development release branch.
Like Albert mentioned depending on what your code is you might also look into some automated tests for some of the shared libraries (which depending on where you are in development really shouldn't be changing all that much or your Dev team is doing a crappy job of organization imho).
You might also talk with your dev team leaders (or who ever manages them) and discuss where they view QA and what QA can do to help them the best. Ask: Does your dev team have a set cut off time before releases? Do you test every single line of code? Are there places that you might be spending too much detailed time testing? It shouldn't all fall on QA, QA and dev need to work together to get the product out.
I manage a small team of developers who up till recently have all been working on independent projects.
We have now all come together to work on one holistic project and its really tough. People are changing things, without consulting other programmers and its very difficult to manage.
We are also working in a pure production environment (no test/production)
I am a bit lost on where to start, I am looking at products like Team Foundation Server or their is a product by Red gate called "Source control"
Does anyone know any good books on the above subject, I cannot be the only person in this situation? :)
Source Code:
If you're looking for how to use source control (versus how to get your team to a controlled source environment) you should check out Source Control HOWTO by Eric Sink. The disclaimer here is that he has written a source control product but the HOWTO isn't specifically tailored to it.
Database Schema:
If you're looking for how to get your database (structure not data) in source control (since it was mentioned in the title) then you should check out this Coding Horror post: Get your database under version control. It covers some basic concepts and points to a few articles on how to actually implement getting your database in source control.
Source Control Software:
If you are looking for actual source control software, a list of applications can be found here: List of Source Control Systems with Visual Studio Plugins.
Don't make the mistake I did when putting the database definition under source code control by allowing branches and merges. This led developers (and managers) to assume they could have several versions of a table in the same database, leading to programmers overwriting each others changes, and having incompatible versions of source code and table definition all over the place.
As you might well imagine this lead to a great deal of wasted time and many frayed tempers.
Depending if time or money is more important, do you have the option to being someone in for a short time with the specific task of improving your processes? They would be able to set you up a test environment, install a version-control server and get your guys (and you) up to speed.
The first thing that you need to do is to give every developer their own copy of the database.
Secondly, you need each dev to be able re-create a baseline database from the master.
Once each dev is happy with their changes to the database, you will then need them to create a series of patches for their changes.
Lastly, each one of these patches needs to be able to checked in to source-control, and re-distributed to the other devs.
If you are looking for a good source control server, consider Subversion, with Tortoise SVN as your client.
Take a look at DBSourceTools. (http://dbsourcetools.codeplex.com).
It's specifically designed to help developers use source control for database changes.
This tool will allow you to baseline your database at a specific point, and create a named version (v1).
All files are scripted to disk, and very easy to add to a Subversion repository.
Have a look at the patching process for creating and using patches.
Have fun.
Recently I was tasked with writing up formal procedures for a team based development enviroment. We have several projects with multiple modules each. Right now there are only two programmers, however there are plans to expand to 4-6 programmers. Each programmer will be working on the same project and possibly pages which may cause over writing or error issues.
So far the ideal solution I have thought up is:
Local development (WAMP/VM or some virtual server instance on their own machine). Once a developer has finished their developments, they check it into the CVS Repository and merge it wih other fixes etc.
The CVS version is then deployed to the primary dev server for testing by the devs.
The MySQL DAtabases are kept on the primary dev server and users may remotely connect to it. Any Schema / Data alterations are run through a DB Admin who will notify all devs of any DB Changes (Which should be rare).
Does anyone see an issue with this or have a better solution?
Looks good. Just wanted to highlight this one very important point:
Make sure you have nightly builds in to the dev server. This will help catch problems at very early stages
By the way, while we are at it and in case you arent aware, Joel has a very good 12 point system for Quality of a Software Team
Excerpt
1 Do you use source control?
2 Can you make a build in one step?
3 Do you make daily builds?
4 Do you have a bug database?
5 Do you fix bugs before writing new code?
6 Do you have an up-to-date schedule?
7 Do you have a spec?
8 Do programmers have quiet working conditions?
9 Do you use the best tools money can buy?
10 Do you have testers?
11 Do new candidates write code during their interview?
12 Do you do hallway usability testing?
The model you describe is the one I've seen and used most often. I think each developer having their own local copy is more efficient and less risky. If all the code only exists on the dev server, an outage of the dev server stops all development. You also get less network traffic with a distributed model.
I think you should use a local version of your database not a live version. This will make testing easier and the developers wont have to worry that another developer edits data.
The idea of testing locally, merging changes and then deploying sounds good. The alternative (everyone developing on one server) sounds like a disaster waiting to happen.
You might want to try upgrading from CVS though. Subversion is similar to CVS but more modern. If you want to completely change your way of thinking, you could trying something like Git.