Related
I have been developing a network security application for several years now, as the lead developer at my company. It is a split-architecture design, where one component resides on the customer's network, and the other component in our own cloud. We have developed our own custom versioning system that keeps both sides synchronized at each patch (per customer), but until now it has only allowed incremental changes to be made, and rollbacks are not possible.
We'd like to move to a forkable git-like solution for our code, so that we can develop and test multiple features simultaneously, but the thing that's holding us back from that is our database. We use PostgreSQL (currently 9.3.12), and I've written a custom script to calculate the deltas between the "old" and "new" database structure, each time we "make a patch". It spits out a list of SQL commands necessary to update the "old" database structure to look like the "new", including tables, functions, sequences, triggers, you name it. It's very elegant and pretty much never fails anymore, even with complicated deltas.
However, I realize that in order to have a git-like solution for this (check-out, check-in, merge changes into test and production code, etc.) while also keeping database changes in sync with application code, we'll need to have something a lot more advanced than just "old" vs "new". Note that we don't need to modify database data for the most part, only table structure, which is altered in place on existing customer databases.
So my question is this: Any ideas for a git-like SQL version control system, which allows forking and merging, and can be easily kept in sync with application code changes? Our custom tool is already a bit more advanced than some open-source tools we've looked into (such as sqlt-diff), and tools like Red Gate are a bit out of our price range as a startup (not to mention that I haven't heard anybody mention forking in context with Red Gate). We're open to writing a custom tool, if that's what we need to do, but we're scratching our heads about where to start with something like that. We know how to calculate deltas, but we don't know how to manage all those things across different forks.
Free or open-source tools, frameworks we can adapt, or general guiding principles for building such tools are all appreciated!
One way of solving this problem is with migrations. A couple of lightweight tools, but there are many others:
http://sequel.jeremyevans.net/rdoc/files/doc/migration_rdoc.html
https://flywaydb.org/
Rather than calculating deltas between versions after the fact, migrations can be used to evolve the schema in a controlled way. You can create feature-specific migrations that can be tracked (and forked/merged) along with the rest of your code.
Depending on how fancy you want to get, you may need to extend the default naming/numbering schemes.
The problem
Ok, sorry that my question is somewhat abstract and subjective, but will try to make it as specific as possible. So, the situation I am in is simple - I am remaking a very old MS Access application on a new website using ASP.NET MVC. As currently the MVC site is using SQL Server 2008 (for many well known reasons) I need to find a way to migrate the tables AND the data, because the information in the old database will be used in the new application.
Alright, so far so good, however there are a few problems. The old application is written in a different language, meaning that I want to translate table names, field names, and all other names that are there to English. Furthermore, I will be making some changes on the models themselves (change the type of some fields, add additional fields to some tables, remove old unnecessary ones and more). So technically I'll be 'having my way' with everything.
Researched solutions
With those things in mind I researched for the ways to migrate data from Access database to a SQL Server. Of course, there is a lot of information on the matter, in Stack Overflow alone there are more than a few questions and solutions. So why am I struggling to find the answer ? Well I found a few solutions that will be sufficient to some extend (actually will definitely solve my problems) but I am writing to ask if someone experienced has a better perspective on it than I do. Alright, the solutions and why I am still looking for advice: /I'll be listing just a couple of the most common and popular ones that I found, many of the others share the same capabilities and/or results /
Upsize Wizzard (Access) - this is a tool devised specifically for migrating tables and data from Access. It is my most favourite one for the moment as I find it kind of straightforward to work with and it provides good overall results. I was able to migrate the tables to SQL Server (along with the data of course) which more or less is what I am intending to do. It is fast, it seems like it allows you to migrate indexes, primary keys and even to my knowledge foreign keys (table relationships). The downsides of this tool, however, include that it ignores your queries (which I don't really need honestly) and it doesn't provide a way to change the model, names or types of the properties of the table you migrate - which is the thing I kind of prefer, because I will have to make more than a few changes, adding, renaming, deleting, etc. And then continue with the development process (of the application) which will lead to a few additional minor changes. And finally I would need to apply all changes (migration + all changes) on the production server, which overall is prone to mistakes as I will be doing it by hand (and there are more than a few tables).
SQL Server Migration Assistant (SSMA) - ok, this is a separate tool (not included in Access) with again the same idea - to migrate data from Access to ... possibly everywhere, haven't researched that. Overall it offers more functionality and customizing from the Upsize Wizard, but of course it does it in a more complicated way. I haven't put enough effort to make a migration with this tool yet, as it involves a lot of installations and additional work, but according to my research it provides almost all (if not all) of the functionality I require. The downside however comes with the naming. As I mentioned it allows you to apply changes on the tables, schema, fields, indexes, keys and probably everything, but the articles advice that I change the names in Access first, as it will be easier and the migration process will run more smoothly. I am not allowed to make changes on the original Access database, as it will remain functional until the publish of the 'renewed' project, and the data inside it is being used, so a mere copy of the file is a solution I am not particularly fond of, because I might loose new records. Also I cant predict the changes I would want to make in the development process (as I said I believe I would want/need to apply some additional changes later on when I find 'weaknesses' in my data design in the development process) so I find it to be a little half baked solution.
Conclusion
The options presented, the way I see them, are two:
Use the Upsize Wizard to migrate the access tables, then write a script that applies the changes I want to make. Then in the development process add any additional changes to the script. When ready to publish on the production server, reapply the migration with the wizard, run the changes script and pray everything is fine.
Get more involved with the SSMA tool and try producing an updated version of the tables with the migration process. (See how efficient the renaming is and decide whether to use copied file to rename and then find a way to migrate only new records or do it all in the SSMA). Then again write a script for the changes that occur in the development process and re-do and apply it all on the production server when ready and then pray everything is fine.
Option I have not yet seen, apply it and then pray everything is fine.
I have researched the matter for a couple of days now, and found a few more solutions that I do not believe are better by the mentioned. However I include the possibility of missing the 'big red X on the map', a practical and easy solution which seems like it was designed specifically for me (though I doubt that a little). Anyway, reducing all the madness that I have written so far to a few simple questions will look like:
Is anyone aware if my conclusions are correct? I am leaning towards option one as it is easier to accomplish.
Has anyone experienced/found a better way to do that, or just found some 'logic-leaps' in my writings as I am overthinking the entire thing a little and may be doing some obvious miscalculation.
Very sorry for asking a trivial question and one that includes decision making that may involve deeper understanding of my project and situation, yet I am working with rather sensitive data and would appreciate feedback, even if only to improve my confidence into the chosen approach.
There is one other tool/method you might want to consider that seems to cater to your specific needs more. This would be to use the data import/export tool that ships with sqlserver to do a complete copy of all data into a temporary location within sql server and then write custom queries to reorganize the names and other changes you want to make. Is a bit more work but you could use the end product as a seed method for your migrations ;) (if you are doing code first anyway)
I'm in the research phase trying to adopt 2012 Database Projects on an existing small project. I'm a C# developer, not a DBA, so I'm not particularly fluent with best practices. I've been searching google and stackoverflow for a few hours now but I still don't know how to handle some key deployment scenarios properly.
1) Over the course of several development cycles, how do I manage multiple versions of my database? If I have a client on v3 of my database and I want to upgrade them to v8, how do I manage this? We currently manage hand-crafted schema and data migration scripts for every version of our product. Do we still need to do this separately or is there something in the new paradigm that supports or replaces this?
2) If the schema changes in such a way that requires data to be moved around, what is the best way to handle this? I assume some work goes in the Pre-Deployment script to preserve the data and then the Post-Deploy script puts it back in the right place. Is that the way of it or is there something better?
3) Any other advice or guidance on how best to work with these new technologies is also greately appreciated!
UPDATE: My understanding of the problem has grown a little since I originally asked this question and while I came up with a workable solution, it wasn't quite the solution I was hoping for. Here's a rewording of my problem:
The problem I'm having is purely data related. If I have a client on version 1 of my application and I want to upgrade them to version 5 of my application, I would have no problems doing so if their database had no data. I'd simply let SSDT intelligently compare schemas and migrate the database in one shot. Unfortunately clients have data so it's not that simple. Schema changes from version 1 of my application to version 2 to version 3 (etc) all impact data. My current strategy for managing data requires I maintain a script for each version upgrade (1 to 2, 2 to 3, etc). This prevents me from going straight from version 1 of my application to version 5 because I have no data migration script to go straight there. The prospect creating custom upgrade scripts for every client or managing upgrade scripts to go from every version to every greater version is exponentially unmanageable. What I was hoping was that there was some sort of strategy SSDT enables that makes managing the data side of things easier, maybe even as easy as the schema side of things. My recent experience with SSDT has not given me any hope of such a strategy existing but I would love to find out differently.
I've been working on this myself, and I can tell you it's not easy.
First, to address the reply by JT - you cannot dismiss "versions", even with declarative updating mechanics that SSDT has. SSDT does a "pretty decent" job (provided you know all the switches and gotchas) of moving any source schema to any target schema, and it's true that this doesn't require verioning per se, but it has no idea how to manage "data motion" (at least not that i can see!). So, just like DBProj, you left to your own devices in Pre/Post scripts. Because the data motion scripts depend on a known start and end schema state, you cannot avoid versioning the DB. The "data motion" scripts, therefore, must be applied to a versioned snapshot of the schema, which means you cannot arbitrarily update a DB from v1 to v8 and expect the data motion scripts v2 to v8 to work (presumably, you wouldn't need a v1 data motion script).
Sadly, I can't see any mechanism in SSDT publishing that allows me to handle this scenario in an integrated way. That means you'll have to add your own scafolding.
The first trick is to track versions within the database (and SSDT project). I started using a trick in DBProj, and brought it over to SSDT, and after doing some research, it turns out that others are using this too. You can apply a DB Extended Property to the database itself (call it "BuildVersion" or "AppVersion" or something like that), and apply the version value to it. You can then capture this extended property in the SSDT project itself, and SSDT will add it as a script (you can then check the publish option that includes extended properties). I then use SQLCMD variables to identify the source and target versions being applied in the current pass. Once you identify the delta of versions between the source (project snapshot) and target (target db about to be updated), you can find all the snapshots that need to be applied. Sadly, this is tricky to do from inside the SSDT deployment, and you'll probably have to move it to the build or deployment pipeline (we use TFS automated deployments and have custom actions to do this).
The next hurdle is to keep snapshots of the schema with their associated data motion scripts. In this case, it helps to make the scripts as idempotent as possible (meaning, you can rerun the scripts without any ill side-effects). It helps to split scripts that can safely be rerun from scripts that must be executed one time only. We're doing the same thing with static reference data (dictionary or lookup tables) - in other words, we have a library of MERGE scripts (one per table) that keep the reference data in sync, and these scripts are included in the post-deployment scripts (via the SQLCMD :r command). The important thing to note here is that you must execute them in the correct order in case any of these reference tables have FK references to each other. We include them in the main post-deploy script in order, and it helps that we created a tool that generates these scripts for us - it also resolves dependency order. We run this generation tool at the close of a "version" to capture the current state of the static reference data. All your other data motion scripts are basically going to be special-case and most likely will be single-use only. In that case, you can do one of two things: you can use an IF statement against the db build/app version, or you can wipe out the 1 time scripts after creating each snapshot package.
It helps to remember that SSDT will disable FK check constraints and only re-enable them after the post-deployment scripts run. This gives you a chance to populate new non-null fields, for example (by the way, you have to enable the option to generate temporary "smart" defaults for non-null columns to make this work). However, FK check constraints are only disabled for tables that SSDT is recreating because of a schema change. For other cases, you are responsible for ensuring that data motion scripts run in the proper order to avoid check constraints complaints (or you manually have disable/re-enable them in your scripts).
DACPAC can help you because DACPAC is essentially a snapshot. It will contain several XML files describing the schema (similar to the build output of the project), but frozen in time at the moment you create it. You can then use SQLPACKAGE.EXE or the deploy provider to publish that package snapshot. I haven't quite figured out how to use the DACPAC versioning, because it's more tied to "registered" data apps, so we're stuck with our own versioning scheme, but we do put our own version info into the DACPAC filename.
I wish I had a more conclusive and exhasutive example to provide, but we're still working out the issues here too.
One thing that really sucks about SSDT is that unlike DBProj, it's currently not extensible. Although it does a much better job than DBProj at a lot of different things, you can't override its default behavior unless you can find some method inside of pre/post scripts of getting around a problem. One of the issues we're trying to resolve right now is that the default method of recreating a table for updates (CCDR) really stinks when you have tens of millions of records.
-UPDATE: I haven't seen this post in some time, but apparently it's been active lately, so I thought I'd add a couple of important notes: if you are using VS2012, the June 2013 release of SSDT now has a Data Comparison tool built-in, and also provides extensibility points - that is to say, you can now include Build Contributors and Deployment Plan Modifiers for the project.
I haven't really found any more useful information on the subject but I've spent some time getting to know the tools, tinkering and playing, and I think I've come up with some acceptable answers to my question. These aren't necessarily the best answers. I still don't know if there are other mechanisms or best practices to better support these scenarios, but here's what I've come up with:
The Pre- and Post-Deploy scripts for a given version of the database are only used migrate data from the previous version. At the start of every development cycle, the scripts are cleaned out and as development proceeds they get fleshed out with whatever sql is needed to safely migrate data from the previous version to the new one. The one exception here is static data in the database. This data is known at design time and maintains a permanent presence in the Post-Deploy scripts in the form of T-SQL MERGE statements. This helps make it possible to deploy any version of the database to a new environment with just the latest publish script. At the end of every development cycle, a publish script is generated from the previous version to the new one. This script will include generated sql to migrate the schema and the hand crafted deploy scripts. Yes, I know the Publish tool can be used directly against a database but that's not a good option for our clients. I am also aware of dacpac files but I'm not really sure how to use them. The generated publish script seems to be the best option I know for production upgrades.
So to answer my scenarios:
1) To upgrade a database from v3 to v8, I would have to execute the generated publish script for v4, then for v5, then for v6, etc. This is very similar to how we do it now. It's well understood and Database Projects seem to make creating/maintaining these scripts much easier.
2) When the schema changes from underneath data, the Pre- and Post-Deploy scripts are used to migrate the data to where it needs to go for the new version. Affected data is essentially backed-up in the Pre-Deploy script and put back into place in the Post-Deploy script.
3) I'm still looking for advice on how best to work with these tools in these scenarios and others. If I got anything wrong here, or if there are any other gotchas I should be aware of, please let me know! Thanks!
In my experience of using SSDT the notion of version numbers (i.e. v1, v2...vX etc...) for databases kinda goes away. This is because SSDT offers a development paradigm known as declarative database development which loosely means that you tell SSDT what state you want your schema to be in and then let SSDT take responsibility for getting it into that state by comparing against what you already have. In this paradigm the notion of deploying v4 then v5 etc.... goes away.
Your pre and post deployment scripts, as you correctly state, exist for the purposes of managing data.
Hope that helps.
JT
I just wanted to say that this thread so far has been excellent.
I have been wrestling with the exact same concerns and am attempting to tackle this problem in our organization, on a fairly large legacy application. We've begun the process of moving toward SSDT (on a TFS branch) but are at the point where we really need to understand the deployment process, and managing custom migrations, and reference/lookup data, along the way.
To complicate things further, our application is one code-base but can be customized per 'customer', so we have about 190 databases we are dealing with, for this one project, not just 3 or so as is probably normal. We do deployments all the time and even setup new customers fairly often. We rely heavily on PowerShell now with old-school incremental release scripts (and associated scripts to create a new customer at that version). I plan to contribute once we figure this all out but please share whatever else you've learned. I do believe we will end up maintaining custom release scripts per version, but we'll see. The idea about maintaining each script within the project, and including a From and To SqlCmd variable is very interesting. If we did that, we would probably prune along the way, physically deleting the really old upgrade scripts once everybody was past that version.
BTW - Side note - On the topic of minimizing waste, we also just spent a bunch of time figuring out how to automate the enforcement of proper naming/data type conventions for columns, as well as automatic generation for all primary and foreign keys, based on naming conventions, as well as index and check constraints etc. The hardest part was dealing with the 'deviants' that didn't follow the rules. Maybe I'll share that too one day if anyone is interested, but for now, I need to pursue this deployment, migration, and reference data story heavily. Thanks again. It's like you guys were speaking exactly what was in my head and looking for this morning.
We have been looking into possible solutions for our SQL Source control. I just came across Red Gates SQL Source control and wondered if anyone has implemented it? I am going to download the trial and give it a shot, but just wanted to see if others have real experience.
As always greatly appreciate the input
--S
I have updated my original post below to reflect changes in the latest versions of SQL Source Control (3.0) and SQL Compare (10.1).
Since this question was asked over a year ago, my response may not be that helpful to you, but for others who may currently be evaluating SSC, I thought I would throw in my two cents. We just started using SQL Source Control (SSC) and overall I am fairly satisfied with it so far. It does have some quirks though, especially if you are working in a shared database environment (as opposed to every developer working locally) and particularly working in a legacy environment where objects in the same database are divided haphazardly between development teams.
To give a brief overview of how we are using the product in our organization, we are working in a shared environment where we all make changes to the same development database, so we attached the shared database to the source control repository. Each developer is responsible for making changes to the objects in the database through SQL Server Management Studio (SSMS), and when they are finished, they can commit their changes to source control. When we are ready to deploy to staging, the build master (me) merges the development branch of the database code to the main (staging) branch and then runs SQL Compare using the main branch repository version of the database as the source and the live staging database as the target, and SQL Compare generates the necessary scripts to deploy the changes made to the staging environment. Staging to production deployments works in similar fashion. One other important point to note is that, given the fact that we are sharing the same database with other development teams, we use a built in feature of SSC that allows you to create filters on database objects by name, type, etc. We manually set up filters on our specific team's objects, excluding all other objects, so that we don't accidentally commit other development team's changes when we do our deployments.
So in general it's a fairly simple product to set up and use and it's really nice because you're always working with live objects in SSMS, as opposed to disconnected script files stored in a separate source repository that run the risk of getting out of sync. It's also nice because SQL Compare generates the deployment scripts for you so you don't have to worry about introducing errors as you would if you were creating the scripts on your own. And as SQL Compare is a very mature and stable product, you can feel pretty confident that it's going to create the proper scripts for you.
With that being said, however, here are some of the quirks that I have run into so far:
SSC is pretty chatty out of the box in terms of communicating with the db server in order to keep track of database items that are out of sync with the source control repository. It polls every few milliseconds and if you add in multiple developers all working against the same database using SSC, you can imagine that our dba's weren't very happy. Fortunately, you can easily reduce your polling frequency to something more acceptable, although at the cost of sacrificing responsive visual notifications of when objects have been changed.
Using the object filtering feature, you can't easily tell from looking at objects in SSMS which objects are included in your filter. So you don’t know for sure if an object is under source control, unlike in Visual Studio, where icons are used to indicate source controlled objects.
The object filtering GUI is very clunky. Due to the fact that we are working in a legacy database environment, there is currently not a clear separation between the objects that our team owns and those owned by other teams, so in order to prevent us from accidentally committing/deploying other teams’ changes, we have set up a filtering scheme to explicitly include each specific object that we own. As you can imagine, this becomes quite cumbersome, and as the GUI to edit the filters is set up to enter one object at a time, it could become quite painful, especially trying to set up your environment for the first time (I ended up writing an application to do this). Going forward, we are creating a new schema for our application to better facilitate object filtering (besides being a better practice anyway).
Using the shared database model, developers are allowed to commit any pending changes to a source controlled database, even if the changes are not theirs. SSC does give you a warning if you try to check in a bunch of changes that these changes might not be yours, but other than that you’re on your own. I actually find this to be one of SSC’s most dangerous “quirks”.
SQL Compare can’t currently share the object filters created by SSC, so you would have to manually create a matching filter in SQL Compare, so there is a danger that these could get out of sync. I just ended up cut-and-pasting the filters from the underlying SSC filter file into the SQL Compare project filter to avoid dealing with the clunky object filtering GUI. I believe that the next version of SQL Compare will allow it to share filters with SSC, so at least this problem is only a short term one. (NOTE: This issue has been resolved in the latest version of SQL Compare. SQL Compare can now use the object filters created by SSC.)
SQL Compare also can’t compare against a SSC database repository when launched directly. It has to be launched from within SSMS. I believe that the next version of SQL Compare will provide this functionality, so again it’s another short term problem. (NOTE: This issue has been resolved in the latest version of SQL Compare.)
Sometimes SQL Compare isn’t able to create the proper scripts to get the target database from one state to another, usually in the case where you are updating the schema of existing tables that aren’t empty, so you currently have to write manual scripts and manage the process yourself. Fortunately, this will be addressed through “migration scripts” in the next release of SSC, and from looking at the early release version of the product, it appears that the implementation of this new feature was well thought out and designed. (NOTE: Migration scripts functionality has been officially released. However, it does not currently support branching. If you want to use migration scripts, you will need to run sql compare against your original development code branch... the one where you checked in your changes... which is pretty clunky and has forced me to modify my build process in a less than ideal way in order to work around this limitation. Hopefully this will be addressed in a future release.)
Overall, I am pretty happy with the product and with Redgate’s responsiveness to user feedback and the direction that the product is taking. The product is very easy to use and well designed, and I feel that in the next release or two the product will probably give us most, if not all, of what we need.
I use SQL Compare for generating scripts when going from dev -> test -> production and it saves me tons of time.
For source control though, we use SVN and ScriptDB (http://scriptdb.codeplex.com/) though. I mainly use source control of SQL scripts for keeping track of changes. I think that rolling back a version of the database seldomly (if ever) works since data may have changed when making structure changes.
This works fine for a few of our current projects (largest is 200 tables and 2000 sprocs). The main reason for doing this though is cost since not all team members have to buy SQL Compare (I avoid adding dependencies to commercial projects unless really needed).
We performed an extensive evaluation of Red Gate's product and found a few major flaws. If you want to look at who changed an object, you can't do it without SysAdmin privileges. The product needs to look at the trace on your server, which requires those rights. I'm on a 5+ person team, and not knowing who had pending changes is what will stop us from using the product.
I just started working for a new company and they use Redgate SQL Source Control for all their projects, amonst them a large and complex one. It does the job well in tandem with TFS. The only drawback from my point of view is that the SQL Server Management Studio integration is highly unstable. Frequent crashes of SQL Server Management Studio happen when the tools are installed.
At my company, our current method of updating the database is to connect using the Server Explorer in VS2005, then modify the stored procedures by opening them and editing. The devs here seem to enjoy that "write and save it like it's code" mentality. It is pretty convenient, how it automatically turns Create into Alter and runs the scripts against the existing database when we need to tweak something.
Recently, this bit us pretty hard during a server crash when we lost a lot of changes that hadn't gotten backed up. I'm pushing to move our SQL development where it belongs: in DB Projects so we can put them into SVN along with the othe code. The alternative is nightly back-ups of the database.
I don't know much about DB projects though, or how the workflow with them is. I'm afraid that if I can't get something of similar utility to their current model, they just won't switch. Any thoughts on maintaining our current working model, but switching over to DB Projects?
If the developers make the rules (and your post sounds like they do), you can only proceed if the new workflow is "better" to them. Being a developer myself, I think that's the way it should be. I've seen some non-developers think up pretty nonsensical development processes, and force them on the developers to everyone's detriment.
If you're thinking about VS DB projects, you'd first test if VS DB actually works with your database. If it does, you'd have to set up a big chance in process: the "true" copy of the database is now in VS DB instead of the database server.
Another way out is to backup the development server regularly. If you back it up daily, and a transaction log backup every hour, it becomes very hard to loose a significant amount of work.
Or create a scheduled job that writes the entire database definition to a text file. (Script all objects in database.) These files are usually very small, so you can keep a long backlog.
Many respected bloggers seem to think storing database definitions in SVN is a good idea. See this coding horror post, or related Stack Overflow related question How do I version my MS SQL database in SVN.
Talk it over with the developers and see what you can agree on.