This may be a pipe dream, but I'm hoping someone knows of a tool which can be configured to compare all or some (keys) of the data in two identical database and merge, perhaps based on relationships.
Specifically looking for one for SQL Server.
I'm not really asking for the best one, but if it exists it would be nice to hear how it is used.
Any other ideas for how to manage the work done or data added in dev and push it out to production without copying the entire database are welcome.
Thanks!
We use this and personally think it's excellent.
http://www.red-gate.com/products/sql-development/sql-data-compare/
There is also another product for the schema side.
http://www.red-gate.com/products/sql-development/sql-compare/
I don't know of a specific tools but you can implement in your process of publications the analysis and executions of delta files, containing the diffs from one verision to another. Magento, Wordpress are using something like this for example. They have something like this
//sql_update_001_002.sql
UPDATE some_table...
DELETE some entries
CREATE a_new_table...
// compere some keys or do other logic.
//etc
Then they have a script that analyses the current version and if needed it executes the corresponding sql.
Navicat allows to make data and structure synchronization between 2 databases (also located on different servers).
In terms of tools I agree with Chris - Redgate's toolset for both schema and data comparisons
If you are also thinking about your overall db dev process - then I have written a blog on the topic which might be of interest.
It also has some links to how others have tackled this subject.
http://michaelbaylon.wordpress.com/category/data-management/database-development/sql-script-management/
Related
I am working using Tableau and have to write down multiple different SQL each time, while making new data sources.
I have to save all changes on SQL for every data source.
Currently I would paste the SQL on notepad and save them on separate folder in my computer, along with description of the changes.
Is there any better way to do this?
Assuming you have permission to create objects in the database, begin by creating database views, As #Nick.McDermaid commented.
Then, instead of using Custom SQL data source in Tableau, just connect to the View as if it were a table.
If you need to track the changes to these SQL views of your data, you will need to learn how to use source control for the .sql files that can be scripted from within SQL Server Management Studio:
Your company or school may have a preferred source control system already in use, in which case you should use that. If they don't, or if you are learning at home, then Git and Subversion are popular open source choices.
There are many courses available on learning platforms like Coursera that will teach you how to learn how to use those systems.
I had similar problem as you.
We ended up writing the queries in SQL Editor SQL Work bench (https://www.sql-workbench.eu/), then managed the code history and performed code peer-review (logic, error check, etc) in team shared space (like confluence).
The reasons we did that is
1) SQL queries are much easy to write on Work Bench
2) Code review is a must! You will find through implementing a review process more mistakes than you could ever think about
3) The shared space is just really convenient as it is accessible by everyone, and all errors are documented. After sometimes you get a lot of visible knowledge accumulated.
I also totally agree with Nick as this is one step to a reporting solution. But developing a whole reporting server is heavy, costly and takes time. Unless management are really convinced of the importance of developing a reporting solution, you may have to get a workaround with queries and Tableau (at least that was the case for us)
A little late to the party, but I would suggest you simply version the tableau workbook. The contents of the workbook are XML, so perfect for versioning using file based tools (Dropbox, One Drive, etc.) or source control (git, etc.). The workbooks themselves are usually quite small, so just make sure to keep the extract data separate if you use it.
Currently, we have a several environments, but mainly we have a live environment and a test environment as most do.
It seems our test environment we are always having issues with having good data in our test environment. Largely because our test sql environment is maintained by a bunch of loose scripts that are not maintained very well. Obviously, one course of action is to maintain these more efficiently, that is not going to happen we have found.
So, I am looking for a tool that would help with this. Has anyone found or used such a tool?
Any advice or direction would be greatly appreciated. This is really a thorn in our developer's sides.
*Update*Our main issue is that test structures often differ from live and we need an automated solution that handles this, i.e. when a table has more columns in test than in live. Updating question to reflect this thank you for your answer.
As far as Transfer of Data is concerned use SSMS import and export data wizard.If source and destination tables names and schema is same Data transfer is a breeze.I use it quite often never faced any problem (use identity insert option if there is any identity col).
and Script generation feature of SSMS is not that bad either. Normally i have to edit 'Create Database' and 'create Users' parts of the script manually.
servers all sql server 2008, and win xp
i have the following task
create a huge database, DONE
distribute it to the 20 waiting servers!!
if there were two or three i would have taken the trouble of creating the db's on all of using sql server managemnt stdio
but i am guessing that there is an efficient way
please note,
only the copy of the database structure, the schema is needed not the values within the cells!
thank you
Or of course you should have been creating the scripts as you went along and putting them in Source control. Then you would have exactly which scripts you needed for this version of the software and be used to doing the same thing for later modifications. You would also script the data inserts for any lookup tables you need to build.
Not having that, you can script the entire database. or use a SQL compare tool. But I strongly urge you to start treating database code like all other code and scripting, storing it in source control and versioning it. Life is so much better when you do that.
What Gabriel McAdams has shown, or, Redgate SQL Compare does this very nicely also.
If you can spare the moolah, using a tool like Red Gate's SQL Packager is an option i have used in the past and it works well!
The tool can do a lot more as well and may not be worth the spend though if you do not need the other features!
In that case, Gabriels'option above is definitely the easiest one to go with!
What would be the best approach for versioning my whole database ?
Creating a file for each database object (table,view,procedsure..) or rather having one file for all DDL scripts and any new change will be put in a separate file ?
What about handling changes made in a Database manager tool ?
I'd like to have a generic solutions for any kind of RDBMS.
Are there any other options ?
I'm a huge VCS fan in general and a big Mercurial booster, but I really think you're going down the wrong path.
VCSs aren't just about iterative changes, the "what", they're also about answering the "who", "when", and "why". For a database those answers are a lot less interesting or hard to provide to the VCS. If you're doing nightly exports and commits the "who" will always be "cron" and the "why" will always be "midnight".
The other thing modern VCSs do really well is helping you merge changes from multiple branches. That's less applicable in the database world. Very seldom do you say "I want this table structure, but this data", and if you do the text/diff merge isn't going to help you much.
The thing that does do "what" and "when" very well is an incremental backup system, and that's probably the better fit.
At work we use Tivoli and at home I use rdiff-backup and duplicity, but there are plenty of great options.
I guess my general rule of thumb is "if it was typed by hand by a human then it does into source control, and if it was generated/exported then it goes in the incremental backups"
Certainly you can make this work, but I don't think it will buy you much over the more traditional backup solutions.
Have a look at this post
If you need generic solution - put everything in the scripts (simple text files) and put under Version Control system (can be used any of VCS).
Grouping similar database objects into scripts will be depend on your requirement.
So you may for example:
Store table/indexes/ in one or several script
Each procedure store in individual script or combine small procedures into one script.
However need to remember one important thing with this approach: don't forget change scripts if you changed table/view/procedure directly in databases and don't create/recreate/compile you db objects in database after changing scripts.
SQL Source Control currently supports SVN and TFS, but Mercurial requests are increasing rapidly and we're hoping to have a story for this very soon.
We use UserVoice to measure demand so please vote accordingly if you're interesting in this: http://redgate.uservoice.com/forums/39019-sql-source-control
The problem: we have one application that has a portion which is used by a very small subset of the total users, and that part of the application is running off of a separate database as well. In a perfect world, the schemas of the two databases would be synced up, but such is not the case. Some migrations have been run on the smaller database, most haven't; and furthermore, there is nothing such as revision number to be able to easily identify which have and which haven't. We would like to solve this quandary for future projects. During a discussion we've come up with the following possible plan of action, and I am wondering if anyone knows of any project which has already solved this problem:
What we would like to do is create an empty database from the schema of the large fully-migrated database, and then move all of the data from the smaller non-migrated database into that empty one. If it makes things easier, it can probably be assumed for the sake of this problem specifically that no migrations have ever removed anything, only added.
Else, if there are other known solutions, I'd like to hear them as well.
You could use a schema comparison tool like Red-Gate's SQL Compare. You can synchronize the changes and not lose any data. I wrote about this and many alternative tools ranging widely in price here:
http://bertrandaaron.wordpress.com/2012/04/20/re-blog-the-cost-of-reinventing-the-wheel/
The nice thing is that most tools have trial versions. So, you can try them our for 14 days (fully functional) and only buy it if it meets your expectations. I can't speak for the other tools, but I've been using RG for years and it is a very capable and reliable tool.
(Updated 2012-06-23 to help prevent link-rot.)
Red-Gate's SQL Compare as Aaron Bertrand mentions in his answer is a very good option. However, if you are not permitted to purchase something, an option is to try something like:
1) For each database, script out all the tables, constraints, indexes, views, procedures, etc.
2) run a DIFF, and go through all the differences and make sure that the small DB can accept them. If not implement any changes (including data) necessary onto the small DB so it can accept the changes.
3) create a new empty database from the schema of the large DB
4) import the data from the small DB into the nee DB.
You could also reverse engineer your database into Visual Studio as a database project. Visual Studio Team Suite Database Edition GDR R2 (I know long name) has the capability to do a schema comparison and data comparison, but the beauty of this approach is that you get all of your database into a nice database project where you can manage change and integrate with source control. This would allow you to build from a common source and deploy consistent changes.