I started at a company as a junior sql developer on a datawarehouse. Ever since I have been going through the code and learning the dimensional models etc. I struggle to see security measures outside of rights that the developer has on the environment.
but if someone would to write code that influences the data in the warehouse in a significant way, update to the wrong values, insert false data, delete records that should be there and hits that code with a commit statement, wouldn't there be a massive impact on the business intelligence aspect of the warehouse? Like if they were to pull data to create statistics and there is bad data, then they will have bad statistics.
We have about 7 billlion records and changes made in this way would be really hard to pick up if it can be seen at all.
Maybe this is a simple question, but I can't really find an answer, since in the datawarehouse you don't have the rigorous relational constraints to check data validity, especially when you move around big data and the database administrators drop the triggers and indexes as well. The transactional side we get the source data from also doesn't keep history (that's our job).
Any views and suggestions on this subject will be highly appreciated, thank you.
When working with databases or writing code in general, mistakes happen. That is why you ALWAYS separate your development environment from your production environment. Most of us also have an intermediate test environment, where new code is tested and data is validated, before the code is deployed to production.
Furthermore, before any deployment, a full backup is taken. That way, if an error is discovered after deployment, a restore of the backup can be made.
Preferably, your development and production environments run on separate, but identical, servers. If that is not possible, at least keep the data in separate databases, and use the security of your database server, to ensure that no one can make changes to the production database, unless a deployment is happening.
Now for the deployment itself, make sure you have a sort of checklist to go over, every time you make a deployment. First step on the checklist should be to backup the existing production environment. Write scripts to automate parts of the deployment, whenever possible. Use tools such as SQL Schema Compare, to identify differences between the development and production database, etc. Ideally, deployment should be a matter of pressing one button, and then everything deploys automagically, and you can go back to developing without worrying.
Related
I'm working with PostgreSQL now for a few months. Now before going live we usually used the live database for almost everything (creating new columns in the live database tables, executing update and insert queries etc.). But now we want to go live and we have to do things differently before we do that. The best way is to have a test database and live database.
Now I created a copy of the live database so we have a test database to run tests on. The problem is that the data is old after 24 hours, so we actually need to create a fresh copy every 24 hours, which is not really smart to do manually.
So my question is, are there people over here who know a proper way to handle this issue?
I think the most ideal way is:
- copy a selection of tables from the live database to the test database (skip tables like users).
- make it possible to add columns, rename them or even delete them and when we deploy a new version of the website, transfer those changes from the test database to the live database (net necassary but would be a good feature).
If your database structure is changing, you do NOT want it automatic. You will blow away dev work and data. You want it manual.
I once managed a team that had a situation similar: multi-TiB database, updated daily, and needing to do testing and development against that up-to-date data. Here was the way we solved it:
In our database, we defined a function called TODAY(). In our live system, this was a wrapper for NOW(). In our test system, it called out to a one-column table whose only row was a date that we could set. This means that our test system was a time machine, that could pretend any date was the current one.
This meant that every function or procedure we wrote had to be time-aware. Should I care about future-scheduled events? How far in the future? This made our functions extremely robust, and made it dead simple to test them against a huge variety of historical data. This helped catch a large number of bugs that we would have never thought would happen, but we saw would indeed occur in our historical data. It's like functional programming for your database!
We would still schedule database updates from a live backup, about every month or so. This had the benefit of more data AND testing our backup/restore procedure. Our DBA would run a "post-test-sync" script that would set permissions for developers, so we were damn sure than anything we ran on the test system would work on the live one as well. This is what helped us build our deployment database scripts.
I have two databases on separate servers (dev and production) I need to move my data from dev to production from multiple tables without affecting the pre-existing data on production. Any idea if SQL Manager support something like this or am I going to have to write a script for it?
My situation in detail:
I have a tool which allow me to create surveys for my company. The tool is located on dev and also on production. Since I don't want to add test data in my production db I am using the dev version of the tool to create my surveys and test them locally. The tool is tied to few tables in my db such as surveys, questions, anwers, results, etc.
My current setup: When I am done with a survey and it is ready to launch, I have to use the production version of the tool to manually redo all of the previous work that i did on production. This is not ideal at all not only because of the time that I have to spend doing it but also risking making mistakes during the manual copying.
What I need to do:
Those tables that I mentioned above, already have production data in them and they are available for my company to use. When I create a new survey I need to transfer only the specific records of the new survey (from all tables) from dev to production without affecting anything that I had there from before.
Use Import and Export Data
Or Add the DEV server as a linked server in your PROD server and then use INSERT/SELECT statements
You can use a database compare tool, for SQL Server I use SQL Delta, wich allows you to automatically create a script to run in the database you wish, http://www.sqldelta.com/
You're not going to find any out-of-the-box solutions for this, but there are tools that can help once you've got a clear idea of what you're trying to accomplish -- in detail. A little time spent at this point to make sure you're really clear on what you expect to have happen will pay huge dividends when you move to production.
The scenario you're describing sounds like you've got some configuration-type data in your database alongside your transactional, or domain data. In other words, you've got changes that need to be promoted from your development environment to production in order for your application to work properly. This isn't unusual, but you've got to be pretty deliberate and very careful when you set up a promotion plan for a scenario like this -- after all, you don't want to push test data to your production system along with your configuration changes. It's critical, therefore, to identify the tables you're going to push from dev to prod and make sure those are the only tables you're pushing in that direction.
You also mentioned something about "without affecting the pre-existing data on production". Can you tell us more about this (maybe an example)? Typically, you'd want to keep specific tables (by convention) set up to move changes in one direction only -- ie, from dev to prod. If you've got tables that need to contain merged changes, you're going to have to apply even more attention to getting this right, because you need to deal with merge errors -- what happens when you've got data to push and it's already present in the target database, for instance?
Once you've got a plan for what you actually want to move, some of the tools mentioned in other answers would probably work, or check out Redgate's tools (like SQL Data Compare) -- they make some really nice products to help with DB management tasks.
---- addendum ----
Based on edits to the question, here are a couple of additional thoughts:
(1) Allow your production surveys to have a "disabled" or "testing" mode, so you don't have to make your data changes in another environment. This allows you to be able to move stuff from dev to production only when actual development changes exist.
(2) Define a "package" mechanism to move a survey from one environment to another. This would allow you to deal with merge conflicts, ID changes, etc., generically and reliably. As a bonus, this would allow you to also move a production survey back to dev for debugging and testing purposes.
The Background
My group has 4 SQL Server Databases:
Production
UAT
Test
Dev
I work in the Dev environment. When the time comes to promote the objects I've been working on (tables, views, functions, stored procs) I make a request of my manager, who promotes to Test. After testing, she submits a request to an Admin who promotes to UAT. After successful user testing, the same Admin promotes to Production.
The Problem
The entire process is awkward for a few reasons.
Each person must manually track their changes. If I update, add, remove any objects I need to track them so that my promotion request contains everything I've done. In theory, if I miss something testing or UAT should catch it, but this isn't certain and it's a waste of the tester's time, anyway.
Lots of changes I make are iterative and done in a GUI, which means there's no record of what changes I made, only the end result (at least as far as I know).
We're in the fairly early stages of building out a data mart, so the majority of the changes made, at least count-wise, are minor things: changing the data type for a column, altering the names of tables as we crystallize what they'll be used for, tweaking functions and stored procs, etc.
The Question
People have been doing this kind of work for decades, so I imagine there have got to be a much better way to manage the process. What I would love is if I could run a diff between two databases to see how the structure was different, use that diff to generate a change script, use that change script as my promotion request. Is this possible? If not, are there any other ways to organize this process?
For the record, we're a 100% Microsoft shop, just now updating everything to SQL Server 2008, so any tools available in that package would be fair game.
I should clarify I'm not necessarily looking for diff tools. If that's the best way to sync our environments then it's fine, but if there's a better way I'm looking for that.
An example doing what I want really well are migrations in Ruby on Rails. Dead simple syntax, all changes are well documented automatically and by default, determining what migrations need to run is almost trivially easy. I'd love if there was something similar to this for SQL Server.
My ideal solution is 1) easy and 2) hard to mess up. Rails Migrations are both; everything I've done so far on SQL Server is neither.
Within our team, we handle database changes like this:
We (re-)generate a script which creates the complete database and check it into version control together with the other changes. We have 4 files: tables, user defined functions and views, stored procedures, and permissions. This is completely automated - only a double-click is needed to generate the script.
If a developer has to make changes to the database, she does so on her local db.
For every change, we create update scripts. Those are easy to create: The developer regenerates the db script of his local db. All the changes are now easy to identify thanks to version control. Most changes (new tables, new views etc) can simply be copied to the update script, other changes (adding columns for example) need to be created manually.
The update script is tested either on our common dev database, or by rolling back the local db to the last backup - which was created before starting to change the database. If it passes, it's time to commit the changes.
The update scripts follow a naming convention so everybody knows in which order to execute them.
This works fairly well for us, but still needs some coordination if several developers modify heavily the same tables and views. This doesn't happen often though.
The important points are:
database structure is only modified by scripts, except for the local developer's db. This is important.
SQL scripts are versioned by source control - the db can be created as it was at any point in the past
database backups are created regularly - at least before making changes to the db
changes to the db can be done quickly - because the scripts for those changes are created relatively easily.
However, if you have a lot of long lasting development branches for your projects, this may not work well.
It is by far not a perfect solution, and some special precautions are to be taken. For example, if there are updates which may fail depending on the data present in a database, the update should be tested on a copy of the production database.
In contrast to rails migrations, we do not create scripts to reverse the changes of an update. But this isn't always possible anyway, at least in respect to the data (the content of a dropped column is lost even if you recreate the column).
Version Control and your Database
The root of all things evil is making changes in the UI. SSMS is a DBA tool, not a developer one. Developers must use scripts to do any sort of changes to the database model/schema. Versioning your metadata and having upgrade script from every version N to version N+1 is the only way that is proven to work reliably. It is the solution SQL Server itself deploys to keep track of metadata changes (resource db changes).
Comparison tools like SQL Compare or vsdbcmd and .dbschema files from VS Database projects are just last resorts for shops that fail to do a proper versioned approach. They work in simple scenarios, but I see them all fail spectacularly in serious deployments. One just does not trust a tool to do a change to +5TB table if the tools tries to copy the data...
RedGate sells SQL Compare, an excellent tool to generate change scripts.
Visual Studio also has editions which support database compares. This was formerly called Database Edition.
Where I work, we abolished the Dev/Test/UAT/Prod separation long ago in favor of a very quick release cycle. If we put something broken in production, we will fix it quickly. Our customers are certainly happier, but in the risk avert corporate enterprise, it can be a hard sell.
There are several tools available for you. One is from Red-Gate called SQL Compare. Awesome and highly recommended. SQL Compare will let you do a diff in schemas between two databases and even build the sql change scripts for you.
Note they have been working on a SQL Server source control product for awhile now as well.
Another (if you're a visual studio shop) is the schema and data compare features that is part of Visual Studio (not sure which versions).
Agree that SQL Compare is an amazing tool.
However, we do not make any changes to the database structure or objects that are not scripted and saved in source control just like all other code. Then you know exactly what belongs in the version you are promoting because you have the scripts for that particular version.
It is a bad idea anyway to make structural changes through the GUI. If you havea lot of data, it is far slower than using alter table at least in SQL Server. You only want to use tested scripts to make changes to prod as well.
I agree with the comments made by marapet, where each change must be scripted.
The problem that you may be experiencing, however, is creating, testing and tracking these scripts.
Have a look at the patching engine used in DBSourceTools.
http://dbsourcetools.codeplex.com
It's been specifically designed to help developers get SQL server databases under source-code control.
This tool will allow you to baseline your database at a specific point, and create a named version (v1).
Then, create a deployment target - and increment the named version to v2.
Add patch scripts to the Patches directory for any changes to schema or data.
Finally, check the database and all patches into source-code control, to distribute with devs.
What this gives you is a repeatable process to test all patches to be applied from v1 to v2.
DBSourceTools also has functionality to help you create these scripts, i.e. schema compare or script data tools.
Once you are done, simply send all of the files in the patches directory to your DBA to upgrade from v1 to v2.
Have fun.
Another "Diff" tool for databases:
http://www.xsqlsoftware.com/Product/Sql_Data_Compare.aspx
Keep database version in a versioning table
Keep script file name that was successfully applied
Keep md5 sum of each sql script that has been applied. It should ignore spaces when calculate md5 sum. Must be effective.
Keep info about who applied a script Keep info about when a script was applied
Database should be verified on application start-up
New sql script should be applied automatically
If md5 sum of a script that was already applied is changed, error should be thrown (in a production mode)
When script have been released it must not be changed. It must be
immutable in a production environment.
Script should be written in a way, so it could be applied to different types of database (see liquibase)
Since most ddl statements are auto-committing on most databases, it is best to have a single ddl statement per SQL script.
DDL sql statement should be run in a way, so it can be executed several times without errors. Really helps in a dev mode, when you may edit script several times. For instance, create a new table, only if it does not exist, or even drop table before creating a new one. It will help you in a dev mode, with a script that has not been released, change it, clear md5 sum for this script, rerun it again.
Each sql script should be run in its own transaction.
Triggers/procedures should be dropped and created after each db
update.
Sql script is kept in a versioning system like svn
Name of a script contains date when it was committed, existing (jira) issue id, small description
Avoid adding rollback functionality in scripts (liquibase allow to do that). It makes them more complicated to write and support. If you use exactly one ddl statement per script, and dml statements are run within a
transaction, even failing a script will not be a big trouble to
resolve it
This is the workflow we have been using succesfully:
Development instance: SQL objects are created/updated/deleted in DB using MSSQL Studio and all operations are saved to scritps we include in each version of our code.
Moving to production: We compare schema between dev and prod db using SQL Schema Compare in Microsoft Visual Studio. We update prod using the same tool.
We maintain a set of change scripts that must be run on the DB when our web application is released. We waste a lot of time and experience some difficultly keeping these updated however, our DBA likes to (rightly) tweak stored procedures and schemas on the live system to maintain system performance.
Every so often we have to rebase our patches to the current schema and stored procedures, however, it is extremely difficult to detect which changes might conflict and work out which of our DBAs' changes we might be clobbering.
How do others manage the need for changes on live DBs against pending changes?
What processes can we put in place to make this process more smooth?
What is the best way to store, manage our schema and apply our/his changesets?
Thanks in advance.
DBAs should not ever tweak procs on prod only. They should also use source control and put the changes onthe other environments so that others making changes are aware of them.
Make any and all DDL changes to the DB schema script based and store then in your source code control. Especially changes your DBA makes - I would suggest getting your base schema and stored procs examined by a db developer and the DBA prior to checking them into your source code control (props to HLGEM for saying it). Moves into prod should be scripted and approved prior to application (ie, if the DBA finds stuff that needs to be changed, have the DBA open a defect and handle via that process).
Lock all such DDL changes away from your developers. The smart guys writing Java and C# should be communicating with your team db "specialist" on how to best accomplish the design goals and needs on the db side.
Limit production tweaks to those highly situation dependent cases, for example, many IT shops have a DBA who will define the physical storage setup based on db deployment scripts your app supplies and this is usually Ok. A wizard with your app to help less experienced people along with a top 10 list of recommendations for setup and basic tuning will go a long way.
We have 18 databases that should have identical schemas, but don't. In certain scenarios, a table was added to one, but not the rest. Or, certain stored procedures were required in a handful of databases, but not the others. Or, our DBA forgot to run a script to add views on all of the databases.
What is the best way to keep database schemas in sync?
For legacy fixes/cleanup, there are tools, like SQLCompare, that can generate scripts to sync databases.
For .NET shops running SQL Server, there is also the Visual Studio Database Edition, which can create change scripts for schema changes that can be checked into source control, and automatically built using your CI/build process.
SQL Compare by Red Gate is a great tool for this.
SQLCompare is the best tool that I have used for finding differences between databases and getting them synced.
To keep the databases synced up, you need to have several things in place:
1) You need policies about who can make changes to production. Generally this should only be the DBA (DBA team for larger orgs) and 1 or 2 backaps. The backups should only make changes when the DBA is out, or in an emergency. The backups should NOT be deploying on a regular basis. Set Database rights according to this policy.
2) A process and tools to manage deployment requests. Ideally you will have a development environment, a test environment, and a production environment. Developers should do initial development in the dev environment, and have changes pushed to test and production as appropriate. You will need some way of letting the DBA know when to push changes. I would NOT recommend a process where you holler to the next cube. Large orgs may have a change control committee and changes only get made once a month. Smaller companies may just have the developer request testing, and after testing is passed a request for deployment to production. One smaller company I worked for used Problem Tracker for these requests.
Use whatever works in your situation and budget, just have a process, and have tools that work for that process.
3) You said that sometimes objects only need to go to a handful of databases. With only 18 databases, probably on one server, I would recommend making each Databse match objects exactly. Only 5 DBs need usp_DoSomething? So what? Put it in every databse. This will be much easier to manage. We did it this way on a 6 server system with around 250-300 DBs. There were exceptions, but they were grouped. Databases on server C got this extra set of objects. Databases on Server L got this other set.
4) You said that sometimes the DBA forgets to deploy change scripts to all the DBs. This tells me that s/he needs tools for deploying changes. S/He is probably taking a SQL script, opening it in in Query Analyzer or Manegement Studio (or whatever you use) and manually going to each database and executing the SQL. This is not a good long term (or short term) solution. Red Gate (makers of SQLCompare above) have many great tools. MultiScript looks like it may work for deployment purposes. I worked with a DBA that wrote is own tool in SQL Server 2000 using O-SQl. It would take an SQL file and execute it on each database on the server. He had to execute it on each server, but it beat executing on each DB. I also helped write a VB.net tool that would do the same thing, except it would also go through a list of server, so it only had to be executed once.
5) Source Control. My current team doesn't use source control, and I don't have enough time to tell you how many problems this causes. If you don't have some kind of source control system, get one.
I haven't got enough reputation to comment on the above answer but the pro version of SQL Compare has a scriptable API. Given that you have to replicate stuff to all of these databases you could use this to make an automated job to either generate the change scripts or to validate that the databases are all in sync. It's also not much more expensive than the standard version.
Aside from using database comparison tools, with 18 databases you should have a DBA, so enforce a policy that only the DBA can change tables at the database level by restricting access to CREATE and ALTER to the DBA only. On both your test and live databases. The dev database shouldn't have this, of course! Make the developers who have been creating or altering the schemas willy-nilly go via the DBA.
Create a single source-controlled DDL/SQL script for each release and only use it to update the databases. The diff tools can be useful but mainly for checking that you haven't made a mistake and getting out of trouble when the policies fail. Combine the DDL, SQL, and stored procedure scripts into a single script so that it's not easy to "forget" to run one of the scripts.
We have got a tool called DB Schema Difftective that can compare and sync database schemas. With our other tool, DB MultiRun you can easily deploy generated (sync) scripts to multiple db servers (project based).
I realize this post is old, but TurnKey is correct. If you are a developer working in a team environment, the best way to maintain a database schema for a large application, is to make updates to a Master Schema in what ever source safe you use. Simply write your own Scripting class and your Database will be perfect every time.