We are using dbt in our project, and we found it's not easy to track "orphan" tables. These are tables or views created by dbt for which there is no matching current model.
These came about when a .sql file got renamed, or a view got converted to a table, etc...
Is there a way to find all of these orphan tables and views? Or at least a way to clear them out.
Ideas:
Use pre or post hooks to maintain a table in the database with all views/tables created by dbt
Schemas?
Note:
We are using MSSQL if it matters
Our team also used SQL Server and we have a macro we run as part of our CI for this. Here is the Pull Request, but we have yet to merge it, so you'll have to copy the macro yourself. The readme also has informatino on how to use it.
I'll update this answer once tsql-utils v0.8.0 is officially released!
Related
I may just be stupid and missing something simple, but how do I make it so when I publish my SSDT project, it either empties or drops the tables in my DB without actually setting the 'CreateNewDatabase' setting to true in my publishing profile.
I have some post deploy inserts that run every time i publish which results in duplicate rows every time.
Just to clarify, every time you deploy you want to clear out the contents of all tables as you have a load of insert statements in the post-deploy scripts to create the data you want and so you want the tables to be empty?
If that is right then that isn’t the way ssdt is normally used, typically the important bit in a database is the data so you wouldn’t want to clear every table on publish so there isn’t anything built in, other than create new database.
That being said if you don’t need any data then you are in a great position, most problems we have is trying to make sure we don’t delete any data by accident :)
There are a couple of approaches, you could change your insert statements to merge statements, there is a proc called sp_generate_merge that you can get from github which generates a merge statement that you can use, this will make your tables look like the data in your post-deploy scripts and is my preferred.
If tables have more than around 2,000 rows then merge statements might not be right so I would just do a delete of truncate table before inserting my data.
Hope I got the question right :)
Ed
I am writing a utility which keeps track of dropped indices or missing indices. I got to know about 2 index tables namely ALL_IND_COLUMNS and ALL_INDEXES which contains all the indices associated to each table in the database. I'm using ALL_IND_COLUMNS because it even contains column names.
Now i want to create a history table which keeps track of all the changes to ALL_IND_COLUMNS. I had thought of writing a trigger so that when there is an insert , delete or update on ALL_IND_COLUMNS the data all be inserted to history table but I heard there will be performance issue if we create a triggers on data dictionary tables. So, I want to know if there is any better alternative to solve this problem in SQL or PL/SQL. Im using using oracle 11g.
Thanks in advance.
Indexes are NOT meant to be created and dropped frequently. Even if you make so frequent changes, then you should be able to track the changes using source code version control.
There are many tools available for VERSION CONTROL. You should install and create required tags and branches for your database objects. Any modification to the database objects should go through database version control.
For example, the scripts that you use to create/drop the indexes, should be in the version control under INDEXES.
Checkout the code/scripts from repository to you local directory
Make necessary modifications
Test it locally
Check in your changes with required description
I personally use Subversion for my database version control.
For more details, read this link Using Source Code Control in Oracle SQL Developer
Read this wiki link about Revision control, also known as version control and source control
I'm writing an application that is using a database (currently MySQL 4) to store data.
It is likely that I will make changes to this in the form of updates later to add additional data. Updating the application is simple, it essentially comes down to overwriting the program files with the new ones. However how do I go about updating the database schema?
The database is remote and so my application might exist in several places, so simply dumping the ALTER and CREATE statements in an installer would result in the changes being made multiple times, and I have been asked explicitly for an automatic solution that allows for the application copies to be updated over a transition period, and for schema updates to be automatic.
I considered examining the schema at start-up to look for missing tables and columns, and adding them as needed, however this does not seem like a clean solution. I also considered putting some kind of “schema version” number on the database, but can’t see any way to do this short of a single row table with an int “Version” column which doesn’t seem a good way either.
I can highly recommend Liquibase. It really does work - I've used it and was very impressed.
Essentially, it keeps its own log of statements run on a database and runs them only if not already run/needed. It is XML driven and allows you to use optional pre- and post-execution statements and conditions. You check your XML files into your source control and invoke it from your build tool. It's even suitable for driving production releases.
It's magic.
Rather than rolling your own system for versioning your database it's probably worth looking into an existing framework that will manage it for you.
I use liquibase and have integrated into my build using the maven plugin. Worth checking out!
Just as you proposed, add a table where you store the current version of the database schema. Then you only have to apply the changes between your last schema update and the new release, and set the new version number accordingly. I've done this to update our production database about 300 times, it just works.
These days I am importing quite a lot of databases from my server and working on them locally. In the process, I am making a number of changes to the table structure and in the process using some complex SQL statements to add the table columns.
Keeping track of everything in a separate file is beginning to be a pain and am wondering if there is a way to do this directly in the SSMS so that I can store the instructions along with the database. Is there any way this can be done or do I have to resort to writing documentation outside SQL Server?
Of course, I can always create a stub table called comments and put everything there but I was looking for a way to associate comments with a particular database or tables. Any suggestions would be greatly appreciated.
SQL-Server handles commenting on database objects through Extended Properties:
http://msdn.microsoft.com/en-us/library/ms190243.aspx
I'm trying to head this one off at the pass. I've got two database servers (DEV and PRD) and I have my database on the DEV server. I am looking to deploy v1 of my application to PRD server.
The question is this: Say in two months, I am ready to ship v1.1 of my application, which introduces two new VIEWS, six new fields (three fields in two tables, each), and an updated version of my sproc that creates records in the tables with new fields. My DEV database has the new schema, but my PRD database has the real data, so I can't simply copy the .mdf file, since I want to keep my PRD data, but include my new schema.
I understand doing the initial creation of tables, views, sprocs via saved .sql files; but what I'm wondering is, is it possible to use SSMS to create the appripriate "alter table" scripts or do I need to manually do this?
I have handled this with a release update SQL script that applies the changes to the previous version.
You either need to code this yourself or use one of the many DBA tools to do database compares and generate a diff script.
There are tools that will do this for you SQL Compare is one of them and one I like the best
Otherwise you have to code these yourself and don't forget to also script the permissions if you recreate the proc (unless you use ALTER PROC in that case permissions are preserved)
Since your database changes should be in scripts that are under source control, you just load them with the version that you are moving to prod just like any other code associated with that version. One you you never under any circumstances do is make changes to the dev (or any other) datbase, using the User interface.
Try the patching engine found in DBSourceTools.
http://dbsourcetools.codeplex.com
DBSourceTools is a utility to help developers get their databases under source control.
Simply point it at a Source Database, and it will script all database objects, incuding data to disk.
Once you have a Target database (v1), you can then place your patch scripts int the patches directory, and DBSourceTools will run these patches in order after re-creating your database.
This is a very effective means of thoroughly testing your change scripts.