It might be super obvious but no one bothered clarifying what or who is actually creating/writing the changesets for liquibase. I read more than a dozen articles related to changesets in liquibase and while I now understand how it works I still wonder if these changesets are generated somewhere by Liquibase ? Or are users supposed to write them by hand ?
And do we agree that the CHANGELOGTABLE, is populated from doing a liquibase update by reading the already existing changesets ? Not the other way around ?
And do we also agree that liquibase doesn't track schema changes, it just computes the desired state of a DB from the changesets ?
Thanks
Edit: I asked many questions, but ultimately I'm just looking for an answer to the title and somehow understand properly how liquibase works.
You write the changeset. And since you can write changesets in sql, its just you writing the database scripts your application needs.
Yes, the DATABASECHANGELOG table is the audit log that gets written after a liquibase update that shows what changesets got executed.
I would recommend taking the fundamentals course provided for free at Liquibase University that covers these very basic concepts. Without it, it will be much harder to be successful using Liquibase. My experience: You can pretty much finish the course in one sitting or maybe an hour each day for a few days.
Related
My company uses liquibase to keep track of database changes. Everyday around 100 new changesets are being added. From what I understand for already executed changesets liquibase computes checksum again and compares it with checksum in databasechangelog table to see whether checksum has changed and gives checksum issue if it is changed.
So after few months when I have large number of changesets already executed, If I add a new changeset doesn't this process of computing checksum of already executed changesets and comparing them make the execution of new changesets slower or cause any performance related issues?
I've never stumbled across this kind of performance issues with liquibase.
But I guess your question raises a couple of more questions:
what do you consider to be "slower"?
when performance starts to become an issue and is it really an issue?
maybe something's wrong with your application's architecture?
Anyway, comparing checksums against DATABASECHANGELOG table shouldn't take a lot of time - it could be couple of seconds, if you have lots and lots of changeSets.
According to liquibase documentation:
Other times, the problem is that liquibase update is taking too long.
Liquibase tries to be as efficient as possible when comparing the
contents of the DATBASECHANGELOG table with the current changelog file
and even if there are thousands of already ran changeSets, an “update”
command should take just seconds to run.
But if these seconds really make an issue, then consider reading this article:
Trimming ChangeLog Files
Any reason why Liquibase's addNotNullConstraint doesn't have an attribute initiallyDeferred?
addForeignKeyConstraint and addUniqueConstraint both have an attribute initiallyDeferred.
I think this is just a missing feature. Liquibase has very few people contributing code, so sometimes things just aren't done yet. My company (Datical) has been making efforts to improve that, but we still rely very heavily on the users of Liquibase to point out where there are shortcomings.
I have created a jira ticket (CORE-3024) to track this.
I am using liquibase 3.2.0 on ORCID, and finding it really useful.
We now have over 200 changeSets on top of the original schema.
These run many times during unit tests because we are using an in memory database (hsqldb).
I would like to 'reset' liquibase by making a new install.xml from the current schema, so that we do not have to run all the changeSets every time.
However, the production database (postgres) has a databasechangelog table with all the old changeSets, so it will try to apply the new install.xml.
How can I start again from a new install.xml without causing problems for production?
Will
Restarting a changeLog from scratch is the same as adding liquibase to an existing project, which is discussed in documentation here
I generally recommend against resetting your changeLog, however, because normally the costs outweigh any benefits in performance. Your 200 changeSet changelog has been fully tested and you know it is correct whereas something regenerated manually or with generateChangeLog can easily have minor differences that can cause problems.
For existing databases, the startup cost of parsing the changelog file and comparing it to the contents of databasechangelog is very low, regardless of the number of changeSets.
For a new database, especially in-memory databases, DDL operations are generally very fast and the speed of going through 200 changeSets to build up your database will probably not be a lot different than building it up in 50 changeSets.
IF there are performance differences, what I've generally seen is that there are a few isolated changeSets that are the problem such as creating an index then dropping it then creating it again. I would recommend looking for any changeSets that may be a problem and carefully removing or combining them vs. a wholesale redo of the changelog.
Is there any way to have data migrations on production database not to be with SQL?
We are using MigratorDotNet and when we build a new funcionality for the application that changes the scheme of the database and we need to do some data updates we have to do this complex and troublesome SQL statements so the data is consistent on production.
Was wondering if there was another way to do this, what are the bests practices to do this? Any ideas on other possible solutions?
We cannot use something like NHibernate because then we have to keep fixing old migrations when the scheme changes, and that can be error prone.
The trick is to use your migration tool and fold said data manipulation statements into the migrations. We use an extended version of the same thing typically for a few projects and it can definitely handle that trick.
If you're already using a migration tool like Migrator.NET then I'd say you're most of the way there. Complex schema/data changes are just a fact of life in the RDBMS world.
Try mite. It let's you do anything that you can do with sql and use sql to do it but have the ability to ensure your database is on the desired version and not risk executing a script that has already run (or miss a script), leaving your database in a consistent state.
If your developers adopt this. Deployments are a simple mite update and then you know problems are product related or data related (but not schema related).
https://github.com/soitgoes/mite
Let me know what you think. I developed this and have been using this with my team for years with great success.
What would be the best approach for versioning my whole database ?
Creating a file for each database object (table,view,procedsure..) or rather having one file for all DDL scripts and any new change will be put in a separate file ?
What about handling changes made in a Database manager tool ?
I'd like to have a generic solutions for any kind of RDBMS.
Are there any other options ?
I'm a huge VCS fan in general and a big Mercurial booster, but I really think you're going down the wrong path.
VCSs aren't just about iterative changes, the "what", they're also about answering the "who", "when", and "why". For a database those answers are a lot less interesting or hard to provide to the VCS. If you're doing nightly exports and commits the "who" will always be "cron" and the "why" will always be "midnight".
The other thing modern VCSs do really well is helping you merge changes from multiple branches. That's less applicable in the database world. Very seldom do you say "I want this table structure, but this data", and if you do the text/diff merge isn't going to help you much.
The thing that does do "what" and "when" very well is an incremental backup system, and that's probably the better fit.
At work we use Tivoli and at home I use rdiff-backup and duplicity, but there are plenty of great options.
I guess my general rule of thumb is "if it was typed by hand by a human then it does into source control, and if it was generated/exported then it goes in the incremental backups"
Certainly you can make this work, but I don't think it will buy you much over the more traditional backup solutions.
Have a look at this post
If you need generic solution - put everything in the scripts (simple text files) and put under Version Control system (can be used any of VCS).
Grouping similar database objects into scripts will be depend on your requirement.
So you may for example:
Store table/indexes/ in one or several script
Each procedure store in individual script or combine small procedures into one script.
However need to remember one important thing with this approach: don't forget change scripts if you changed table/view/procedure directly in databases and don't create/recreate/compile you db objects in database after changing scripts.
SQL Source Control currently supports SVN and TFS, but Mercurial requests are increasing rapidly and we're hoping to have a story for this very soon.
We use UserVoice to measure demand so please vote accordingly if you're interesting in this: http://redgate.uservoice.com/forums/39019-sql-source-control