In Liquibase, how can I rollback changes made to "rerunnable" (runOnChange=true) changesets? - liquibase

In my liquibase project, I have an sql view that is defined within a file called create_myview.sql.
Each time a change is made to the view, i.e. a column is dropped or renamed, instead of creating a new changeset that contains an ALTER statement, the view definition itself in this file is changed.
This file has the runOnChange attribute set to true so that when changes are made, the view is dropped and recreated the next time liquibase update is run. As such the same file can be run over and over when its contents change, i.e. it is "rerunnable".
Since previous definitions of the view are overwritten with each change, rolling back to a previous version presents a challenge and I am unable to work out the best way to do this.
Currently the only way that I am storing the previous view definitions is through git branches, i.e. with each view change I create a new branch.
Ideally I would like to checkout an old branch and be able to rollback to the definition that is currently checked out. I would like to be able to hop between versions easily.
Other non-rerunnable changesets defined in my project use a --rollback tag that is specified in the sql file and provides the inverse sql operation to each table change i.e. if a column is added to a table using an alter statement, then the inverse sql statement specified looks like:
-- rollback ALTER TABLE x DROP COLUMN name;
The equivalent of using this tag in my rerunnable view file would be to copy the entire previous view definition into this rollback tag, which doesn't seem like it would be best practice.
However, liquibase rollback doesn't seem to work without the --rollback tag being specified.
The file structure:
myDB/
changelog/
ver1/
rerunnable/
create_myview.sql
rerunnable.myDB.xml
myDB.changelog-root.xml
This is how I would make changes and update my view from one version to the next.
checkout new branch
git checkout -b c-01
add changes to view
create_myview.sql:
-- changeset john:c-01 runOnChange:true
DROP VIEW my_view IF EXISTS my_view;
CREATE my_view AS
SELECT name, date
FROM my_table;
...
update the changeset attributes
rerunnable.myDB.xml
<changeSet author="john" id="c-01">
<tagDatabase tag="1.0.0"/>
</changeSet>
Run liquibase update.
Next, an update is made to the view when the date column is dropped.
git checkout -b c-02
-- changeset john:c-02 runOnChange:true
DROP VIEW my_view IF EXISTS my_view;
CREATE my_view AS
SELECT name
FROM my_table;
...
<changeSet author="john" id="c-02">
<tagDatabase tag="2.0.0"/>
</changeSet>
liquibase update
At this point, the view in the database is up to date with the latest update, without the date column, and the databasechangelog looks like:
id
author
orderexecuted
md5sum
description
tag
c-01
john
1
83h8hs...
tagDatabase
1.0.0
c-01
john
2
ln9n2b1...
sql
c-02
john
3
ib309bd...
tagDatabase
2.0.0
c-02
john
4
lmxo21...
sql
From this point I am unable to rollback to how the view was at c-01/1.0.0.
The behaviour that I expect/hope is possible would be something like:
check out branch c-01
the old view definition is now in the working directory
run liquibase rollback or liquibase update
the view in the database is dropped and recreated with the c-01 schema (with the date column).
The changelog only has the first 2 lines.
Unfortunately, liquibase update does nothing, and liquibase rollback specifies that I need a --rollback statement.
would the runAlways attribute work here as a good solution?

For those not on the Liquibase forums, I answered this question there:
https://forum.liquibase.org/t/how-do-i-manage-rolling-back-changes-made-to-rerunnable-runonchange-true-changelogs-that-contain-stored-logic/7780

Related

liquibase checks that changesets have not been changed, but doesn't check that they've not been removed: is there a setting to fix this?

Liquibase does a pretty good job in keeping the applied changesets consistent with their source-folder.
If you modify a changeset that has already been applied to the db, Liquibase refuses to do anything, even operations that are not related with the modified changeset.
The rule enforced here is: anything that has been applied to the database must be unapplied before changing it (this is a usual workflow during development).
The problem of 'orphaned' changesets
Unfortunately this rule doesn't apply if you just delete the changeset completely.
In this case there will be what I call an 'orphaned' changeset, that is a record on the DATABASECHANGELOG table (and the database object, of course) with liquibase not complaining at all of a missing changeset in the source.
I expected an error at least when you 'bump' into the missing changeset, that is when you try to rollback it, but liquibase simply seems to ignore its presence, it skips it and rolls back the next one. This can be a problem.
The question is: can we change this liquibase behavior via settings? Is this design needed for some use case that I haven't thought of?
I think what I'm asking should be clear enough, however here is an example as demonstration.
Example
<databaseChangeLog ... >
<include relativeToChangelogFile="true" file="CS1.sql"/>
<include relativeToChangelogFile="true" file="CS2.sql"/>
</databaseChangeLog>
My changesets are SQL based files like this:
-- liquibase formatted sql
-- changeset agostinox:-1
CREATE TABLE T1 (
X INT
)
-- rollback DROP TABLE T1
And CS2.sql is the same for the T2 table.
Now i can apply my changesets like this:
PS > liquibase update
...
Liquibase Version: 4.19.0
Liquibase Open Source 4.19.0 by Liquibase
...
Running Changeset: CS1.sql::-1::agostinox
Running Changeset: CS2.sql::-1::agostinox
Liquibase command 'update' was executed successfully.
And on my db, the DATABASECHANGELOG has the following content:
ID
AUTHOR
FILENAME
DATEEXECUTED
ORDEREXECUTED
EXECTYPE
MD5SUM
-1
agostinox
CS1.sql
2023-01-18 18:52:08.476689
1
EXECUTED
8:d966f9ba2b90eaea9b917a6d93962eff
-1
agostinox
CS2.sql
2023-01-18 18:52:08.666667
2
EXECUTED
8:7f2a735fa83b196a0c72885c95362b81
So far so good. Now, now I get to the point.
Let's mess with the CS1.sql, by adding a comment:
-- liquibase formatted sql
-- changeset agostinox:-1
CREATE TABLE T1 (
X INT --Added comment, very harmless but enough to annoy liquibase :-)
)
-- rollback DROP TABLE T1
Now, I try to rollback the last changeset.
PS > liquibase rollbackcount 1
...
Unexpected error running Liquibase: Validation Failed:
1 changesets check sum
CS1.sql::-1::agostinox was: 8:d966f9ba2b90eaea9b917a6d93962eff but is now:
8:2cea5484e81eb542fa94bb67ba2ffdf5
For more information, please use the --log-level flag
You can see that liquibase complains about CS1.sql been changed even if we are not even rolling back CS1.sql but CS2.sql. Actually this change blocks any further operation, so it have to be reverted in order to break the deadlock.
However, if you just remove the changeset, liquibase doesn't complain anymore:
<databaseChangeLog ... >
<!--REMOVED <include relativeToChangelogFile="true" file="CS1.sql"/> -->
<include relativeToChangelogFile="true" file="CS2.sql"/>
</databaseChangeLog>
PS > liquibase rollbackcount 1
...
Rolling Back Changeset: CS2.sql::-1::agostinox
Liquibase command 'rollbackCount' was executed successfully.
But even if liquibase says that everything is successfull, it is not really the case, infact the table DATABASECHANGELOG now contains the 'orphaned' changeset (and the database contains the T1 table since also the rollback code is gone).
ID
AUTHOR
FILENAME
DATEEXECUTED
ORDEREXECUTED
EXECTYPE
MD5SUM
-1
agostinox
CS1.sql
2023-01-18 18:52:08.476689
1
EXECUTED
8:d966f9ba2b90eaea9b917a6d93962eff
For db people, it seems like liquibase does a left join between changesets source folder and records in DATABASECHANGELOG (that is: all the items on the left set are taken and are associated with the matching items on the right set); this way liquibase can see applied migrations and check if their checksum matches. It can also see not-yet-applied migrations (changesets in source folder without a DATABASECHANGELOG record associated) in order to apply them in the next 'update' call.
But it can't see records from DATABASECHANGELOG (the right set) that don't have an associated changeset in the source folder.
This parallel with join operation well explains the liquibase behavior, so let's use it for describing what a better behavior might be.
Liquibase should do a full-join, that is: also the items on the right side that don't have a source changeset associated are considered. Those records indicate an anomaly that is possibly just a 'major version' of the one that you have when a checksum doesn't match. Indeed you can easily think of a null file as a particolar case of checksum, the checksum of null being something necessarily different from the one in the table. Therefore the existence of those non associated record should give the same kind of error. This is what i would expected to truly ensure consistence between source an DATABASECHANGELOG.
I've been curious about this also. I'd recommend opening a github issue so a Liquibase employee can address why they don't do this check. I'm guessing this was done on purpose.
https://github.com/liquibase/liquibase/issues

Liquibase Update Command doesn't drop elements (tables, functions, procedure...) from my database despite SQL script absent from my solution

I use liquibase tool to manage a postgres database. I work as the following :
I have a solution composed of different folders containing SQL scripts responsible for schema creation, tables creations, types creation, procedures creation, etc... Then, I have a ChangeLog file in xml format, containing the following informations :
includeAll path="#{Server.WorkingDirectory}#/02 - Schema" relativeToChangelogFile="false"
includeAll path="#{Server.WorkingDirectory}#/03 - Types" relativeToChangelogFile="false
includeAll path="#{Server.WorkingDirectory}#/04 - Tables" relativeToChangelogFile="false"
includeAll path="#{Server.WorkingDirectory}#/05 - Fonctions" relativeToChangelogFile="false"
includeAll path="#{Server.WorkingDirectory}#/06 - Stored Procedures" relativeToChangelogFile="false"
I run liquibase via command line :
liquibase --changeLogFile=$(Changelog.File.Name) --driver=$(Driver.Name) --classpath=$(Driver.Classpath) --url=$(BDD.URL) --username=$(BDD.Login) --password=$(BDD.Password) update
This enable Liquibase to take all the SQL scripts in the different folders listed in the changelogFile, compare it with the current database at url $(BDD.URL), and generate a delta script containing all the SQL queries to be executed to have a database corresponding to my solution.
This works well when I add new scripts (new tables or procedures) or modify existing scripts, my database is correctly updated by the command line, as expected. BUT it does not do anything when I delete a script from my solution.
To be more factual, here is what I want :
I have a SQL file containing the query "CREATE TABLE my_table" located in the folder "04 - Tables".
I execute the update command above, and it creates the table "my_table" in my database.
I finally do not want this table in my database any more. Thus I would like to simply remove the corresponding SQL script from my solution, and then run again the "update" command to simply remove my table in my database, generating automatically a "DROP TABLE my_table" by the liquibase "update" command. But this is not working as Liquibase doesn't record any change when I remove a sql file (whereas it does when I add or modify a file).
Does anyone know a solution to this ? Is there a specific command to drop an element when there is no "CREATE" query for this element, in a SQL solution ?
Many thanks in advance for you help :)
You will need to explicitly write a script to drop the table.
Other option is to rollback the change IF YOU HAVE Specified the Rollback SQL as part of your original SQL script.
There is a Pro Version option to rollback a single update , with free / community version, you can rollback last few changes in sequence
ex; I did "liquibase rollbackCount 5" will rollback the last 5 changes that were applied ONLY IF I HAD Coded the rollback sql needed as part of my script.
My Sql script sample that included the code to rollback is
--rollback drop TABLE test.user1 ; drop table test.cd_activity;
CREATE TABLE test.user1 (
user_type_id int NOT NULL
);
CREATE TABLE test.cd_activity (
activity_id Integer NOT NULL
userid int);

Flyway always execute repeatable migrations

Is it possible to execute repeatable migrations in flyway even when the checksum is the same? The problem I am facing is having a view which extends another table with additional rows and the view doesn't get updated automatically.
An example here:
R__person_view.sql
CREATE OR REPLACE VIEW person_view AS
SELECT p.*, e.name FROM person p, entity e
WHERE /* not relevant here ... */;
If this migration is executed at first it will work fine. If I add another migration, where I modify the table person, the changes are not adapted, because the view migration checksum did not change.
Yes, from Flyway 6.3.0 it has been possible to have repeatable migrations run each time using the timestamp placeholder in a comment, ensuring that Flyway sees this as being changed afresh each time. For example:
R__UtilityProcedures.sql
-- ${flyway:timestamp}
create or replace procedure my_important_proc

Run rollback when checksum is changed for changeset in Liquibase

I see an attribute runOnChange that re-runs changeset when it is changed. But is it possible to apply a rollback for this changeset before re-applying automatically?
For example, I have a script that is called from changeset. I made some changes there and want to re-apply, but before it rollback need to be called and after it new version of the script should be applied.
Thank you!
There is no feature to automatically roll back a changeSet on checksum change. Not sure if it is possible in general because if the configuration has changed you don't know what the old value was to roll back.
Depending on what you are doing in your script and your database, can roll back the changes manually in the script and use the liquibase runOnChange="true" changeSet flag.
For example, if you have a script that creates a stored procedure, you could use <changeSet runOnChange="true"> and then define your procedure as "CREATE OR REPLACE"
If you have a script that defines a view, you could add a <sql>IF EXISTS VIEW_NAME DROP VIEW VIEW_NAME</sql>

liquibase does not add script to history when changeset fails

When a changeset is marked as failOnError:false, does liquibase record it as having been applied when it fails?
For example, we have a script that performs a pre-emptive drop table in one changeset and then creates the table in the next changeset. When the script is first run, the drop table statement fails as expected and then the table is created successfully. However, the changeset that attempted the drop table is not added to the databasechangelog table.
Is that expected behavior?
That is the current behavior currently. Depending on the reason for the failure, it can make sense to either continue to retry it or to not.
I created https://liquibase.jira.com/browse/CORE-1766 to add the feature to mark it as failed and not try again.
Currently, your best option would be to add a precondition to the dropTable changeSet with onFail="MARK_RAN"