How does liquibase rollback works with springboot application? Would appreciate your inputs.
Here is what i tried - I am creating TableA and TableB in Oracle within a changeset.
TableB already exists in the database, I expect liquibase to rollback TableA as changeset fails while creating TableB but liquibase creating TableA and failing with the below error and never executes rollback block, which is strange:
Caused by: liquibase.exception.DatabaseException: ORA-00955: name is already used by an existing object
Liquibase Configuration:
<changeSet id="rollback" author="test_user">
<validCheckSum>any</validCheckSum>
<sqlFile path="db/changelog/changes/DML/ddl.sql"/>
<sqlFile path="db/changelog/changes/DML/ddl.sql"/>
<rollback> drop table TABLEA;</rollback>
<rollback> drop table TABLEB;</rollback>
</changeSet>
ddl.sql
CREATE TABLE TABLEA
(
TEST_COL VARCHAR2(100)
);
dml.sql
CREATE TABLE TABLEB
(
TEST_COL VARCHAR2(100)
);
Looks like you have a wrong perception of what the rollback is for. It's not for "reverting one big transaction within a whole changeSet". It's for reinstating some past schema state.
According to Oracle documentation
Oracle Database issues an implicit COMMIT before and after any data
definition language (DDL) statement.
So after every create table statement execution, Oracle commits the transaction.
If the error happens during the execution of a changeSet, liquibase can rollback only a failed transaction, which it does.
Check out this thread
Liquibase does not automatically roll back changes made during a deploy if there was an error. Rather, it can roll a database back to a previous schema state when you ask it to. In some cases, it can do that without you explicitly saying how to do the rollback.
It would be better to think of the rollback command in Liquibase as
'undo deploys' rather than a transactional rollback.
Also, check out this liquibase-rollback article on how to use rollbacks.
Related
In my liquibase project, I have an sql view that is defined within a file called create_myview.sql.
Each time a change is made to the view, i.e. a column is dropped or renamed, instead of creating a new changeset that contains an ALTER statement, the view definition itself in this file is changed.
This file has the runOnChange attribute set to true so that when changes are made, the view is dropped and recreated the next time liquibase update is run. As such the same file can be run over and over when its contents change, i.e. it is "rerunnable".
Since previous definitions of the view are overwritten with each change, rolling back to a previous version presents a challenge and I am unable to work out the best way to do this.
Currently the only way that I am storing the previous view definitions is through git branches, i.e. with each view change I create a new branch.
Ideally I would like to checkout an old branch and be able to rollback to the definition that is currently checked out. I would like to be able to hop between versions easily.
Other non-rerunnable changesets defined in my project use a --rollback tag that is specified in the sql file and provides the inverse sql operation to each table change i.e. if a column is added to a table using an alter statement, then the inverse sql statement specified looks like:
-- rollback ALTER TABLE x DROP COLUMN name;
The equivalent of using this tag in my rerunnable view file would be to copy the entire previous view definition into this rollback tag, which doesn't seem like it would be best practice.
However, liquibase rollback doesn't seem to work without the --rollback tag being specified.
The file structure:
myDB/
changelog/
ver1/
rerunnable/
create_myview.sql
rerunnable.myDB.xml
myDB.changelog-root.xml
This is how I would make changes and update my view from one version to the next.
checkout new branch
git checkout -b c-01
add changes to view
create_myview.sql:
-- changeset john:c-01 runOnChange:true
DROP VIEW my_view IF EXISTS my_view;
CREATE my_view AS
SELECT name, date
FROM my_table;
...
update the changeset attributes
rerunnable.myDB.xml
<changeSet author="john" id="c-01">
<tagDatabase tag="1.0.0"/>
</changeSet>
Run liquibase update.
Next, an update is made to the view when the date column is dropped.
git checkout -b c-02
-- changeset john:c-02 runOnChange:true
DROP VIEW my_view IF EXISTS my_view;
CREATE my_view AS
SELECT name
FROM my_table;
...
<changeSet author="john" id="c-02">
<tagDatabase tag="2.0.0"/>
</changeSet>
liquibase update
At this point, the view in the database is up to date with the latest update, without the date column, and the databasechangelog looks like:
id
author
orderexecuted
md5sum
description
tag
c-01
john
1
83h8hs...
tagDatabase
1.0.0
c-01
john
2
ln9n2b1...
sql
c-02
john
3
ib309bd...
tagDatabase
2.0.0
c-02
john
4
lmxo21...
sql
From this point I am unable to rollback to how the view was at c-01/1.0.0.
The behaviour that I expect/hope is possible would be something like:
check out branch c-01
the old view definition is now in the working directory
run liquibase rollback or liquibase update
the view in the database is dropped and recreated with the c-01 schema (with the date column).
The changelog only has the first 2 lines.
Unfortunately, liquibase update does nothing, and liquibase rollback specifies that I need a --rollback statement.
would the runAlways attribute work here as a good solution?
For those not on the Liquibase forums, I answered this question there:
https://forum.liquibase.org/t/how-do-i-manage-rolling-back-changes-made-to-rerunnable-runonchange-true-changelogs-that-contain-stored-logic/7780
Liquibase does a pretty good job in keeping the applied changesets consistent with their source-folder.
If you modify a changeset that has already been applied to the db, Liquibase refuses to do anything, even operations that are not related with the modified changeset.
The rule enforced here is: anything that has been applied to the database must be unapplied before changing it (this is a usual workflow during development).
The problem of 'orphaned' changesets
Unfortunately this rule doesn't apply if you just delete the changeset completely.
In this case there will be what I call an 'orphaned' changeset, that is a record on the DATABASECHANGELOG table (and the database object, of course) with liquibase not complaining at all of a missing changeset in the source.
I expected an error at least when you 'bump' into the missing changeset, that is when you try to rollback it, but liquibase simply seems to ignore its presence, it skips it and rolls back the next one. This can be a problem.
The question is: can we change this liquibase behavior via settings? Is this design needed for some use case that I haven't thought of?
I think what I'm asking should be clear enough, however here is an example as demonstration.
Example
<databaseChangeLog ... >
<include relativeToChangelogFile="true" file="CS1.sql"/>
<include relativeToChangelogFile="true" file="CS2.sql"/>
</databaseChangeLog>
My changesets are SQL based files like this:
-- liquibase formatted sql
-- changeset agostinox:-1
CREATE TABLE T1 (
X INT
)
-- rollback DROP TABLE T1
And CS2.sql is the same for the T2 table.
Now i can apply my changesets like this:
PS > liquibase update
...
Liquibase Version: 4.19.0
Liquibase Open Source 4.19.0 by Liquibase
...
Running Changeset: CS1.sql::-1::agostinox
Running Changeset: CS2.sql::-1::agostinox
Liquibase command 'update' was executed successfully.
And on my db, the DATABASECHANGELOG has the following content:
ID
AUTHOR
FILENAME
DATEEXECUTED
ORDEREXECUTED
EXECTYPE
MD5SUM
-1
agostinox
CS1.sql
2023-01-18 18:52:08.476689
1
EXECUTED
8:d966f9ba2b90eaea9b917a6d93962eff
-1
agostinox
CS2.sql
2023-01-18 18:52:08.666667
2
EXECUTED
8:7f2a735fa83b196a0c72885c95362b81
So far so good. Now, now I get to the point.
Let's mess with the CS1.sql, by adding a comment:
-- liquibase formatted sql
-- changeset agostinox:-1
CREATE TABLE T1 (
X INT --Added comment, very harmless but enough to annoy liquibase :-)
)
-- rollback DROP TABLE T1
Now, I try to rollback the last changeset.
PS > liquibase rollbackcount 1
...
Unexpected error running Liquibase: Validation Failed:
1 changesets check sum
CS1.sql::-1::agostinox was: 8:d966f9ba2b90eaea9b917a6d93962eff but is now:
8:2cea5484e81eb542fa94bb67ba2ffdf5
For more information, please use the --log-level flag
You can see that liquibase complains about CS1.sql been changed even if we are not even rolling back CS1.sql but CS2.sql. Actually this change blocks any further operation, so it have to be reverted in order to break the deadlock.
However, if you just remove the changeset, liquibase doesn't complain anymore:
<databaseChangeLog ... >
<!--REMOVED <include relativeToChangelogFile="true" file="CS1.sql"/> -->
<include relativeToChangelogFile="true" file="CS2.sql"/>
</databaseChangeLog>
PS > liquibase rollbackcount 1
...
Rolling Back Changeset: CS2.sql::-1::agostinox
Liquibase command 'rollbackCount' was executed successfully.
But even if liquibase says that everything is successfull, it is not really the case, infact the table DATABASECHANGELOG now contains the 'orphaned' changeset (and the database contains the T1 table since also the rollback code is gone).
ID
AUTHOR
FILENAME
DATEEXECUTED
ORDEREXECUTED
EXECTYPE
MD5SUM
-1
agostinox
CS1.sql
2023-01-18 18:52:08.476689
1
EXECUTED
8:d966f9ba2b90eaea9b917a6d93962eff
For db people, it seems like liquibase does a left join between changesets source folder and records in DATABASECHANGELOG (that is: all the items on the left set are taken and are associated with the matching items on the right set); this way liquibase can see applied migrations and check if their checksum matches. It can also see not-yet-applied migrations (changesets in source folder without a DATABASECHANGELOG record associated) in order to apply them in the next 'update' call.
But it can't see records from DATABASECHANGELOG (the right set) that don't have an associated changeset in the source folder.
This parallel with join operation well explains the liquibase behavior, so let's use it for describing what a better behavior might be.
Liquibase should do a full-join, that is: also the items on the right side that don't have a source changeset associated are considered. Those records indicate an anomaly that is possibly just a 'major version' of the one that you have when a checksum doesn't match. Indeed you can easily think of a null file as a particolar case of checksum, the checksum of null being something necessarily different from the one in the table. Therefore the existence of those non associated record should give the same kind of error. This is what i would expected to truly ensure consistence between source an DATABASECHANGELOG.
I've been curious about this also. I'd recommend opening a github issue so a Liquibase employee can address why they don't do this check. I'm guessing this was done on purpose.
https://github.com/liquibase/liquibase/issues
Which is the best way to rollback this liquibase script?
<changeSet author="me" id="drop_column_example">
<dropColumn tableName="BLABLA">
<column name="example"/>
</dropColumn>
</changeSet>
Usually when I drop a table, I restore all the data from a duplicate temporary table created before the drop, but how can I manage this?
PS: I need to restore all the old data in that column.
Liquibase provides commands to undo changes made to database. The intention of a rollback script is to return the database to a previous specified point in time.
There are two categories of Liquibase operations, resulting in a different generation of a rollback statement:
1. Automatic - where migration can deterministically generate steps required for rolling back
2. Manual - where we need to issue a rollback command because migration instruction cannot be used to identify the statement deterministically
For example, the rollback of a “create table” statement would be to “drop” the created table. This can be determined without a doubt, and therefore the rollback statement can be autogenerated.
On the other hand, the rollback statement for a “drop table” command is not possible to be determined. It is not possible to determine the last state of the table, and therefore the rollback statement can't be autogenerated. These types of migration statements require a manual rollback instructions.
Read more about it here and here.
In your case, rollback statement for a “drop column” command is not possible to be determined. You will manually have to write an “alter table ADD column” statement for your table.
For your example, the query will be :
ALTER TABLE BLABLA ADD COLUMN example DATATYPE;
So the complete changeset would look like this :
<changeSet author="me" id="drop_column_example">
<dropColumn tableName="BLABLA">
<column name="example"/>
</dropColumn>
<rollback>
ALTER TABLE BLABLA ADD COLUMN example COLUMN_DATATYPE;
</rollback>
</changeSet>
To restore data in the column, you will have to create a changeset to copy and insert the data from a duplicate temporary table (similar to the way you mentioned for “drop table”) to this created column. Make sure you execute your data insertion in column after the rollback query for “drop column” is executed successfully.
You could either create a separate changeset or could include the logic to copy and insert old data for restoring into new column in the above changeset as below:
<changeSet author="me" id="drop_column_example">
<dropColumn tableName="BLABLA">
<column name="example"/>
</dropColumn>
<rollback>
ALTER TABLE BLABLA ADD COLUMN example COLUMN_DATATYPE;
<!-- Logic to restore old data goes here OR include the changeset created separately for data restoration here (After creating the dropped column) -->
</rollback>
</changeSet>
Added logic to copy the old data from temporary table to new created table -
INSERT INTO `New_Table` (`example`) VALUES
(SELECT `example` FROM `Temporary_Table` WHERE
`New_Table`.`Some_Common_Column` = `Temporary_Table`.`Some_Common_Column`);
You need to maintain the old data into a table in such a way that you can insert it at the time of restoration with proper mapping to the primary key column or some column which is common between the newly created table and backup table. Without having such structure you won't be able to map the required column data for data restoration in new table column.
Trying to figure out how rollbacks work with formatted SQL and Liquibase.
Working from the quickstart on the Liquibase site, I was able to create a table in my database. Then started working on the rollback. Just as a manual test, once my table was created, I tagged the change in the DATABASECHANGELOG table. Ran Liquibase rollback <mytag> and it said it completed it successfully, but the table wasn't modified.
--liquibase formatted sql
--changeset user:1
create table addresses (
address VARCHAR(45),
city VARCHAR(45),
zip VARCHAR(10)
);
--rollback drop table addresses;
Liquibase said the rollback completed successfully, but the table wasn't dropped.
The tagging concept in Liquibase seems to confuse many people. Tagging is used to mark a known good state, so if you tagged it after deploying a change that created a table, and then said rollback to that tag, it did things 'correctly' in its way of thinking.
Here is the tiny bit of documentation on that (I added the emphasis on after):
Tag
Specifying a tag to rollback to will roll back all change-sets that
were executed against the target database after the given tag was
applied. See the “command line” documentation for how to tag your
database.
To test the rollback of the table in your example, you would need to use the rollbackCount command or the rollbackToDate command.
Is it OK to manually delete rows from Databasechangelog?
Btw, rollback to a TAG is not an option in my case.
Deleting rows from the table will simulate a rollback without actually performing one.... Leaving your database in an unknown state (the whole point is that liquibase manages the schema for you).
Why not just rollback a specified number of changesets instead?
liquibase rollbackCount 5