First of all, I'm using an npm library called liquibase to run Liquibase within Node.
So, I'm trying to use diffChangeLog to compare two databases and generate a changelog of their differences. The changelog is normally generated, but when I try to run an update using the generated changelog Liquibase responds with a validation error. Here's the output:
node_modules/liquibase/lib/liquibase-4.0.0/liquibase --changeLogFile=examples/common/migration.xml --url="jdbc:mysql://127.0.0.1:3306/mySchema?useJvmCharsetConverters=true" --username=migration --password=migration --classpath=lib/migrator/drivers/mysql-connector-java-8.0.22.jar --changeLogFile=examples/DatabaseDiff/diffChangeLog.xml update
Liquibase Community 4.0.0-beta1 by Datical
Starting Liquibase at 17:54:32 (version 4.0.0-beta1 #6 built at 2020-04-20 18:23+0000)
Unexpected error running Liquibase: Validation Failed:
16 changes have validation failures
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-1::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-2::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-3::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-4::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-5::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-6::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-7::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-8::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-9::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-10::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-11::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-12::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-13::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-14::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-15::joao (generated)
columns is empty, examples/DatabaseDiff/diffChangeLog.xml::1618001652047-16::joao (generated)
For more information, please use the --logLevel flag
And here is part of the changelog that is mentioned in the output:
<changeSet author="joao (generated)" id="1618001652047-2">
<createTable tableName="FeaturedProducer"/>
</changeSet>
<changeSet author="joao (generated)" id="1618001652047-3">
<createTable tableName="PreVerification"/>
</changeSet>
<changeSet author="joao (generated)" id="1618001652047-4">
<createTable tableName="Producer"/>
</changeSet>
I also tried to use the SQL format for the generated changelog (named changeLog.mysql.sql), but it creates a bad SQL syntax, for example:
CREATE TABLE USER ();
This raises a syntax error from MySQL. Which is right, I've tried running this query manually and the error is the same.
My guess is that Liquibase is not allowing me to create empty columns, but why?
I don't understand what's going on and there are not many results on Google.
Can anyone give me a hand?!
Well, after a while we've found out the problem.
To debug the issue I raised two local databases with Docker and tried the same operation, it worked just fine.
So, it occurred to us that the reason we were getting this error could be due to user permissions. I've seen some people talking about it, that Liquibase can spit unusual errors when the user has insufficient permissions.
So I gave my user all permissions for that schema and it worked! Actually, we just went to another error, but it was definitely progress!
The error was just like this:
liquibase.exception.LiquibaseException: liquibase.command.CommandExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at liquibase.integration.commandline.CommandLineUtils.doDiffToChangeLog(CommandLineUtils.java:250)
at liquibase.integration.commandline.Main.doMigration(Main.java:1285)
at liquibase.integration.commandline.Main$1.lambda$run$0(Main.java:314)
at liquibase.Scope.lambda$child$0(Scope.java:149)
at liquibase.Scope.child(Scope.java:160)
at liquibase.Scope.child(Scope.java:148)
at liquibase.Scope.child(Scope.java:127)
at liquibase.Scope.child(Scope.java:173)
at liquibase.Scope.child(Scope.java:177)
at liquibase.integration.commandline.Main$1.run(Main.java:313)
at liquibase.integration.commandline.Main$1.run(Main.java:169)
at liquibase.Scope.child(Scope.java:160)
at liquibase.Scope.child(Scope.java:134)
at liquibase.integration.commandline.Main.run(Main.java:169)
at liquibase.integration.commandline.Main.main(Main.java:148)
Caused by: liquibase.command.CommandExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at liquibase.command.AbstractCommand.execute(AbstractCommand.java:24)
at liquibase.integration.commandline.CommandLineUtils.doDiffToChangeLog(CommandLineUtils.java:248)
... 14 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at liquibase.diff.output.changelog.DiffToChangeLog.print(DiffToChangeLog.java:193)
at liquibase.diff.output.changelog.DiffToChangeLog.print(DiffToChangeLog.java:86)
at liquibase.command.core.DiffToChangeLogCommand.run(DiffToChangeLogCommand.java:63)
at liquibase.command.AbstractCommand.execute(AbstractCommand.java:19)
... 15 more
Caused by: java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at liquibase.diff.output.changelog.DiffToChangeLog$1.run(DiffToChangeLog.java:176)
at liquibase.Scope.lambda$child$0(Scope.java:149)
at liquibase.Scope.child(Scope.java:160)
at liquibase.Scope.child(Scope.java:148)
at liquibase.Scope.child(Scope.java:127)
at liquibase.diff.output.changelog.DiffToChangeLog.print(DiffToChangeLog.java:119)
... 18 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
at java.base/java.util.Objects.checkIndex(Objects.java:372)
at java.base/java.util.ArrayList.get(ArrayList.java:459)
at liquibase.change.ColumnConfig.<init>(ColumnConfig.java:154)
at liquibase.change.AddColumnConfig.<init>(AddColumnConfig.java:16)
at liquibase.diff.output.changelog.core.MissingIndexChangeGenerator.fixMissing(MissingIndexChangeGenerator.java:70)
at liquibase.diff.output.changelog.ChangeGeneratorChain.fixMissing(ChangeGeneratorChain.java:48)
at liquibase.diff.output.changelog.ChangeGeneratorFactory.fixMissing(ChangeGeneratorFactory.java:95)
at liquibase.diff.output.changelog.DiffToChangeLog.generateChangeSets(DiffToChangeLog.java:279)
at liquibase.diff.output.changelog.DiffToChangeLog.printNew(DiffToChangeLog.java:203)
at liquibase.diff.output.changelog.DiffToChangeLog$1.run(DiffToChangeLog.java:125)
... 23 more
Really bad logging btw...
This MissingIndexChangeGenerator.fixMissing(MissingIndexChangeGenerator.java:70) part caught my attention. Surfing through the web I found barely anything, so we thought: Let's check the source code for that line. On their GitHub, we've found the exact line of the exception and everything became clearer.
That error meant that Liquibase failed to retrieve some table indexes.
Taking a look at our database we were reminded that some indexes were actually foreign keys to tables on another schema. And the user we were using didn't have permission on that other schema. We gave the user all permissions for that second schema too, and BAM, it finally worked!
The lesson here is: Liquibase has really bad logging... Oh, and also, never give up!
Related
Liquibase does a pretty good job in keeping the applied changesets consistent with their source-folder.
If you modify a changeset that has already been applied to the db, Liquibase refuses to do anything, even operations that are not related with the modified changeset.
The rule enforced here is: anything that has been applied to the database must be unapplied before changing it (this is a usual workflow during development).
The problem of 'orphaned' changesets
Unfortunately this rule doesn't apply if you just delete the changeset completely.
In this case there will be what I call an 'orphaned' changeset, that is a record on the DATABASECHANGELOG table (and the database object, of course) with liquibase not complaining at all of a missing changeset in the source.
I expected an error at least when you 'bump' into the missing changeset, that is when you try to rollback it, but liquibase simply seems to ignore its presence, it skips it and rolls back the next one. This can be a problem.
The question is: can we change this liquibase behavior via settings? Is this design needed for some use case that I haven't thought of?
I think what I'm asking should be clear enough, however here is an example as demonstration.
Example
<databaseChangeLog ... >
<include relativeToChangelogFile="true" file="CS1.sql"/>
<include relativeToChangelogFile="true" file="CS2.sql"/>
</databaseChangeLog>
My changesets are SQL based files like this:
-- liquibase formatted sql
-- changeset agostinox:-1
CREATE TABLE T1 (
X INT
)
-- rollback DROP TABLE T1
And CS2.sql is the same for the T2 table.
Now i can apply my changesets like this:
PS > liquibase update
...
Liquibase Version: 4.19.0
Liquibase Open Source 4.19.0 by Liquibase
...
Running Changeset: CS1.sql::-1::agostinox
Running Changeset: CS2.sql::-1::agostinox
Liquibase command 'update' was executed successfully.
And on my db, the DATABASECHANGELOG has the following content:
ID
AUTHOR
FILENAME
DATEEXECUTED
ORDEREXECUTED
EXECTYPE
MD5SUM
-1
agostinox
CS1.sql
2023-01-18 18:52:08.476689
1
EXECUTED
8:d966f9ba2b90eaea9b917a6d93962eff
-1
agostinox
CS2.sql
2023-01-18 18:52:08.666667
2
EXECUTED
8:7f2a735fa83b196a0c72885c95362b81
So far so good. Now, now I get to the point.
Let's mess with the CS1.sql, by adding a comment:
-- liquibase formatted sql
-- changeset agostinox:-1
CREATE TABLE T1 (
X INT --Added comment, very harmless but enough to annoy liquibase :-)
)
-- rollback DROP TABLE T1
Now, I try to rollback the last changeset.
PS > liquibase rollbackcount 1
...
Unexpected error running Liquibase: Validation Failed:
1 changesets check sum
CS1.sql::-1::agostinox was: 8:d966f9ba2b90eaea9b917a6d93962eff but is now:
8:2cea5484e81eb542fa94bb67ba2ffdf5
For more information, please use the --log-level flag
You can see that liquibase complains about CS1.sql been changed even if we are not even rolling back CS1.sql but CS2.sql. Actually this change blocks any further operation, so it have to be reverted in order to break the deadlock.
However, if you just remove the changeset, liquibase doesn't complain anymore:
<databaseChangeLog ... >
<!--REMOVED <include relativeToChangelogFile="true" file="CS1.sql"/> -->
<include relativeToChangelogFile="true" file="CS2.sql"/>
</databaseChangeLog>
PS > liquibase rollbackcount 1
...
Rolling Back Changeset: CS2.sql::-1::agostinox
Liquibase command 'rollbackCount' was executed successfully.
But even if liquibase says that everything is successfull, it is not really the case, infact the table DATABASECHANGELOG now contains the 'orphaned' changeset (and the database contains the T1 table since also the rollback code is gone).
ID
AUTHOR
FILENAME
DATEEXECUTED
ORDEREXECUTED
EXECTYPE
MD5SUM
-1
agostinox
CS1.sql
2023-01-18 18:52:08.476689
1
EXECUTED
8:d966f9ba2b90eaea9b917a6d93962eff
For db people, it seems like liquibase does a left join between changesets source folder and records in DATABASECHANGELOG (that is: all the items on the left set are taken and are associated with the matching items on the right set); this way liquibase can see applied migrations and check if their checksum matches. It can also see not-yet-applied migrations (changesets in source folder without a DATABASECHANGELOG record associated) in order to apply them in the next 'update' call.
But it can't see records from DATABASECHANGELOG (the right set) that don't have an associated changeset in the source folder.
This parallel with join operation well explains the liquibase behavior, so let's use it for describing what a better behavior might be.
Liquibase should do a full-join, that is: also the items on the right side that don't have a source changeset associated are considered. Those records indicate an anomaly that is possibly just a 'major version' of the one that you have when a checksum doesn't match. Indeed you can easily think of a null file as a particolar case of checksum, the checksum of null being something necessarily different from the one in the table. Therefore the existence of those non associated record should give the same kind of error. This is what i would expected to truly ensure consistence between source an DATABASECHANGELOG.
I've been curious about this also. I'd recommend opening a github issue so a Liquibase employee can address why they don't do this check. I'm guessing this was done on purpose.
https://github.com/liquibase/liquibase/issues
Using Flyway-core:4.1.2 for database-migration. Added a new DDL file for flyway to execute. Flyway executes the DDL correctly and makes the corresponding changes to tables and columns. (We're adding a table and altering some previous columns in the new DDL). But, flyway fails to register this attempt to schema_version table: I get the following error:
Current version of schema [dbo]: 2.1
Unable to insert row for version '3.0' in metadata table [dbo].[schema_version]
Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.dbsupport.FlywaySqlException:
Message : Cannot insert the value NULL into column 'installed_on', table 'dbo.schema_version'; column does not allow nulls. INSERT fails.
Flyway successfully executes the DDL however, fails to logs it to the schema_version table due to NULL on installed_on. Any help will be greatly appreciated. Thanks in advance. !
In my case the error was that the database table flyway_schema_history had column installed_on defined like DATETIME NOT NULL while it should have been DATETIME DEFAULT GETDATE() NOT NULL.
The issue was resolved when I manually altered the column to include the default value definition.
My company has an number of databases which were created over a period of last 3 years, and i have noticed that the oldest and the youngest of them have the column set properly, while the ones from around 1.5 years have the column defined without the default. Perhaps it was a bug in some older versions of Flyway?
I wanted to change the data type of one field from string to date. So i dropped the table in db. Then modified the liquibase file and ran the application. now it complains with the following message.
liquibase.exception.ValidationFailedException: Validation Failed:
So after that I reverted the liquibase file changes and ran the application. This time no error but it is not creating the table.
Please help me how to solve this issue.
I assume the failed validation was an error about checksums. This happens when you modify a changeset which was already executed and try to execute it again.
Liquibase keeps all executed changesets in a table called databasechangelog, so it can find out which changesets can be skipped during execution.
To execute a changeset again, delete the corresponding from this table before, and run Liquibase again.
When using Liquibase, you shouldn't (in general) modify the database outside of Liquibase - the main exception being if you are a developer working on your own private development database. If you are in that state (working on your own private database), then when you modify the database outside of Liquibase (i.e. dropping a table) you will also need to delete the row in the DATABASECHANGELOG table that corresponds to the table create statement so that when you re-run liquibase update it will re-create the table.
When a changeset is marked as failOnError:false, does liquibase record it as having been applied when it fails?
For example, we have a script that performs a pre-emptive drop table in one changeset and then creates the table in the next changeset. When the script is first run, the drop table statement fails as expected and then the table is created successfully. However, the changeset that attempted the drop table is not added to the databasechangelog table.
Is that expected behavior?
That is the current behavior currently. Depending on the reason for the failure, it can make sense to either continue to retry it or to not.
I created https://liquibase.jira.com/browse/CORE-1766 to add the feature to mark it as failed and not try again.
Currently, your best option would be to add a precondition to the dropTable changeSet with onFail="MARK_RAN"
I am trying to modify the data type on a number of columns from int to bigint using:
<modifyDataType tableName="ACCESS_HISTORY" columnName="ID" newDataType="${LongType}"/>
Where I have defined LongType to be:
<property name="LongType" value="bigint" dbms="mssql"/>
I noticed in the Liquibase JIRA (https://liquibase.jira.com/browse/CORE-1062) that there is a known issue from 2011 relating to the PKs and FKs that need to be dropped. Dropping and re-creating the keys isn't really an option for me.
The error I got was:
liquibase.exception.MigrationFailedExc
eption: Migration failed for change set ChangeColumnTypes.xml::4-4-060
-2::thof:
Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TA
BLE [dbo].[ACCESS_HISTORY] ALTER COLUMN [ID] BIGINT: The object 'PK_Access_Histo
ry' is dependent on column 'ID'.:
Caused By: Error executing SQL ALTER TABLE [dbo].[ACCESS_HISTORY] ALTE
R COLUMN [ID] BIGINT: The object 'PK_Access_History' is dependent on column 'ID'
I was wondering if anyone had got around this? Thanks in advance
I'm afraid to say that there is no way to currently do this unless you drop the PK constraint. It looks like the bug you're talking about though is scheduled to be fixed in version 3 of liquibase, however there is no release date for this version.
Sorry!
As Liquibase is open source you could always fix the bug yourself. :)