I’m using Liquibase via the Gradle-Liquibase (v 1.1.1) plugin. I have the following changeset …
<changeSet id="create_my_stored_proc" author="davea" dbms="mysql" runAlways="true">
<sqlFile endDelimiter="//" path="src/main/resources/scripts/create_my_stored_proc.sql" stripComments="true"/>
</changeSet>
Is it possible to set something such that checksums are ignored for this changeset only? The underlying procedure is in a state of flux that could be repeatedly updated and rather than create a new changeset each time, I would like the existing one to run upon every Liquibase build.
You can disable the checks per changeSet using the <validCheckSum> tag with known good values.
For example, if the previous changeset had a checksum like 8:b3d6a29ce3a75940858cd093501151d1 and you wanted to tweak that changeSet (but not re-apply it where this step has already succeeded) then you could use something like this:
<changeSet author="me" id="mychangeset">
<validCheckSum>8:b3d6a29ce3a75940858cd093501151d1</validCheckSum>
<sqlFile ... />
</changeSet>
"RunAlways will still throw a checksum error by default, but you can always use runOnChange=true or any to change that."
Have a look at this ticket raised in liquibase: https://liquibase.jira.com/browse/CORE-2506
So, you could do:
<changeSet id="create_my_stored_proc" author="davea" dbms="mysql" runAlways="true">
<validCheckSum>any</validCheckSum>
<sqlFile endDelimiter="//" path="src/main/resources/scripts/create_my_stored_proc.sql" stripComments="true"/>
</changeSet>
You can add the runAlways and/or runOnChange attributes.
Related
Is it possible to generate change logs, merging change sets together?
Like this:
<changeSet author="Author" id="1">
<dropNotNullConstraint columnDataType="varchar(255)" columnName="name" tableName="table"/>
<dropNotNullConstraint columnDataType="varchar(255)" columnName="phone" tableName="table"/>
<dropNotNullConstraint columnDataType="bigint" columnName="email" tableName="table"/>
</changeSet>
instead of this:
<changeSet author="Author" id="1">
<dropNotNullConstraint columnDataType="varchar(255)" columnName="name" tableName="table"/>
</changeSet>
<changeSet author="Author" id="2">
<dropNotNullConstraint columnDataType="varchar(255)" columnName="phone" tableName="table"/>
</changeSet>
<changeSet author="Author" id="3">
<dropNotNullConstraint columnDataType="bigint" columnName="email" tableName="table"/>
</changeSet>
If so, can you choose a custom scope? (e.g. all changes in a given table are grouped together, etc.). If not, is there a 'best practice' reason for avoiding grouping them manually?
If you mean command generateChangeLog then no, there is no option to put all changes in one changeLog. However you can create that programatically if you need.
Liquibase should changes per changeSet because each changeSet is executed in one transaction read here. So if you include a lot of changes inside one changeSet and execution of that changeSet fails, it would be harder for you to find on which change it failed. It's better to handle change per changeSet.
note: sometimes I'm also using multiple changes per changeSet (for example remarks (comments) for table or DML (data) modifications) but I prefer to have one DDL per changeSet.
I want to update very old changeset because there is a bug in that. I have updated that changesetid and After running it I got checksum error i.e checksum got modified. Now I have added validCheckSum Tag with this new checkSum. After it is getting successfully I cant not see new changes. Is there any way to update that change set which has already been executed.
Each changeSet tag is uniquely identified by the combination of the “id” tag, the “author” tag so please check whether id and author name already exists or not in DATABASECHANGELOG table. If the combination exists then change the id in your xml file then run the build
If still not works then add runOnChange property in <ChangeSet> and try it
runOnChange - Executes if any changes found it already existing changeset
Example of using runOnChange in changeset with sqlFile tag .
I had a script for Oracle 18 using IDENTITY column type and than have to change it for Oracle 10 which doesn't support IDENTITY column type
<changeSet id="2" author="auto" runOnChange="true">
<sqlFile dbms="oracle" encoding="utf8" endDelimiter=";" path="oracle/230.sql" relativeToChangelogFile="true" splitStatements="true" stripComments="true" />
</changeSet>
If it is a very old changeset as you have stated you should consider not changing the old changeset but add a new changeset instead which fixes the bug since it allows you better control over the versions you have. This is one of the main reasons of using liquibase.
I have a changeset to create an index.
<changeSet author="hilland" id="x-NC-U-y" runOnChange="true">
<createIndex indexName="NC-U-y" tableName="x" unique="true">
<column name="y"/>
</createIndex>
<rollback>
<dropIndex .../>
</rollback>
<modifySql>
<append value=" INCLUDE ( [a],[b]) WITH (DATA_COMPRESSION=page)"/>
</modifySql>
</changeSet>
The problem is that an older version of the index might exist (it will on some targets (dev server, with an older version of this index; no includes), will not on others (eg a fresh deploy to an empty database).
Is there a better way to resolve this so that the script becomes universal, than to include a prefix which says
<sql>IF select {sys.indexes.stuff} is not null {drop the index}</sql>?
Ideally there would be a precondition to drop-if-exists the existing index so that it can be recreated, but i don't think that can be done.
What is the best way to handle this situation?
an alternate solution i thought of would be to have a separate changest "if index v1 exists, drop it" then have a separate v2 changeset, but then that would make rollback hard.
I realize that the run on change condition for the changeset makes rollback hard, but the only rollback i'm concerned with in this case is to and from an empty database, although a pattern which easily accommodates rollbacks to previous version would be welcome also.
You can execute changesets dependent on a precondition. Your changeset could look like this:
<changeSet>
<preConditions onFail="MARK_RAN">
<indexExists indexName="NC-U-y" />
</preConditions>
<dropIndex indexName="NC-U-y" />
</changeSet>
See also the Liquibase Documentation.
Testers are updating data through the app
I am working with an application where the app is changing rapidly, but at the same time, the testers need to build test and certification data. This data is being created by accessing the app directly, rather than writing SQL statements.
So, I have changesets coming in from the developers and data changes being applied through the application by testers at the same time.
I have wired up liquibase to handle running the changesets written by the developers, but I'm having difficulties figuring out the cleanest way to track and preserve data changes by the testers.
Possible workflow
Based on using liquibase through the entire process, I'm thinking I need a workflow like:
Start with the latest clean database
Run liquibase update
Snapshot or tag the database for differencing later
Let the testers hack away through the app.
If the testers approve the changes, upon promotion:
generate a data diff as a changeset
include the data changeset in the master update list
on this VM, record the changeset as already ran
commit to scm
Repeat
Questions
Is there a way to use liquibase to get a true data diff, and not a full data export? It seems that generateChangeLog is the only tool in the diffs that allows setting the --diff-type="data" flag and value, but the documentation also makes that seem that it won't diff, it just dumps all of the data.
If yes,
can you provide a sample call? I have the url & referenceURL figured out and stored in a liquibase.properties file, I just need to know which command and flags to pass.
can it be used against a tag instead of having to create a backup of the database? (step 3 in the workflow)
If no,
has anyone seen a good tutorial or howto showing the orchestration between liquibase and dbUnit updates?
How do you handle the situation where the data export no longer fits the schema? For example, FullName split => FirstName and LastName; liquibase can handle this, but I would think that I would need to orchestrate running updates between liquibase and dbUnit, otherwise the diff of dbUnit will be invalid?
Any guidance, tips, past experiences or gotchas to watch out for would be greatly appreciated.
No, liquibase does not support a data diff, only a full data dump.
I have seen http://ljnelson.github.io/liquiunit/ which may help you with dbunit integration, but using dbunit or any other data load tool to manage your data will run into schema incompatibilities like you suggest.
What I would suggest doing is to have "test data load" changeSets that are added into your changeLog to build up your test data as you go along.
For example:
<changeSet id="1" author="x">
<createTable name="a"../>
</changeSet>
<changeSet id="2" author="x">
<createTable name="b"../>
</changeSet>
<changeSet id="3" author="x" context="test">
<sqlFile path="data-dump.1.sql">
</changeSet>
<changeSet id="4" author="x">
<renameColumn oldColumnName="s" newColumnName="t"../>
</changeSet>
<changeSet id="5" author="x" context="test">
<sqlFile path="data-dump.2.sql">
</changeSet>
You see it creates an initial structure and then loads a round of QA's data into the database with the database structure as it is after changeset 2. Notice the use of contexts so the test data isn't loaded into production.
After the test data are more structure changes and then another round of additional QA data. The new data doesn't re-create data-dump.1.sql but is in addition to it. Since data-dump.1.sql is always ran before changeSet 4, it doesn't have to be updated as the schema changes.
The big problem, though, is how to extract your test data as QA is building it up. If they are adding it through your application, the easiest approach may be to use something like p6spy to automatically collect all the SQL executed in your application and then just copy it into your data-dump.X.sql files.
As an alternative to my other answer, you can have a full dump of your database (use liquibase generateChangeLog diffTypes=data or your standard database backup tool) which you create from the QA-built database and move along your changelog file.
Step 1:
Create your database
<changeSet id="1" author="x">
<createTable name="a"../>
</changeSet>
<changeSet id="2" author="x">
<createTable name="b"../>
</changeSet>
Step 2
QA creates database then makes a backup which is included in the changelog file.
<changeSet id="1" author="x">
<createTable name="a"../>
</changeSet>
<changeSet id="2" author="x">
<createTable name="b"../>
</changeSet>
<changeSet id="testdata-1" author="x" context="test">
<sqlFile path="data-dump.sql">
</changeSet>
Step 3
There are schema changes that don't require new test data, keep adding to the changelog file. The test data is migrated along and matches your needed schema.
<changeSet id="1" author="x">
<createTable name="a"../>
</changeSet>
<changeSet id="2" author="x">
<createTable name="b"../>
</changeSet>
<changeSet id="testdata-1" author="x" context="test">
<sqlFile path="data-dump.sql">
</changeSet>
<changeSet id="4" author="x">
<renameColumn oldColumnName="s" newColumnName="t"../>
</changeSet>
Step 4
When QA needs additional data, they add what they want using the app then make a new full backup and move the testdata changeset to the end
<changeSet id="1" author="x">
<createTable name="a"../>
</changeSet>
<changeSet id="2" author="x">
<createTable name="b"../>
</changeSet>
<changeSet id="4" author="x">
<renameColumn oldColumnName="s" newColumnName="t"../>
</changeSet>
<changeSet id="testdata-1" author="x" context="test">
<sqlFile path="data-dump.sql">
</changeSet>
This process does require you to drop your dev/test databases to get the new test data or you will get insert conflicts, but depending on your workflow it may work for you.
I couldn't immediately see, in the current documentation I am looking at:
Will the following work in Liquibase, and use the included SQL file for the rollback?
<changeSet author="username" id="85138">
<sql splitStatements="false">
UPDATE some_table ...;
</sql>
<rollback>
<include file="path/to/rollback.sql"/>
</rollback>
</changeSet>
If I've missed the appropriate spot in the documentation, feel free to point out the relevant section.
What you can do is use sqlFile:
<changeset ..>
<sql>...</sql>
<rollback>
<sqlFile path="rollback.sql" />
</rollback>
</changeset>
As mentioned by another post, include is used for changelog composition only - to include other changelogs.
No, that will not work. The include tag is intended for including further changelog XML files, not for including arbitrary information.
It's documented here: http://www.liquibase.org/manual/include
It doesn't say you "can't" specifically, but I tried it - and liquibase will yell at you for it. =)