Liquibase - Error on SYSTEM.DATABASECHANGELOGLOCK while re-executing a migration - liquibase

I'm using Liquibase 3.0.2, Ant task updateDatabase and change sets defined directly inside SQL scripts using comments like
--liquibase formatted sql
--changeset com.noemalife:1 dbms:oracle
etc.
The first run works fine, all change sets are executed and DB objects (oracle) are deployed. I can see DATABASECHANGELOG and DATABASECHANGELOGLOCK tables filled up.
Then I try to re-run the Ant task with the same exact configuration, expecting Liquibase to say something like "Ok, all is already deployed, nothig to do here."
But I get this instead:
C:\Users\dmusiani\Desktop\liquibase-test>ant migrate
Buildfile: build.xml
migrate:
[copy] Copying 1 file to C:\Users\dmusiani\Desktop\liquibase-test
BUILD FAILED
liquibase.exception.LockException: liquibase.exception.DatabaseException: Error executing SQL CREATE
TABLE SYSTEM.DATABASECHANGELOGLOCK (ID INTEGER NOT NULL, LOCKED NUMBER(1) NOT NULL, LOCKGRANTED TIM
ESTAMP, LOCKEDBY VARCHAR2(255), CONSTRAINT PK_DATABASECHANGELOGLOCK PRIMARY KEY (ID)); on jdbc:oracl
e:thin:#localhost:1521:WBMDINSERT INTO SYSTEM.DATABASECHANGELOGLOCK (ID, LOCKED) VALUES (1, 0): ORA-
00955: nome giÓ utilizzato da un oggetto esistente
at liquibase.lockservice.LockServiceImpl.acquireLock(LockServiceImpl.java:122)
at liquibase.lockservice.LockServiceImpl.waitForLock(LockServiceImpl.java:62)
at liquibase.Liquibase.update(Liquibase.java:123)
at liquibase.integration.ant.DatabaseUpdateTask.executeWithLiquibaseClassloader(DatabaseUpda
teTask.java:45)
at liquibase.integration.ant.BaseLiquibaseTask.execute(BaseLiquibaseTask.java:70)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:357)
at org.apache.tools.ant.Target.performTasks(Target.java:385)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
at org.apache.tools.ant.Main.runBuild(Main.java:758)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
Caused by: liquibase.exception.DatabaseException: Error executing SQL CREATE TABLE SYSTEM.DATABASECH
ANGELOGLOCK (ID INTEGER NOT NULL, LOCKED NUMBER(1) NOT NULL, LOCKGRANTED TIMESTAMP, LOCKEDBY VARCHAR
2(255), CONSTRAINT PK_DATABASECHANGELOGLOCK PRIMARY KEY (ID)); on jdbc:oracle:thin:#localhost:1521:W
BMDINSERT INTO SYSTEM.DATABASECHANGELOGLOCK (ID, LOCKED) VALUES (1, 0): ORA-00955: nome giÓ utilizza
to da un oggetto esistente
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:56)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:98)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:64)
at liquibase.database.AbstractJdbcDatabase.checkDatabaseChangeLogLockTable(AbstractJdbcDatab
ase.java:771)
at liquibase.lockservice.LockServiceImpl.acquireLock(LockServiceImpl.java:95)
... 21 more
Caused by: java.sql.SQLException: ORA-00955: nome giÓ utilizzato da un oggetto esistente
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:113)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:754)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:210)
at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:963)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1192)
at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1731)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1701)
at liquibase.executor.jvm.JdbcExecutor$1ExecuteStatementCallback.doInStatement(JdbcExecutor.
java:86)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:49)
... 25 more
Total time: 1 second
C:\Users\dmusiani\Desktop\liquibase-test>
It seems to me that Liquibase is trying to re-create the DATABASECHANGELOGLOCK table.
I have this problem when I run Liquibase with the Oracle "system" user (my patch cares about creating a couple of other users, thus for testing purposes I use system directly to do that).
The other strange thing is that after the system's patch run succesfully, in the lock table I can still see the lock is active.
When I run other patches in other schemas(ex. the ones created by the system's patch), I have the patch completing successfully and the lock released in the lock table; relaunching that patch behaves as expected: Liquibase detects the patch is already in place ad does nothing.
This said, now my doubt is if Liquibase has problems, in the system schema, in detecting the lock table is already existing (and fails trying to deploy it) or if there is some kind of locking/commit problem.
Any suggestion is welcome
Thanks
Davide

I'm facing the same issue as you.
From what I see from the sources, when running as SYSTEM, the following condition (DatabaseSnapshot#include) is evaluated to true.
if (database.isSystemObject(example)) {
return null;
}
Because of that, the creation will always be attempted.
I'm gonna work further on a patch and keep you updated.
And here is a patch proposal.

Related

DBT RUN - Getting Database Error using VS Code BUT Not Getting Database Error using DBT Cloud

I'm using DBT connected to Snowflake. I use DBT Cloud, but we are moving to using VS Code for our DBT project work.
I have an incremental DBT model that compiles and works without error when I issue the DBT RUN command in DBT Cloud. Yet when I attempt to run the exact same model from the same git branch using the DBT RUN command from the terminal in VS Code I get the following error:
Database Error in model dim_cifs (models\core_data_warehouse\dim_cifs.sql)
16:45:31 040050 (22000): SQL compilation error: cannot change column LOAN_MGMT_SYS from type VARCHAR(7) to VARCHAR(3) because reducing the byte-length of a varchar is not supported.
The table in Snowflake defines this column as VARCHAR(50). I have no idea why DBT is attempting to change the data length or why it only happens when the command is run from VS Code Terminal. There is no need to make this DDL change to the table.
When I view the compiled SQL in the Target folder there is nothing that indicates a DDL change.
When I look in the logs I find the following, but don't understand what is triggering the DDL change:
describe table "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS"
16:45:31.354314 [debug] [Thread-9 (]: SQL status: SUCCESS 36 in 0.09 seconds
16:45:31.378864 [debug] [Thread-9 (]:
In "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS":
Schema changed: True
Source columns not in target: []
Target columns not in source: []
New column types: [{'column_name': 'LOAN_MGMT_SYS', 'new_type': 'character varying(3)'}]
16:45:31.391828 [debug] [Thread-9 (]: Using snowflake connection "model.xxxxxxxxxx.dim_cifs"
16:45:31.391828 [debug] [Thread-9 (]: On model.xxxxxxxxxx.dim_cifs: /* {"app": "dbt", "dbt_version": "1.1.1", "profile_name": "xxxxxxxxxx", "target_name": "dev", "node_id": "model.xxxxxxxxxx.dim_cifs"} */
alter table "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS" alter "LOAN_MGMT_SYS" set data type character varying(3);
16:45:31.546962 [debug] [Thread-9 (]: Snowflake adapter: Snowflake query id: 01a5bc8d-0404-c9c1-0000-91b5178ac72a
16:45:31.548895 [debug] [Thread-9 (]: Snowflake adapter: Snowflake error: 040050 (22000): SQL compilation error: cannot change column LOAN_MGMT_SYS from type VARCHAR(7) to VARCHAR(3) because reducing the byte-length of a varchar is not supported.
Any help is greatly appreciated.

MongoDB: updateOne() duplicate key exception

I am trying to save a record in cosmos db using com.mongodb.client.MongoCollection.updateOne() with UPSERT flag "true". But I am getting duplicate key _id error and on retry the same object saves into db. I am unable to figure out the root cause of this error.
Below are the environmental details
Azure cosmos version 3.6
mongo driver version 2.1.6
Unique constraint on all index field is set to false
Code
mongoCollection.updateOne(filter, new Document("$set", doc), updateOptions.upsert(true));
Exception
E11000 duplicate key error collection: my-db.myCollection. Failed _id or unique index constraint.
com.mongodb.MongoWriteException:
at com.mongodb.client.internal.MongoCollectionImpl.executeSingleWriteRequest (MongoCollectionImpl.java967)
at com.mongodb.client.internal.MongoCollectionImpl.executeUpdate (MongoCollectionImpl.java951)
at com.mongodb.client.internal.MongoCollectionImpl.updateOne (MongoCollectionImpl.java613)
at com.xyz.util.myclass.myMethod (myClass.java162)
at com.xyz.util.myclass.myMethod (myClass.java73)
at com.xyz.process.myclass.myMethod (myClass.java135)
at com.xyz.process.myclass.myMethod (myClass.java87)
at com.xyz.process.myclass.myMethod (myClass.java51)
at com.xyz.springcloudflow.myclass.myMethod (myClass.java34)
at sun.reflect.GeneratedMethodAccessor129.invoke
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java43)
at java.lang.reflect.Method.invoke (Method.java498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke (InvocableHandlerMethod.java171)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke (InvocableHandlerMethod.java120)
at org.springframework.cloud.stream.binding.StreamListenerMessageHandler.handleRequestMessage (StreamListenerMessageHandler.java55)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal (AbstractReplyProducingMessageHandler.java123)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage (AbstractMessageHandler.java169)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch (AbstractDispatcher.java115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch (UnicastingDispatcher.java132)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch (UnicastingDispatcher.java105)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend (AbstractSubscribableChannel.java73)
at org.springframework.integration.channel.AbstractMessageChannel.send (AbstractMessageChannel.java453)
at org.springframework.integration.channel.AbstractMessageChannel.send (AbstractMessageChannel.java401)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend (GenericMessagingTemplate.java187)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend (GenericMessagingTemplate.java166)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend (GenericMessagingTemplate.java47)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send (AbstractMessageSendingTemplate.java109)
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage (MessageProducerSupport.java205)
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny (KafkaMessageDrivenChannelAdapter.java369)
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$400 (KafkaMessageDrivenChannelAdapter.java74)
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage (KafkaMessageDrivenChannelAdapter.java431)
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage (KafkaMessageDrivenChannelAdapter.java402)
at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambda$onMessage$0 (RetryingMessageListenerAdapter.java120)
at org.springframework.retry.support.RetryTemplate.doExecute (RetryTemplate.java287)
at org.springframework.retry.support.RetryTemplate.execute (RetryTemplate.java211)
at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage (RetryingMessageListenerAdapter.java114)
at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage (RetryingMessageListenerAdapter.java40)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage (KafkaMessageListenerContainer.java1275)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage (KafkaMessageListenerContainer.java1258)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener (KafkaMessageListenerContainer.java1219)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords (KafkaMessageListenerContainer.java1200)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener (KafkaMessageListenerContainer.java1120)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener (KafkaMessageListenerContainer.java935)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke (KafkaMessageListenerContainer.java751)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run (KafkaMessageListenerContainer.java700)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java511)
at java.util.concurrent.FutureTask.run (FutureTask.java266)
at java.lang.Thread.run (Thread.java748)

Junit HSQLDB - user lacks privilege or object not found - THIS_.oh-ordnbr

I am getting an exception when my column name contains hyphen "-"
Entity : this is the entity name.
#Entity
#Table(name = "RequestHeader")
public class RequestHeader implements Serializable {
....
....
#Column(name = "`oh-ordnbr`")
private Integer ohOrdnbr;
Schema definition: This is the query for schema creation.
CREATE MEMORY TABLE PUB.REQUESTHEADER(
REQUESTID INTEGER,
IMUSERID INTEGER,
REQUESTDATE DATE,
REQUESTTIME INTEGER,
REQUESTSTATUS VARCHAR(19),
REQUESTTYPE VARCHAR(22),
HEADERINSTRUCTIONS VARCHAR(5150),
DATEFORMAT VARCHAR(20),
TIMEFORMAT VARCHAR(20),
LANGUAGEID INTEGER,
"OH-ORDNBR" INTEGER,
"OH-TRCNSTAMP" INTEGER,
ISPICKUPLIST BIT(1),
CONSTRAINT "RQH-1" PRIMARY KEY(REQUESTID)
);
The error is below:
Exception Stack: Error message which I have received by running the Junit.
Caused by: org.hsqldb.HsqlException: user lacks privilege or object not found: THIS_.oh-ordnbr
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.ExpressionColumn.checkColumnsResolved(Unknown Source)
at org.hsqldb.QueryExpression.resolve(Unknown Source)
at org.hsqldb.ParserDQL.compileCursorSpecification(Unknown Source)
at org.hsqldb.ParserCommand.compilePart(Unknown Source)
at org.hsqldb.ParserCommand.compileStatement(Unknown Source)
at org.hsqldb.Session.compileStatement(Unknown Source)
at org.hsqldb.StatementManager.compile(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
Could some one help me in fixing this?
The reason for the object not found error is the fact that the oh-ordnbr column is defined to be case sensitive (this is due to the double quotes you put around it).
You have two possible solutions:
Do not use dashes (hyphens) in your SQL. It is bad practice anyway, use underscores instead.
Update the JPA annotation as follows:
#Column(name = "`OH-ORDNBR`")
private Integer ohOrdnbr;
I strongly recommend using underscores instead of dashes, you never know what weirdness different JPA implementations might have when using the second solution.
Check your configuration file has correct provider, I fixed same issue by providing org.hibernate.ejb.HibernatePersistence provider.
If HSQL Scripts are run via Java and if any new table query is followed by select/alter/update queries it always throws "user lacks privilege object". In reality the create queries after execution with Java Statements never gets commited into the DB and it is not persisted and we run our select/alter/update and it results in the exception. If we run the queries line by line and commit for each line, we could avoid this error.

Failing jdbc INSERT INTO statement to MS Access database

statement.executeUpdate("INSERT INTO countrylookup (Country, DialCode) VALUES('Iran', '957')")
Running this statement gives me no error output in the console, but when I check the database no update/insert is made. What could be the reason for this?
The access to the database itself is successful, and fetching values with a statement such as SELECT * FROM countrylookup succeeds.
I tried the preparedStatement approach aswell with the exact same result. The file is not open when I execute the command.
UPDATE: Stacktrace: (first row in Swedish means "INSERT INTO-expression contains the following unknown fieldname: 'Pa_RaM000'. Please check that the name is rightly spelled and try again.)
Exception in thread "main" java.sql.SQLException: [Microsoft][Drivrutin f?r ODBC Microsoft Access] INSERT INTO-uttrycket inneh?ller f?ljande ok?nda f?ltnamn: 'Pa_RaM000'. Kontrollera att namnet ?r r?ttstavat och f?rs?k igen.
at sun.jdbc.odbc.JdbcOdbc.createSQLException(Unknown Source)
at sun.jdbc.odbc.JdbcOdbc.standardError(Unknown Source)
at sun.jdbc.odbc.JdbcOdbc.SQLExecute(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcPreparedStatement.execute(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcPreparedStatement.executeUpdate(Unknown Source)
at MDBAccessor.insertValueIntoField(MDBAccessor.java:43)
at TestRunner.main(TestRunner.java:28)
Is dialcode numeric? If so, remove the quotes from the value.
VALUES('Iran', 957)
In order for the INSERT INTO statement to actually be reflected in the database you have to call connection.commit() followed by connection.close(). Another similar thread describing this: Java General Error On Insert...???

Trouble running liquibase with different agent

I need to execute the same db-changelog by ant and then by spring. I hope that ant will run the changelog and when spring run, it will not do anything and just stop normally. Ant run the db-changelog successfully and then spring run but it throws an exception, part of the stack trace :
Reason: liquibase.exception.JDBCException: Error executing SQL CREATE TABLE action (action_id int8 NOT NULL, action_name VARCHAR(255), version_no int8, reason_required BOOLEAN, comment_required BOOLEAN, step_id int8, CONSTRAINT action_pkey PRIMARY KEY (action_id)):
Caused By: Error executing SQL CREATE TABLE action (action_id int8 NOT NULL, action_name VARCHAR(255), version_no int8, reason_required BOOLEAN, comment_required BOOLEAN, step_id int8, CONSTRAINT action_pkey PRIMARY KEY (action_id)):
Caused By: ERROR: relation "action" already exists; nested exception is org.springframework.beans.factory.BeanCreationException....
Any help will much appreciated.
Regards,
It does sound like it is trying to run the changelog again. Each changeSet in the changeLog is identified by a combination of the id, author, and the changelog path/filename. If you run "select * from databasechangelog" you can see the values used.
Your problem may be that you are referencing the changelog file differently from ant and spring therefore generating different filename values. Usually you will want to include them in the classpath so no matter where and how you run them they have the same path (like "com/example/db.changelog.xml")
I ran into this same problem and was able to fix it by altering the filename column of DATABASECHANGELOG to reference the spring resource path. In my case, I was using a ServletContextResource under the WEB-INF directory:
update DATABASECHANGELOG set FILENAME = 'WEB-INF/path/to/changelog.xml' where FILENAME = 'changelog.xml'