I'm trying to run liquibase to generate a changelog of my current database, but I get the following error and not sure how to get around it.
Unexpected error running Liquibase: liquibase.exception.DatabaseException: Connection could not be created to jdbc:postgres://kbjoxiifqvunac:ef1f6ba0b9633d5951c6f4a28897fe390fdf2a7f4189320569d303531899e843#ec2-54-217-235-159.eu-west-1.compute.amazonaws.com:5432/d1pdhars1kb6ep with driver org.postgresql.Driver. Possibly the wrong driver for the given database URL
liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Connection could not be created to jdbc:postgres://kbjoxiifqvunac:ef1f6ba0b9633d5951c6f4a28897fe390fdf2a7f4189320569d303531899e843#ec2-54-217-235-159.eu-west-1.compute.amazonaws.com:5432/d1pdhars1kb6ep with driver org.postgresql.Driver. Possibly the wrong driver for the given database URL
at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:135) ~[liquibase.jar:na]
at liquibase.integration.commandline.Main.doMigration(Main.java:977) [liquibase.jar:na]
at liquibase.integration.commandline.Main.run(Main.java:209) [liquibase.jar:na]
at liquibase.integration.commandline.Main.main(Main.java:132) [liquibase.jar:na]
Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Connection could not be created to jdbc:postgres://kbjoxiifqvunac:ef1f6ba0b9633d5951c6f4a28897fe390fdf2a7f4189320569d303531899e843#ec2-54-217-235-159.eu-west-1.compute.amazonaws.com:5432/d1pdhars1kb6ep with driver org.postgresql.Driver. Possibly the wrong driver for the given database URL
at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:259) ~[liquibase.jar:na]
at liquibase.database.DatabaseFactory.openDatabase(DatabaseFactory.java:149) ~[liquibase.jar:na]
at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:100) ~[liquibase.jar:na]
... 3 common frames omitted
Caused by: liquibase.exception.DatabaseException: Connection could not be created to jdbc:postgres://kbjoxiifqvunac:ef1f6ba0b9633d5951c6f4a28897fe390fdf2a7f4189320569d303531899e843#ec2-54-217-235-159.eu-west-1.compute.amazonaws.com:5432/d1pdhars1kb6ep with driver org.postgresql.Driver. Possibly the wrong driver for the given database URL
at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:254) ~[liquibase.jar:na]
... 5 common frames omitted
I have run the following command
liquibase --driver=org.postgresql.Driver --changeLogFile=changelog.xml --url=jdbc:postgres://user:password#ec2-54-217-235-159.eu-west-1.compute.amazonaws.com:5432/dbhost generateChangeLog
I have omitted the classpath flag since I put the postgres drivers inside the lib folder of my liquibase, and per documentation it should scan it.
I'm running on windows, liquibase version 3.8.0 and postgres-drivers 42.1.1. Any ideas? Thanks in advance for any help
I think your driver jar should be in the lib directory rather than the bin directory. If you look at these lines from liquibase.bat you can see that it adds all jar files in the bin directory that start with liquibase, and then ALL jar files that are in the lib directory.
set CP=.
for /R %LIQUIBASE_HOME% %%f in (liquibase*.jar) do set CP=!CP!;%%f
for /R %LIQUIBASE_HOME%\lib %%f in (*.jar) do set CP=!CP!;%%f
Related
We write our custom changes in Kotlin. But we cannot execute these custom changes using the Liquibase CLI, because the kotlin-stdlib is missing from the classpath.
Here's my liquibase.properties file:
changelog-file:db/changelog.yaml
driver:org.postgresql.Driver
url:jdbc:postgresql://localhost:5432/project
username:postgres
password:postgres
classpath:build/classes/kotlin/main:src/main/resources
includeSystemClasspath:true
In src/main/resources we keep the changelog.yaml and the changesets, while in build/classes/kotlin/main we have the custom changes written in Kotlin.
Running liquibase update --log-level=debug return the following output:
[...]
Caused by: java.lang.ClassNotFoundException: kotlin.jvm.internal.Intrinsics
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 70 more
Caused by: liquibase.exception.ValidationFailedException: Validation Failed:
2 changes have validation errors
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.UnexpectedLiquibaseException: java.lang.reflect.InvocationTargetException
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CustomChangeException: liquibase.exception.UnexpectedLiquibaseException: java.lang.reflect.InvocationTargetException
If I hard code the path to the kotlin-stdlib in the liquibase.properties file, then gradle update works fine.
classpath:build/classes/kotlin/main:src/main/resources:/Home/user/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.6.21/11ef67f1900634fd951bad28c53ec957fabbe5b8/kotlin-stdlib-1.6.21.jar
But we cannot hard code it in liquibase.properties as the path to the kotlin-stdlib would be different for each machine. I was expecting that exporting it in the system CLASSPATH would solve this issue:
export CLASSPATH=/Home/user/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.6.21
This doesn't seem to work, even when we specified includeSystemClasspath:true in the liquibase.properties file.
Using the LIQUIBASE_CLASSPATH env variable requires a Liquibase Pro or Labs license.
Liquibase Community detected and ignored the following environment variables:
- LIQUIBASE_CLASSPATH
To configure Liquibase with environment variables requires a Liquibase Pro or Liquibase Labs license. Get a free trial at https://liquibase.com/trial. Options include the liquibase.licenseKey in the defaults file, adding a flag in the CLI, and more. Learn more at https://docs.liquibase.com.
Been using Liquibase a long time, and love it. However, on the project I'm currently working on, I'm encountering an issue I don't seem to be able to find the cause.
I can run every changelog/feature/*.xml file from the command line (using update) and everything succeeds.
I can run every db.changelog-*.xml file (except master) from the command line (using update) and everything succeeds.
But when I try and run the db.changelog-master file, I get:
Starting Liquibase at Tue, 14 Apr 2020 11:49:47 CDT (version 3.8.0 built at 2019-08-15T20:38:06Z)
Liquibase Community 3.8.0 by Datical
Unexpected error running Liquibase: Unknown reason
liquibase.exception.ChangeLogParseException: liquibase.exception.SetupException
at liquibase.parser.core.xml.AbstractChangeLogParser.parse(AbstractChangeLogParser.java:25) ~[liquibase.jar:na]
at liquibase.Liquibase.getDatabaseChangeLog(Liquibase.java:217) ~[liquibase.jar:na]
at liquibase.Liquibase.update(Liquibase.java:190) ~[liquibase.jar:na]
at liquibase.Liquibase.update(Liquibase.java:179) ~[liquibase.jar:na]
at liquibase.integration.commandline.Main.doMigration(Main.java:1223) [liquibase.jar:na]
at liquibase.integration.commandline.Main.run(Main.java:209) [liquibase.jar:na]
at liquibase.integration.commandline.Main.main(Main.java:132) [liquibase.jar:na]
Caused by: liquibase.exception.SetupException: null
at liquibase.changelog.DatabaseChangeLog.includeAll(DatabaseChangeLog.java:531) ~[liquibase.jar:na]
at liquibase.changelog.DatabaseChangeLog.handleChildNode(DatabaseChangeLog.java:398) ~[liquibase.jar:na]
at liquibase.changelog.DatabaseChangeLog.load(DatabaseChangeLog.java:308) ~[liquibase.jar:na]
at liquibase.parser.core.xml.AbstractChangeLogParser.parse(AbstractChangeLogParser.java:23) ~[liquibase.jar:na]
... 6 common frames omitted
Caused by: java.lang.NullPointerException: null
at java.util.AbstractCollection.addAll(AbstractCollection.java:343) ~[na:1.8.0_242]
at liquibase.integration.commandline.CommandLineResourceAccessor.list(CommandLineResourceAccessor.java:44) ~[liquibase.jar:na]
at liquibase.resource.CompositeResourceAccessor.list(CompositeResourceAccessor.java:40) ~[liquibase.jar:na]
at liquibase.changelog.DatabaseChangeLog.includeAll(DatabaseChangeLog.java:506) ~[liquibase.jar:na]
... 9 common frames omitted
For more information, please use the --logLevel flag
I did find an old view and trigger that were created during research for the project for a different schema, which were never run in this one (neither was in this DATABASECHANGELOG). They've been deleted, but the problem still exists.
Running:
macOS Catalina 10.15.3
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (Zulu 8.44.0.11-CA-macosx) (build
1.8.0_242-b20)
OpenJDK 64-Bit Server VM (Zulu 8.44.0.11-CA-macosx) (build
25.242-b20, mixed mode)
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0
64bit Production PL/SQL Release 11.2.0.2.0 Production
CORE 11.2.0.2.0 Production
TNS for Linux: Version 11.2.0.2.0 Production
NLSRTL Version 11.2.0.2.0 Production
Liquibase 3.8.0
Tried everything I can think of. Anyone have any thoughts?
**UPDATE APR-29-2020**
I have a little more information - I've discovered WHAT is happening, and the plan to fix this isn't working.
The what:
Apparently Liqiubase changed the includeAll path somewhere between version 3.4.2 and 3.5.5:
See 'Upgrade from 3.4.2 to 3.5.5 or higher'
I'd copied our liquibase setup from a legacy system running 3.4.2. In the db.changelog-master.xml file, it has <includeAll path="/features"/> (leading slash). However, on the new project, I upgraded to 3.8.0. And the includeAll is what's throwing the exception.
The new value is <includeAll path="features/"/> (trailing slash). The format for the FILENAME column in DATABASECHANGELOG is the same, i.e. features/20191024-LD-1.xml
In the documentation, there's a command line maintenance command that looks perfectly designed for this fix: cleanCheckSums. Per the website:
Removes current checksums from database. On next update changesets
that have already been deployed will have their checksums recomputed,
and changesets that have not been deployed will be deployed.
Challenge is that yes, cleanCheckSums nulls the MD5SUM column, the EXECTYPE column is EXECUTED, and each row has a populated DEPLOYMENT_ID, but the next update command tries to re-run all of the changesets.
Here's the plan we created for the fix:
Steps
Change the includeAll parameter path in db.changelog-master.xml: <includeAll path="features/"/>
Run liquidate clearCheckSums:
$LIQUIBASE_HOME/liquibase \ --driver=oracle.jdbc.OracleDriver \ --classpath=/data/apps/liquibase-3.8.0/ojdbc7.jar \ --url={pathToDatabase} \ --username={dbUserName} \ --password={dbUserPass} \ clearCheckSums
SQL to fix the database column filename to the expected format
Appears to be exactly the same format:
old: features/20191024-LD-1.xml
new test: features/20200429-deleteme.xml
Run master update:
$LIQUIBASE_HOME/liquibase \ --driver=oracle.jdbc.OracleDriver \ --classpath=/data/apps/liquibase-3.8.0/ojdbc7.jar \ --url={pathToDatabase} \ --changeLogFile=db.changelog-master.xml \ --username={dbUserName} \ --password={dbUserPass} \ update
Anyone know what I'm missing?
Please try the following command:
liquibase changelogSync
This command will create corresponding rows to the db.changelog-master.xml changeSets in the DATABASECHANGELOG tracking table and consider those changeSets as deployed.
I added a sql file to my project and now I am receiving the following error:
nested exception is org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 1.0.9 (update)
Here is my sql file I'm adding:
ALTER TABLE `episodes`
ADD COLUMN `rating` TINYINT(1) NULL DEFAULT NULL;
I added the same query to MySQL Workbench and it's working fine, so I think the error here is beyond the sql file added
UPDATE: full stack trace below
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 1.0.9 (update)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1708)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:581)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:503)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:304)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1089)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:859)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
at com.nbcuni.cds.Application.main(Application.java:12)
Caused by: org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 1.0.9 (update)
at org.flywaydb.core.Flyway.doValidate(Flyway.java:1286)
at org.flywaydb.core.Flyway.access$100(Flyway.java:71)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:1176)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:1168)
at org.flywaydb.core.Flyway.execute(Flyway.java:1655)
at org.flywaydb.core.Flyway.migrate(Flyway.java:1168)
at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:66)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1767)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1704)
... 18 common frames omitted
UPDATE: When I run mvn:flyway-validate, I receive the following error:
org.flywaydb.core.api.FlywayException: Unable to connect to the database. Configure the url, user and password!
I'm not sure where to configure it? It's already set in my applications.properties file? And w/o this version the spring application is working fine
I guess the problem is the error in migration history saved on your database.
Try deleting the line that failed and try again. If the error won't vanish, the problem is somehow in your statement.
I just had the same issue :)
below resolved the validate problem for me:
mvn flyway:validate -Dflyway.configFile=myFlywayConfig.properties
The error happen if there is any record is inserted into the 'flyway_schema_history' with failed status or part of the failed migration might already run. Just need to remove failed recored from the 'flyway_schema_history' then run "flyway migrate" again.
Query to flyway_schema_history table
select * from your_database_name.flyway_schema_history
You will get something like this:
Remove the record with success =0 and run again "flyway migrate".
Note, that this table might have a different name depending on the configuration
I am creating oozie workflow for hive create table command.
I have added hive-site.xml in hdfs location.
I am getting below error:-
Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, com/facebook/fb303/FacebookService$Iface
java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
at java.lang.ClassLoader.defineClass1(Native Method)
This might be because you are missing Thrift jar or version mismatch.
Refer the following
Error while executing program with Hive JDBC
I am trying to get a Spark/Shark cluster up but keep running into the same problem. I have followed the instructions on https://github.com/amplab/shark/wiki/Running-Shark-on-a-Cluster and addressed Hive as stated.
Here are the details, any help would be great.
I already installed the following package:
Spark/Shark 1.0.0
Apache Hadoop 2.4.0
Apache Hive 0.13
Scala 2.9.3
Java 7
I configure ~/spark/conf/spark-env.sh as:
export HADOOP_HOME=/path/to/hadoop/
export HIVE_HOME=/path/to/hive/
export MASTER=spark://xxx.xxx.xxx.xxx:7077
export SPARK_HOME=/path/to/spark
export SPARK_MEM=4g
export HIVE_CONF_DIR=/path/to/hive/conf/
source $SPARK_HOME/conf/spark-env.sh
When start spark with "./spark-withinfo", I get the following errors:
-hiveconf hive.root.logger=INFO,console
Starting the Shark Command Line Client
14/07/07 16:26:57 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead
14/07/07 16:26:57 [main]: WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead
Logging initialized using configuration in jar:file:/path/to/hive/lib/hive-exec-0.13.0.jar!/hive-log4j.properties
14/07/07 16:26:57 [main]: INFO SessionState:
Logging initialized using configuration in jar:file:/path/to/hive/lib/hive-exec-0.13.0.jar!/hive-log4j.properties
14/07/07 16:26:57 [main]: INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:344)
at shark.SharkCliDriver$.main(SharkCliDriver.scala:128)
at shark.SharkCliDriver.main(SharkCliDriver.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2444)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:338)
... 2 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1137)
... 7 more
Caused by: java.lang.NoSuchFieldError: METASTOREINTERVAL
at org.apache.hadoop.hive.metastore.RetryingRawStore.init(RetryingRawStore.java:78)
at org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:60)
at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:413)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:401)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:439)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:325)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:285)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4102)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
... 12 more
I guess Spark can not find some libs ton connect metastore in Hive, but I have been stacked here for a couple days and don't know how to solve it. BTW, I use MYSQL for hive metadata, and everything works well in hive.
Any help is appreciated. Thanks in advance.
You may need to add mysql connector jar file before you start spark...
In my case, I added mysql connector jar like below.
$SPARK_HOME/bin/compute-classpath.sh
CLASSPATH=$CLASSPATH:/opt/big/hive/lib/mysql-connector-java-5.1.25-bin.jar