Liquibase diff without indexes fails when diffChangeLog declared - indexing

i'm new to liquibase and i was playing around with the diff command. It works perfectly fine but recently found this and i can't figure out why it's not functioning in this specific context.
so the main problem is that i want to compare two databases but without indexes. these are dynamically generated on primary keys and get different names, but are in fact equivalent. liquibase does not understand so i want to run diff without indexes.
so i add this to my pom.xml:
<diffTypes>tables, views, columns, primaryKeys, foreignKeys, uniqueconstraints</diffTypes>
it runs as expected, liquibase does not compare indexes.
in the next step, i want to generate the diff as changelog, so i add a diffChangeLog-File
<diffTypes>tables, views, columns, primaryKeys, foreignKeys, uniqueconstraints</diffTypes>
<diffChangeLogFile>src/main/diffs/diff_test.xml</diffChangeLogFile>
when running liquibase:diff, it fails:
[ERROR] Failed to execute goal org.liquibase:liquibase-maven-plugin:3.4.1:diff (default-cli) on project liquibase_artifactID: Error setting up or running Liquibase: liquibase.command.CommandExecutionException: liquibase.exception.UnexpectedLiquibaseException: Could not resolve MissingObjectChangeGenerator dependencies due to dependency cycle. Dependencies:
[ERROR] [] -> Catalog -> []
[ERROR] [] -> Schema -> []
[ERROR] [Index] -> ForeignKey -> []
[ERROR] [] -> UniqueConstraint -> []
[ERROR] [] -> Column -> []
[ERROR] [] -> Table -> []
[ERROR] [] -> PrimaryKey -> []
[ERROR] [] -> View -> []
[ERROR] -> [Help 1]
Why does liquibase act like this? Is it "illegal" to generate a diffChangeLog without indexes?
When including indexes to diffTypes it works, but the generated Changelog is unusable because liquibase wants to change the indexes with createIndex and dropIndex. But these statements are not executable (it fails to drop an index on primary keys and it can't create an index when it already exists).
Any ideas how to generate a usable changelog without indexes? Or did i just miss something?

The answer to the question is there in the exception message:
Could not resolve MissingObjectChangeGenerator dependencies due to dependency cycle.
It then lists the dependencies.
Internally, Liquibase is generating a directed graph of dependencies and making sure that the dependencies are all satisfied. If you would like to see the code that does this, see the class DiffToChangeLog and its internal private class DependencyGraph

Related

DBT RUN - Getting Database Error using VS Code BUT Not Getting Database Error using DBT Cloud

I'm using DBT connected to Snowflake. I use DBT Cloud, but we are moving to using VS Code for our DBT project work.
I have an incremental DBT model that compiles and works without error when I issue the DBT RUN command in DBT Cloud. Yet when I attempt to run the exact same model from the same git branch using the DBT RUN command from the terminal in VS Code I get the following error:
Database Error in model dim_cifs (models\core_data_warehouse\dim_cifs.sql)
16:45:31 040050 (22000): SQL compilation error: cannot change column LOAN_MGMT_SYS from type VARCHAR(7) to VARCHAR(3) because reducing the byte-length of a varchar is not supported.
The table in Snowflake defines this column as VARCHAR(50). I have no idea why DBT is attempting to change the data length or why it only happens when the command is run from VS Code Terminal. There is no need to make this DDL change to the table.
When I view the compiled SQL in the Target folder there is nothing that indicates a DDL change.
When I look in the logs I find the following, but don't understand what is triggering the DDL change:
describe table "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS"
16:45:31.354314 [debug] [Thread-9 (]: SQL status: SUCCESS 36 in 0.09 seconds
16:45:31.378864 [debug] [Thread-9 (]:
In "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS":
Schema changed: True
Source columns not in target: []
Target columns not in source: []
New column types: [{'column_name': 'LOAN_MGMT_SYS', 'new_type': 'character varying(3)'}]
16:45:31.391828 [debug] [Thread-9 (]: Using snowflake connection "model.xxxxxxxxxx.dim_cifs"
16:45:31.391828 [debug] [Thread-9 (]: On model.xxxxxxxxxx.dim_cifs: /* {"app": "dbt", "dbt_version": "1.1.1", "profile_name": "xxxxxxxxxx", "target_name": "dev", "node_id": "model.xxxxxxxxxx.dim_cifs"} */
alter table "DEVELOPMENT_DW"."DBT_XXXXXXXX"."DIM_CIFS" alter "LOAN_MGMT_SYS" set data type character varying(3);
16:45:31.546962 [debug] [Thread-9 (]: Snowflake adapter: Snowflake query id: 01a5bc8d-0404-c9c1-0000-91b5178ac72a
16:45:31.548895 [debug] [Thread-9 (]: Snowflake adapter: Snowflake error: 040050 (22000): SQL compilation error: cannot change column LOAN_MGMT_SYS from type VARCHAR(7) to VARCHAR(3) because reducing the byte-length of a varchar is not supported.
Any help is greatly appreciated.

dbt relationship test compilation error: test definition dictionary must have exactly one key

I'm a new user of dbt, trying to write a relationship test:
- name: PROTOCOL_ID
tests:
- relationships:
to: ref('Animal_Protocols')
field: id
I am getting this error:
Compilation Error
Invalid test config given in models/Animal_Protocols/schema.yml:
test definition dictionary must have exactly one key, got [('relationships', None), ('to', "ref('Animal_Protocols')"), ('field', 'id')] instead (3 keys)
#: UnparsedNodeUpdate(original_file_path='model...ne)
"unique" and "not-null" tests in the same file are working fine, but I have a similar error with "accepted_values".
I am using dbt cli version 0.21.0 with Snowflake on MacOS Big Sur 11.6.
You are very close! I'm 96% sure that this is an indentation issue -- the #1 pain point of working with YAML. The solution is that both to and field need to be indented below the relationships key as opposed to at the same level.
See the Tests dbt docs page for an example
- name: PROTOCOL_ID
tests:
- relationships:
to: ref('Animal_Protocols')
field: id

liquibase-hibernate shows all tables as "unexpected"

I followed these steps to get liquibase-hibernate working. I hope I correctly understood the instructions in the wiki.
Our hibernate entities are declared in the file applicationContext.xml. We do not have a hibernate.cfg.xml. My liquibase properties are:
url=jdbc:postgresql://localhost:1234/MY_DATABASE
username=user
password=pass
referenceUrl=hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect
The thing is no matter what I enter as somePackage, liquibase shows everything (tables, columns, constraints) as "unexpected". Liquibase "finds" somePackage even if it does not exist.
liquibase diff
INFO 09.08.17 10:41: liquibase-hibernate: Reading hibernate configuration hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect
INFO 09.08.17 10:41: liquibase-hibernate: Found package somePackage
And the comparison result is like
Reference Database: null # hibernate:spring:somePackage?dialect=org.hibernate.dialect.PostgreSQLDialect (Default Schema: HIBERNATE)
Comparison Database: postgres # jdbc:postgresql://localhost:1234/MY_DATABASE (Default Schema: public)
Compared Schemas: HIBERNATE -> public
Product Name:
Reference: 'Hibernate'
Target: 'PostgreSQL'
Product Version:
Reference: '4.3.11.Final'
Target: '9.5.4'
Missing Catalog(s): NONE
Unexpected Catalog(s): NONE
Changed Catalog(s):
HIBERNATE
name changed from 'HIBERNATE' to 'MY_DATABASE'
Missing Column(s): NONE
[...]
Unexpected Table(s):
activityentity
addressentity
advertisemententity
advertisementusageentity
[...]
I really don't know what's going on or whether I'm doing something wrong. Any help would be appreciated.

Why does mapping to a List inside lifted query fail with "Slick does not know how to map the given types."?

I am trying to map to a List inside a Slick lifted query, and I get a compilation error:
No matching Shape found.
[error] Slick does not know how to map the given types.
[error] Possible causes: T in Table[T] does not match your * projection. Or you use an unsupported type in a Query (e.g. scala List).
[error] Required level: scala.slick.lifted.FlatShapeLevel
[error] Source type: Seq[String]
[error] Unpacked type: T
[error] Packed type: G
Why is it that in a non-Slick map operation, I can map to any type, whereas in a Slick query, I can map to simple Scala types, but not to a Scala List?
Try to
import slick.driver.(yourDB)Driver.api._

Hive persmission denied error

I have two ids us_id1 and us_id2, both ids are part of group alldev.
With us_id1 I can go and view all directories in HDFS, and I can execute all Hive commands
With us_id2 I can view all directories and log into hive.
But when I run a simple command like show databases; I get the following error.
on database directory /datalake/sample_dev/metastore_db in READ ONLY mode
Database Class Loader started - derby.database.classpath=''
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
What is missing for us_id2?