I would like to generate csv files and loaddata changeset for some tables.
I use this command line:
$LB_HOME/liquibase --logLevel=DEBUG --changeLogFile=${TABLE}.xml \
--url=jdbc:oracle:thin:#local:1521/ORCL --username=TEST --password=TEST \
--dataOutputDirectory=csv --diffTypes=data \
--includeObjects="table:$TABLE" generateChangeLog
After a very long list of lines like this:
DEBUG [liquibase.util.DependencyUtil$DependencyGraph]:
Potential StackOverflowException. Pro-actively removing with incoming nodes
I get this error:
ERROR [liquibase.integration.commandline.Main]: Unexpected error running Liquibase: Unknown reason
java.lang.StackOverflowError: null
I put includeObjects="table:$TABLE" with only one table, why liquibase reads all object dependencies?
Any suggestion?
As per Liquibase Documentation, includeObjects is not valid parameter
here is the Link: https://docs.liquibase.com/commands/community/generatechangelog.html
Can you try running just generateChangeLog command without data and see if it works first?
Related
I have a csv file on my computer. I would like to load this CSV file into a BigQuery table.
I'm using the following command from a terminal:
bq load --apilog=./logs --field_delimiter=$(printf ';') --skip_leading_rows=1 --autodetect dataset1.table1 mycsvfile.csv myschema.json
The command in my terminal doesn't give any output. In the GCP interface, I see no job being created, which makes me think the request doesn't even reach GCP.
In the logs file (from the --apilog parameter) I get informations about the request being made, and it ends with this:
INFO:googleapiclient.discovery:URL being requested: POST https://bigquery.googleapis.com/upload/bigquery/v2/projects/myproject/jobs?uploadType=resumable&alt=json
and that's it. No matter how long I wait, nothing happens.
You are mixing --autodetect with myschema.json, something like the following shoud work:
bq load --apilog=logs \
--source_format=CSV \
--field_delimiter=';' \
--skip_leading_rows=1 \
--autodetect \
dataset.table \
mycsvfile.csv
If you continue having issues, please post the content of the apilog, the line you shared doesn't seem to be an error. There should be more than one line and normally contains the error in a json structure, for instance:
"reason": "invalid",
"message": "Provided Schema does not match Table project:dataset.table. Field users is missing in new schema"
I'm not sure why you are using
--apilog=./logs
I did not find this in the bq load documentation, please clarify.
Based on that, maybe the bq load command could be the issue, you can try with something like:
bq load \
--autodetect \
--source_format=CSV \
--skip_leading_rows= 1 \
--field_delimiter=';'
dataset1.table1 \
gs://mybucket/mycsvfile.csv \
./myschema.json
If it fails, please check your job list to get the job created, then use bq show to view the information about that job, there you should find an error messag which can help you to determine the cause of the issue.
I'm trying to diff two databases from the command line in linux. I'm hoping that someone can tell me what I'm doing wrong... my command is
./liquibase
--driver=com.mysql.cj.jdbc.Driver \
--url=jdbc:mysql://{it'svalid}:3306/db_schema_1?useLegacyDateTimeCcxle=false&serverTimezone=UTC \
--username={it'svalid} \
--password={it'svalid} \
diff \
--referenceUrl=jdbc:mysql://{it'svalid}:3306/db_schema_2?useLegacyDateTimeCcxle=false&serverTimezone=UTC \
--referenceUsername={it'svalid}
--referencePassword={it'svalid} \
--changeLogFile=databaseChangeLog.sql
I'm getting this in response:
java -cp
".:/home/someguy/needit/liquibase/liquibase.jar:/home/someguy/needit/liquibase/lib/:/home/someguy/needit/liquibase/lib/jcl-over-slf4j-1.7.25.jar:/home/someguy/needit/liquibase/lib/jul-to-slf4j-1.7.25.jar:/home/someguy/needit/liquibase/lib/log4j-over-slf4j-1.7.25.jar:/home/someguy/needit/liquibase/lib/logback-1.1.jar:/home/someguy/needit/liquibase/lib/logback-classic-1.2.3.jar:/home/tkrueger/needit/liquibase/lib/logback-core-1.2.3.jar:/home/someguy/needit/liquibase/lib/mysql-connector-java-8.0.15.jar:/home/someguy/needit/liquibase/lib/slf4j-api-1.7.25.jar:/home/someguy/needit/liquibase/lib/snakeyaml-1.18.jar" liquibase.integration.commandline.Main
"--driver=com.mysql.cj.jdbc.Driver
--url=jdbc:mysql://{it'svalid}:3306/db_schema_1 --username={it'svalid} --password={it'svalid} diff --referenceUrl=jdbc:mysql://{it'svalid}:3306/db_schema_2 --referenceUsername={it'svalid} --referencePassword={it'svalid} --changeLogFile=databaseChangeLog.sql" Starting Liquibase at Thu, 02 May 2019 14:21:56 CDT (version 3.6.3 built at 2019-01-29 11:34:48) To
display the help, please pass the '--help' option on the command line.
Unexpected error running Liquibase: Unexpected value
'"--driver=com.mysql.cj.jdbc.Driver' (options must start with a '--')
liquibase.exception.CommandLineParsingException: Unexpected value
'"--driver=com.mysql.cj.jdbc.Driver' (options must start with a '--')
at liquibase.integration.commandline.Main.parseOptions(Main.java:778)
at liquibase.integration.commandline.Main.run(Main.java:185)
at liquibase.integration.commandline.Main.main(Main.java:137)
I've dumped most of the parameters into a liquibase.properties file, to no avail... I thought I found someone else dealing with the leading doublequote on the rejected parameter string (i.e., it appears to be parsing the parameters as "-- instead of simply --), but I'm struggling to understand how to get around that while doing a diff, by putting all my parameters into a liquibase.properties file.
You should check what end of line you have. I had same problem. Resolution was to change from windows eol to linux.
Without getting too much into the weeds, I have a Pentaho PDI job with multiple sub-transformations and sub-jobs (ETL from MySQL to Postgres). This job runs exactly as expected from Spoon, no errors, but when I run the job--with the following command--I am met with an endless loop error at the first step where a parameter would need to be defined and passed from within the job (the named params from the command seem to integrate fine). The command I am using is as follows:
sudo /bin/sh kitchen.sh \
-rep=KettleFileRepo \
-dir=M2P \
-job=ETL-M2P \
-level=Rowlevel \
-param:MY.PAR.LOADTYPE=full \
-param:MY.PAR.TABLELIST=table1 \
-param:MY.PAR.TENANTS=tenant1 \
/
Has anyone run into this type of issue with a discrepancy between Spoon and Kitchen? Is there some sort of config or command line option that I am missing? I am running version 6.0.1.0-386 on OS X 10.11.4.
If you think more details would be beneficial please let me know and I can provide whatever is necessary.
I am not aware of any discrepancy between Spoon and Kitchen. Are you sure, its not something in the ETL that causing the loop. I would suggest to go through your ETL in detail.
Another thing you can try to debug is run only part of the job in kitchen and keep adding more as you see success.
I'm using Hue for PIG scripts on amazon EMR. I am using the declare and default statements as mentioned in the documentation.
I have some %default and %declare statements and it looks like they are
not preprocessed within Hue. Therefore, although the parameters are defined
in my script, the editor keeps popping in a parameter configuration window. If I leave the parameter blank, the job fails with an error.
Sample Script
%declare OUTPUT_FOLDER 'testingOutput01';
ts = LOAD 's3://testbucket1/input/testdata-00000.gz' USING PigStorage('\t');
STORE ts INTO 's3://testbucket1/$OUTPUT_FOLDER' USING PigStorage('\t');
Upon execution, it shows the pop-up window asking for values for OUTPUT_FOLDER. If I leave it blank it fails with the following error:
2015-06-23 20:15:54,908 [main] ERROR org.apache.pig.Main - ERROR 2997:
Encountered IOException. org.apache.pig.tools.parameters.ParseException:
Encountered "<EOF>" at line 1, column 12.
Was expecting one of:
<IDENTIFIER> ...
<OTHER> ...
<LITERAL> ...
<SHELLCMD> ...
Is that the expected behavior? Is this a known issue or am I missing something?
Configuration details:
AMI version:3.7.0
Hadoop distribution:Amazon 2.4.0
Applications:Hive 0.13.1, Pig 0.12.0, Impala 1.2.4, Hue
The same behavior is seen with default instead of declare.
If you need any clarifications then please do comment on this question. I will update it as needed.
Hue does not support %declare with a default statement. It will be fixed with: https://issues.cloudera.org/browse/HUE-2508
The current temporary workaround is to put any value in the popup.
I am trying to execute command on my command prompt:
liquibase --driver=com.mysql.jdbc.Driver --classpath=E:\mysqljar\mysql.jar --changeLogFile=E:\1.xml --url="jdbc:mysql://localhost:3306/abc" --username=root --password=root generateChangeLog
But I am getting this error:
Liquibase Update Failed: Empty result set, expected one row
SEVERE 24/9/13 6:29 PM:liquibase: Empty result set, expected one row
liquibase.exception.DatabaseException: Error getting jdbc:mysql://localhost:3306/abc view with liquibase.statement.core.GetViewDefinitionStatement#53330681
at liquibase.snapshot.jvm.JdbcDatabaseSnapshotGenerator.readView(JdbcDatabaseSnapshotGenerator.java:168)
at liquibase.snapshot.jvm.JdbcDatabaseSnapshotGenerator.readViews(JdbcDatabaseSnapshotGenerator.java:304)
at liquibase.snapshot.jvm.JdbcDatabaseSnapshotGenerator.createSnapshot(JdbcDatabaseSnapshotGenerator.java:241)
at liquibase.snapshot.DatabaseSnapshotGeneratorFactory.createSnapshot(DatabaseSnapshotGeneratorFactory.java:69)
at liquibase.diff.Diff.compare(Diff.java:63)
at liquibase.integration.commandline.CommandLineUtils.doGenerateChangeLog(CommandLineUtils.java:145)
at liquibase.integration.commandline.Main.doMigration(Main.java:760)
at liquibase.integration.commandline.Main.main(Main.java:134)
Caused by: liquibase.exception.DatabaseException: Empty result set, expected one row
at liquibase.util.JdbcUtils.requiredSingleResult(JdbcUtils.java:124)
at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:159)
at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:167)
at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:163)
at liquibase.database.AbstractDatabase.getViewDefinition(AbstractDatabase.java:748)
at liquibase.snapshot.jvm.JdbcDatabaseSnapshotGenerator.readView(JdbcDatabaseSnapshotGenerator.java:166)
... 7 more
Could anyone help me to interpret this?
I had the same problem.
My fix:
There needs to be an entry in the database change log lock table; it needs to have id=1, locked=false and the rest of the values set to null.
I'm getting the same error also using MySQL and tried v3.0.0 and 3.0.5 of liquibase. The error was the same when I tried to do a migrate as well as a generateChangeLog.
./liquibase --logLevel=debug --changeLogFile=./db.changelog-test-v0.1.xml --username=abc --password=abc99 --url="jdbc:mysql://localhost:3306/test" migrate
FYI, here is the select statement that it had trouble executing for the 'migrate' command:
select view_definition from information_schema.views where table_name='patient_info' and table_schema='test'
The information_schema.views table is empty.