We are deploying sql query's using liquibase automation tool.
liquibase version: 4.3.0
We have created DB files in SQL format and used the "liquibase validate" command to validate the syntax error. But I didn't get expected result.
Ex: dem1.sql, demo2.sql
I have found the below doc's for validation,
https://docs.liquibase.com/commands/validate.html?msclkid=72d2d27da9cf11ecae4cc3525df5294e
As per the above document, "liqubase validate" command won't support for SQL format files.
Can any one help me to validate the SQL format file with liquibase or any other ways (like shell script or jenkins pipeline stage) before deploy the sql query's to my DB.
One possible approach for syntax validation could be to utilise a Database test-container matching your runtime environment during integration coverage.
That way your LiquiBase changelogs can be executed during test-setup and syntax is implicitly validated.
See example here (https://www.baeldung.com/spring-boot-testcontainers-integration-test)
Related
I want to validate SQL scripts (No Syntax errors of SQL) in Azure DevOps.
I have added one task MSBuild#1 which validates my SQL scripts but gives a lot of non-required errors like Only One Statement is required per batch. Batch Separator, such as 'Go' required between statements.
I have written one update statement still it gives me above mentioned error and when I search for such an error people are recommended not to build a database project.
Is there any other alternative way of compiling my SQL scripts which will give only required syntax errors?
You can try:
Unload SQL project in Visual studio
In your *.SqlProj file, you under <PropertyGroup>, add:
<SuppressTSqlWarnings>SQL71006</SuppressTSqlWarnings>
SQL71006 is the code for the warning/error statement you are getting MSBuild - Only One Statement is required per batch. Batch Separator, such as 'Go' required between statement.
For multiple error/warning codes suppression:
In visual studio SQL project, go to project properties, click on Build.
Under "Suppress Transact-SQL warnings", type warning numbers (without 'SQL') in comma separated format.
Hope this helps:)
I have a project based on ASPNETZERO, which uses DotNet Core EF code-first migrations. It has an Azure Devops pipeline to build and deploy it to Azure Web Apps and Azure SQL PaaS database. The SQL migrations are created by Entity Framework Core Migrations Script Generator.
After the most recent commit, the "Release" job failed when running the SQL migration scripts. It gave a series of errors indicating that a number of columns were invalid. Those columns no longer exist in the database, my new commit renames those columns, but no subsequent migrations refer to the old columns. No other information was reported to the logs.
As part of my troubleshooting I connected Visual Studio to the Azure SQL PaaS deployed database, and ran UPDATE-DATABASE. It successfully applied all migrations. I committed the branch again, and again the deployment step threw the "Invalid Column" errors when applying migrations.
To get more information I ran the Devops build task on a remote agent on my own PC so that I could grab the .sql file that it outputs. I executed that .sql against the Azure SQL database in SSMS and it threw the Invalid Column errors I saw in the Azure logs. In SSMS I was able to see precisely where the errors were occurring because the error output takes you to the offending code.
What happened was that in some earlier migrations I had done some data transforms using migrationBuilder.Sql("Raw SQL command string"), and those raw SQL commands were using the columns which I have now, much later, renamed. This was no problem for the Visual Studio UPDATE-DATABASE processor because the SQL command string is not parsed. It is just handed through to SQL at runtime, and even then, only if that migration is required. But when the Azure SQL Database deployment generates the .sql output script it turns that SQL string into raw SQL like this:
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'<MigrationName>')
BEGIN
UPDATE LinkPersonLMSs
set EntitlementStartDate = (SELECT EnrolmentStart from People where People.id = PersonId)
END;
... and that query is subject to the SQL query analyser validating the column names before executing it even if the conditional check means the inner query won't execute. If the column "EntitlementStartDate" is renamed or removed in a subsequent migration (as it was in my case), the SQL script causes an error when the SQL query analyser parses it to prepare an execution plan.
The Solution
I went back through all my migrations and found every place where I had used migrationBuilder.Sql("Raw SQL command string"), and instead made it migrationBuilder.Sql("exec('Raw SQL command string')"), so that the exec statement would become the raw SQL, and the payload SQL command string would be rendered as a literal string in the resulting SQL deployment script, and therefore not subject to parsing unless it actually gets executed.
Note: Any SQL literal symobols inside the command string then need to be escaped, so every single quote (') needs to be changed to two single quotes (''). But that's the only change that should be required.
Now my DEVOPS deployment runs without error, as does UPDATE-DATABASE.
im using Liquibase to manage my database in SQLdeveloper. Now, we have a PLSQL script that runs fine by itself. It also makes the changes in the database. Now, if we want Liquibase to run this PLSQL script it gets the following error: ORA-00922: missing or invalid option.
Is it even possible for Liquibase to run external PLSQL scripts?
There are two potential issues you are running into:
You are using <sqlFile> but not specifying splitStatements="false". By default, Liquibase will split the SQL on semicolons because that is what the JDBC driver needs normally, but in a PL/SQL script you can have a single CREATE PROCEDURE or something similar which contains semicolons but is still one statement. You can also use <createProcedure> instead of sqlFile which does not split statements.
Your script has sqlplus-specific functionality in it. SQLPlus and SQLDeveloper do not simply pass the SQL strings straight through to the database, they have their own functionality to sometimes modify what is actually executed. JDBC and therefore Liquibase does not have all the same functionality and so if you are using it in your scripts they will not work. If this is the case, the best approach is to use a changeSet with <executeCommand> to make a call to sqldeveloper or sqlplus.
I found 2 issues per se when I ran the updateSQL commandline in Liquibase
The last statement in Liquibase updateSQL output viz Insert into DBCHANGELOG table does not commit automatically when the sql is run via sqlplus commandline
As a result of this, though the changeset gets executed, the DBCHANGELOG table does not have the insert statement to record it. So when I run the updateSQL once again, the last changeset is once again created in the SQL output which is incorrect.
Liquibase does not validate / check syntax errors in SQL.
As a result of this, even if the changeset SQL fails, the insert to DBChangeLog table for the changeset succeeds which is incorrect. Is there a way that the insert statement following the changeset be stopped / failed if the changeset SQL actually failed ?
Any help is greatly appreciated... we are this close to getting Liquibase implemented... !!
To answer the question in your subject line, no, Liquibase cannot validate the SQL. Liquibase supports many different databases, and each has different SQL syntax.
If you can, stop using the SQL generated by updateSQL to actually do the updates, and use Liquibase itself to do the updates. That way Liquibase can detect errors and behave more properly. I recommend that if DBAs are scared of Liquibase touching the database that teams use the generateSQL as a pre-check to see what Liquibase will do, but let Liquibase do its job.
I also find best practice of Liquibase is not to use a SQL script but to manually write the Liquibase XML file for the change.
I've tried using the ExecuteCommand tag to lunch sqlplus or sqlcmd (as I know my target database) and it has a bug which as for now it is closed?! (but this is open source, so I can't complain :) )
Having said that, I found that working on XML to specify the changes causes many other challenges, for example:
1. Making sure that every change was included in the changelog xml file. I've heard many organizations who forget to add the file to the changelog.
2. Making sure the file for the specific change is always in sync with the file-based version control. Imagine what will happen if it doesn't - which happens to many of my customers...
3. Wasted time spent on merging changelogs between different environments (branches, UAT - critical fixes, sandboxes, etc...)
I am trying to migrate from oracle db to mysql or postgres using Liquibas. I have generated the sql file using Liquibase but, the syntax is not right there is a lot of issue with the generated sql. If anyone has any solution please do let me know thank you.
The best approach is to use the generateChangeLog function to create an XML changeSet description of your oracle database. Go through the generated changelog to make sure everything expected is there, and make any changes to the file as needed such as data type changes.
Once the changelog is correct, you can run the XML changelog directly against your mysql or postgresql database or use updateSQL to generate the SQL liquibase would use. Liquibase will create the correct database-specific SQL when it runs a changelog against a given database.