I want to validate SQL scripts (No Syntax errors of SQL) in Azure DevOps.
I have added one task MSBuild#1 which validates my SQL scripts but gives a lot of non-required errors like Only One Statement is required per batch. Batch Separator, such as 'Go' required between statements.
I have written one update statement still it gives me above mentioned error and when I search for such an error people are recommended not to build a database project.
Is there any other alternative way of compiling my SQL scripts which will give only required syntax errors?
You can try:
Unload SQL project in Visual studio
In your *.SqlProj file, you under <PropertyGroup>, add:
<SuppressTSqlWarnings>SQL71006</SuppressTSqlWarnings>
SQL71006 is the code for the warning/error statement you are getting MSBuild - Only One Statement is required per batch. Batch Separator, such as 'Go' required between statement.
For multiple error/warning codes suppression:
In visual studio SQL project, go to project properties, click on Build.
Under "Suppress Transact-SQL warnings", type warning numbers (without 'SQL') in comma separated format.
Hope this helps:)
Related
We are deploying sql query's using liquibase automation tool.
liquibase version: 4.3.0
We have created DB files in SQL format and used the "liquibase validate" command to validate the syntax error. But I didn't get expected result.
Ex: dem1.sql, demo2.sql
I have found the below doc's for validation,
https://docs.liquibase.com/commands/validate.html?msclkid=72d2d27da9cf11ecae4cc3525df5294e
As per the above document, "liqubase validate" command won't support for SQL format files.
Can any one help me to validate the SQL format file with liquibase or any other ways (like shell script or jenkins pipeline stage) before deploy the sql query's to my DB.
One possible approach for syntax validation could be to utilise a Database test-container matching your runtime environment during integration coverage.
That way your LiquiBase changelogs can be executed during test-setup and syntax is implicitly validated.
See example here (https://www.baeldung.com/spring-boot-testcontainers-integration-test)
I have a project based on ASPNETZERO, which uses DotNet Core EF code-first migrations. It has an Azure Devops pipeline to build and deploy it to Azure Web Apps and Azure SQL PaaS database. The SQL migrations are created by Entity Framework Core Migrations Script Generator.
After the most recent commit, the "Release" job failed when running the SQL migration scripts. It gave a series of errors indicating that a number of columns were invalid. Those columns no longer exist in the database, my new commit renames those columns, but no subsequent migrations refer to the old columns. No other information was reported to the logs.
As part of my troubleshooting I connected Visual Studio to the Azure SQL PaaS deployed database, and ran UPDATE-DATABASE. It successfully applied all migrations. I committed the branch again, and again the deployment step threw the "Invalid Column" errors when applying migrations.
To get more information I ran the Devops build task on a remote agent on my own PC so that I could grab the .sql file that it outputs. I executed that .sql against the Azure SQL database in SSMS and it threw the Invalid Column errors I saw in the Azure logs. In SSMS I was able to see precisely where the errors were occurring because the error output takes you to the offending code.
What happened was that in some earlier migrations I had done some data transforms using migrationBuilder.Sql("Raw SQL command string"), and those raw SQL commands were using the columns which I have now, much later, renamed. This was no problem for the Visual Studio UPDATE-DATABASE processor because the SQL command string is not parsed. It is just handed through to SQL at runtime, and even then, only if that migration is required. But when the Azure SQL Database deployment generates the .sql output script it turns that SQL string into raw SQL like this:
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'<MigrationName>')
BEGIN
UPDATE LinkPersonLMSs
set EntitlementStartDate = (SELECT EnrolmentStart from People where People.id = PersonId)
END;
... and that query is subject to the SQL query analyser validating the column names before executing it even if the conditional check means the inner query won't execute. If the column "EntitlementStartDate" is renamed or removed in a subsequent migration (as it was in my case), the SQL script causes an error when the SQL query analyser parses it to prepare an execution plan.
The Solution
I went back through all my migrations and found every place where I had used migrationBuilder.Sql("Raw SQL command string"), and instead made it migrationBuilder.Sql("exec('Raw SQL command string')"), so that the exec statement would become the raw SQL, and the payload SQL command string would be rendered as a literal string in the resulting SQL deployment script, and therefore not subject to parsing unless it actually gets executed.
Note: Any SQL literal symobols inside the command string then need to be escaped, so every single quote (') needs to be changed to two single quotes (''). But that's the only change that should be required.
Now my DEVOPS deployment runs without error, as does UPDATE-DATABASE.
I've been dealing for trying to migrate a really big database to an earlier SQL Server with multiple ways, I started doing a .bak file, but I found that it is not compatible and it should be the same SQL Server version.
Then I chose the task - generate Script, to create a .sql file with all schema and data, but the file was 24gb big! Even though the file was really big by using the sqlcmd I managed to execute it. But it never finished executing successfully, it threw multiple types of errors, like:
Msg 156, Level 15, State 1:
Incorrect syntax near the keyword '...'
Msg 105, Level 15, State 1
Unclosed quotation mark after the character string
Then I found this comment with 2 solutions https://stackoverflow.com/a/27623706/3192041, I tried the first one but still throwing the second error, I tried the second one and It worked! everything was now running smoothly, but then I got another error...
This error:
Sqlcmd: Error: Internal error at ReadText (Reason: An attempt was made
to move the file pointer before the beginning of the file).
So now the issue has something to do with sqlcmd command??
Should I continue trying to migrate the database with the generated script? is there a better way of this and making it compatible with an earlier SQL Server version?
Things to clarify
I first created a script only with schema information, but when I tried to generate a separate script for data only the SSMS was throwing an error. So with this way I can't or I don't know how to export all data with an easy way. I know you could export data for each of the tables, but the database has more than 200 tables and this is not viable.
Also the script takes more than one hour, and maybe a lot more than that time if the process would finish correctly.
Finally after also trying with a bacpac file, that also didn't let me to create because of a bunch of errors of windows users, external object references, and more...
The best answer to solve this, is by creating a .dacpac file. The dacpac file from SSMS 2012 to the latest versions, you can now include the data of all your tables.
And to solve the incompatibility issue, you can use the AllowIncompatiblePlatform property to allow deployment to different versions of SQL Server when publishing to the target server.
so first you need to extract using the SqlPackage.exe from your bin folder of the sql server, in my case this is the folder: C:\Program Files (x86)\Microsoft SQL Server\130\DAC\bin
then run the command with the Extract action:
SqlPackage /Action:Extract /SourceDatabaseName:"<database-name>" /SourceServerName:"<server-name>" /SourceUser:"sa" /SourcePassword:"<password>" /TargetFile:"<dacpac-file-path>" /p:ExtractAllTableData=True
Then in the other server run this command in correct bin folder of the SqlPackage.exe program:
SqlPackage /Action:Publish /SourceFile:" <dacpac file path>\filename.dacpac" /TargetDatabaseName:"<database name>" /TargetServerName:"<ServerName>" /TargetUser:"<username>" /TargetPassword:"<password>(if needed)" /p:AllowIncompatiblePlatform=true /p:CreateNewDatabase=true
And If you want to create the database from scrath.
/p:CreateNewDatabase=true
I hope this helps anyone with this problem, with big databases, and importing from a bigger sql server version.
I'm working to make some fact tables (taking some data from some resources, doing some transformations and putting them in a table). My main dilemma is that I can't run any SQL query other than select, update, and insertion. As soon as i try:
exec someProcedure
or a conditional statement (if #part1 ...) or even (create table ...) I take errors. Opening the task to build my SQL statements and find problems it gives errors ranging from (The Set SQL construct or statement is not supported.) to (The EXEC SQL construct or statement is not supported.).
I looked for numerous topics here on stackoverflow but none were actually addressing me problem.
Thanks,
You can see a view of what I'm facing in this picture :
I expect to run my SQL commands as usual in SSIS.
Try changing the SQL Source Type from Direct Input to Stored Procedure and just specify the stored procedure name instead of Exec stored procedure
Also make sure that you have selected the relevant TargetServerVersion from the project configuration:
How to change TargetServerVersion of my SSIS Project
Based on your comments, you are using SQL Server 2012 with Visual Studio 2010 which are not compatible.
You have to use Visual Studio 2012 or 2015+ (backward compatibility added). You can refer to the SSIS tag wiki for more info:
https://stackoverflow.com/tags/ssis/info
This works perfectly in a SSMS 2008 query window:
use [HSS Maintenance]
exec HSS_Guest.[olap storedprocedure]
exec HSS_Guest.[MAKE OLAP_CUBE]
But when I build an Agent Job and then test by using "Start Job at Step..." it always fails.
I have tried Operating System (CmdExec) "Type" as well as T-SQL and even SSAS I believe.
Am I having a Syntax issue or not using the correct "Type"?
Using full rights on an Enterprise edition & Server 2008.
I have Googled this to death and do not believe that I can't just build this simply via the GUI.
Please show me how dumb I am!
Note - I have tried to generate the Script from the Job window and get an error that "There is no Action to be Scripted"
Edit the SQL Server Agent job that contains the step that has the SQL code you posted. Remove the "use [HSS Maintenance]" line and instead use the drop down menu on the step to choose this database.