I'm also new to DevOps, but let me see if I can explain my situation, also if anyone has any advice on doing post deployment scripts and can share their experiences I will be most grateful.
I use SQL Source control from Redgate, Git and Azure DevOps. I have managed to get the build and deployment working perfectly. Additionally we would like to make some data changes so we have additional (Data Change) scripts we wish to run. Linking these tables as static data is not an option as we would end up having most tables linked and end up with extremely timeous build and deploy times.
The script inserts data to a table, thereby triggering the insert trigger which calls a procedure to write to an audit table. The actual error from the deployment log is:
Task : DbUp Migration
2021-01-25T12:52:17.2629353Z Description : Runs SQL Server change scripts, and only those which have not been run already.
2021-01-25T12:52:17.2629405Z Version : 2.1.4
2021-01-25T12:52:17.2629671Z Author : Johan Classon
2021-01-25T12:52:17.2629721Z Help : [More Information](https://github.com/johanclasson/vso-agent-tasks)
2021-01-25T12:52:17.2629775Z ==============================================================================
2021-01-25T12:52:19.6610608Z Beginning database upgrade
2021-01-25T12:52:19.6689401Z Checking whether journal table exists..
2021-01-25T12:52:19.6728756Z Journal table does not exist
2021-01-25T12:52:19.7775348Z Executing Database Server script '001 - EX27605 - rdl.sql'
2021-01-25T12:52:19.7908913Z Checking whether journal table exists..
2021-01-25T12:52:19.7929056Z Creating the [dbo].[_SchemaVersions] table
2021-01-25T12:52:19.8077238Z The [dbo].[_SchemaVersions] table has been created
2021-01-25T12:52:20.2264323Z ifExists - rdl_Rule_Definition_Lookup - rdl_Code = "9900226e"
2021-01-25T12:52:20.2277308Z insert - rdl_Rule_Definition_Lookup - rdl_Code = "9900226e"
2021-01-25T12:52:20.2805780Z SQL exception has occured in script: '001 - EX27605 - rdl.sql'
2021-01-25T12:52:20.2942912Z ##[error]Script block number: 0; Block line 74; Message: Trig_After_Ins_Upd_Del_rdl_Rule_Definition_Lookup
2021-01-25T12:52:20.4528752Z ##[error]System.Data.SqlClient.SqlException (0x80131904): p_dte_Audit_Backend: (Line: 108) [dte_admin]
So the 1st error refers to line 74 of the trigger, if I read this correctly which is a commit on a procedure call to write to the audit table, line 108 of my procedure sets the userID:
DECLARE #user VARCHAR(40)
SET #user =
CASE
WHEN suser_sname() = 'NT SERVICE\SQLAgent$'+##servicename THEN 'dte_admin'
WHEN suser_sname() = 'NT SERVICE\SQLSERVERAGENT' THEN 'dte_admin'
ELSE suser_sname()
END
--###############
select #udt_Audit = udt_Audit
from udt_User_Detail (nolock)
where udt_User_Id = #user--suser_sname()
--select #udt_Audit = udt_Audit from udt_User_Detail
--where udt_User_Id = suser_sname()
if ##rowcount = 0
begin
declare #udt_User_Id varchar(30)
set #udt_User_Id = suser_sname()
The last line being line 108.
Important to note that dte_admin is a login on the SQL Server, it is a sysadmin account and mapped to the database, the dte_admin user is also a user at database level and exists as a user in my user table udt_user_detail.
During the build and deployment I am using variables and they are set to dte_admin username and password.
My question is then, why does this fail with this user?
Deployment Pipeline result
Related
I have a SSIS package in which we want to see if the DataLoadStatusID == 2 then go to the path that copies the file and deletes from the initial import folder. This part works as expected.
And if we load the same file again, due to duplicate values the DataLoadStatusID !=2, then go to the path where the file does not get copied or deleted. The file should remain in the initial import folder.** This is not working when I re-load the same file the DataLoadStatusID !=2 does not work.**
AND
I tired changing the Result Set to Single Row
Set the Result Set to DataLoadStatusID
Set the Parameter. One thing to note is that the DataLoadId is the only parameter setup in the stored procedure. But the DataLoadStatusId is setup as a variable in the SSIS package only. DataLoadStatusID is not setup as a parameter in the Stored Porcedure. But we have a update statement in the stored procedure.
-- If there were validation errors (not just warnings, mark the data load as failed and exit procedure
DECLARE #ErrorCount INT = 0
SELECT #ErrorCount = COUNT(*) from dbo.ClaimLoadValidation where DataLoadId = #DataLoadId and ValidationStatusId = 2
IF (#ErrorCount > 0)
BEGIN
UPDATE dbo.DataLoad set DataLoadStatusId = 3 where DataLoadId = #DataLoadId
RETURN
END
-- If everything worked, mark record as successful
UPDATE dbo.DataLoad set DataLoadStatusId = 2 where DataLoadId = #DataLoadId
Once I run the package it gives me below error:
"[Execute SQL Task] Error: Executing the query "execute uspLoadClaimHartford ?" failed with the following error: "Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done. Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly."
Are you sure the DataLoadStatus variable is != 2. Maybe put a breakpoint and watch on it to check it
I am having a fully running procedure on snowflake with no errors at all when calling it:
call ADD_MONTHLY_OBSERVATION_VALUES('#test_azure_blob_stg/Monthly_Report.csv', 'GENERIC_CSV_FORMAT');
I wanted to wrap this command into a task and schedule it to a specific time like so:
CREATE OR REPLACE TASK ADD_MONTHLY_OBSERVATION_VALUES_TASK
WAREHOUSE = 'DEV_WH'
TIMESTAMP_INPUT_FORMAT = 'YYYY-MM-DD HH24'
//SCHEDULE = 'USING CRON 0 6-7 * * SUN,MON,TUE,WED,THU Asia/Dacca'
//Schedule for each minute
SCHEDULE = 'USING CRON * * * * * UTC'
AS
call ADD_MONTHLY_OBSERVATION_VALUES('#test_azure_blob_stg/Monthly_Report.csv', 'GENERIC_CSV_FORMAT');
And then I resumed the task to work:
ALTER TASK ADD_MONTHLY_OBSERVATION_VALUES_TASK RESUME;
When I checked the history of the task:
STATE: SUCCEEDED
ERROR_CODE: NULL
ERROR_MESSAGE: NULL
QUERY_START_TIME: 2021-01-27 16:00:06.198 -0800
COMPLETED_TIME: 2021-01-27 16:00:24.902 -0800
RETURN_VALUE: NULL
Actually the procedure will return a string DONE when everything have been successfully added/updated.
When running:
show tasks;
The procedure's state is started.
Why the procedure is not being executed when called using a task?
There was new data uploaded to the staged file from Azure, so the procedure should detect the new submissions and start the insert process.
If you run the procedure outside the task, you have to use the task owner role to ensure correct testing. If your procedure works outside the task with task owner privileges, it should also work within the task.
So... I assume there is a problem with your access/rights management. SQL statements executed by the task can only operate on Snowflake objects on which the role has the required privileges. --> You have to grant more privileges on the objects within your procedure to your task owner role.
See more here: https://docs.snowflake.com/en/sql-reference/sql/create-task.html
I have two service programs: mySrvPgm and myErr
mySrvPgm has a procedure which contains:
/free
...
exec sql INSERT INTO TABLE VALUES(:RECORD_FMT);
if sqlError() = *ON;
//handle error
endif;
...
/end-free
myErr has a procedure sqlError:
/free
exec sql GET DIAGNOSTICS CONDITION 1
:state = RETURNED_SQLSTATE;
...
/end-free
Background info: I am using XMLSERVICE to call the given procedure in mySrvPgm from PHP. I am not using a persistent connection. myErr is bound-by-reference via a binding directory used by mySrvPgm. Its activation is set to *IMMED, its activation group is set to *CALLER.
The problem: Assume there is an error on the INSERT statement in mySvrPgm. The first time sqlError() is called it will return SQLSTATE 00000 despite the error. All subsequent calls to sqlError() return the expected SQLSTATE.
A workaround: I added a procedure to myErr called initSQL:
/free
exec sql SET SCHEMA MYLIB;
/end-free
If I call initSQL() before the INSERT statement in mySrvPgm, sqlError() functions correctly. It doesn't have to be SET SCHEMA, it can be another GET DIAGNOSTICS statement. However, if it does not contain an executable SQL statement it does not help.
The question: I believe the myErr service program is activating properly and has the correct scope, but I am wondering if there is something more I need to do to activate the SQL part of it. Is there some way to set it up so SQL auto-initializes when the service program is activated, or do I have to execute an unneeded SQL statement in order to get it started?
There is some more background information available here.
Thank you for reading.
What version an release of the OS? Are you upto date on PTFs?
Honestly, seems to me that it's possibly a bug. Or the manual(s) need clarification.. I'd open a PMR.
I've created a logon trigger in MS SQL that needs to check dm_exec_sessions for a login. This login is the user listed in the connection string and has owner right to the database. If the login is verified, I need it to update a specific table and send an email.
So far I've done just the following piece and it disabled my web site. The error I get is: "Logon failed for login 'dev' due to trigger execution. Changed database context to 'mydatabase'. Changed language setting to us_english."
Any idea what I did wrong? Thanks,
Risho
CREATE TRIGGER TDY_Assets_Notification
ON ALL SERVER WITH EXECUTE AS 'dev'
FOR LOGON
AS
BEGIN
IF ORIGINAL_LOGIN()='dev' AND
(SELECT COUNT(*) FROM sys.dm_exec_sessions
WHERE is_user_process = 1 AND
original_login_name = 'dev') > 1
UPDATE Assets_TDY
SET Suspense = 1, Warning = 1
WHERE (Date_Returned IS NULL) AND (GETDATE() >= DATEADD(day, 3, Date_Return))
END
Try connecting with the Dedicated Administrator Connection. That will give you an opportunity to disable the faulty logon trigger.
DISABLE TRIGGER [TDY_Assets_Notification] ON ALL SERVER
Essentially I have a job which runs in BIDS and as as a stand lone package and while it runs under the SQL Server Agent it doesn't complete properly (no error messages though).
The job steps are:
1) Delete all rows from table;
2) Use For each loop to fill up table from Excel spreasheets;
3) Clean up table.
I've tried this MS page (steps 1 & 2), didn't see any need to start changing from Server side security.
Also SQLServerCentral.com for this page, no resolution.
How can I get error logging or a fix?
Note I've reposted this from Server Fault as it's one of those questions that's not pure admin or programming.
I have logged in as the proxy account I'm running this under, and the job runs stand alone but complains that the Excel tables are empty?
Here's how I managed tracking "returned state" from an SSIS package called via a SQL Agent job. If we're lucky, some of this may apply to your system.
Job calls a stored procedure
Procedure builds a DTEXEC call (with a dozen or more parameters)
Procedure calls xp_cmdshell, with the call as a parameter (#Command)
SSIS package runs
"local" SSIS variable is initialized to 1
If an error is raised, SSIS "flow" passes to a step that sets that local variable to 0
In a final step, use Expressions to set SSIS property "ForceExecutionResult" to that local variable (1 = Success, 0 = Failure)
Full form of the SSIS call stores the returned value like so:
EXECUTE #ReturnValue = master.dbo.xp_cmdshell #Command
...and then it gets messy, as you can get a host of values returned from SSIS . I logged actions and activity in a DB table while going through the SSIS steps and consult that to try to work things out (which is where #Description below comes from). Here's the relevant code and comments:
-- Evaluate the DTEXEC return code
SET #Message = case
when #ReturnValue = 1 and #Description <> 'SSIS Package' then 'SSIS Package execution was stopped or interrupted before it completed'
when #ReturnValue in (0,1) then '' -- Package success or failure is logged within the package
when #ReturnValue = 3 then 'DTEXEC exit code 3, package interrupted'
when #ReturnValue in (4,5,6) then 'DTEXEC exit code ' + cast(#Returnvalue as varchar(10)) + ', package could not be run'
else 'DTEXEC exit code ' + isnull(cast(#Returnvalue as varchar(10)), '<NULL>') + ' is an unknown and unanticipated value'
end
-- Oddball case: if cmd.exe process is killed, return value is 1, but process will continue anyway
-- and could finish 100% succesfully... and #ReturnValue will equal 1. If you can figure out how,
-- write a check for this in here.
That last references the "what if, while SSIS is running, some admin joker kills the CMD session (from, say, taskmanager) because the process is running too long" situation. We've never had it happen--that I know of--but they were uber-paranoid when I was writing this so I had to look into it...
Why not use logging built into SSIS? We send our logs toa database table and then parse them out to another table in amore user friendly format and can see every step of everypackage that was run. And every error.
I did fix this eventually, thanks for the suggestions.
Basically I logged into Windows with the proxy user account I was running and started to see errors like:
"The For each file enumerator is empty"
I copied the project files across and started testing, it turned out that I'd still left a file path (N:/) in the properties of the For Each loop box, although I'd changed the connection properties. Easier once you've got error conditions to work with. I also had to recreate the variable mapping.
No wonder people just recreate the whole package.
Now fixed and working!