Azure database copy, wait for ready state - sql

I am making new Azure databases now and then and without any human interaction, by using a template database and azure copy database t-sql script.
After reading a few caveats I wonder what is the best way to wait for the copied database to be ready.
I have posted my own answer but it could end up in an infinite loop if the database copy fails.
What is the best way to be sure that the copying is done. This is very important as you are only allowed one copy database command at a time. I'm using an Azure worker with a c# queue that creates the databases as nessesary.

This is how I'm doing it now. This should work but unfortunatly errors when copying are rare.
declare
#sName varchar(max),
#sState varchar(max),
#iState int,
#sDbName varchar(50)
set #sDbName = 'myDbName'
set #iState = 7 --copying
while #iState = 7
begin
WAITFOR DELAY '00:00:05'
SELECT #sName = name, #iState = state, #sState = state_desc FROM sys.databases WHERE name = #sDbName
end
SELECT name, state,state_desc FROM sys.databases WHERE name = #sDbName
This code is actually run from a Azure worker in c#
Edit:
If you want more granular approch use sys.dm_database_copies
But otherwise remember that this sql returns the state as integer, and its only successfull if the state ==0 Othervise the copy has failed.
the states are:
0 = ONLINE
1 = RESTORING
2 = RECOVERING
3 = RECOVERY_PENDING
4 = SUSPECT
5 = EMERGENCY
6 = OFFLINE
7 = COPYING (Applies to Windows Azure SQL Database)
Windows Azure SQL Database returns the states 0, 1, 4, and 7.

Related

Executing SQL Script causes error in procedure called by trigger

I'm also new to DevOps, but let me see if I can explain my situation, also if anyone has any advice on doing post deployment scripts and can share their experiences I will be most grateful.
I use SQL Source control from Redgate, Git and Azure DevOps. I have managed to get the build and deployment working perfectly. Additionally we would like to make some data changes so we have additional (Data Change) scripts we wish to run. Linking these tables as static data is not an option as we would end up having most tables linked and end up with extremely timeous build and deploy times.
The script inserts data to a table, thereby triggering the insert trigger which calls a procedure to write to an audit table. The actual error from the deployment log is:
Task : DbUp Migration
2021-01-25T12:52:17.2629353Z Description : Runs SQL Server change scripts, and only those which have not been run already.
2021-01-25T12:52:17.2629405Z Version : 2.1.4
2021-01-25T12:52:17.2629671Z Author : Johan Classon
2021-01-25T12:52:17.2629721Z Help : [More Information](https://github.com/johanclasson/vso-agent-tasks)
2021-01-25T12:52:17.2629775Z ==============================================================================
2021-01-25T12:52:19.6610608Z Beginning database upgrade
2021-01-25T12:52:19.6689401Z Checking whether journal table exists..
2021-01-25T12:52:19.6728756Z Journal table does not exist
2021-01-25T12:52:19.7775348Z Executing Database Server script '001 - EX27605 - rdl.sql'
2021-01-25T12:52:19.7908913Z Checking whether journal table exists..
2021-01-25T12:52:19.7929056Z Creating the [dbo].[_SchemaVersions] table
2021-01-25T12:52:19.8077238Z The [dbo].[_SchemaVersions] table has been created
2021-01-25T12:52:20.2264323Z ifExists - rdl_Rule_Definition_Lookup - rdl_Code = "9900226e"
2021-01-25T12:52:20.2277308Z insert - rdl_Rule_Definition_Lookup - rdl_Code = "9900226e"
2021-01-25T12:52:20.2805780Z SQL exception has occured in script: '001 - EX27605 - rdl.sql'
2021-01-25T12:52:20.2942912Z ##[error]Script block number: 0; Block line 74; Message: Trig_After_Ins_Upd_Del_rdl_Rule_Definition_Lookup
2021-01-25T12:52:20.4528752Z ##[error]System.Data.SqlClient.SqlException (0x80131904): p_dte_Audit_Backend: (Line: 108) [dte_admin]
So the 1st error refers to line 74 of the trigger, if I read this correctly which is a commit on a procedure call to write to the audit table, line 108 of my procedure sets the userID:
DECLARE #user VARCHAR(40)
SET #user =
CASE
WHEN suser_sname() = 'NT SERVICE\SQLAgent$'+##servicename THEN 'dte_admin'
WHEN suser_sname() = 'NT SERVICE\SQLSERVERAGENT' THEN 'dte_admin'
ELSE suser_sname()
END
--###############
select #udt_Audit = udt_Audit
from udt_User_Detail (nolock)
where udt_User_Id = #user--suser_sname()
--select #udt_Audit = udt_Audit from udt_User_Detail
--where udt_User_Id = suser_sname()
if ##rowcount = 0
begin
declare #udt_User_Id varchar(30)
set #udt_User_Id = suser_sname()
The last line being line 108.
Important to note that dte_admin is a login on the SQL Server, it is a sysadmin account and mapped to the database, the dte_admin user is also a user at database level and exists as a user in my user table udt_user_detail.
During the build and deployment I am using variables and they are set to dte_admin username and password.
My question is then, why does this fail with this user?
Deployment Pipeline result

synchronization between 2 applications pooling a SQL table

I have 2 instances of a VB.NET application each running on their own dedicated servers. The said application runs a While true loop with a 5s sleep on IDLE (IDLE is when the Table doesn't have any ProcessQuery to be treated). On each iteration, the application questions a table in the SQL Database to know if there is anything it could process.
The problem is that i sometimes encounter the problem where both of the instances are "taking" the same ProcessQuery.
I'm using EntityFramework6. I have looked into EntityState but i don't think it does exactly what i'm trying to accomplish.
I was wondering what would be my solution to have perfect parallel instances. It's not impossible at some point i have 12 instances running on 12 machines.
Thanks!
Dim conn As New Info_IndusEntities()
Dim DemandeWilma As WilmaDemandes = conn.WilmaDemandes.Where(Function(x) x.Site = 'LONDON' AndAlso x.Statut = 'toProcess').OrderBy(Function(x) x.RequestDate).FirstOrDefault
If Not IsNothing(DemandeWilma) Then
DemandeWilma.Statut = Statuts.EnTraitement.ToString
DemandeWilma.ServerName = Environment.MachineName
DemandeWilma.ProcessDate = DateTime.Now
conn.SaveChanges()
Return DemandeWilma
end if
UPDATE (21/06/19)
I found an article that I find interesting.
I started by adding a column to my Table :
UPDATED (21/06/19)
I then refreshed my model and changed the Concurrency Check property of RowVersion column in my ORM :
When I tested the update, here's the log of EF6 :
UPDATE [dbo].[WilmaDemandes] SET [Statut] = #0, [ServerName] = #1,
[DateDebut] = #2 WHERE (([ID] = #3) AND ([RowVersion] = #4)) SELECT
[RowVersion] FROM [dbo].[WilmaDemandes] WHERE ##ROWCOUNT > 0 AND [ID]
= #3
-- #0: 'EnTraitement' (Type = String, Size = 20)
-- #1: 'TRB5995' (Type = String, Size = 20)
-- #2: '2019-06-25 7:31:01 AM' (Type = DateTime2)
-- #3: '124373' (Type = Int32)
-- #4: 'System.Byte[]' (Type = Binary, Size = 8)
-- Executing at 2019-06-25 7:31:24 AM -04:00
-- Completed in 95 ms with result: SqlDataReader
Closed connection at 2019-06-25 7:31:24 AM -04:00
Exception thrown:
'System.Data.Entity.Infrastructure.DbUpdateConcurrencyException' in
EntityFramework.dll
UPDATED (25/06/19)
The problems, as explained in this post, starts when you are using DB-First instead of Code-First. Your property will get overwritten silently as soon as you update the model. Some people back then coded a console app workaround that they run on pre-build. I'm not sure i'm quite ready to take this solution as final solution.
Interesting tutorial on how to test optimistic concurrency and ways to resolve such an exception.
Add an "owner" column to your queue table
Your application updates one record (TOP 1) and sets the owner value to their identifier (WHERE Owner IS NULL)
Now your application goes back and reads their owned rows and processes them
It's a simple pattern and it works great. If any processes happen to take ownership 'simultaneously', only one will actually get the reservation.
I'm not very good at LINQ so here's a brute force method, multiline for clarity:
// First try reserving a row
conn.Database.ExecuteSqlCommand(
"WITH UpdateTop1 AS
(SELECT TOP 1 * FROM WilmaDemandes
WHERE Owner IS NULL
AND Site = 'LONDON'
ORDER BY RequestDate)
UPDATE UpdateTop1 SET Owner='ThisApplication'"
);
// See if we got one
Dim DemandeWilma As WilmaDemandes =
conn.WilmaDemandes.
Where(x => x.Owner=='ThisApplication').FirstOrDefault
// If we got a row, process it. Otherwise Idle and repeat
There's also no reason that you must reserve one row. You could reserve all the free rows and work your way through them. Meanwhile other processes will pick up any subsequently arriving rows
Personally I would refactor your status column and make it NULL for new records ready to be processed, otherwise it's the worker ID that has reserved it.
It also helps to add things like timestamp columns to record when the row was reserved etc.

SQL Server OLEDB error with no details

I have a SQL Server Agent job running, which uses a stored procedure to do several operations, then exports some data to an xls spreadsheet and emails that spreadsheet. Most of the time it works, but several times a month the job fails with the error:
OLE DB provider 'Microsoft.Jet.OLEDB.4.0' reported an error. The provider did not give any information about the error. [SQLSTATE 42000] (Error 7399). The step failed.
Thanks, Microsoft, for the detailed error message. Anyway, the short term fix is usually to simply re-run the job. Usually this works, but in rarer cases it does not, and I must restart the SQL Server instance.
Here is how my code interacts with OLEDB:
Insert into OPENROWSET('Microsoft.Jet.OLEDB.4.0', 'Excel 5.0;Database=\\Excel\POStatus\POStatus.xls;',
'SELECT * FROM [POStatus$]')
Select --Tons of columns with tons of math and functions
FROM --5 tables joined together (left joins)
WHERE -- Tons of where conditions
Order by --Case statement for custom sorting
Set #vCommand = 'copy \\Excel\POStatus\POStatus.xls \\Excel\POStatus\POStatus_' + #vDate + '.xls'
EXEC master..xp_cmdshell #vCommand , NO_OUTPUT
... omitted for brevity...
Set #nvSubject = ' POStatus ' + #vDate
Set #nvMessage = ' This is an automated message, please respond to the IS department, thank you '
Set #nvMessage = #nvMessage + char(13) + char(10)
Set #nvAttachments = '\\Excel\POStatus\POStatus_' + #vDate + '.xls'
Exec master..xp_sendmail
#recipients = #nvRecipients , #copy_recipients = #nvCopy_recipients ,
#subject = #nvSubject , #message = #nvMessage ,
#query = #nvQuery , #width = #iWidth , #attachments = #nvAttachments
So, what is the cause of this, and how can I prevent it?
When you call OPENROWSET, it load the DLL for OLEDB provider for Excel. These operations happens inside the SQL Server Stack memory. When a lot of call to older DLL (ActiveX/COM) are made, it's not impossible that your stack might get full. You can resolve this in 2 ways:
1) You make these operations outside a TSQL code. You can use a SSIS package for example but you have to change your code not to use OPENROWSET. You can use the DTS Wizard to do most of the job. This is my personnal recommandation !
2) You can also set SQL Server to have a bigger memory Stack by using the -g command parameter. I think 256 MB is the default and you can set it to 512. To do that you need to open the SQL Server Configuration Manager. Depending on your version, you should have a place to change the "Startup Parameters"

"Uninitialized subscription" error in replication monitor

I'm using SQL Server 2012 and trying to implement transactional replication. Im using the system stored procedures to create the publications and subscriptions. I was successful in creating these things, but when i check the replication monitor, it shows "Uninitialized subscription".
When I check the synchronization status on the subscription, i found this log
Date 6/20/2012 7:36:33 PM
Log Job History (HYDHTC0131320D-PublisherDB-PublicationOne-HYDHTC0131320D\MSS-ReplicationSubscri-7C1D7509-C8A6-4073-A901-0433A2B6D2D3)
Step ID 1
Server HYDHTC0131320D\MSSQLSERVER2
Job Name HYDHTC0131320D-PublisherDB-PublicationOne-HYDHTC0131320D\MSS-ReplicationSubscri-7C1D7509-C8A6-4073-A901-0433A2B6D2D3
Step Name Run agent.
Duration 00:07:41
Sql Severity 0
Sql Message ID 0
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0
Message
2012-06-20 14:14:13.986 Copyright (c) 2008 Microsoft Corporation
2012-06-20 14:14:13.986 Microsoft SQL Server Replication Agent: distrib
2012-06-20 14:14:13.986
2012-06-20 14:14:13.986 The timestamps prepended to the output lines are expressed in terms of UTC time.
2012-06-20 14:14:13.986 User-specified agent parameter values:
-Publisher HYDHTC0131320D
-PublisherDB PublisherDB
-Publication PublicationOne
-Distributor HYDHTC0131320D
-SubscriptionType 2
-Subscriber HYDHTC0131320D\MSSQLSERVER2
-SubscriberSecurityMode 1
-SubscriberDB ReplicationSubscriberDB
-Continuous
-XJOBID 0xDFE51AEC7F9E3F42A450CE8874B662CD
-XJOBNAME HYDHTC0131320D-PublisherDB-PublicationOne-HYDHTC0131320D\MSS-ReplicationSubscri-7C1D7509-C8A6-4073-A901-0433A2B6D2D3
-XSTEPID 1
-XSUBSYSTEM Distribution
-XSERVER HYDHTC0131320D\MSSQLSERVER2
-XCMDLINE 0
-XCancelEventHandle 000005F8
-XParentProcessHandle 00000560
2012-06-20 14:14:13.986 Startup Delay: 619 (msecs)
2012-06-20 14:14:14.606 Connecting to Subscriber 'HYDHTC0131320D\MSSQLSERVER2'
2012-06-20 14:14:14.656 Connecting to Distributor 'HYDHTC0131320D'
2012-06-20 14:14:14.671 Parameter values obtained from agent profile:
-bcpbatchsize 2147473647
-commitbatchsize 100
-commitbatchthreshold 1000
-historyverboselevel 1
-keepalivemessageinterval 300
-logintimeout 15
-maxbcpthreads 1
-maxdeliveredtransactions 0
-pollinginterval 5000
-querytimeout 1800
-skiperrors
-transactionsperhistory 100
2012-06-20 14:14:14.683 Agent message code 21040. Publication '' does not exist.
How do I solve this issue?
I received the same error. My fix was to explicitly define the Job_login and job_password which I had as null to start with.
EXEC sp_addpullsubscription_agent
#publisher = #publisher,
#publisher_db = #publicationDB,
#publication = #publication,
#distributor = #publisher,
#job_login = $(Login),
#job_password = $(Password);
It seems there is an error in your replication setup scripts.
I suspect the error is in the call to sp_addpushsubscription_agent (if it is push subscription) or sp_addpullsubscription_agent (if it is pull subscription). Specifically the #publication parameter is wrong as the Distribution Agent is stating that the specified Publication " does not exist.
Please review your script and try again.
I faced same problem and fixed by doing the following :
-Subscriber Job owner is same as Publication User
-Subscriber User has been added to the subscriber users list and added
to the sysadmin server role
Just a few notes as I managed to get a pull subscriber up and running with a backup initialisation:
Make sure you have the agent account of the publisher, distributor and subscriber as sysadmin logins across all instances, example subscriber must be on the distributor and publisher.
Create linked servers for all components on each instance, example distributor and publisher on the subscriber.
Then take a full backup of the source database and restore without recovery on the subscription instance.
Create the publication component on the publisher database (sp_addsubscription) and make sure you have #sync_type = N'replication support only'.
Then take a diff backup of the source database and restore with recovery on the subscription instance.
Then create the pull subscription on the subscription instance (sp_addpullsubscription), note that if the source is a mirrored instance, use the original instance value here (#publisher). If you the principal on the "mirror" instance, meta data will be created correct and the distributor contains the logic to connect the log reader to either of the mirror instances.
Now you will have a problem with primary keys etc failing as the distributor started capturing data before the differential backup ... no problem. Stop the subscriber job and then add a new profile to the distributor agent with setting the -SkipErrors parameter to "2601:2627". This will skip all primary key violation transactions and continue processing. Remember to select this agent profile then and hit "OK".
Restart the subscription job and monitor the subscription as it start to catch up on the transactions.
When caught up, stop the job, change the agent profile back to the default and restart the job.
Hope this help anyone still struggling with pull subscribers based on backup initialisation ... note that at no point did I use any backup configurations within the replication configuration. Also stated that the pull subscription must not initialise (#immediate_sync = 0).
Here is the scripts:
-----------------BEGIN: Script to be run at Publisher 'DB001\OLTP'-----------------
use [DB1]
go
exec sp_addsubscription #publication = N'DB1', #subscriber = N'DB002\OLTP', #destination_db = N'DB1', #sync_type = N'replication support only', #subscription_type = N'pull', #update_mode = N'read only'
GO
-----------------END: Script to be run at Publisher 'DB001\OLTP'-----------------
-----------------BEGIN: Script to be run at Subscriber 'DB002\OLTP'-----------------
use [DB1]
exec sp_addpullsubscription #publisher = N'DB001\OLTP', #publication = N'DB1', #publisher_db = N'DB1', #independent_agent = N'True', #subscription_type = N'pull', #description = N'', #update_mode = N'read only', #immediate_sync = 0
exec sp_addpullsubscription_agent #publisher = N'DB001\OLTP', #publisher_db = N'DB1', #publication = N'DB1', #distributor = N'DB003\DIST', #distributor_security_mode = 1, #distributor_login = N'', #distributor_password = null, #enabled_for_syncmgr = N'False', #frequency_type = 64, #frequency_interval = 0, #frequency_relative_interval = 0, #frequency_recurrence_factor = 0, #frequency_subday = 0, #frequency_subday_interval = 0, #active_start_time_of_day = 0, #active_end_time_of_day = 235959, #active_start_date = 20170327, #active_end_date = 99991231, #alt_snapshot_folder = N'\\DB001\Replication', #working_directory = N'', #use_ftp = N'False', #job_login = null, #job_password = null, #publication_type = 0
GO
-----------------END: Script to be run at Subscriber 'DB002\OLTP'-----------------

SQL Server Agent 2005 job runs but no output

Essentially I have a job which runs in BIDS and as as a stand lone package and while it runs under the SQL Server Agent it doesn't complete properly (no error messages though).
The job steps are:
1) Delete all rows from table;
2) Use For each loop to fill up table from Excel spreasheets;
3) Clean up table.
I've tried this MS page (steps 1 & 2), didn't see any need to start changing from Server side security.
Also SQLServerCentral.com for this page, no resolution.
How can I get error logging or a fix?
Note I've reposted this from Server Fault as it's one of those questions that's not pure admin or programming.
I have logged in as the proxy account I'm running this under, and the job runs stand alone but complains that the Excel tables are empty?
Here's how I managed tracking "returned state" from an SSIS package called via a SQL Agent job. If we're lucky, some of this may apply to your system.
Job calls a stored procedure
Procedure builds a DTEXEC call (with a dozen or more parameters)
Procedure calls xp_cmdshell, with the call as a parameter (#Command)
SSIS package runs
"local" SSIS variable is initialized to 1
If an error is raised, SSIS "flow" passes to a step that sets that local variable to 0
In a final step, use Expressions to set SSIS property "ForceExecutionResult" to that local variable (1 = Success, 0 = Failure)
Full form of the SSIS call stores the returned value like so:
EXECUTE #ReturnValue = master.dbo.xp_cmdshell #Command
...and then it gets messy, as you can get a host of values returned from SSIS . I logged actions and activity in a DB table while going through the SSIS steps and consult that to try to work things out (which is where #Description below comes from). Here's the relevant code and comments:
-- Evaluate the DTEXEC return code
SET #Message = case
when #ReturnValue = 1 and #Description <> 'SSIS Package' then 'SSIS Package execution was stopped or interrupted before it completed'
when #ReturnValue in (0,1) then '' -- Package success or failure is logged within the package
when #ReturnValue = 3 then 'DTEXEC exit code 3, package interrupted'
when #ReturnValue in (4,5,6) then 'DTEXEC exit code ' + cast(#Returnvalue as varchar(10)) + ', package could not be run'
else 'DTEXEC exit code ' + isnull(cast(#Returnvalue as varchar(10)), '<NULL>') + ' is an unknown and unanticipated value'
end
-- Oddball case: if cmd.exe process is killed, return value is 1, but process will continue anyway
-- and could finish 100% succesfully... and #ReturnValue will equal 1. If you can figure out how,
-- write a check for this in here.
That last references the "what if, while SSIS is running, some admin joker kills the CMD session (from, say, taskmanager) because the process is running too long" situation. We've never had it happen--that I know of--but they were uber-paranoid when I was writing this so I had to look into it...
Why not use logging built into SSIS? We send our logs toa database table and then parse them out to another table in amore user friendly format and can see every step of everypackage that was run. And every error.
I did fix this eventually, thanks for the suggestions.
Basically I logged into Windows with the proxy user account I was running and started to see errors like:
"The For each file enumerator is empty"
I copied the project files across and started testing, it turned out that I'd still left a file path (N:/) in the properties of the For Each loop box, although I'd changed the connection properties. Easier once you've got error conditions to work with. I also had to recreate the variable mapping.
No wonder people just recreate the whole package.
Now fixed and working!