I have a stored procedure that essentially rebuilds a pivot table. It builds the new data in a temp table, then truncates the permanent table and inserts the new data, and finally drops the temp table.
When I execute the stored proc (or the T-SQL code directly) in Management Studio, it takes about a minute. While I know this isn't the most efficient of processes, I'm OK with it.
My problem comes in when I try to schedule this task to run every 20 minutes or so. When I setup a SQL Server Agent Job to execute the stored proc, its now taking almost an hour and a half... that's right, 90 TIMES SLOWER!
I found this post: SQL Server Agent Job Running Slow, which seems to be a similar issue, but set nocount on doesn't seem to have any effect whether I call it at the beginning of the stored proc or before the exec command in the SQL Agent Job. My query doesn't use any cursors, though I am doing a cross apply on a table valued function (which also doesn't use any cursors).
I'm obviously missing something, but I don't even know where to start on this. I thought by creating the stored proc I would have avoided these types of issues.
For reference, the stored proc looks something like the following:
create table #temp
(
ID int,
data1 float,
data2 float
)
insert into #temp(ID, data1, data2)
select t.ID, d.data1, d.data2
from tbl1 t
cross apply dbo.getInterestingData(t.ID, t.param1) d
where d.useMe = 1
truncate table dataPivot
insert into dataPivot(ID, data1, data2)
select ID, data1, data2
from #temp
drop table #temp
Related
I am supposed to do incremental load and using below structure.
Do the statements execute in sequence i.e. TRUNCATE is never executed before first two statements which are getting data:
#newData = Extract ... (FROM FILE STREAM)
#existingData = SELECT * FROM dbo.TableA //this is ADLA table
#allData = SELECT * FROM #newData UNION ALL SELECT * FROM #existingData
TRUNCATE TABLE dbo.TableA;
INSERT INTO dbo.TableA SELECT * FROM #allData
To be very clear: U-SQL scripts are not executed statement by statement. Instead it groups the DDL/DML/OUTPUT statements in order and the query expressions are just subtrees to the inserts and outputs. But first it binds the data during compilation to their names, so your SELECT from TableA will be bound to the data (kind of like a light-weight snapshot), so even if the truncate is executed before the select, you should still be able to read the data from table A (note that permission changes may impact that).
Also, if your script fails during the execution phase, you should have an atomic execution. That means if your INSERT fails, the TRUNCATE should be undone at the end.
Having said that, why don't you use INSERT incrementally and use ALTER TABLE REBUILD periodically instead of doing the above pattern that reads the full table on every insertion?
We have a tables in Db2, That which we need to get that table to MS SQL server (only for read), And I want it to be in sync for every 15 minutes (one way from DB2 to SQL Server). Can you suggest the best approach?
Have a SQL Agent job execute an SSIS package every 15 minutes.
I know that all the time MERGE is the right option to sync the tables in the SQL. But I am not sure, whether we can use it in linked servers also. Anyway, after some research I got this task accomplished by using the merge join. Merge will update, insert, delete what ever required. But it will take a little bit more time to update the table for every 15 min, when the job runs. So, you can create a #Temptable to insert the transactions that were done from the lastjob done.You can use the datetime stamp in that source table to retrieve the transactions that were done from the last job done(15min). If you don't have the date time in source table, you can use the audit table for that source table(if applicable).
(JLT table have 3 columns (last_job_end)( cur_job_start)(some job identity). JLT is the job log table we need to create in linked server to get the last job end and cur job start time, We need to update last job end every time at the end of query in JOB. As well as cur job start in the beginning of the job )
SELECT *
INTO #TEMPtable
FROM OPENQUERY([DB2], 'Select * from source_table
where some_id_column in
(select some_id_column
from audit_table AT, Job_log_table JLT
where datetime > last_job_end
and datetime <= cur_job_start
and c_job = ''some_job_id'')’)`
If you don't have the audit table and you have the datetime in Source.
SELECT *
INTO #TEMPtable
FROM OPENQUERY([DB2], 'Select *
from source_table s, JOB_CYCLE_TABLE pr
where s.DATETIME <= pr.cur_job_start
and s.DATETIME > pr.last_job_end
and pr.c_job = ''some_job_id''')
I am working in SQL Server 2012 Management studio.
In a SQL query window, an insert into a #table is happening. It is expected to insert somewhere around 80 million rows with 3 INT columns each.
The query execution is going on.
Is there a way that I can track the no of rows in the #table?
Since you cannot run two queries in the same window simultaneously and temp tables are not accessible in other sessions if they are declared with a single #, you should try defining it with a double # in your insert query.
Then you could try querying it using WITH(NOLOCK).
Open a new query window on the same db and try
SELECT COUNT(*)
FROM ##YourTableName WITH(NOLOCK)
This will get dirty reads, but i do not think it would be a problem in your case as you would like a rough measure on where your INSERT is.
One method is to query the DMVs using the temp table object id. You can get the local temp table object id from the session that created it using this query:
SELECT OBJECT_ID(N'tempdb..#table', 'U');
Then run the script below in another windows, supplying the object_id value from the above query (-1180342868 in this example):
DECLARE #object_id int = -1180342868;
SELECT SUM(rows)
FROM tempdb.sys.partitions
WHERE
object_id = #object_id
AND index_id IN(0,1);
Of course, this method assumes you had the foresight to get the temp table object id before running the insert. If the query is currently running, you could run the script below and make an educated guess as to which object might be the temp table being loaded.
USE tempdb;
SELECT OBJECT_NAME(object_id), SUM(rows)
FROM tempdb.sys.partitions
WHERE
index_id IN(0,1)
AND OBJECTPROPERTYEX(object_id, 'IsUserTable') = 1
GROUP BY
OBJECT_NAME(object_id);
Be aware that this might not be a reliable way to track the load progress. Much depends on the query plan particulars. It could be that the costly operators are earlier in the plan and the actual insert won't occur until the last minute.
If you wish to run the query to count rows in another window or outside the scope where the table was declared, please use a global temp table.
For Example,
CREATE TABLE ##table(
a int,
b int,
c int)
And the in another window you can run, this will work
SELECT COUNT(*) FROM ##table WITH (NOLOCK)
How do I lock a global temporary table in a stored procedure that's getting created and populated by a SELECT INTO statement? For example:
SELECT *
INTO ##TempEmployee
FROM Employee
This stored procedure is executed for generating reports and it's there in every client database (multi-tenant architecture using different DB per client). I do not want data in this global temporary table to be shared between clients when the report is generated concurrently. I don't have a choice but to use global temp table because I use it for generating columns on the fly using PIVOT.
Why not include it inside a transaction block like
begin transaction
SELECT *
INTO ##TempEmployee
FROM Employee
Try this,
WorkDummySQL
create table rr(id integer,name varchar(20))
insert into rr values(1,'aa')
select * from rr
Tempdb
select * into ##ta from WorkDummySQL.dbo.rr
I'm trying to create a SQL script with a delay.
I could use:
blah blah
WAITFOR DELAY '00:30:00'
blah blah
but using a script like this requires that I sit through the 30 minutes.
Leaving the website will cancel the script.
The problem is that I want to change something in a table, and then change it back automatically 30 minutes later.
The person making the change will leave the webpage, so any kind of client-side script is out of the question.
Nor can I wait for the person to return and make the change if 30 minutes have passed, the change must happen after 30 minutes regardless.
Is there anyway to do this without making a service or any other program on the server?
Using only ASP/SQL programming.
If this is impossible, how do I make a service or program on the server to make this change?
Must be able to start this from the website using ASP.
I personally would not approach the situation this way. I don't know exactly what your data structure is, or why you need to change something for 30 minutes, but I would use a 'Change' table.
So you might have something like
MainTable (ID, Column1, Column2, Column3, Column4);
ChangeTable (ID, Column1, Column2, Column3, Column4, CreatedDateTime);
Whenever you make your change instead of updating your main table you can simply insert the values you would be updating to into the ChangeTable (I'm assuming SQL-Server based on WAITFOR).
I would then make a view like so:
CREATE VIEW dbo.MainView
AS
SELECT m.ID,
Column1 = ISNULL(c.Column1, m.Column1),
Column2 = ISNULL(c.Column2, m.Column2),
Column3 = ISNULL(c.Column3, m.Column3)
FROM dbo.MainTable m
OUTER APPLY
( SELECT TOP 1 c.Column1, c.Column2, c.Column3
FROM dbo.ChangeTable c
WHERE c.ID = m.ID
AND c.CreatedDate >= DATEADD(MINUTE, -30, GETDATE())
ORDER BY c.CreatedDate DESC
) c;
Then refer to this throughout the website.
If space is an issue you could set up a nightly Job to delete any old entries, e.g. set the following to run at 00:30
DELETE ChangeTable
WHERE CreatedDate < CAST(GETDATE() AS DATE);
Personally, although others on this board would never think it is professional... I use website uptime monitoring services to run scripts. For example this one will hit an asp page of your choosing every 30 minutes http://www.serviceuptime.com/free_monitoring.php.
For ease of development I use different monitoring services to load scripts. There are more professional ways of doing so such as making a VBS script and running it through the Task Manager which I do for long running scripts but for simple things like checking a mail queue every so often I just use a monitoring service to load the page often enough to do what I want.
Figured it out by using SQL Server Agent and SQL procedures.
This is basically how my code is built up now:
Make the temporary change in the table
UPDATE table SET column = 'temp_value' WHERE column = 'normal_value'
Check if the procedure is there, if so, delete it. Create a procedure to revert the changes in the table.
IF EXISTS ( SELECT *
FROM sys.objects
WHERE object_id = OBJECT_ID(N'myRevertProcedure')
AND type IN ( N'P', N'PC' ) )
DROP PROCEDURE myRevertProcedure
CREATE PROCEDURE myRevertProcedure
AS
BEGIN
WAITFOR DELAY '00:30:00'
UPDATE table SET column = 'normal_value' WHERE column = 'temp_value'
END
I've created a job in the SQL Server Agent that runs the following:
IF EXISTS ( SELECT *
FROM sys.objects
WHERE object_id = OBJECT_ID(N'myRevertProcedure')
AND type IN ( N'P', N'PC' ) )
BEGIN
EXEC MyProc
DROP PROCEDURE myRevertProcedure
END
The reason the job does not simply revert the change itself is because the user shall set the delay.
If the delay were allways to be 30 mins, I could've made the job run the following:
IF EXISTS (SELECT * FROM table WHERE column = 'temp_value')
BEGIN
WAITFOR DELAY '00:30:00'
UPDATE table SET column = 'normal_value' WHERE column = 'temp_value'
END
By doing this, I would not need any procedure.
BTW: The job runs every few seconds.