We have a tables in Db2, That which we need to get that table to MS SQL server (only for read), And I want it to be in sync for every 15 minutes (one way from DB2 to SQL Server). Can you suggest the best approach?
Have a SQL Agent job execute an SSIS package every 15 minutes.
I know that all the time MERGE is the right option to sync the tables in the SQL. But I am not sure, whether we can use it in linked servers also. Anyway, after some research I got this task accomplished by using the merge join. Merge will update, insert, delete what ever required. But it will take a little bit more time to update the table for every 15 min, when the job runs. So, you can create a #Temptable to insert the transactions that were done from the lastjob done.You can use the datetime stamp in that source table to retrieve the transactions that were done from the last job done(15min). If you don't have the date time in source table, you can use the audit table for that source table(if applicable).
(JLT table have 3 columns (last_job_end)( cur_job_start)(some job identity). JLT is the job log table we need to create in linked server to get the last job end and cur job start time, We need to update last job end every time at the end of query in JOB. As well as cur job start in the beginning of the job )
SELECT *
INTO #TEMPtable
FROM OPENQUERY([DB2], 'Select * from source_table
where some_id_column in
(select some_id_column
from audit_table AT, Job_log_table JLT
where datetime > last_job_end
and datetime <= cur_job_start
and c_job = ''some_job_id'')’)`
If you don't have the audit table and you have the datetime in Source.
SELECT *
INTO #TEMPtable
FROM OPENQUERY([DB2], 'Select *
from source_table s, JOB_CYCLE_TABLE pr
where s.DATETIME <= pr.cur_job_start
and s.DATETIME > pr.last_job_end
and pr.c_job = ''some_job_id''')
Related
I have a range partitioned table in my database, it is range partitioned by a date column: transaction_date, with 1 partition per 1 month.
Now my problem is:
When running SQL statement to read data from the table,
select col1,col2 from mytable where ID=1
My table is very large so it takes a long time for the SQL to finish.
However, there is another ETL job to insert (append) data to the table at the same time, the insert operation cannot start until the read SQL finishes.
Any suggestions I can avoid this issue while reading data? Also are there any IBM official documents regarding this problem?
** EDIT 1:
$ db2level
DB21085I This instance or install (instance name, where applicable:
"db2inst1") uses "64" bits and Db2 code release "SQL11011" with level
identifier "0202010F".
Informational tokens are "DB2 v11.1.1.1", "s1610100100",
"DYN1610100100AMD64", and Fix Pack "1".
Product is installed at "/opt/ibm/db2/v11.1".
$ db2set -all
[i] DB2COMM=TCPIP
[i] DB2AUTOSTART=TRUE
[i] DB2OPTIONS=+c
[g] DB2FCMCOMM=TCPIP4
[g] DB2SYSTEM=<server hostname>
[g] DB2INSTDEF=db2inst1
** EDIT 2:
For the select and load SQL statement, I am not specifying any isolation level.
For the ETL job, it is an IBM DataStage job, the ETL insert is a bulk load append operation to insert data to a pre-existing range.
You may use the MON_LOCKWAITS administrative view to check what's happening during such a lock situation. You may optionally format a lock with the MON_FORMAT_LOCK_NAME function to get more details on this as well.
SELECT
W.*
--, F.NAME, F.VALUE
FROM SYSIBMADM.MON_LOCKWAITS W
--, TABLE(MON_FORMAT_LOCK_NAME(W.LOCK_NAME)) F
--WHERE W.REQ_APPLICATION_HANDLE = XXX -- if you know the holder's handle to reduce the amount of data returned
ORDER BY W.LOCK_NAME
;
I want to find the sql server agent jobs that operate upon a particular table. For ex:
I have a table called TAB1 which is updated daily by a job called SAJ1. I need a query to extract this information.
You can start with this select if the job step in T-SQL then it will find the text what you will declare as table name.
use msdb
Declare #table_name varchar(50)
set #table_name='Test'
select j.name,js.command from dbo.sysjobs j
inner join
dbo.sysjobsteps js
on j.job_id=js.job_id
where command like '%'+#table_name+'%'
One of the options available is the table msdb.dbo.sysjobsteps that enlists all the jobs and their steps. You can go into the Command column of the output and look for required information.
select * from msdb.dbo.sysjobsteps;
I have a table which has two attributes: ID & Datetime when that record was created.
How can I make a trigger (procedure?) to delete a record after, say, 1 day?
I want a task which executes itself every X time instead of manually having to do it.
I suggest you to use SQL Server agent and write a stored procedure that deletes every rows that date is passed one day.
You can find how to use Sql server agent jobs here in this link.
And the stored procedure like :
CREATE PROCEDURE DeleteRows()
AS
BEGIN
DELETE FROM mytable
WHERE (DATEDIFF(DAY, DATEADD(day, -1, DateColum), GETDATE())) >= 1
END
Edit :
The number 1 in where statement is days. you can change it to what you want to use.
The statement to delete a record older than one day is:
DELETE FROM tableName WHERE DATEDIFF(day, getdate(), dateColumn) < -1
You would need to cron that statement using whatever language you have available, php for instance.
That said of course it's hard to imagine a scenario where you would want to be deleting records in the first place ;)
You could create a SQL Job to run after one day (or daily as you want.) and execute a specified stored procedure that carries a simple delete statement.
Follow the next topic:-
http://technet.microsoft.com/en-us/library/ms190268.aspx
I need to insert records into a new table from an existing table. I used the following query to do this:
Insert into Newtable
Select * from Oldtable where date1 = #date
This query works most of the time, but in one scenario I get 10 million records to be inserted for the date1 value. In this case I'm getting the following error message:
Error : The transaction log for database "tempDB" is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases
Should I break the query into parts and insert them sequentially, or is there a way to do this with the current query?
This is, perhaps, a distasteful suggestion. But, you can try exporting the data to a file and then inserting using bulk-insert, with database logging set to SIMPLE or BULK-LOGGED.
More information is at http://msdn.microsoft.com/en-us/library/ms190422.aspx.
I have a stored procedure that essentially rebuilds a pivot table. It builds the new data in a temp table, then truncates the permanent table and inserts the new data, and finally drops the temp table.
When I execute the stored proc (or the T-SQL code directly) in Management Studio, it takes about a minute. While I know this isn't the most efficient of processes, I'm OK with it.
My problem comes in when I try to schedule this task to run every 20 minutes or so. When I setup a SQL Server Agent Job to execute the stored proc, its now taking almost an hour and a half... that's right, 90 TIMES SLOWER!
I found this post: SQL Server Agent Job Running Slow, which seems to be a similar issue, but set nocount on doesn't seem to have any effect whether I call it at the beginning of the stored proc or before the exec command in the SQL Agent Job. My query doesn't use any cursors, though I am doing a cross apply on a table valued function (which also doesn't use any cursors).
I'm obviously missing something, but I don't even know where to start on this. I thought by creating the stored proc I would have avoided these types of issues.
For reference, the stored proc looks something like the following:
create table #temp
(
ID int,
data1 float,
data2 float
)
insert into #temp(ID, data1, data2)
select t.ID, d.data1, d.data2
from tbl1 t
cross apply dbo.getInterestingData(t.ID, t.param1) d
where d.useMe = 1
truncate table dataPivot
insert into dataPivot(ID, data1, data2)
select ID, data1, data2
from #temp
drop table #temp