I have to create a table through ETL in redshift. First, I am loading the data in a temporary table and then transacting it to the main table. The approach below is good when I want to update a data for a particular period. If I have just created the main_table and it doesn't have any data then how do the TRANSACTION and DELETE statements work for initial upload of data in main_table. I am new to ETL process.
BEGIN TRANSACTION;
DELETE FROM main_table
USING temporary_table
WHERE main_table.id = temporary_table.id
AND main_table.time_stamp = temporary_table.time_stamp
INSERT INTO main_table
SELECT * FROM temporary_table ;
END TRANSACTION;
Related
During the ETL we do the following operations:
begin transaction;
drop table if exists target_tmp;
create table target_tmp like target;
insert into target_tmp select * from source_a inner join source_b on ...;
analyze table target_tmp;
drop table target;
alter table target_tmp rename to target;
commit;
The SQL command is performed by AWS Data Pipeline, if this is important.
However, the pipelines sometimes fail with the following error:
ERROR: table 111566 dropped by concurrent transaction
Redshift supports serializable isolation. Does one of the commands break isolation?
Yes that works, but if generating the temp table takes a while you can expect to see that error for other queries while it runs. You could try generating the temp table in a separate transaction (transaction may not be needed unless you worry about updates to the source tables). Then do a quick rotation of the table names so there is much less time for contention:
-- generate target_tmp first then
begin;
alter table target rename to target_old;
alter table target_tmp rename to target;
commit;
drop table target_old;
I am having two databases. When a new customer record is added into fDebtor table in database ABC, a new record should be appended in fDebtor table in database DEF as well.
Can some one help me to write a stored procedure for this?
there should be a validation to check whether the record is already exists in next table also
create trigger TRG_Insert_fDebtor on ABC.fDebtor
ON AFTER Insert
as
begin
SET NOCOUNT ON;
--- checking for the record exist in the DEF database
IF (select count(*) from DEF.fDebtor where exists(select * from inserted)) > 0
begin
----- insert record in your other database DEF
insert into DEF.fDebtor values ()
end
end
Try trigger like that made changes as you need that's just approach you can use.
I have found a trigger example for Creation and Modification of the record but the question is, should I create those two triggers for each table or is there any way to run them on each update and insert regardless of the table name. Of course the names of the fields will be unique for each table for instance "CreationDate", "LastUpdate". Actually first question should have been, is creating a trigger for such a case a correct practice or should I handle it on code behind?
Here is the trigger that I have found on the internet;
CREATE TRIGGER tr[TableName]CreateDate ON [TableName]
FOR INSERT
AS
UPDATE [TableName] SET [TableName].Created=getdate()
FROM [TableName] INNER JOIN Inserted ON [TableName].[UniqueID]= Inserted.[UniqueID]
GO
CREATE TRIGGER tr[TableName]LastModifiedDate ON [TableName]
FOR UPDATE
AS
UPDATE [TableName] SET [TableName].LastModified=getdate()
FROM [TableName] INNER JOIN Inserted ON [TableName].[UniqueID]= Inserted.[UniqueID]
Triggers can be created on DML (Tables, Views events) or DDL (Create, Alter, Drop etc). You can not create a generic trigger which applies to all tables, you need to specify the table name.
You could create a script which automates the Trigger scripts creation for all tables if need be.
More info on: http://msdn.microsoft.com/en-us/library/ms189799.aspx
Just give your trigger the option to run for INSERT AND UPDATE
CREATE TRIGGER [dbo].[tr_TableName] ON [dbo].[TableName] FOR INSERT,DELETE,UPDATE AS
BEGIN
/*
Do stuff here.
*/
Select * from Inserted
Select * from deleted
END
I have a table Which has more than 1 million records, I have created a stored Procedure to insert data in that table, before Inserting the data I need to truncate the table but truncate is taking too long.
I have read on some links that if a table is used by another person or some locks are applied then truncate takes too long time but here I am the only user and I have applied no locks on that.
Also no other transactions are open when I tried to truncate the table.
As my database is on SQL Azure I am not supposed to drop the indexes as it does not allow me to insert the data without an index.
Drop all the indexes from the table and then truncate, if you want to insert the data then insert data and after inserting the data recreate the indexes
When deleting from Azure you can get into all sorts of trouble, but truncate is almost always an issue of locking. If you can't fix that you can always do this trick when deleting from Azure.
declare #iDeleteCounter int =1
while #iDeleteCounter > 0
begin
begin transaction deletes;
with deleteTable as
(
select top 100000 * from mytable where mywhere
)
delete from deleteTable
commit transaction deletes
select #iDeleteCounter = count(1) from mytable where mywhere
print 'deleted 100000 from table'
end
I have a table in a SQL db that I want to remove the data from? I want to keep the columns though.
e.g. my table has 3 columns, Name, Age, Date. I don't want to remove these, i just want to remove the data.
Should I should Truncate, Delete or Drop?
Don't drop - it will delete the data and the definition.
If you delete - the data is gone and auto-increment values go on from the last value.
If you truncate - then it is like you just did create the table. No data and all counters resetted
Truncate is very fast - like quick format of the table. It does not require any extra space when deleting. You can not rollback his operation. You can not specify conditions. This is best choice for deleting all data from table.
Delete is much slower and you need extra space for this operation, because you must be able to rollback the data. If you need to delete all data, use truncate. If you need to specify conditions, use delete.
Drop table - you can delete data from table by dropping and creating it again like you would truncate it, but it is slower and you can have some other problems, like foreign key dependencies. I definitely don't recommend this operation.
delete from TableName should do the trick
DROPing the table will remove the colums too.
Delete: The DELETE Statement is used to delete rows from a table.
Truncate: The SQL TRUNCATE command is used to delete all the rows from the table and free the space containing the table.
*Drop:* The SQL DROP command is used to remove an object from the database. If you drop a table, all the rows in the table is deleted and the table structure is removed from the database
Truncate the table. That would be good option in your case
We can rollback the data in conditions of Delete, Truncate & Drop.
But must be used Begin Transaction before executing query Delete, Drop & Truncate.
Here is example :
Create Database Ankit
Create Table Tbl_Ankit(Name varchar(11))
insert into tbl_ankit(name) values('ankit');
insert into tbl_ankit(name) values('ankur');
insert into tbl_ankit(name) values('arti');
Select * From Tbl_Ankit
/*======================For Delete==================*/
Begin Transaction
Delete From Tbl_Ankit where Name='ankit'
Rollback
Select * From Tbl_Ankit
/*======================For Truncate==================*/
Begin Transaction
Truncate Table Tbl_Ankit
Rollback
Select * From Tbl_Ankit
/*======================For Drop==================*/
Begin Transaction
Drop Table Tbl_Ankit
Rollback
Select * From Tbl_Ankit