I need help optimizing my stored procedure. This is for our fact table, and currently the stored procedure truncates the table, and then loads the data back in. I want to get rid of truncating and instead append new rows or delete rows by a last_update column which currently does not exist. There also is a last_update table with one column, which changes at every stored procedure run, but I'd rather the last_update be a column in the table itself, rather than a separate column.
I've created a trigger that should update the last_updated column with the current date when the stored procedure runs, but I would also like to get rid of truncating and instead append/delete rows as well. The way the stored procedure is currently structured is making it difficult for me to figure out how best to do it.
The stored procedure begins by adding data into 2 temp tables, then adds the data from the two temp tables into a 3rd temp table, then truncates the current FACT TABLE and then the 3rd temp table finally inserts into the FACT table.
--CLEAR LAST UPDATE TABLE
TRUNCATE TABLE ADM.LastUpdate;
--SET NEW LAST UPDATE TIME
INSERT INTO ADM.LastUpdate(TABLE_NAME, UPDATE_TIME)
VALUES('FactBP', CONVERT(VARCHAR, GETDATE(), 100)+' (CST)');
--CHECK TO SEE IF TEMP TABLES EXISTS THEN DROP
IF OBJECT_ID('tempdb.dbo.#TEMP_CARTON', 'U') IS NOT NULL
DROP TABLE #TEMP_CARTON;
IF OBJECT_ID('tempdb.dbo.#TEMP_ORDER', 'U') IS NOT NULL
DROP TABLE #TEMP_ORDER;
--CREATE TEMP TABLES
SELECT *
INTO #TEMP_CARTON
FROM [dbo].[FACT_CARTON_V];
SELECT *
INTO #TEMP_ORDER
FROM [dbo].[FACT_ORDER_V];
--CHECK TO SEE IF DATA EXISTS IN #TEMP_CARTON AND #TEMP_ORDER
IF EXISTS(SELECT * FROM #TEMP_CARTON)
AND EXISTS(SELECT * FROM #TEMP_ORDER)
--CODE HERE joins the data from #TEMP_CARTON and #TEMP ORDER and puts it into a 3rd temp table #TEMP_FACT.
--CLEAR ALL DATA FROM FACTBP
TRUNCATE TABLE dbo.FactBP;
--INSERT DATA FROM TEMP TABLE TO FACTBP
INSERT INTO dbo.FactBP
SELECT
[SOURCE]
,[DC_ORDER_NUMBER]
,[CUSTOMER_PURCHASE_ORDER_ID]
,[BILL_TO]
,[CUSTOMER_MASTER_RECORD_TYPE]
,[SHIP_TO]
,[CUSTOMER_NAME]
,[SALES_ORDER]
,[ORDER_CARRIER]
,[CARRIER_SERVICE_ID]
,[CREATE_DATE]
,[CREATE_TIME]
,[ALLOCATION_DATE]
,[REQUESTED_SHIP_DATE]
,[ADJ_REQ_SHIP]
,[CANCEL_DATE]
,[DISPATCH_DATE]
,[RELEASED_DATE]
,[RELEASED_TIME]
,[PRIORITY_ORDER]
,[SHIPPING_LOAD_NUMBER]
,[ORDER_HDR_STATUS]
,[ORDER_STATUS]
,[DELIVERY_NUMBER]
,[DCMS_ORDER_TYPE]
,[ORDER_TYPE]
,[MATERIAL]
,[QUALITY]
,[MERCHANDISE_SIZE_1]
,[SPECIAL_PROCESS_CODE_1]
,[SPECIAL_PROCESS_CODE_2]
,[SPECIAL_PROCESS_CODE_3]
,[DIVISION]
,[DIVISION_DESC]
,[ORDER_QTY]
,[ORDER_SELECTED_QTY]
,[CARTON_PARCEL_ID]
,[CARTON_ID]
,[SHIP_DATE]
,[SHIP_TIME]
,[PACKED_DATE]
,[PACKED_TIME]
,[ADJ_PACKED_DATE]
,[FULL_CASE_PULL_STATUS]
,[CARRIER_ID]
,[TRAILER_ID]
,[WAVE_NUMBER]
,[DISPATCH_RELEASE_PRIORITY]
,[CARTON_TOTE_COUNT]
,[PICK_PACK_METHOD]
,[RELEASED_QTY]
,[SHIP_QTY]
,[MERCHANDISE_STYLE]
,[PICK_WAREHOUSE]
,[PICK_AREA]
,[PICK_ZONE]
,[PICK_AISLE]
,EST_DEL_DATE
FROM #TEMP_FACT;
Currently, since I've added the last_updated column into my FACT TABLE and created a trigger, I don't actually pass any value via the stored procedure for it, so I get an error
An object or column name is missing or empty.
I am not sure as to where I'm supposed to pass any value for the LAST_UPDATED column.
Here is the trigger I've created for updating the last_updated column:
CREATE TRIGGER last_updated
ON dbo.factbp
AFTER UPDATE
AS
UPDATE dbo.factbp
SET last_updated = GETDATE()
FROM Inserted i
WHERE dbo.factbp.id = i.id
The first thing I would try is to create primary keys on the two temp tables #TEMP_CARTON and #TEMP_ORDER and use the intersect command to get the rows that are common to both tables:
select * from #TEMP_CARTON
intersect
SELECT * FROM #TEMP_ORDER
Figured out the answer. I just had to put "null" for the last_updated value during Insert, and then the Trigger took care of adding the timestamp on its own.
Related
In my SQL Server table, I have a table whose PK is GUID with lots of records already.
Now I want to add records which only needs to change the COMMON_ID and COMMON_ASSET_TYPE column of some existing records.
select * from My_Table where COMMON_ASSET_TYPE = "ASSET"
I am writing sql to copy above query result, changing COMMON_ID value to new GUID value and COMMON_ASSET_TYPE value from "ASSET" to "USER", then insert the new result into My_Table.
I do not know how to write it since now I feel it is a trouble to insert records manually.
Update:
I have far more columns in table and most of them are not nullable, I want to keep all these columns' data for new records except above two columns.Is there any way if I do not have to write all these column names in sql?
Try to use NEWID if you want to create new guid:
INSERT INTO dbo.YourTable
(
COMMON_ID,
COMMON_ASSET_TYPE
)
select NEWID(), 'User' as Common_Asset_Type
from My_Table
where COMMON_ASSET_TYPE = "ASSET"
UPDATE:
As a good practice I would suggest to write all column names explicitly to have a clean and clear insert statement. However, you can use the following construction, but it is not advisable in my opinion:
insert into table_One
select
id
, isnull(name,'Jon')
from table_Two
INSERT INTO My_Table (COMMON_ID,COMMON_LIMIT_IDENTITY, COMMON_CLASS_ID,COMMON_ASSET_TYPE)
SELECT NEWID(), COMMON_LIMIT_IDENTITY, COMMON_CLASS_ID,'USER'
FROM My_Table
WHERE COMMON_ASSET_TYPE = 'ASSET'
If I've understood correctly you want to take existing records in your table, modify them, and insert them as new records in the same table.
I'll assume ID column contains the the GUID?
I'd first create a temporary table
CREATE TABLE #myTempTable(
ID UNIQUEIDENTIFIER,
Name varchar(max),
... etc
);
Fill this temp table with the records to change with your SELECT statement.
Change the records in the temp table using UPDATE statement.
Finally, Insert those "new" records back into the primary table. with INSERT INTO SELECT statement.
You will probably have to sandwitch the INSERT INTO SELECT with IDENTITY_INSERT (on/off):
SET IDENTITY_INSERT schema_name.table_name ON
SET IDENTITY_INSERT schema_name.table_name OFF
IDENTITY_INSERT "Allows explicit values to be inserted into the identity column of a table."
I've created a Stored Procedure that refreshes the data in a table. It first re-loads the entire table. Next, several filters are applied. (Example: the column 'Model' must equal 'W'; all rows with model 'B' are deleted.) This happens after the table has been loaded (and not during) because I want to log how many rows are deleted because of each individual filter. After the filters have been applied, some columns contain the same value in every row (the other values were deleted in the filtering process). These columns are now useless, so I want to delete them.
This seems to be problematic for SQL Server. When given the command to execute the SP, it indicates that the columns it is supposed to remove in its final step do not currently exist and refuses to run. That is technically correct, the columns currently don't exist, but they will be created by the SP itself.
Some mockup code:
CREATE PROCEDURE dbo.Procedure AS (
DROP TABLE dbo.Table
SELECT * INTO dbo.Table FROM dbo.View
INSERT INTO dbo.Log VALUES (GETDATE(),(SELECT COUNT(1) FROM dbo.Table))
DELETE FROM dbo.Table WHERE Model <> 'W'
INSERT INTO dbo.Log VALUES (GETDATE(),(SELECT COUNT(1) FROM dbo.Table))
ALTER TABLE dbo.Table DROP COLUMN Model
)
Error code when executing:
[2016-09-02 12:25:20] [S0001][207] Invalid column name 'Model'.
How do I circumvent this problem and get the SP to run?
If I understand correctly, you can use dynamic SQL:
exec sp_executesql 'ALTER TABLE dbo.Table DROP COLUMN Model';
Syntax to remove any column from table in SQL Server is
ALTER TABLE TableName DROP COLUMN ColumnName ;
This may be cause for issue.
Can you check one more time for the existency of the column 'Model' exists in the view.
because i have tried with the same scenario and its works for me..
Okay first a little bit of background, I've inherited maintaining a Database on MSSQL 2000.
In the Database there's a massive collection of interconnected tables, through Foreign keys.
What I'm attempting to do is to rebuild each table in a sorted fashion that will eliminate gaps in the IDENT column of the table.
On one table in particular I have the following columns:
RL_ID, RL_FK_RaidID, RL_FK_MemberID, RL_FK_ItemID, RL_ItemValue, RL_Notes, RL_IsUber, RL_IsWishItem, RL_LootModifier, RL_WishItemValue, RL_WeightedLootValue
It uses RL_ID as the IDENT column which currently reports 32620 by using DBCC CHECKIDENT (Table)
There is, however, only 12128 rows of information in this table.
So I tried a simple script to copy all the information in a sorted fashion into a new table:
INSERT INTO Table_1
SELECT RL_ID, RL_FK_RaidID, RL_FK_MemberID, RL_FK_ItemID, RL_ItemValue, RL_Notes, RL_IsUber, RL_IsWishItem, RL_LootModifier, RL_WishItemValue, RL_WeightedLootValue
FROM RaidLoot
ORDER BY RL_ID
Then Delete all the rows from the source table with:
TRUNCATE TABLE (RaidLoot)
Verify the IDENT is 1 with:
DBCC CHECKIDENT (RaidLoot)
Now copy the Data back into the Original table from Row 1 to the end:
SET IDENTITY_INSERT RaidLoot ON
INSERT INTO RaidLoot (RL_ID, RL_FK_RaidID, RL_FK_MemberID, RL_FK_ItemID, RL_ItemValue, RL_Notes, RL_IsUber, RL_IsWishItem, RL_LootModifier, RL_WishItemValue, RL_WeightedLootValue)
SELECT RL_ID, RL_FK_RaidID, RL_FK_MemberID, RL_FK_ItemID, RL_ItemValue, RL_Notes, RL_IsUber, RL_IsWishItem, RL_LootModifier, RL_WishItemValue, RL_WeightedLootValue
FROM Table_1
ORDER BY RL_ID
SET IDENTITY_INSERT RaidLoot OFF
Now verify that I only have the 12128 rows of data:
DBCC CHECKIDENT (RaidLoot)
(Note: I end up with 32620 again since it never did renumber the RL_ID, it just put them back into the same spots leaving the gaps). So where / how can I get it to Renumber the RL_ID column starting from 1 so that when it writes the data back to the original table I don't have the gaps?
The only other solution I can see is a heartache process of Manually changing each row RL_ID in the Table_1 before I write it back to the Original table. While this isn't impossible. I have another table that has approx 306,000 rows of data, but the IDENT report lists out as 450,123, so I'm hoping there is an easier way to automate the renumbering process.
If you really have to do this (seems like a great waste of time to me), you will have to adjust all of the foreign key references as well.
Consider the strategy of adding a NewID column for each table and populate the new column sequentially. Then you can use this NewID column in the queries needed to adjust the foreign keys. Very messy nonetheless unless you can come up with a consistent pattern to do so.
Since you can query the metadata to determine foreign keys, etc. this is certainly possible, and definitely should be considered seriously if you really do have lots of tables.
ADDED
There is a simple way to populate the NewID column
declare #id int
set #id = 0
update MyTable set NewID=#id, #id=#id+1
It is not obvious that this works, but it does.
I don't think it has to do with RL_ID being referenced by other tables in the schema - if I set up a single table test, the identity will always show up as the max number in the identity field:
CREATE TABLE #temp (id INT IDENTITY(1,1), other VARCHAR(1))
INSERT INTO #temp
( other )
VALUES ( -- id - int
'a' -- other - varchar(1)
),('b'),('c'),('d'),('e')
SELECT *
FROM #temp
SELECT *
INTO #holder
FROM #temp
WHERE other = 'C'
TRUNCATE TABLE #temp
SET IDENTITY_INSERT #temp ON
INSERT INTO #temp
( id, other )
SELECT id ,
other
FROM #holder
DBCC CHECKIDENT (#temp)
DROP TABLE #temp
DROP TABLE #holder
So your new identity is 32620 because that is the MAX(RL_ID)
I want to delete new record if the same record created before.
My columns are date, time and MsgLog. If date and time are same, I want to delete new one.
I need help .
You can check in the table whether that value exists or not in the column using a query. If it exists, you can show message that a record already exists.
To prevent such kind of erroneous additions you can add restriction to your table to ensure unique #Date #Time pairs; if you don't want to change data structure (e.g. you want to add records with such restrictions once or twice) you can exploit insert select counstruction.
-- MS SQL version, check your DBMS
insert into MyTable(
Date,
Time,
MsgLog)
select #Date,
#Time,
#MsgLog
where not exists(
select 1
from MyTable
where (#Date = Date) and
(#Time = Time)
)
P.S. want to delete new one equals to do not insert new one
You should create a unique constraint in the DB level to avoid invalid data no matter who writes to your DB.
It's always important to have your schema well defined. That way you're safe that no matter how many apps are using your DB or even in case someone just writes some inserts manually.
I don't know which DB are you using but in MySQL can use to following DDL
alter table MY_TABLE add unique index(date, time);
And in Oracle you can :
alter table MY_TABLE ADD CONSTRAINT constaint_name UNIQUE (date, time);
That said, you can also (not instead of) do some checks before inserting new values to avoid dealing with exceptions or to improve performance by avoiding making unnecessary access to your DB (length \ nulls for example could easily be dealt with in the application level).
You can avoid deleting by checking for duplicate while inserting.
Just modify your insert procedure like this, so no duplicates will entered.
declare #intCount as int;
select #intCount =count(MsgLog) where (date=#date) and (time =#time )
if #intCount=0
begin
'insert procedure
end
> Edited
since what you wanted is you need to delete the duplicate entries after your bulk insert. Think about this logic,
create a temporary table
Insert LogId,date,time from your table to the temp table order by date,time
now declare four variables, #preTime,#PreDate,#CurrTime,#CurrDate
Loop for each items in temp table, like this
while
begin
#pkLogID= ' Get LogID for the current row
select #currTime=time,#currDate=date from tblTemp where pkLogId=#pkLogID 'Assign Current values
'Delete condition check
if (#currDate=#preDate) and (#currTime=#preTime)
begin
delete from MAINTABLE where pkLogId=#pkLogID
end
select #preDate=#currDate,#preTime=#currTime 'Assign current values as preValues for next entries
end
The above strategy is we sorted all entries according to date and time, so duplicates will come closer, and we started to compare each entry with its previous, when match found we deleting the duplicate entry.
I have a sql statement to insert data into a table for archiving, but I need a merge statement to run on a monthyl basis to update the new table(2) with any data that changed in the old table(1) that should now be moved into archive.
Part of the issue is to remove the moved data from the old table. My insert is not doing that, but I need to have it to where the saved data is purged from the original table.
Is there a single sql statement that will move data out of one table into another in this way? Or does it need to be a two step operation?
the initial statement moved data depending on age and a few other relative factors.
insert is:
INSERT /*+ append */
INTO tab1
SELECT *
FROM tab2
WHERE (Postingdate < TO_DATE ('2001/07/01', 'yyyy/mm/dd')
OR jobname IS NULL)
AND STATUS <> '45';
All help appreciated...
The merge statement will let you do this in one statement by adding a delete statement in the update clause. See Oracle Documentation on Merge.
I think you should try this with a partition table. My idea is to create table which have range partition on date:
create table(id number primary key,name varchar,J_date date )
partition by range(J_date)(PARTITION one_mnth VALUES LESS THAN(sysdate-30)),
partition by range(J_date)(PARTITION one_mnth VALUES LESS THAN(maxvalue)));
then move that partition in to another table and and truncate that partition