I have a query that I need to run more than once a day. This query is importing data from a database to another.
The target table structure is:
Id Date Department Location PersonId Starttime EndTime State
1 2012-01-01 2 5 200 12:00:00.000 15:00:00.000 2
An application can also insert data to the target table. The records that are inserted by the application may not be updated also when this record exists in the source(temp table) table with another state.
To make this possible I have an solution created. I will create an new column in the target table with a second state so that I can check.
Id Date Department Location PersonId Starttime EndTime State StateSource
1 2012-01-01 2 5 200 12:00:00.000 15:00:00.000 2 2
Some Requirements:
If a record is added by the application than StateSource will be NULL. Means that this record may not be deleted, updated or inserted again from the source table.
If a Record is updated by the application than the value State and StateSource will be different. In this case I do not update this record.
I will update if the state from the sourcetable and targettable are not same and the values from target table State = StateSource.
I will INSERT a record when this is not exists in the target table. When records already exists do not insert (no matter if this is added by the application or my query on the first run).
I will delete the records from the target when they are no more exists in my sourcetable and State=StateSource.
I already have the following queries. I have decided to make 3 statements.
--Delete Statement first
Delete from t
from TargetTable t LEFT JOIN SourceTable s ON t.Id=s.Id
and t.Date=s.Date
and t.departments=s.Department
and t.PersonId=s.PersonId
and t.State=t.StateSource
--Just delete if a date is no more exists from the source table and this records is NOT
--changed by the application (t.State=t.StateSource)
--Update statement second
Update t
set t.State = s.State
From Targettable t INNER JOIN SourceTable s ON t.Id=s.Id
and t.Date=s.Date
and t.departments=s.Department
and t.PersonId=s.PersonId
The problem here is:
--when I have State 2 already in the targettable and in my sourcetable i have
--another state then the state in the targettable changes. This would not be the case.
--Insert Statement thirth
insert into TargetTable (Id, Date, Department, Location, PersonId, Starttime, EndTime,State, StateSource)
select Id, Date, Department, Location, PersonId, Starttime, EndTime,State, StateSource
from SourceTable s
WHERE Date not in (select Date
from TargetTable t
where t.id=s.id
and t.PersonId=s.PersonId
and t.date=s.date
and t.department=s.department)
--I have no idea about how the insert should be because the application also can
--insert records. When a record exists then no insert. What to do with the State?
Remember that the states that are changed by the application are leading.
Can anyone help me with the desired result?
you may use a merge statement.. something like this...
with target_T as (select * from UR_TARGET_TABLE
where statesource is not null) -- to dont change the data inserted from application...
merge target_T as TARGET
using UR_SOURCE_TABLE as SOURCE
on SOURCE.id = TARGET.id -- id is unique? anyway, put your primary key here...
when matched and TARGET.state = TARGET.statesource then --if inserted/updated from application, will not change data
update set TARGET.state = SOURCE.state
,TARGET.statesource = SOURCE.state --important update it together to be different from an application update
--, other collumns that you have to set...
--should use another when matched then update if need to change something on inserted/updated from application data
when not matched by TARGET then
insert (Id, Date, Department, Location, PersonId, Starttime, EndTime,State, StateSource)
values(SOURCE.Id, SOURCE.Date, SOURCE.Department, SOURCE.Location, SOURCE.PersonId, SOURCE.Starttime, SOURCE.EndTime,SOURCE.State, SOURCE.StateSource);
if you set an sample with declaring your tables and inserting some data...
I should help more, with a code that really works.. not just a sample...
Related
I have two tables a stage table and a target table. I want my target table to hold valid CustomerScore values. Currently, we insert into staging and load to our target table. We do not want to load invalid values(-8.0000). However, if there is a customerNumber with a valid value in our target table we would like to decommission numbers by giving it a customerScore of (-8.0000). This should be the only time this value makes it into the target table, so a record for that CustomerNumber has to already be in the target for this to update that record currently in the target table. My create statement is below
CREATE TABLE stg.CustomerAppreciation (
CustomerId INT identity(1, 1)
,CustomerNumber VARCHAR(50)
,CustomerScore DECIMAL(5, 4)
);
CREATE TABLE ods.CustomerAppreciation (
CustomerId INT identity(1, 1)
,CustomerNumber VARCHAR(50)
,CustomerScore DECIMAL(5, 4)
);
Currently, my target table has two records, each value below belongs to my create table fields.
1 123 0.8468
2 143 1.0342
Now say we want to decommission CustomerID = 2 because there is a record been inserted into staging as
3 143 -8.0000
The target table should now be updated on this CustomerNumber. Making my target table look like:
1 123 0.8468
2 143 -8.0000
This should be the only time we allow -8.0000 into the table when a CustomerNumber already exists. If a customerNumber does not exists in the target table and for some reason -8.0000 is seen in staging it should not be allowed in. How would I write an update query that updates a record in my target table only if that scenario exists and prevents -8.0000 from coming in if it does not exist?
Assuming the staging table only contains one row per customer number (if not, group it to show the highest customer Id), you can use a merge to perform this function. Without checking exact syntax, something like this:
MERGE ods.CustomerAppreciation AS Target
USING (SELECT * FROM stg.CustomerAppreciation WHERE CustomerScore >= 0) AS Source ON Target.CustomerNumber = Source.CustomerNumber
WHEN MATCHED
-- choose your match criteria here
--AND Source.CustomerId > Target.CustomerId
AND NOT EXISTS (SELECT Target.* INTERSECT SELECT Source.*)
THEN UPDATE
SET Target.CustomerScore = Source.CustomerScore;
Not sure if I fully understand the specifics but here is some syntax that should help to at least get you started ...
BEGIN;
MERGE ods.CustomerAppreciation AS X
USING (SELECT CustomerNumber,CustomerScore FROM stg.CustomerAppreciation) AS Y (CustomerNumber,CustomerScore)
ON (X.CustomerNumber = Y.CustomerNumber)
WHEN MATCHED /*AND Y.CustomerNumber = '-8.0000'*/ THEN
UPDATE SET CustomerScore = Y.CustomerScore
WHEN NOT MATCHED BY X /*AND Y.CustomerNumber = '-8.0000'*/ THEN
INSERT (CustomerNumber,CustomerScore)
VALUES (Y.CustomerNumber,Y.CustomerScore)
OUTPUT $action, inserted.* INTO #MyTempTable;
END;
[UPDATE: 2/20/19]
I figured out a pretty trivial solution to solve this problem.
CREATE TRIGGER TriggerClaims_History on Claims
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON
INSERT INTO Claims_History
SELECT name, status, claim_date
FROM Claims
EXCEPT SELECT name, status, claim_date FROM Claims_History
END
GO
I am standing up a SQL Server database for a project I am working on. Important info: I have 3 tables - enrollment, cancel, and claims. There are files located on a server that populate these tables every day. These files are NOT deltas (i.e. each new file placed on server every day contains data from all previous files) and because of this, I am able to simply drop all tables, create tables, and then populate tables from files each day. My question is regarding my claims table - since tables will be dropped and created each night, I need a way to keep track of all the different status changes.
I'm struggling to figure out the best way to go about this.
I was thinking of creating a claims_history table that is NOT dropped each night. Essentially I'd want my claims_history table to be populated each time an initial new record is added to the claims table. Then I'd want to scan the claims table and add a row to the claims_history table if and only if there was a change in the status column (i.e. claims.status != claims_history.status).
Day 1:
select * from claims
id | name | status
1 | jane doe | received
select * from claims_history
id | name | status | timestamp
1 | jane doe | received | datetime
Day 2:
select * from claims
id | name | status
1 | jane doe | processed
select * from claims_history
id | name | status | timestamp
1 | jane doe | received | datetime
1 | jane doe | processed | datetime
Is there a SQL script that can do this? I'd also like to automatically have the timestamp field populate in claims_history table each time a new row is added (status change). I know I could write a python script to handle something like this, but i'd like to keep it in SQL if at all possible. Thank you.
Acording to your questions you need to create a trigger after update of the column claims.status and it very simple to do that use this link to know and see how to do a simple trigger click here create asimple sql server trigger
then as if there is many problem to manipulate dateTime in a query a would suggest you to use UNIX time instead of using datetime you can use Long or bigInt UNix time store the date as a number to know the currente time simple use the query SELECT UNIX_TIMESTAMP()
A very common approach is to use a staging table and a production (or final) table. All your ETLs will truncate and load the staging table (volatile) and then you execute an Stored Procedure that adds only the new records to your final table. This requires that all the data you handle this way have some form of key that identifies unequivocally a row.
What happens if your files suddenly change format or are badly formatted? You will drop your table and won't be able to load it back until you fix your ETL. This approach will save you from that, since the process will fail while loading the staging table and won't impact the final table. You can also keep deleted records for historic reasons instead of having them deleted.
I prefer to separate the staging tables into their proper schema, for example:
CREATE SCHEMA Staging
GO
CREATE TABLE Staging.Claims (
ID INT,
Name VARCHAR(100),
Status VARCHAR(100))
Now you do all your loads from your files into these staging tables, truncating them first:
TRUNCATE TABLE Staging.Claims
BULK INSERT Staging.Claims
FROM '\\SomeFile.csv'
WITH
--...
Once this table is loaded you execute a specific SP that adds your delta between the staging content and your final table. You can add whichever logic you want here, like doing only inserts for new records, or inserting already existing values that were updated on another table. For example:
CREATE TABLE dbo.Claims (
ClaimAutoID INT IDENTITY PRIMARY KEY,
ClaimID INT,
Name VARCHAR(100),
Status VARCHAR(100),
WasDeleted BIT DEFAULT 0,
ModifiedDate DATETIME,
CreatedDate DATETIME DEFAULT GETDATE())
GO
CREATE PROCEDURE Staging.UpdateClaims
AS
BEGIN
BEGIN TRY
BEGIN TRANSACTION
-- Update changed values
UPDATE C SET
Name = S.Name,
Status = S.Status,
ModifiedDate = GETDATE()
FROM
Staging.Claims AS S
INNER JOIN dbo.Claims AS C ON S.ID = C.ClaimID -- This has to be by the key columns
WHERE
ISNULL(C.Name, '') <> ISNULL(S.Name, '') AND
ISNULL(C.Status, '') <> ISNULL(S.Status, '')
-- Insert new records
INSERT INTO dbo.Claims (
ClaimID,
Name,
Status)
SELECT
ClaimID = S.ID,
Name = S.Name,
Status = S.Status
FROM
Staging.Claims AS S
WHERE
NOT EXISTS (SELECT 'not yet loaded' FROM dbo.Claims AS C WHERE S.ID = C.ClaimID) -- This has to be by the key columns
-- Mark deleted records as deleted
UPDATE C SET
WasDeleted = 1,
ModifiedDate = GETDATE()
FROM
dbo.Claims AS C
WHERE
NOT EXISTS (SELECT 'not anymore on files' FROM Staging.Claims AS S WHERE S.ClaimID = C.ClaimID) -- This has to be by the key columns
COMMIT
END TRY
BEGIN CATCH
DECLARE #v_ErrorMessage VARCHAR(MAX) = ERROR_MESSAGE()
IF ##TRANCOUNT > 0
ROLLBACK
RAISERROR (#v_ErrorMessage, 16, 1)
END CATCH
END
This way you always work with dbo.Claims and the records are never lost (just updated or inserted).
If you need to check the last status of a particular claim you can create a view:
CREATE VIEW dbo.vClaimLastStatus
AS
WITH ClaimsOrdered AS
(
SELECT
C.ClaimAutoID,
C.ClaimID,
C.Name,
C.Status,
C.ModifiedDate,
C.CreatedDate,
DateRanking = ROW_NUMBER() OVER (PARTITION BY C.ClaimID ORDER BY C.CreatedDate DESC)
FROM
dbo.Claims AS C
)
SELECT
C.ClaimAutoID,
C.ClaimID,
C.Name,
C.Status,
C.ModifiedDate,
C.CreatedDate,
FROM
ClaimsOrdered AS C
WHERE
DateRanking = 1
I want to record student movement when they swipe their cards on the door. Every time a card id and name is presented, I want to check if its a new combination which is not already in the students table. If not present then insert a new record with FirstEntry and LastEntry being the same date and cardid and name being the one that is sent in to the respective parameters. If combination already present then only update the [LastEntry] field. At the moment my query is only updating existing records, which means only the [LastEntry] field gets updated. For update this is normal. However, it doesn't insert a new record for a new entry. CardID and Name are the composite primary keys here on the table.
I don't want to use the IF Exists(Select...) method for acheiving
this.
I want to use MERGE only.
This action is to be done on the same table without using any temp tables
After performing the action I want to see the affected records in either the update or insert scenarios. Only the following fields to be shown [cardid, name, FirstEntry, LastEntry] and not the other fields in the table.
The table has other columns which I don't want to show or return.
Here is my statement which only does updates
MERGE INTO
Students AS T
USING
(SELECT cardid, name, FirstEntry, LastEntry from Students where cardid = #id and name = #name) AS S
ON
(S.cardid = T.cardid
AND
S.name = T.name)
WHEN MATCHED THEN
UPDATE SET LastEntry = #DT
WHEN NOT MATCHED THEN
INSERT (cardid, name, FirstEntry, LastEntry) VALUES(#id, #name, #DT, #DT )
OUTPUT $action, S.cardid, S.name, S.FirstEntry, S.LastEntry;
Along with updates I want it to effect inserts as well for new records
Your using clause shouldn't be querying the Students table. That's where you put the new values you want to insert or update. It should look like this:
MERGE INTO
Students AS T
USING
(SELECT #id as cardid, #name as name, #DT as FirstEntry, #DT as LastEntry) AS S
ON
(S.cardid = T.cardid
AND
S.name = T.name)
WHEN MATCHED THEN
UPDATE SET LastEntry = s.LastEntry
WHEN NOT MATCHED THEN
INSERT (cardid, name, FirstEntry, LastEntry) VALUES(s.cardid, s.name, s.FirstEntry, s.LastEntry )
OUTPUT $action, inserted.cardid, inserted.name, inserted.FirstEntry, inserted.LastEntry;
EDIT
Your output clause should probably be using inserted.<col_name> instead of S.<col_name> for the returned column names to see the new inserted or updated values. You can also use deleted.<col_name> if you want to see the value before it got updated (will be null for inserts).
First time creating a Database trigger. I have a child table that when its cost column is updated, I need its parent table to also update its cost column to reflect the change.
Here is my sorry attempt so far. That's obviously not working. I am having a problem figuring out how to extract the total cost as a variable and store it in the parent table.
My current approach assumes a static id vaule at the moment. I am not entirely sure how to dynamically determine the id value of the row that was updated.
CREATE TRIGGER ParentCost_Update
ON ChildTable
AFTER INSERT, UPDATE
AS
SELECT SUM(Cost) AS TotalCost FROM ChildTable where parent_id=2080
UPDATE ParentTable
SET Cost=TotalCost
where id=parent_id;
GO
This current script is returning an error
Msg 207, Level 16, State 1, Procedure ParentCost_Update, Line 9
Invalid column name 'TotalCost'.
You need to be careful in triggers, in that there could be more than one row updated. Therefore, you need to do row-based handling.
To obtain the newly inserted / updated row, use the inserted and deleted pseudo rows.
You will almost certainly also going to need to implement a deleted trigger as well, viz, if a row is removed from a child table that the parent will need to be recalculated.
Here's a row based take, using a CTE to map your two-step process above , as follows:
CREATE TRIGGER ParentCost_Update
ON ChildTable
AFTER INSERT, UPDATE, DELETE
AS
SET NOCOUNT ON;
WITH cteParentsAffected AS
(
SELECT ins.parent_id
FROM inserted ins
UNION
SELECT del.parent_id
FROM deleted del
)
, cteTotal AS
(
SELECT ct.parent_id, SUM(ct.Cost) AS TotalCost
FROM ChildTable ct
INNER JOIN cteParentsAffected par
ON ct.parent_id = par.parent_id
GROUP BY ct.parent_id
)
UPDATE pt
SET Cost=cte.TotalCost
FROM ParentTable pt
INNER JOIN cteTotal cte
ON id=cte.parent_id;
GO
With a SqlFiddle here
Try this
Update parenttable set total= (select sum(total) from childtable c where c.id= parent table.id)
Where id in (select id from inserted)
Change the table and column names.
I have created two tables customersrc and customertemp with the columns:
customertemp
ID name age addr cityid isactive
34 Gi 24 Chennai 1 1
customersrc
CustomerId CustomerName CustomerAge CustomerAddress
1 Gi 24 madurai
2 Pa 23 Tirupur
3 MI 27 Tirupur
Now I need to insert pa and mi data value to the temp table bcz it is not matched with the rows of customertemp. And the row gi data will be updated which was matched.
I used the following MERGE statement
DECLARE #cityid INT SET #cityid=1
MERGE Temp.dbo.customersrc as src_customer
USING ( SELECT CustomerName,CustomerAge,CustomerAddress FROM customertemp) as temp_customer
ON src_customer.name=temp_customer.CustomerName
AND
src_customer.cityid=#cityid
WHEN MATCHED THEN
UPDATE SET
src_customer.age=temp_customer.CustomerAge,
src_customer.addr=temp_customer.CustomerAddress,
src_customer.isactive=1
WHEN NOT MATCHED BY SOURCE THEN
UPDATE SET src_customer.isactive=0 ; -- here i need the insert statement to insert in another table
Questions:
is it possible to write insert statement inside the when not matched by source query?
if it is not possible then how to achieve this using merge?
in a simple set theory I need to put the customersrc(table_B)-customertemp (table_A). B-A value into the another or temp table.
One of the main usages of the MERGE statement is to perform so called "UPSERTS" (Update matching records, insert new records), so it is definitely possible to do what you want. Just add the following to the last part of your MERGE statement:
WHEN NOT MATCHED BY TARGET THEN
INSERT (name, age, addr, cityid, isactive)
VALUES (CustomerName, CustomerAge, CustomerAddress, #cityid, 1)
If you also need to insert data into a 3rd table, depending on whether rows are updated or inserted, you can use the OUTPUT clause of the merge statement. Check out the documentation: http://technet.microsoft.com/en-us/library/ms177564.aspx
Me: Why do you want to insert to another table?
You: To show the user who are not in the customertemp table.
So your requirement is not to insert into another table. Your requirement is to get the missing users.
You could do that with a dummy UPDATE (SET SomeCol = SomeCol) and OUTPUT. But that is a hack that I would try to avoid.
It is probably easier to do this in two statements. Here's how you'd get the missing rows:
SELECT temp_customer.*
FROM (SELECT CustomerName,CustomerAge,CustomerAddress FROM customertemp) as temp_customer
LEFT JOIN customersrc ON src_customer.name=temp_customer.CustomerName AND src_customer.cityid=#cityid
WHERE customersrc.cityid IS NULL