Related
I'm currently in a scenario where we have a clunky, old, monolith of a database powering one or two of our less user-friendly systems (this is controlled by a third-party, so I can't change that).
Going forward, I'm looking to push the necessary data into a new (better structured) database and implement a mechanism to keep the data in sync between the two.
One of the quirky traits of this old database is that, instead of just using null values, it uses empty strings and treats them as if they're null.
In the new database, I want to use nulls where there's no data (because I can't think of any good reasons not to).
My issue is when I'm pulling the data from the old database to the new one (using MERGE statements), I'm using a NULLIF([myCol], '') to check there's actually any data - and just treating it as a null if not.
For example, when syncing the data for students I'm planning on doing something along the lines of (the name of the databases and schemas isn't relevant to the question so I'm anonymizing the structure a bit):
USE [NewDB];
DROP TABLE IF EXISTS #myTempTable;
SELECT *
INTO #myTempTable
FROM [OldDB].[schemaName].[Students];
MERGE [schema].[Person] p
USING #myTempTable s
ON TRIM(s.STUD_ISN) = p.StudentDBID
--WHEN MATCHED and not equal, update
WHEN NOT MATCHED
THEN INSERT (
[Surname] ,
[PreferredSurname] ,
[FormerSurname] ,
[Forename] ,
[MiddleNames] ,
[PreferredForename] ,
[DoB] ,
[Gender] ,
[GenderIdentity] ,
[Title] ,
[Deceased] ,
[DeceasedDate] ,
[StudentDBID])
VALUES (
NULLIF(TRIM([s].[STUD_Surname]), '') ,
NULLIF(TRIM([s].[STUD_Preferred_Surname]), '') ,
NULLIF(TRIM([s].STUD_Former_Surname), '') ,
NULLIF(TRIM([s].[STUD_Forename_1]), '') ,
NULLIF(TRIM([s].[STUD_Forename_2]), '') ,
NULLIF(TRIM([s].[STUD_Known_As]), '') ,
[s].[STUD_DOB] ,
NULLIF(TRIM([s].[STUD_Gender]), '') ,
NULLIF(TRIM([s].STUD_Gender_Identity), '') ,
NULLIF(TRIM([s].STUD_Title), '') ,
[s].STUD_Deceased ,
[s].STUD_Date_of_Death ,
TRIM([s].STUD_ISN)
);
Now, obviously this current query only deals with the following scenario:
The old database has data and the new one doesn't
Because I'm having some grief with the logic for the following scenario:
The new database has the corresponding record, but it needs updating.
For the text based fields there's 3 different scenarios where the "new" data would need updating:
Neither value is null/empty, but they're not equal
The record in the "old" database is null/empty and the "new" one isn't
The record in the "new" database is null and the "old" one isn't
I was hoping to do something relatively simple like:
WHEN MATCHED AND (
(NULLIF(TRIM([s].[STUD_Surname]), '') IS NOT NULL
AND p.Surname IS NULL)
OR (NULLIF(TRIM([s].[STUD_Surname]), '') IS NULL
AND p.Surname IS NOT NULL)
OR (NULLIF(TRIM([s].[STUD_Surname]), '') != p.Surname)
) -- do the same for the other columns
THEN UPDATE SET
Surname = s.STUD_Surname
But, NULLIF(TRIM([s].[STUD_Surname]), '') IS NOT NULL is apparently not valid?
Is my best option to just switch out the NULLIF for an ISNULL in the WHEN MATCHED clause... Or am I missing something that'll make it more efficient?
The checks for the update can be simplified.
Since the source temp table doesn't have NULL's.
You just need to compare the trimmed values.
MERGE Person t
USING #tmpTable s
ON s.STUD_ISN = t.StudentDBID
WHEN NOT MATCHED THEN
INSERT (
StudentDBID
, Forename
, PreferredForename
, Surname
)
VALUES (
NULLIF(TRIM(s.STUD_ISN), '')
, NULLIF(TRIM(s.STUD_Forename), '')
, NULLIF(TRIM(s.STUD_Forename_1), '')
, NULLIF(TRIM(s.STUD_Surname), '')
)
WHEN MATCHED
AND (
COALESCE(t.Forename, '') != TRIM(s.STUD_Forename)
OR COALESCE(t.PreferredForename, '') != TRIM(s.STUD_Forename_1)
OR COALESCE(t.Surname, '') != TRIM(s.STUD_Surname)
)
THEN
UPDATE SET
UpdatedOn = SYSDATETIME()
, Forename = NULLIF(TRIM(s.STUD_Forename), '')
, PreferredForename = NULLIF(TRIM(s.STUD_Forename_1), '')
, Surname = NULLIF(TRIM(s.STUD_Surname), '')
;
SELECT *
FROM Person
StudentDBID | Forename | PreferredForename | Surname | UpdatedOn | CreatedOn
:---------- | :------- | :---------------- | :------ | :-------------------------- | :--------------------------
B9701 | Bob | Bobby | Modest | 2022-01-14 12:58:05.9862269 | 2022-01-14 12:58:05.9862269
J0402 | Jane | JD | Doe | 2022-01-14 12:58:05.9862269 | 2022-01-14 12:58:05.9862269
J0504 | Jim | Jimmy | Savage | null | 2022-01-14 12:58:05.9862269
J7201 | John | null | Doe | null | 2022-01-14 12:58:05.9862269
M0803 | Mike | null | Hammer | 2022-01-14 12:58:05.9862269 | 2022-01-14 12:58:05.9862269
Demo on db<>fiddle here
In the end, my solution was to apply the various functions within the SELECT .... Into .... block. Like so:
ALTER PROCEDURE [NGSync].[spFullStudentSync]
AS
BEGIN
SET NOCOUNT ON;
DROP TABLE IF EXISTS #students
SELECT [STUD_ISN],
NULLIF(TRIM([STUD_Student_ID]), '') AS [STUD_Student_ID] ,
NULLIF(TRIM([STUD_Surname]), '') AS [STUD_Surname] ,
NULLIF(TRIM([STUD_Forename_1]), '') AS [STUD_Forename_1] ,
NULLIF(TRIM([STUD_Forename_2]), '') AS [STUD_Forename_2] ,
NULLIF(TRIM([STUD_Known_As]), '') AS [STUD_Known_As] ,
[STUD_DOB] ,
NULLIF(TRIM([STUD_Gender]) , '') AS [STUD_Gender],
NULLIF(TRIM([STUD_Title]) , '') AS [STUD_Title],
NULLIF(TRIM([STUD_Ethnicity]) , '') AS [STUD_Ethnicity],
NULLIF(TRIM([STUD_LDDHP]) , '') AS [STUD_LDDHP],
NULLIF(TRIM([STUD_Home_Telephone_No] ) , '') AS [STUD_Home_Telephone_No] ,
NULLIF(TRIM([STUD_Daytime_Telephone] ) , '') AS [STUD_Daytime_Telephone] ,
NULLIF(TRIM([STUD_Mobile_Telephone] ) , '') AS [STUD_Mobile_Telephone] ,
NULLIF(TRIM([STUD_EMail_Address] ) , '') AS [STUD_EMail_Address] ,
NULLIF(TRIM([STUD_Former_Surname] ) , '') AS [STUD_Former_Surname] ,
CAST( CASE WHEN NULLIF(TRIM([STUD_Deceased] ) , '') = 'D' THEN 1 else 0 end AS bit ) AS [STUD_Deceased] ,
NULLIF(TRIM([STUD_Deletion_Flag] ) , '') AS [STUD_Deletion_Flag] ,
[STUD_Delete_Merge] ,
NULLIF(TRIM([STUD_Photo_filename] ) , '') AS [STUD_Photo_filename] ,
NULLIF(TRIM([STUD_Nationality] ) , '') AS [STUD_Nationality] ,
[STUD_Date_of_Entry_in_UK] ,
NULLIF(TRIM([STUD_Student_Type_FESR] ) , '') AS [STUD_Student_Type_FESR] ,
[STUD_School_ISN] ,
NULLIF(TRIM([STUD_Home_LEA] ) , '') AS [STUD_Home_LEA] ,
NULLIF(TRIM([STUD_Employer_Code] ) , '') AS [STUD_Employer_Code] ,
NULLIF(TRIM([STUD_Religion] ) , '') AS [STUD_Religion] ,
NULLIF(TRIM([STUD_Location] ) , '') AS [STUD_Location] ,
[STUD_TPS_Include] ,
[STUD_QOE_Complete] ,
NULLIF(TRIM([STUD_UCAS_Application_Code] ) , '') AS [STUD_UCAS_Application_Code] ,
[STUD_MIAP_Consent_Status] ,
[STUD_MIAP_Verification_Type] ,
NULLIF(TRIM([STUD_MIAP_Other_Verification] ) , '') AS [STUD_MIAP_Other_Verification] ,
NULLIF(TRIM([STUD_Bank_Sort_Code] ) , '') AS [STUD_Bank_Sort_Code] ,
NULLIF(TRIM([STUD_Bank_AC_No] ) , '') AS [STUD_Bank_AC_No] ,
[STUD_Bank_ISN] ,
NULLIF(TRIM([STUD_Bank_Postcode] ) , '') AS [STUD_Bank_Postcode] ,
NULLIF(TRIM([STUD_Bank_AC_Holders_Name] ) , '') AS [STUD_Bank_AC_Holders_Name] ,
NULLIF(TRIM([STUD_Current_Tutor_Group] ) , '') AS [STUD_Current_Tutor_Group] ,
NULLIF(TRIM([STUD_Current_PostCode] ) , '') AS [STUD_Current_PostCode] ,
NULLIF(TRIM([STUD_Doctor_Name] ) , '') AS [STUD_Doctor_Name] ,
NULLIF(TRIM([STUD_Doctor_Telephone] ) , '') AS [STUD_Doctor_Telephone] ,
[STUD_ULN] ,
NULLIF(TRIM([STUD_College_Email_Address] ) , '') AS [STUD_College_Email_Address] ,
[STUD_Date_Due_to_Leave_UK] ,
[STUD_UK_Residence] ,
NULLIF(TRIM([STUD_Parish] ) , '') AS [STUD_Parish] ,
NULLIF(TRIM([STUD_Area] ) , '') AS [STUD_Area] ,
[STUD_Bus_Pass] ,
NULLIF(TRIM([STUD_Bus_Route] ) , '') AS [STUD_Bus_Route] ,
[STUD_Eng_1st_Lang] ,
NULLIF(TRIM([STUD_Language] ) , '') AS [STUD_Language] ,
[STUD_High_Achiever] ,
[STUD_Is_Staff] ,
[STUD_Is_Staff_ISN] ,
[STUD_Excluded] ,
[STUD_RUI_3_no_contact] ,
[STUD_RUI_1_courses] ,
[STUD_RUI_2_surveys] ,
[STUD_PMC_C1_post] ,
[STUD_PMC_C2_phone] ,
[STUD_PMC_C3_email] ,
[STUD_Created_Date] ,
NULLIF(TRIM([STUD_Created_User] ) , '') AS [STUD_Created_User] ,
NULLIF(TRIM([STUD_Created_Prog] ) , '') AS [STUD_Created_Prog] ,
[STUD_Modified_Date] ,
NULLIF(TRIM([STUD_Modified_User] ) , '') AS [STUD_Modified_User] ,
NULLIF(TRIM([STUD_Modified_Prog] ) , '') AS [STUD_Modified_Prog] ,
[STUD_OK_to_use_Image] ,
NULLIF(TRIM([STUD_Sexual_Orientation] ) , '') AS [STUD_Sexual_Orientation] ,
NULLIF(TRIM([STUD_Gender_Identity] ) , '') AS [STUD_Gender_Identity] ,
[STUD_Visa_Proof_Produced] ,
[STUD_Visa_Proof_Produced_Date] ,
NULLIF(TRIM([STUD_Visa_Proof_Produced_Details] ) , '') AS [STUD_Visa_Proof_Produced_Details] ,
NULLIF(TRIM([STUD_Visa_Type] ) , '') AS [STUD_Visa_Type] ,
[STUD_Visa_Expiry_Date] ,
[STUD_Visa_Letter_Issued] ,
[STUD_Asylum_Seeker] ,
[STUD_Refugee] ,
[STUD_Entered_UK_for_Education] ,
[STUD_Restrictions_on_Stay] ,
NULLIF(TRIM([STUD_Add_To_Portal_Title_Bar] ) , '') AS [STUD_Add_To_Portal_Title_Bar] ,
[STUD_RUI_5] ,
NULLIF(TRIM([STUD_Visa_Reference]) , '') AS [STUD_Visa_Reference],
NULLIF(TRIM([STUD_Visa_Note] ) , '') AS [STUD_Visa_Note] ,
[STUD_FA_Bank_ISN] ,
NULLIF(TRIM([STUD_FA_Bank_Sort_Code] ) , '') AS [STUD_FA_Bank_Sort_Code] ,
NULLIF(TRIM([STUD_FA_Bank_AC_No] ) , '') AS [STUD_FA_Bank_AC_No] ,
NULLIF(TRIM([STUD_FA_Bank_AC_Holders_Name]) , '') AS [STUD_FA_Bank_AC_Holders_Name],
NULLIF(TRIM([STUD_Marital_Status] ) , '') AS [STUD_Marital_Status] ,
NULLIF(TRIM([STUD_Country_of_Birth] ) , '') AS [STUD_Country_of_Birth] ,
NULLIF(TRIM([STUD_On_Supervision] ) , '') AS [STUD_On_Supervision] ,
[STUD_S_IraqDEAS] ,
[STUD_S_Pending_SCN] ,
NULLIF(TRIM([STUD_Primary_LLDDCode] ) , '') AS [STUD_Primary_LLDDCode] ,
NULLIF(TRIM([STUD_A2C_Language] ) , '') AS [STUD_A2C_Language] ,
NULLIF(TRIM([STUD_Signature_Filename] ) , '') AS [STUD_Signature_Filename] ,
[STUD_S_ILA_Expiry_Date] ,
[STUD_Date_Of_Death] ,
[STUD_S_SQA_Exception] ,
NULLIF(TRIM([STUD_S_SQA_VID] ) , '') AS [STUD_S_SQA_VID] ,
[STUD_S_SQA_Registration_Date] ,
NULLIF(TRIM([STUD_FA_Bank_AC_Ref] ) , '') AS [STUD_FA_Bank_AC_Ref] ,
NULLIF(TRIM([STUD_Bank_AC_Ref] ) , '') AS [STUD_Bank_AC_Ref] ,
[STUD_Anonymised] ,
NULLIF(TRIM([STUD_Quarantine_Status] ) , '') AS [STUD_Quarantine_Status] ,
[STUD_KeycloakID] ,
NULLIF(TRIM([STUD_Preferred_Surname] ) , '') AS [STUD_Preferred_Surname] ,
[STUD_Exclude_From_Balancing] ,
NULLIF(TRIM([STUD_Corresp_Preference] ) , '') AS [STUD_Corresp_Preference] ,
NULLIF(TRIM([STUD_HESA_ID] ) , '') AS [STUD_HESA_ID] ,
NULLIF(TRIM([STUD_FEPUS_ID] ) , '') AS [STUD_FEPUS_ID] ,
NULLIF(TRIM([STUD_PEV_PIN] ) , '') AS [STUD_PEV_PIN] ,
NULLIF(TRIM([STUD_Photo_GUID_Filename] ) , '') AS [STUD_Photo_GUID_Filename]
INTO #students
FROM [NG].[dbo].[STUDstudent];
MERGE [people].[Person] AS [p]
USING #students AS [s]
ON [s].[STUD_ISN] = [p].[StudentDBID]
WHEN MATCHED
AND (
[s].[STUD_Surname] != [p].[Surname]
OR [s].[STUD_Preferred_Surname] != [p].[PreferredSurname]
OR [s].[STUD_Former_Surname] != [p].[FormerSurname]
OR [s].[STUD_Forename_1] != [p].[Forename]
OR [s].[STUD_Forename_2] != [p].[MiddleNames]
OR [s].[STUD_Known_As] != [p].[PreferredForename]
OR [s].[STUD_DoB] != [p].[DoB]
OR [s].[STUD_Gender] != [p].[Gender]
OR [s].[STUD_Gender_Identity] != [p].[GenderIdentity]
OR [s].[STUD_Title] != [p].[Title]
OR [s].[STUD_Deceased] != [p].[Deceased]
OR [s].[STUD_Date_of_Death] != [p].[DeceasedDate]
OR [s].[STUD_Is_Staff_ISN] != [p].[StaffDBID]
)
THEN UPDATE SET
[Surname] = [s].[STUD_Surname] ,
[PreferredSurname] = [s].[STUD_Preferred_Surname] ,
[FormerSurname] = [s].[STUD_Former_Surname],
[Forename] = [s].[STUD_Forename_1] ,
[MiddleNames] = [s].[STUD_Forename_2] ,
[PreferredForename] = [s].[STUD_Known_As],
[DoB] = [s].[STUD_DoB] ,
[Gender] = [s].[STUD_Gender] ,
[GenderIdentity] = [s].[STUD_Gender_Identity] ,
[Title] = [s].[STUD_Title] ,
[Deceased] = [s].[STUD_Deceased],
[StaffDBID] = [s].[STUD_Is_Staff_ISN]
WHEN NOT MATCHED
THEN INSERT (
[Surname] ,
[PreferredSurname] ,
[FormerSurname] ,
[Forename] ,
[MiddleNames] ,
[PreferredForename] ,
[DoB] ,
[Gender] ,
[GenderIdentity] ,
[Title] ,
[Deceased] ,
[DeceasedDate] ,
[StudentDBID] ,
[StaffDBID])
VALUES (
[s].[STUD_Surname] ,
[s].[STUD_Preferred_Surname] ,
[s].[STUD_Former_Surname] ,
[s].[STUD_Forename_1] ,
[s].[STUD_Forename_2] ,
[s].[STUD_Known_As] ,
[s].[STUD_DOB] ,
[s].[STUD_Gender] ,
[s].[STUD_Gender_Identity] ,
[s].[STUD_Title] ,
[s].[STUD_Deceased] ,
[s].[STUD_Date_of_Death] ,
[s].[STUD_ISN] ,
[s].[STuD_IS_Staff_ISN]
);
END
It could probably be made more efficient but, considering it's going to part of a nightly task and run when people aren't using the system, it handles ~35000 records in ~2 seconds so it's "efficient enough".
You can use the Coalesce Function available in SQL to achieve the desired result
It will return the first Non-Null value from the passed parameters
SELECT COALESCE( NULL ,'First Non Null','Alex')
COALESCE Keyword
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I will like to understand what does exactly the /*+ APPEND PARALLEL(TEST,12) */, it is a improve but I'm not really sure what it does.
--FIRST SQL
insert into TEST
select ORDER_DATE , ORDER_NO , ORDER_INV_LINE , CUSTOMER_NO , ORDER_INV_LINE_TYPE , ORDER_INV_LOC_CD , CUST_REF_NO , GROUP_ACCT_NO , SELL_STYLE , RCS_CD , GRADE
, INV_STYLE_NO , DISCOUNT_CD , CREDITED_SELL_CO , DELI_VEHICLE_CD , QUANTITY , GROSS_AMT , REBATE_NET_AMT , nvl(TERM_SAVG_AMT,TERMS_AVG_AMT) , TERMS_AMT , UNIT_PRICE , DISCOUNT_AMT
, COMM_LOAD , DELIVERED_FRT_AMT , CREDITED_DISTRICT_ID , INVOICE_NO , INVOICE_DATE , INVOICE_MONTH , SELL_COLOR , WIDTH_FT , WIDTH_IN , LENGTH_FT , LENGTH_IN , ROLL_NO
, ACTUAL_DUTY , GST_AMT , BROKERAGE_FEE , CRED_REGION_ID , TERMS_PCT , CRED_TERRITORY_ID , WHSE_UPCHARGE , OVERBILL_A_AMOUNT , OVERBILL_B_AMOUNT , OVERBILL_C_AMOUNT , OVERBILL_D_AMOUNT
, OVERBILL_E_AMOUNT , OVERBILL_F_AMOUNT , OVERBILL_G_AMOUNT , OVERBILL_H_AMOUNT , OVERBILL_I_AMOUNT , TERMS_CD , ORDER_LINE_STATUS_CD , NET_UNIT_PRICE , INV_FOB_COST , NET_SALES_AMT_CCA
, NET_UNIT_PRICE_CCA , INVOICE_PAID_FLAG , DISC_FLAG , NVL(BUILDER_NO,BUILDER_NUMBER) , BUILDER_NAME , SUB_DIVISION , BLOCK_NBR , LOT , PROJECT_NAME , INV_PRICING_UOM , PRO_ROLL_OVB , PRO_CUT_OVB , EFF_DATE
, EXP_DATE, CCA_PROGRAM, OVBG_FLAG, REBATE_NET_AMTCN, sysdate as ARCHIVE_DATE, ENDUSER_CODE, ENDUSER_NAME, SELL_BUSINESS_GRP, SALES_MIX_GRP, BUSINESS_GRP_CAT, MIX_GRP_CAT, BDF_GROUP
FROM SCHEMA.prestg_order_invoices poi
WHERE NOT EXISTS (
SELECT 1
FROM SCHEMA.TEST ar
WHERE ar.order_no = poi.order_no
and nvl(ar.invoice_no, 'XYZ') = nvl(poi.invoice_no, 'XYZ')
and ar.order_inv_line = poi.order_inv_line)
----
--SQL MODIFIED
insert /*+ APPEND PARALLEL(TEST,12) */ into TEST
select /*+ PARALLEL(poi,12) */ ORDER_DATE , ORDER_NO , ORDER_INV_LINE , CUSTOMER_NO , ORDER_INV_LINE_TYPE , ORDER_INV_LOC_CD , CUST_REF_NO , GROUP_ACCT_NO , SELL_STYLE , RCS_CD , GRADE
, INV_STYLE_NO , DISCOUNT_CD , CREDITED_SELL_CO , DELI_VEHICLE_CD , QUANTITY , GROSS_AMT , REBATE_NET_AMT , nvl(TERM_SAVG_AMT,TERMS_AVG_AMT) , TERMS_AMT , UNIT_PRICE , DISCOUNT_AMT
, COMM_LOAD , DELIVERED_FRT_AMT , CREDITED_DISTRICT_ID , INVOICE_NO , INVOICE_DATE , INVOICE_MONTH , SELL_COLOR , WIDTH_FT , WIDTH_IN , LENGTH_FT , LENGTH_IN , ROLL_NO
, ACTUAL_DUTY , GST_AMT , BROKERAGE_FEE , CRED_REGION_ID , TERMS_PCT , CRED_TERRITORY_ID , WHSE_UPCHARGE , OVERBILL_A_AMOUNT , OVERBILL_B_AMOUNT , OVERBILL_C_AMOUNT , OVERBILL_D_AMOUNT
, OVERBILL_E_AMOUNT , OVERBILL_F_AMOUNT , OVERBILL_G_AMOUNT , OVERBILL_H_AMOUNT , OVERBILL_I_AMOUNT , TERMS_CD , ORDER_LINE_STATUS_CD , NET_UNIT_PRICE , INV_FOB_COST , NET_SALES_AMT_CCA
, NET_UNIT_PRICE_CCA , INVOICE_PAID_FLAG , DISC_FLAG , NVL(BUILDER_NO,BUILDER_NUMBER) , BUILDER_NAME , SUB_DIVISION , BLOCK_NBR , LOT , PROJECT_NAME , INV_PRICING_UOM , PRO_ROLL_OVB , PRO_CUT_OVB , EFF_DATE
, EXP_DATE, CCA_PROGRAM, OVBG_FLAG, REBATE_NET_AMTCN, sysdate as ARCHIVE_DATE, ENDUSER_CODE, ENDUSER_NAME, SELL_BUSINESS_GRP, SALES_MIX_GRP, BUSINESS_GRP_CAT, MIX_GRP_CAT, BDF_GROUP
FROM SCHEMA.prestg_order_invoices poi
WHERE NOT EXISTS (
SELECT 1
FROM SCHEMA.TEST ar
WHERE ar.order_no = poi.order_no
and nvl(ar.invoice_no, 'XYZ') = nvl(poi.invoice_no, 'XYZ')
and ar.order_inv_line = poi.order_inv_line)
APPEND or PARALLEL hints invoke direct path load. This means blocks are allocated from above the HWM (high water mark). That is, blocks that do not, and never have had any rows in them. For that reason, Oracle does not generate UNDO. (There's no need for a 'before image', since the 'before image is that the block didn't exist in the segment.) Redo is still generated for direct path load, unless NOLOGGING is also set.
it isn't necessarily always faster in general. It does a direct path load to disk - bypassing the buffer cache. There are many cases - especially with smaller sets - where the direct path load to disk would be far slower than a conventional path load into the cache.
Also, you cannot query a table after direct pathing into it until you commit or rollback. And also consider the fact that one and only one user can direct path into a table at a time. It would cause all modifications to serialize. No one else could insert/update/delete or merge into this table until the transaction that direct paths commits.
I have a situation where an order can contain multiple license purchases - and if the order does contain multiple licenses, I want to display the license descriptions in a single cell with the values separated by commas. If we were on SQL 2017, I could use STRING_AGG but we are on SQL 2016 so I am trying the tried and true STUFF / FOR XML Path method.
From the screenshot below, Customer 4341073 had two license purchases on Order ID 18519173:
When I add the STUFF / FOR XML Path to the T-SQL, I am not able to achieve the desired result of showing the license description in the same record - each license still has it's own row.
SELECT x.CustomerID ,
x.ATOLicenseTypeID ,
x.ATOLicense ,
x.AuthorizationBeginDate ,
x.AuthorizationEndDate ,
x.OrderID ,
x.OrderDate ,
STUFF ( (
SELECT ',' + lt.description
FROM dbo.LicenseTypes AS lt
--INNER JOIN #XMLPATH ON lt.id = x.OrderLicenseTypeID
WHERE lt.id = x.OrderLicenseTypeID
--GROUP BY ',' + lt.description
FOR XML PATH ( '' )
) , 1 , 1 , '' ) AS Licenses
FROM #XMLPATH AS x
--GROUP BY x.CustomerID ,
-- x.ATOLicenseTypeID ,
-- x.ATOLicense ,
-- x.AuthorizationBeginDate ,
-- x.AuthorizationEndDate ,
-- x.OrderID ,
-- x.OrderDate ,
-- x.OrderLicenseTypeID;
I've tried different ways to join the sub-query to the outer query and added and removed GROUP BY to achieve the desired result but nothing is working for me.
Any suggestions on where I am going wrong with this query?
Sample dataset:
DROP TABLE IF EXISTS #XMLPATH;
CREATE TABLE #XMLPATH
(
CustomerID INT ,
ATOLicenseTypeID INT ,
ATOLicense VARCHAR (500) ,
AuthorizationBeginDate DATE ,
AuthorizationEndDate DATE ,
OrderID INT ,
OrderDate DATETIME ,
OrderLicenseTypeID INT
);
INSERT INTO #XMLPATH
VALUES ( 4341073, 52, 'Temporary Resident Fishing', N'2019-01-07T00:00:00', N'2019-01-07T00:00:00', 18519136, N'2019-01-07T12:01:55.317', 2141 ) ,
( 4341073, 52, 'Temporary Resident Fishing', N'2019-01-07T00:00:00', N'2019-01-07T00:00:00', 18519173, N'2019-01-07T12:34:13.107', 204 ) ,
( 4341073, 52, 'Temporary Resident Fishing', N'2019-01-07T00:00:00', N'2019-01-07T00:00:00', 18519173, N'2019-01-07T12:34:13.107', 2141 );
SELECT * FROM #XMLPATH;
SELECT x.CustomerID ,
x.ATOLicenseTypeID ,
x.ATOLicense ,
x.AuthorizationBeginDate ,
x.AuthorizationEndDate ,
x.OrderID ,
x.OrderDate ,
STUFF ( (
SELECT ',' + lt.description
FROM dbo.LicenseTypes AS lt
--INNER JOIN #XMLPATH ON lt.id = x.OrderLicenseTypeID
WHERE lt.id = x.OrderLicenseTypeID
--GROUP BY ',' + lt.description
FOR XML PATH ( '' )
) , 1 , 1 , '' ) AS Licenses
FROM #XMLPATH AS x
GROUP BY x.CustomerID ,
x.ATOLicenseTypeID ,
x.ATOLicense ,
x.AuthorizationBeginDate ,
x.AuthorizationEndDate ,
x.OrderID ,
x.OrderDate ,
x.OrderLicenseTypeID;
In order to get all rows of one OrderID as one result-row, you must not include the separating information (the OrderLicenseTypeID) into the GROUP BY. But then you have the issue you've encountered: You cannot use this ID within your FOR XML construct.
The trick is (as your out-commented trials show), to add the source table to the sub-select and filter there with a grouped column. But you have to use different aliases to deal with them as two different sets. Try this:
(I had to add one more temp table to test this...)
SELECT x.CustomerID ,
x.ATOLicenseTypeID ,
x.ATOLicense ,
x.AuthorizationBeginDate ,
x.AuthorizationEndDate ,
x.OrderID ,
x.OrderDate ,
STUFF ( (
SELECT ',' + lt.description
FROM #XMLPATH x2
INNER JOIN #LicenseTypes AS lt ON lt.id=x2.OrderLicenseTypeID
WHERE x2.OrderID = x.OrderID --you might need to add more columns here....
--in most cases we want to add an ORDER BY
FOR XML PATH ( '' )
) , 1 , 1 , '' ) AS Licenses
FROM #XMLPATH AS x
GROUP BY x.CustomerID ,
x.ATOLicenseTypeID ,
x.ATOLicense ,
x.AuthorizationBeginDate ,
x.AuthorizationEndDate ,
x.OrderID ,
x.OrderDate;
Btw: Starting with v2017 there is STRING_AGG(), which makes this much easier...
I am using a CTE to explode out a Bill of Materials and need to locate all those materials that have recursive components.
What I was attempting, was to limit the number of cycles (levels) deep, by setting BOM_Level in the child node to a maximum bound:
exec pr_sys_drop_object '#BOMExploded'
;with BOM
as
(
select
Prod_Plant_CD
, Demand_Plant_CD
, material_cd
, component_cd
, component_quantity
, component_quantity AS Calculated_Component_Quantity
, BOM_Level
, Demand_Quantity
, CONVERT(float,1) AS Produced_Ratio
, Material_CD AS Demand_Material_CD
from #firstLevel a
UNION ALL
SELECT
b.Plant_CD as 'Prod_Plant_CD'
, a.Demand_Plant_CD
, b.Material_CD
, b.Component_CD
, b.component_quantity
, b.component_quantity
, a.BOM_Level + 1
, a.Demand_Quantity
, a.Produced_Ratio * a.Component_Quantity -- Produced Quantity for the current level = Produced Quantity (level -1) * Component_Quantity (level -1)
, a.Demand_Material_CD
FROM BOM a
inner join #BOM_ProdVersion_Priority b
on a.component_cd = b.material_cd
inner join #base_routes c
on a.Demand_Plant_CD = c.Recipient_Plant_CD
and b.Plant_CD = c.Source_Plant_CD
and c.Material_CD = b.Material_CD -- Need to have material_cd to link
where b.Material_CD != b.Component_CD
and b.Component_Quantity > 0
and BOM_Level < 5 -- Give the max number of levels deep we are allowed to cyncle to
)
select *
into #BOMExploded
from BOM
OPTION (MAXRECURSION 20)
Using this method however, would require a post-process to locate when the cycling on the recursive component level started, then back trace.
How can a CTE recursive query be stopped given a certain condition?
ie. when top-level material_cd = component_cd for a deeper BOM_Level
If I understand you correctly, you don't need to stop at a certain depth/level, or rather you want to stop at a certain level, but you also need to stop in case you start cycling through materials repeatedly.
In the case of the following recursive path: mat_1->mat_2->mat_3->mat_1, you would want to stop before that last mat_1 starts cycling again to mat_2 and so on.
If that's correct, then your best bet is to add a Path field to your recursive query that tracks each term that it finds as it moves down the recursive path:
exec pr_sys_drop_object '#BOMExploded'
;with BOM
as
(
select
Prod_Plant_CD
, Demand_Plant_CD
, material_cd
, component_cd
, component_quantity
, component_quantity AS Calculated_Component_Quantity
, BOM_Level
, Demand_Quantity
, CONVERT(float,1) AS Produced_Ratio
, Material_CD AS Demand_Material_CD
, CAST(material_cd AS VARCHAR(100)) AS Path
from #firstLevel a
UNION ALL
SELECT
b.Plant_CD as 'Prod_Plant_CD'
, a.Demand_Plant_CD
, b.Material_CD
, b.Component_CD
, b.component_quantity
, b.component_quantity
, a.BOM_Level + 1
, a.Demand_Quantity
, a.Produced_Ratio * a.Component_Quantity -- Produced Quantity for the current level = Produced Quantity (level -1) * Component_Quantity (level -1)
, a.Demand_Material_CD
, a.Path + '|' + b.material_cd
FROM BOM a
inner join #BOM_ProdVersion_Priority b
on a.component_cd = b.material_cd
inner join #base_routes c
on a.Demand_Plant_CD = c.Recipient_Plant_CD
and b.Plant_CD = c.Source_Plant_CD
and c.Material_CD = b.Material_CD -- Need to have material_cd to link
where b.Material_CD != b.Component_CD
and b.Component_Quantity > 0
and BOM_Level < 5 -- Give the max number of levels deep we are allowed to cyncle to
and a.path NOT LIKE '%' + b.material_cd + '%'
)
select *
into #BOMExploded
from BOM
OPTION (MAXRECURSION 20)
Now you have a path that is pipe delimited and you can test your current material_cd to see if it's already in the path. If it is, then you end that leg of the recursion to prevent cycling.
Updating to include a version where we capture material_cd cycles and only report those at the end of the recursion:
exec pr_sys_drop_object '#BOMExploded'
;with BOM
as
(
select
Prod_Plant_CD
, Demand_Plant_CD
, material_cd
, component_cd
, component_quantity
, component_quantity AS Calculated_Component_Quantity
, BOM_Level
, Demand_Quantity
, CONVERT(float,1) AS Produced_Ratio
, Material_CD AS Demand_Material_CD
, CAST(material_cd AS VARCHAR(100)) AS Path
, CAST(NULL AS CHAR(5)) as Cycle_Flag
, 0 as Cycle_Depth
from #firstLevel a
UNION ALL
SELECT
b.Plant_CD as 'Prod_Plant_CD'
, a.Demand_Plant_CD
, b.Material_CD
, b.Component_CD
, b.component_quantity
, b.component_quantity
, a.BOM_Level + 1
, a.Demand_Quantity
, a.Produced_Ratio * a.Component_Quantity -- Produced Quantity for the current level = Produced Quantity (level -1) * Component_Quantity (level -1)
, a.Demand_Material_CD
, a.Path + '|' + b.material_cd
, CASE WHEN a.path NOT LIKE '%' + b.material_cd + '%' then 'Cycle' END AS Cycle_Flag,
, CASE WHEN a.path NOT LIKE '%' + b.material_cd + '%' then a.Cycle_Depth + 1 END as Cycle_Depth
FROM BOM a
inner join #BOM_ProdVersion_Priority b
on a.component_cd = b.material_cd
inner join #base_routes c
on a.Demand_Plant_CD = c.Recipient_Plant_CD
and b.Plant_CD = c.Source_Plant_CD
and c.Material_CD = b.Material_CD -- Need to have material_cd to link
where b.Material_CD != b.Component_CD
and b.Component_Quantity > 0
and a.cycle_depth < 2 --stop the query if we start cycling, but only after we capture at least one full cycle
)
select *
into #BOMExploded
from BOM
WHERE cycle_flag IS NOT NULL
OPTION (MAXRECURSION 20)
This will capture cycle_depth which is a counter that measures how deep into a cycle we get. We stop the recursion after we get to cycle_depth of 1 so the cycle can be captures in the final select.
If you could please help with this. The code generates an error:
Msg 8156, Level 16, State 1, Line 236
The column 'Classification_Value_Id' was specified multiple times for 'piv1'.
I am doing this on SQL Server. The steps for the code is as follows:
1. Unpivot the data from the source table DB.[dbo].[Classification] into one column
2. Join this unpivoted data to a table called DB.dbo.[Classification_Value] to return/add the column 'cv.Classification_Name' to the data set
3. Pivot this dataset (This is the part returning the error)
CODE:
SELECT
activityCode
, actjvPartnerRef
, actMonth
, actSalesChannel
, addCBPCharge
, agentId
, appType
, areaCode
--SELECT
--polRef,[Arrangement_Id],UnpivotedData.Classification_Value_Id,UnpivotedData.Classification_Scheme_Id,ColValues, ColNames,cv.Classification_Name
FROM
(
SELECT top 10
[polRef]
, [Arrangement_Id]
, [Classification_Scheme_Id]
, [Classification_Value_Id]
-- ,[Arrangement_Classification_Type_Id]
-- ,[Effective_TimeStamp]
-- ,[End_date]
, CAST((ISNULL([character_measure],'')) AS NVARCHAR(MAX)) AS character_measure
, CAST((ISNULL([datetime_measure],'')) AS NVARCHAR(MAX)) AS datetime_measure
, CAST([decimal_measure] AS NVARCHAR(MAX)) AS decimal_measure
, CAST((ISNULL([integer_measure],'')) AS NVARCHAR(MAX)) AS integer_measure
, CAST((ISNULL([logical_measure],'')) AS NVARCHAR(MAX)) AS logical_measure
, CAST((ISNULL([charmax_measure],'')) AS NVARCHAR(MAX)) AS charmax_measure
, CAST((ISNULL([long_measure],'')) AS NVARCHAR(MAX)) AS long_measure
FROM DB.[dbo].[Classification]
) AS SrcDataConverted
UNPIVOT
(
ColValues FOR ColNames IN
(
character_measure
, datetime_measure
, decimal_measure
, integer_measure
, logical_measure
, charmax_measure
, long_measure
)
) AS UnpivotedData
LEFT JOIN DB.dbo.[Classification_Value] cv
ON cv.[Classification_Scheme_Id] = UnpivotedData.[Classification_Scheme_Id]
AND cv.Classification_Value_Id = UnpivotedData.Classification_Value_Id
PIVOT
(MAX(ColValues) for Classification_Name in (
activityCode
, actjvPartnerRef
, actMonth
, actSalesChannel
, addCBPCharge
, agentId
, appType
, areaCode
)) AS piv1;
Any help would be much appreciated
Thank you
StuarLC:
An additional derived table needs to wrap the results of the UNPIVOT before commencing the re-PIVOT, as the join introduces a duplicated Classification_Value_Id and Classification_Scheme_id, which is needed for the join.
select
activityCode
, actjvPartnerRef
, actMonth
, actSalesChannel
, addCBPCharge
, agentId
, appType
, areaCode
from (
SELECT polRef
, [Arrangement_Id]
, UnpivotedData.Classification_Value_Id
, UnpivotedData.Classification_Scheme_Id
, ColValues
, ColNames
, Classification_Name
FROM (
SELECT [polRef]
, [Arrangement_Id]
, [Classification_Scheme_Id]
, [Classification_Value_Id]
, CAST((ISNULL([character_measure],'')) AS NVARCHAR(MAX)) AS character_measure
, CAST((ISNULL([datetime_measure],'')) AS NVARCHAR(MAX)) AS datetime_measure
, CAST([decimal_measure] AS NVARCHAR(MAX)) AS decimal_measure
, CAST((ISNULL([integer_measure],'')) AS NVARCHAR(MAX)) AS integer_measure
, CAST((ISNULL([logical_measure],'')) AS NVARCHAR(MAX)) AS logical_measure
, CAST((ISNULL([charmax_measure],'')) AS NVARCHAR(MAX)) AS charmax_measure
, CAST((ISNULL([long_measure],'')) AS NVARCHAR(MAX)) AS long_measure
FROM DB.[dbo].[Classification]
) AS SrcDataConverted
UNPIVOT
(
ColValues FOR ColNames IN
(
character_measure
, datetime_measure
, decimal_measure
, integer_measure
, logical_measure
, charmax_measure
, long_measure
)
) AS UnpivotedData
LEFT JOIN
DB.dbo.[Classification_Value] cv
ON cv.[Classification_Scheme_Id] = UnpivotedData.[Classification_Scheme_Id]
AND cv.Classification_Value_Id = UnpivotedData.Classification_Value_Id
) as src
PIVOT
(
MAX(ColValues) for Classification_Name in (
activityCode
, actjvPartnerRef
, actMonth
, actSalesChannel
, addCBPCharge
, agentId
, appType
, areaCode
)
) AS piv1;