I have a number of row base on plan and plan detail, I want to find out the same row set with other detail like plan 1 has 3 rows of data in detail table so need to find out the same rows for another plan. I have some sample data with this post may be more helpful to understand my problem. below is the Sample data Image, iwan to group by the record but not on a single row base on the full row set base on
PlanId, MinCount, MaxCount and CurrencyId
my expected data is below
I had tried to do with Some lengthy process like append all data in a single row and compare with other data, but it seems very lengthy process and takes to much time for 100 records I have an approx 20000 records in an actual database so not a good solution, please suggest me some thought
This would be trivial in MS Sql Server 2017 by using STRING_AGG.
But in 2012 one of the methods is to use the FOR XML trick.
SELECT PlanId, StairCount, MinCount, MaxCount, CurrencyId,
STUFF((
SELECT CONCAT(',', t1.AccountId)
FROM YourTable t1
WHERE t1.PlanId = t.PlanId
AND t1.StairCount = t.StairCount
AND t1.MinCount = t.MinCount
AND t1.MaxCount = t.MaxCount
AND t1.CurrencyId = t.CurrencyId
ORDER BY t1.AccountId
FOR XML PATH('')), 1, 1, '') AS AccountIdList
FROM YourTable t
GROUP BY PlanId, StairCount, MinCount, MaxCount, CurrencyId
Test here
Your desired result is rather hard to produce in SQL Server. The simplest method requires two levels of string concatenation:
with t as (
select a.account_id,
stuff( (select '[' +
convert(varchar(255), staircount) + ',' +
convert(varchar(255), mincount) + ',' +
convert(varchar(255), maxcount) + ',' +
convert(varchar(255), currencyid) +
']'
from t
where t.account_id = a.account_id
order by staircount
for xml path ('')
), 1, 1, '') as details
from (select distinct account_id from t) a
)
select d.details,
stuff( (select cast(varchar(255), account_id) + ','
from t
where t.details = d.details
for xml path ('')
), 1, 1, '') as accounts
from (select distinct details from t) d;
This isn't exactly your output, but it might be good enough for your problem.
Related
I have a situation where I need to compare two tables where t1.ColumnA = t2.ColumnA and t1.ColumnB<>t2.ColumnB. The caveat to this problem is that t2.ColumnB is using a "FOR XML PATH" to concatenate like values from another table (linked server using OPENQUERY). This all has to be done within a View.
To concatenate the rows, I am using the following code:
SELECT DISTINCT
CAST(A.CHECK_NUMBER AS nvarchar) [CHECK_NUMBER]
,(
SELECT B.INVOICE_NUMBER + '|'
FROM OPENQUERY([SERVER], 'SELECT * FROM CHECK_LISTING ') B
WHERE B.CHECK_NUMBER = A.CHECK_NUMBER
AND (NULLIF (B.INVOICE_NUMBER, '') IS NOT NULL)
FOR XML PATH('')
) [INVOICE_NUMBER]
, (
SELECT LTRIM(RTRIM(B.PURCHASE_ORDER_ID)) + '|'
FROM OPENQUERY([SERVER], 'SELECT * FROM CHECK_LISTING ') B
WHERE B.CHECK_NUMBER = A.CHECK_NUMBER
AND (NULLIF (B.PURCHASE_ORDER_ID, '') IS NOT NULL)
FOR XML PATH('')
) [PURCHASE_ORDER_ID]
FROM OPENQUERY([SERVER], 'SELECT * FROM CHECK_LISTING ') AS A
This works perfectly and concatenates just like it needs to. My problem is that my next view I created was to run that view against a local table to see the difference in the INVOICE_NUMBER.
SELECT
A.EntryID,
A.Check#,
A.CheckDate,
A.CheckAmount,
A.VendorID,
A.VendorName,
B.INVOICE_NUMBER,
A.Invoice#
FROM dbo.APChecks AS A
LEFT JOIN
dbo.CHECKS_Step2 AS B
ON A.Check#=B.CHECK_NUMBER
WHERE (A.Invoice# != B.INVOICE_NUMBER)
When I try to run this, the query takes at least 25+ minutes. I had to stop the query manually. Some concatenated values are longer than 1000 chars. I was told that it was not possible to INDEX on a dynamic entry like an XML PATH.
Any suggestions? Thank you in advance.
I'm building a query and the latest step involved adding a SELECT COUNT(*) FROM [modification] sub-clause to allow me to detect the last row of the query, but it destroys the performance:
SELECT CONCAT(
IIF(row_number() OVER (ORDER BY forecastId) % 50000 = 1,
CONCAT('INSERT INTO modification (userId, epochTime, ',
'forecastId, description, auxText, auxDate) VALUES ('),
' ('),
userId, ',',
epochTime, ',',
forecastId, ',',
'''', dbo.encode4MySql(description), ''',',
'''', dbo.encode4MySql(auxText), ''',',
'''', CONVERT(VARCHAR(20), auxDate, 120), ''')',
IIF(row_number() OVER (ORDER BY forecastId) % 50000 = 0
OR row_number() OVER (ORDER BY forecastId) =
(SELECT COUNT(*) FROM modification),
'; COMMIT;', ','))
FROM modification
ORDER BY forecastId;
If you can't see what I'm doing, I'm building INSERT () VALUES (),(),(),... statements of 50000 rows at a time.
Please restrict suggestions for completely alternative approaches to the comments. I'm looking for a way to find the last row number here without it slowing the query down massively - which it does.
I'm not massively familiar with query plans but can post one here if it helps. I've tried a lot of things from related questions here, but nothing that I can get to work.
One other option would be to order by forecastId desc in the final or:
IIF(row_number() OVER (ORDER BY forecastId) % 50000 = 0
OR row_number() OVER (ORDER BY forecastId desc) = 1,
'; COMMIT;', ','))
You have a rather complicated expression, so SQL Server may not optimize it. Move the logic to the FROM clause:
FROM (SELECT m.*, COUNT(*) OVER () as num_rows
FROM modification m
) m
And then use num_rows in the rest of the query.
Assuming you don't want to change the current design, you could just add an extra UNION ALL step at the end. By looking at your query, it looks like the only purpose of changing the query is to add a COMMIT at the end.
CURRENT QUERY
UNION ALL
SELECT 'COMMIT;';
Let me know if that works for you.
***********UPDATE*********
I thought this query is easier to troubleshoot. See if it will perform any better. You would have to plugin the CTE part for your table.
SELECT BusinessEntityID,JobTitle,HireDate INTO dbo.TestTable FROM [HumanResources].[Employee]
SELECT TOP 0 BusinessEntityID,JobTitle,HireDate INTO dbo.TestTable2 FROM [HumanResources].[Employee]
SET NOCOUNT ON
WITH CTE AS (SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) Col,
' (' +
CAST(BusinessEntityID AS VARCHAR(20)) +
', ''' +
JobTitle +
''', ''' +
CONVERT(VARCHAR(20), HireDate, 120) +
''')' Query
FROM TestTable
)
SELECT
CASE
WHEN COL % 50 = 0 THEN ', ' + Query + ' COMMIT;'
WHEN COL % 50 = 1 THEN ' INSERT INTO dbo.TestTable2 (BusinessEntityID, JobTitle, HireDate) VALUES ' + Query
ELSE ', ' + Query
END
FROM CTE
UNION ALL SELECT 'COMMIT;'
I have an interesting problem. I have two T-SQL STUFF statements. In this example I have used union. This displays the results in two rows.
What I’d like to do is to combine these results so that it returns a single row. I am using SQL Server 2008 R2. The solution should be able to copy with either stuff statement returning null.
Is there a way to achieve this?
Here is the code
SELECT
(SELECT
STUFF((SELECT (' ' + mod_orders.mod_no + '<br>')
FROM mod_serials
INNER JOIN mod_orders ON mod_serials.mod_id = mod_orders.mod_id
INNER JOIN mod_order_items ON mod_orders.mod_id = mod_order_items.mod_id AND mod_order_items.item_id = 170
WHERE
(mod_serials.serial_id = 62 AND
mod_serials.date_implemented IS NOT NULL)
ORDER BY mod_serials.serial_id
FOR XML PATH(''), TYPE).value('.', 'nvarchar(max)'), 1, 1, ''))
UNION
SELECT
(SELECT
STUFF((SELECT (' ' + char_data_mv.char_value + '<br>')
FROM char_data_mv
WHERE (char_data_mv.object_id = 62 AND char_data_mv.char_id = 177)
ORDER BY char_data_mv.row_id
FOR XML PATH(''), TYPE ).value('.','nvarchar(max)'), 1, 1, '')) as [Impl]
Example data:
Row1 - 00001<br> 00005<br>
Row2 - PMB 12345<br>
Combined results would be:
00001<br> 00005<br> PMB 12345<br>
The UNION statement returns all of the rows from the first table, and then all of the rows from the second table together in one big table - so this is why you are getting 2 rows.
Instead, you can just add 2 strings together with the '+' operator. You also want to return a string if the other is null, and adding nulls begets nulls. So use the ISNULL function to give an empty string instead.
SELECT
ISNULL((SELECT STUFF
((SELECT
(' ' + mod_orders.mod_no + '<br>')
FROM mod_serials
INNER JOIN mod_orders ON mod_serials.mod_id = mod_orders.mod_id
INNER JOIN mod_order_items ON mod_orders.mod_id = mod_order_items.mod_id AND mod_order_items.item_id = 170
WHERE (mod_serials.serial_id = 62 AND mod_serials.date_implemented IS NOT NULL)
ORDER BY mod_serials.serial_id FOR XML PATH(''), TYPE ).value('.', 'nvarchar(max)'), 1, 1, '')),'')
+
ISNULL((SELECT STUFF
((SELECT
(' ' + char_data_mv.char_value + '<br>')
FROM char_data_mv
WHERE (char_data_mv.object_id = 62 AND char_data_mv.char_id = 177)
ORDER BY char_data_mv.row_id FOR XML PATH(''), TYPE ).value('.','nvarchar(max)'), 1, 1, '')),'') as [Impl]
Note that you need to remove bothe the UNION and also the first SELECT after it - which is creating the second table.
I'm trying to control the output of this query to return the values sorted from lower to higher values. As you can see on the last line of the results below I'm getting 16,15,1 as the result, but I need to get 1,15,16 as the result.
Below is the script I'm currently using. I would appreciate any help I can get. Thank you in advance.
SELECT STUFF ((SELECT ',' + CONVERT(varchar(max), UserId)
FROM MessageRecipients mr2
WHERE mr1.ConversationId = mr2.ConversationId FOR xml path('')), 1, 1,'') AS 'Result'
FROM MessageRecipients mr1
GROUP BY ConversationId
This is the result I get with the current script (column name 'result'):
1,19
1,15
16,15,1
Just add an ORDER BY to the inner SELECT:
SELECT STUFF (
(SELECT ',' + CONVERT(varchar(max), UserId)
FROM MessageRecipients mr2
WHERE mr1.ConversationId = mr2.ConversationId
ORDER BY UserId
FOR xml path(''))
, 1, 1,'') AS 'Result'
FROM MessageRecipients mr1
GROUP BY ConversationId
I have a table that contains text field with placeholders. Something like this:
Row Notes
1. This is some notes ##placeholder130## this ##myPlaceholder##, #oneMore#. End.
2. Second row...just a ##test#.
(This table contains about 1-5k rows on average. Average number of placeholders in one row is 5-15).
Now, I have a lookup table that looks like this:
Name Value
placeholder130 Dog
myPlaceholder Cat
oneMore Cow
test Horse
(Lookup table will contain anywhere from 10k to 100k records)
I need to find the fastest way to join those placeholders from strings to a lookup table and replace with value. So, my result should look like this (1st row):
This is some notes Dog this Cat, Cow. End.
What I came up with was to split each row into multiple for each placeholder and then join it to lookup table and then concat records back to original row with new values, but it takes around 10-30 seconds on average.
You could try to split the string using a numbers table and rebuild it with for xml path.
select (
select coalesce(L.Value, T.Value)
from Numbers as N
cross apply (select substring(Notes.notes, N.Number, charindex('##', Notes.notes + '##', N.Number) - N.Number)) as T(Value)
left outer join Lookup as L
on L.Name = T.Value
where N.Number <= len(notes) and
substring('##' + notes, Number, 2) = '##'
order by N.Number
for xml path(''), type
).value('text()[1]', 'varchar(max)')
from Notes
SQL Fiddle
I borrowed the string splitting from this blog post by Aaron Bertrand
SQL Server is not very fast with string manipulation, so this is probably best done client-side. Have the client load the entire lookup table, and replace the notes as they arrived.
Having said that, it can of course be done in SQL. Here's a solution with a recursive CTE. It performs one lookup per recursion step:
; with Repl as
(
select row_number() over (order by l.name) rn
, Name
, Value
from Lookup l
)
, Recurse as
(
select Notes
, 0 as rn
from Notes
union all
select replace(Notes, '##' + l.name + '##', l.value)
, r.rn + 1
from Recurse r
join Repl l
on l.rn = r.rn + 1
)
select *
from Recurse
where rn =
(
select count(*)
from Lookup
)
option (maxrecursion 0)
Example at SQL Fiddle.
Another option is a while loop to keep replacing lookups until no more are found:
declare #notes table (notes varchar(max))
insert #notes
select Notes
from Notes
while 1=1
begin
update n
set Notes = replace(n.Notes, '##' + l.name + '##', l.value)
from #notes n
outer apply
(
select top 1 Name
, Value
from Lookup l
where n.Notes like '%##' + l.name + '##%'
) l
where l.name is not null
if ##rowcount = 0
break
end
select *
from #notes
Example at SQL Fiddle.
I second the comment that tsql is just not suited for this operation, but if you must do it in the db here is an example using a function to manage the multiple replace statements.
Since you have a relatively small number of tokens in each note (5-15) and a very large number of tokens (10k-100k) my function first extracts tokens from the input as potential tokens and uses that set to join to your lookup (dbo.Token below). It was far too much work to look for an occurrence of any of your tokens in each note.
I did a bit of perf testing using 50k tokens and 5k notes and this function runs really well, completing in <2 seconds (on my laptop). Please report back how this strategy performs for you.
note: In your example data the token format was not consistent (##_#, ##_##, #_#), I am guessing this was simply a typo and assume all tokens take the form of ##TokenName##.
--setup
if object_id('dbo.[Lookup]') is not null
drop table dbo.[Lookup];
go
if object_id('dbo.fn_ReplaceLookups') is not null
drop function dbo.fn_ReplaceLookups;
go
create table dbo.[Lookup] (LookupName varchar(100) primary key, LookupValue varchar(100));
insert into dbo.[Lookup]
select '##placeholder130##','Dog' union all
select '##myPlaceholder##','Cat' union all
select '##oneMore##','Cow' union all
select '##test##','Horse';
go
create function [dbo].[fn_ReplaceLookups](#input varchar(max))
returns varchar(max)
as
begin
declare #xml xml;
select #xml = cast(('<r><i>'+replace(#input,'##' ,'</i><i>')+'</i></r>') as xml);
--extract the potential tokens
declare #LookupsInString table (LookupName varchar(100) primary key);
insert into #LookupsInString
select distinct '##'+v+'##'
from ( select [v] = r.n.value('(./text())[1]', 'varchar(100)'),
[r] = row_number() over (order by n)
from #xml.nodes('r/i') r(n)
)d(v,r)
where r%2=0;
--tokenize the input
select #input = replace(#input, l.LookupName, l.LookupValue)
from dbo.[Lookup] l
join #LookupsInString lis on
l.LookupName = lis.LookupName;
return #input;
end
go
return
--usage
declare #Notes table ([Id] int primary key, notes varchar(100));
insert into #Notes
select 1, 'This is some notes ##placeholder130## this ##myPlaceholder##, ##oneMore##. End.' union all
select 2, 'Second row...just a ##test##.';
select *,
dbo.fn_ReplaceLookups(notes)
from #Notes;
Returns:
Tokenized
--------------------------------------------------------
This is some notes Dog this Cat, Cow. End.
Second row...just a Horse.
Try this
;WITH CTE (org, calc, [Notes], [level]) AS
(
SELECT [Notes], [Notes], CONVERT(varchar(MAX),[Notes]), 0 FROM PlaceholderTable
UNION ALL
SELECT CTE.org, CTE.[Notes],
CONVERT(varchar(MAX), REPLACE(CTE.[Notes],'##' + T.[Name] + '##', T.[Value])), CTE.[level] + 1
FROM CTE
INNER JOIN LookupTable T ON CTE.[Notes] LIKE '%##' + T.[Name] + '##%'
)
SELECT DISTINCT org, [Notes], level FROM CTE
WHERE [level] = (SELECT MAX(level) FROM CTE c WHERE CTE.org = c.org)
SQL FIDDLE DEMO
Check the below devioblog post for reference
devioblog post
To get speed, you can preprocess the note templates into a more efficient form. This will be a sequence of fragments, with each ending in a substitution. The substitution might be NULL for the last fragment.
Notes
Id FragSeq Text SubsId
1 1 'This is some notes ' 1
1 2 ' this ' 2
1 3 ', ' 3
1 4 '. End.' null
2 1 'Second row...just a ' 4
2 2 '.' null
Subs
Id Name Value
1 'placeholder130' 'Dog'
2 'myPlaceholder' 'Cat'
3 'oneMore' 'Cow'
4 'test' 'Horse'
Now we can do the substitutions with a simple join.
SELECT Notes.Text + COALESCE(Subs.Value, '')
FROM Notes LEFT JOIN Subs
ON SubsId = Subs.Id WHERE Notes.Id = ?
ORDER BY FragSeq
This produces a list of fragments with substitutions complete. I am not an MSQL user, but in most dialects of SQL you can concatenate these fragments in a variable quite easily:
DECLARE #Note VARCHAR(8000)
SELECT #Note = COALESCE(#Note, '') + Notes.Text + COALSCE(Subs.Value, '')
FROM Notes LEFT JOIN Subs
ON SubsId = Subs.Id WHERE Notes.Id = ?
ORDER BY FragSeq
Pre-processing a note template into fragments will be straightforward using the string splitting techniques of other posts.
Unfortunately I'm not at a location where I can test this, but it ought to work fine.
I really don't know how it will perform with 10k+ of lookups.
how does the old dynamic SQL performs?
DECLARE #sqlCommand NVARCHAR(MAX)
SELECT #sqlCommand = N'PlaceholderTable.[Notes]'
SELECT #sqlCommand = 'REPLACE( ' + #sqlCommand +
', ''##' + LookupTable.[Name] + '##'', ''' +
LookupTable.[Value] + ''')'
FROM LookupTable
SELECT #sqlCommand = 'SELECT *, ' + #sqlCommand + ' FROM PlaceholderTable'
EXECUTE sp_executesql #sqlCommand
Fiddle demo
And now for some recursive CTE.
If your indexes are correctly set up, this one should be very fast or very slow. SQL Server always surprises me with performance extremes when it comes to the r-CTE...
;WITH T AS (
SELECT
Row,
StartIdx = 1, -- 1 as first starting index
EndIdx = CAST(patindex('%##%', Notes) as int), -- first ending index
Result = substring(Notes, 1, patindex('%##%', Notes) - 1)
-- (first) temp result bounded by indexes
FROM PlaceholderTable -- **this is your source table**
UNION ALL
SELECT
pt.Row,
StartIdx = newstartidx, -- starting index (calculated in calc1)
EndIdx = EndIdx + CAST(newendidx as int) + 1, -- ending index (calculated in calc4 + total offset)
Result = Result + CAST(ISNULL(newtokensub, newtoken) as nvarchar(max))
-- temp result taken from subquery or original
FROM
T
JOIN PlaceholderTable pt -- **this is your source table**
ON pt.Row = T.Row
CROSS APPLY(
SELECT newstartidx = EndIdx + 2 -- new starting index moved by 2 from last end ('##')
) calc1
CROSS APPLY(
SELECT newtxt = substring(pt.Notes, newstartidx, len(pt.Notes))
-- current piece of txt we work on
) calc2
CROSS APPLY(
SELECT patidx = patindex('%##%', newtxt) -- current index of '##'
) calc3
CROSS APPLY(
SELECT newendidx = CASE
WHEN patidx = 0 THEN len(newtxt) + 1
ELSE patidx END -- if last piece of txt, end with its length
) calc4
CROSS APPLY(
SELECT newtoken = substring(pt.Notes, newstartidx, newendidx - 1)
-- get the new token
) calc5
OUTER APPLY(
SELECT newtokensub = Value
FROM LookupTable
WHERE Name = newtoken -- substitute the token if you can find it in **your lookup table**
) calc6
WHERE newstartidx + len(newtxt) - 1 <= len(pt.Notes)
-- do this while {new starting index} + {length of txt we work on} exceeds total length
)
,lastProcessed AS (
SELECT
Row,
Result,
rn = row_number() over(partition by Row order by StartIdx desc)
FROM T
) -- enumerate all (including intermediate) results
SELECT *
FROM lastProcessed
WHERE rn = 1 -- filter out intermediate results (display only last ones)