I have a whole SWIFT message with fees in one "cell" as SwiftMessage.Body. So the whole message is in one string. What I need to do is, extract certain data from it. Using SSRS and MS SQL The message looks something like this:
.....(FEE 1)
:20C::PCOM//C22033100734330
:20C::PREF//FC22033100734330
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//NEUR50
:99A::DAAC//001
.....(FEE 2)
:20C::PCOM//C22033100734331
:20C::PREF//FC22033100734331
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//EUR40
:99A::DAAC//002
.....(FEE n)
there can be any number of fees, not just 2
The result should be extracting :20C::PREF// and :19A::AMCO//:
ID
Amount
FC22033100734330
-50
FC22033100734331
40
what I have right now
SELECT
SUBSTRING(swf.SwiftMessage.Body, (CHARINDEX(':20C::PREF', swf.SwiftMessage.Body) + 12)
, CHARINDEX(':22H::PNTP',swf.SwiftMessage.Body) - CHARINDEX(':20C::PREF', swf.SwiftMessage.Body) - 12) as REF1,
SUBSTRING(swf.SwiftMessage.Body, (CHARINDEX(':19A::AMCO', swf.SwiftMessage.Body) + 12)
, CHARINDEX(':99A::DAAC',swf.SwiftMessage.Body) - CHARINDEX(':19A::AMCO', swf.SwiftMessage.Body) - 12) as AMT
FROM
swf.SwiftMessage
so with this I am somehow able to extract data I need (the amount is in format currency+amt which I can deal with later hopefuly). The main problem right now is how to deal with the fact, that there might be more fees, than just one so I need to make some kind of a loop? that will go through the whole string and find every :20C::PREF// and :19A::AMCO// values.
First of all, I would suggest doing this splitting at a different place, either when inserting the data into the database, or after retrieving it, perhaps using the language that the program used to communicate with the DB.
If you really must do the splitting using SQL alone, you have a few options to loop over the Data. They either use Recursion, classic WHILE loops or use the STRING_SPLIT function, as mentioned in this post.
If you happen to use SQL Server 2016 or later, I suggest you use STRING_SPLIT, something along the lines of
SELECT value FROM STRING_SPLIT('.....(FEE 1):20C::PCOM//C22033100734330:20C::PREF//FC22033100734330:22H::PNTP//SEFP:24B::ACTV//NEWP:19A::AMCO//NEUR50:99A::DAAC//001.....(FEE 2):20C::PCOM//C22033100734331:20C::PREF//FC22033100734331:22H::PNTP//SEFP:24B::ACTV//NEWP:19A::AMCO//EUR40:99A::DAAC//002.....(FEE n)', ':99A::DAAC');
this together with CROSS APPLY whould get you a whole step closer to parsing the data for each message:
SELECT value FROM swf.SwiftMessage CROSS APPLY STRING_SPLIT(Body,':99A::DAAC')
then just substring each value
Here is an option which parses the string via JSON to maintain the SEQUENCE. Then it becomes a small matter of a conditional aggregation
Note: this is keyed on EUR. If you have other currencies, it would be a small tweak
Example
Declare #S varchar(max)='.....(FEE 1)
:20C::PCOM//C22033100734330
:20C::PREF//FC22033100734330
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//NEUR50
:99A::DAAC//001
.....(FEE 2)
:20C::PCOM//C22033100734331
:20C::PREF//FC22033100734331
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//EUR40
:99A::DAAC//002'
Select ID = max( case when value like ':20C::PREF//%' then substring(value,13,100) end)
,Amt = max( case when value like ':19A::AMCO//%' then try_convert(decimal(15,4),replace(replace(substring(value,13,100),'NEUR','-'),'EUR','')) end)
From (
Select *
,Grp = sum( case when value like '.....(FEE%' then 1 end ) over (order by convert(int,[key]))
From OpenJSON( '["'+replace(string_escape(replace(#S,char(13)+char(10),'|||'),'json'),'|||','","')+'"]' )
) A
Group By Grp
Results
ID Amt
FC22033100734330 -50.0000
FC22033100734331 40.0000
Give this a try:
DECLARE #swifty NVARCHAR(MAX) = '.....(FEE 1)
:20C::PCOM//C22033100734330
:20C::PREF//FC22033100734330
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//NEUR50
:99A::DAAC//001
.....(FEE 2)
:20C::PCOM//C22033100734331
:20C::PREF//FC22033100734331
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//EUR40
:99A::DAAC//002
.....(FEE 3)
:20C::PCOM//C22033100734332
:20C::PREF//FC22033100734332
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//USD9999
:99A::DAAC//001
.....(FEE 4)
:20C::PCOM//C22033100734333
:20C::PREF//FC22033100734333
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//GBP1
:99A::DAAC//001
.....(FEE 5)
:20C::PCOM//C22033100734334
:20C::PREF//FC22033100734334
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//NGBP300
:99A::DAAC//001
.....(FEE 6)
:20C::PCOM//C22033100734335
:20C::PREF//FC22033100734335
:22H::PNTP//SEFP
:24B::ACTV//NEWP
:19A::AMCO//NUSD325412254
:99A::DAAC//001
'
SELECT FeeID,
MAX(CASE WHEN value LIKE CHAR(10)+':20C::PREF//%' THEN SUBSTRING(value,CHARINDEX('//',value)+2,LEN(value)) END) AS ID,
MAX(CASE WHEN value LIKE CHAR(10)+':19A::AMCO//N%' THEN SUBSTRING(value,CHARINDEX('//',value)+3,3)
WHEN value LIKE CHAR(10)+':19A::AMCO//[^N]%' THEN SUBSTRING(value,CHARINDEX('//',value)+2,3) END) AS Currency,
MAX(CASE WHEN value LIKE CHAR(10)+':19A::AMCO//N%' THEN SUBSTRING(value,CHARINDEX('//',value)+6,LEN(value))*-1
WHEN value LIKE CHAR(10)+':19A::AMCO//[^N]%' THEN SUBSTRING(value,CHARINDEX('//',value)+5,LEN(value)) END) AS Fee
FROM (
SELECT Value, ((ROW_NUMBER() OVER (ORDER BY (SELECT 1))-1) / 7)+1 AS FeeID
FROM STRING_SPLIT(#swifty,CHAR(13))
) A
WHERE value LIKE CHAR(10)+':19A::AMCO//%'
OR value LIKE CHAR(10)+':20C::PREF//%'
GROUP BY FeeID
This shouldn't be reliant on the values being in any specific format or type, just that they are referenced as :19A:: and :20C::
FeeID ID Currency Fee
-------------------------------------------
1 FC22033100734330 EUR -50
2 FC22033100734331 EUR 40
3 FC22033100734332 USD 9999
4 FC22033100734333 GBP 1
5 FC22033100734334 GBP -300
6 FC22033100734335 USD -325412254
Related
Suppose I have a table like this in SQL Server 2017, let's call it "maps_and_cups"
some_code
quantity
big_map
6
tiny_map
5
big_cup
10
tiny_cup
4
I would like to know the best way to group the maps and cups into one, in this way.
some_code
quantity
maps
11
cups
14
I know that it is using "if" and "case", adding and comparing if it is a tiny_map, a big_map, and so on, I have seen several examples but I cannot make it compile.
You can indeed use a case when expression. For instance:
with base as
(select case some_code when 'big_map' then 'maps'
when 'tiny_map' then 'maps'
when 'big_cup' then 'cups'
when 'tiny_cup' then 'cups'
else 'other'
end grp,
quantity
from maps_and_cups)
select grp, sum(quantity) quantity from base group by grp;
However, if you're going to list each and every code explicitly, you might as well create a reference table for it:
some_code
grp
big_map
maps
tiny_map
maps
big_cup
cups
tiny_cup
cups
...and then join that table into your query:
select grp, sum(quantity)
from maps_and_cups a left join ref_maps_cups b on a.some_code = b.some_code
group by grp;
You can solve this task using "case" and "charindex" functions, like this:
declare
#t table (some_code varchar (20), quantity int)
insert into #t
values
('big_map', 6),
('tiny_map', 5),
('big_cup',10),
('tiny_cup', 4)
select
case
when charindex ('map', some_code)>0 then 'map'
when charindex ('cup', some_code)>0 then 'cup'
end some_code
,sum(quantity) quantity
from #t
group by
case
when charindex ('map', some_code)>0 then 'map'
when charindex ('cup', some_code)>0 then 'cup'
end
OUTPUT:
If you just want the right three characters for aggregating, you can use right():
select right(some_code, 3) + 's', sum(quantity)
from maps_and_cups
group by right(some_code, 3) + 's';
You are creating a problem for yourself as you're (probably) breaking the first normal form by storing non atomic values in the field "some_code". (Some field name i'd say. ;)
Why not separating the value into [ type ] and [ size ] ?
I have a continuous variable SQL table:
x
1 622.108
2 622.189
3 622.048
4 622.758
5 622.191
6 622.677
7 622.598
8 622.020
9 621.228
10 622.690
...
and I try to get a simple frequency table, e.g. with 3 buckets, like this:
bucket n
[621.228-621.738[ 1
[621.738-622.248[ 5
[622.248-622.758] 4
Seems easy but I cannot manage to make it in SQL (I am running it on a Cloudera Impala engine).
I have looked into dense_rank() or ntile() without success.
Any idea ?
You can use window functions to divide the range into three equal parts and then use arithmetic:
select min_x + range * (row_number() over (order by min(x)) - 1) as bucket_hi,
min_x + range * row_number() over (order by min(x)) as bucket_hi,
count(*)
from (select t.*,
min(x) over () as min_x,
max(x) over () as max_x,
0.000001 + max(x) over () - min(x) over () as range
from t
) t
group by floor((x - min_x) / range)), min_x, range
There are at least two problems with your question:
You have not provided any code to show us what you have tried. It really is good sometimes to just work out the problem yourself. Nevertheless, I found the problem interesting and decided to play.
Your range blocks overlap. If, for example, you were to have the value 621.738 in your list, which bucket would contain it? [621.228-621.738] or [621.738-622.248]?
There are also at least three problems with my answer, so I don't expect you to accept this. However, maybe it will get you started. Hopefully, this disclaimer will keep me from getting down voted. :-)
The answer is in T-SQL. Sorry, it's what I have to work with.
The answer is not generic. It always creates three and only three buckets.
It only works if the data type limits the result to 3 decimal places.
Remember, this is only one possible solution, and in my mind a very weak one at that.
With those disclaimers, here's what I wrote:
SELECT
'[' + STR( RANGES.RANGESTART, 7, 3 )
+ ' - '
+ STR( RANGES.RANGEEND, 7, 3 ) + ']' AS 'BUCKET'
,COUNT(*) AS 'N'
FROM
( SELECT
VALS.MINVAL + (CAST( CNT.INC AS DECIMAL(7,3) ) * VALS.RANGEWIDTH) AS 'RANGESTART'
,CASE WHEN CNT.INC < 2
THEN VALS.MINVAL + (CAST( CNT.INC + 1 AS DECIMAL(7,3) ) * VALS.RANGEWIDTH) - 0.001
ELSE VALS.MINVAL + (CAST( CNT.INC + 1 AS DECIMAL(7,3) ) * VALS.RANGEWIDTH)
END AS 'RANGEEND'
FROM
( SELECT
MIN(CURVAL) AS 'MINVAL'
,MAX(CURVAL) AS 'MAXVAL'
,(MAX(CURVAL) - MIN(CURVAL)) / 3 AS 'RANGEWIDTH'
FROM
MYVALUE ) VALS
CROSS JOIN (VALUES (0), (1), (2) ) CNT(INC)
) RANGES
INNER JOIN MYVALUE V
ON V.CURVAL BETWEEN RANGES.RANGESTART AND RANGES.RANGEEND
GROUP BY
RANGES.RANGESTART
,RANGES.RANGEEND
ORDER BY 1
;
In the above, your values would be in the CURVAL column of the MYVALUE table.
Good luck. I hope this helps you on your way.
I am trying to create an expression in qlik sense to get the count the distinct number of ID's where each prod is added up and is greater than 0.
Qlik sense expression so far, but wrong:
sum(aggr(count (DISTINCT ID), PROD1 + PROD2 + PROD3 ))
I'm not too sure how to add to the expression where to add >0 and the year month.
Working sql:
select count(distinct ID) as Number
from tb1 x
where (x.Prod1 + x.Prod2 + x.Prod3)> 0
x.Year = '2016/05'
Any help would be great,
Thanks.
The easiest way is with an if statement, your code
select count(distinct ID) as Number
from tb1 x
where (x.Prod1 + x.Prod2 + x.Prod3)> 0
x.Year = '2016/05'
becomes
count (DISTINCT, if((PROD1 + PROD2 + PROD3)>0,ID)))
If you have a date field in your database, you'll need to create a YearMonth field from your date (Date(mydate, 'YYYYMM') as YearMonth) in the data model script and then put this in your expression:
count({<Prod1={'>0'}, Prod2={'>0'}, Prod3={'>0'}, YearMonth={'201605'}>}[distinct ID])
If your field Year in the database is already a yearmonth field, you can do this (but I recommend the first method):
count({<Prod1={'>0'}, Prod2={'>0'}, Prod3={'>0'}, Year={'2016/05'}>}[distinct ID])
You should read this help section from the Qlik site, it's about set analysis
in your script add the calc field:
rangesum(Prod1,Prod2,Prod3) as Prod_Total
"rangesum" also converts null to 0! if Prod1,Prod2 or Prod3 is null you will get 0 as a total.
In the chart use this calc:
count({<Prod_Total={'>0'}>} Distinct ID)
I have a table with a certain flag called FL_virtual, if this flag equals 1 i need to get my stock in a special way using a function
now i want to make a select statement with it but depending on this flag i need to adjust my select to use a certain function instead of a subquery
so presume i start with this select statement
select product_name,..(other options from the product table),
(select sum(qy_stock) from STOCK where warehouse_id = 1) as 'qy_stock_internal',
(select sum(qy_stock) from STOCK where warehouse_id = 2) as qy_stock_external
From product
now i need to change the subquery (qy_stock) with a call to a function when the fl_virtual flag is 1
so that it becomes like this
select product_name,..(other options from the product table),
FN_GET_stock_PRODUCT(1) as qy_stock_internal,
FN_GET_stock_PRODUCT(2) as qy_stock_external
from product
so i thought a simple if then else structure will do but for some reason i can't get it to work
this is how i thought it would look
select product_name,..(other options from the product table),
IF fl_virtual > 0 THEN
(select sum(qy_stock) from STOCK where warehouse_id = 1) as 'qy_stock_internal',
(select sum(qy_stock) from STOCK where warehouse_id = 2) as qy_stock_external
ELSE
FN_GET_stock_PRODUCT(1) as qy_stock_internal,
FN_GET_stock_PRODUCT(2) as qy_stock_external
END IF
but it doesn't work , anyone got an idea?
You're close - just use CASE instead of IF (you have to repeat your condition, since you cannot easily return two columns from a single CASE (see P.S.):
select product_name,
..(other options from the product table),
(CASE
WHEN fl_virtual > 0 THEN
(select sum(qy_stock) from STOCK where warehouse_id = 1)
ELSE
FN_GET_stock_PRODUCT(1)
END) as qy_stock_internal,
(CASE
WHEN fl_virtual > 0 THEN
(select sum(qy_stock) from STOCK where warehouse_id = 2)
ELSE
FN_GET_stock_PRODUCT(2)
END) as qy_stock_external
P.S.: It is possible to return multiple values from a single CASE, e.g. using Object Types, but that's stuff for a different question :-)
Best way is to use a union select with those 2 different queries and contitions.
Btw. the names stock and warehouse look like a typical school work examin.
Ok, the title is probably confusing.
I need to use a case statement to calculate the amount of tax to deduct offa value.
This reduced value will be used in a Scalar function to calculate another value in a CTE in a Stored procedure..
How can I re-use this calculated column inside the scalar function?? I have tried the following.
paidminusIPT = case when country = "ireland" then (paid - 1) else paid end,
dbo.fnEarnedPrem(a,b,c,d,paidminusipt,e,f)
please note, this isn't the exact code, just a quick reference..
Is this possible to do?
Regards.
You should use full expression instead of just column name:
paidminusIPT = case when country = "ireland" then (paid - 1) else paid end,
dbo.fnEarnedPrem(a,b,c,d,case when country = "ireland" then (paid - 1) else paid end,e,f)
or use CTE
;WITH C as(
SELECT a,b,c,d,e,f,paidminusIPT = case when country = "ireland" then (paid - 1) else paid end FROM SomeTable)
Select *,dbo.fnEarnedPrem(a,b,c,d,paidminusipt,e,f) FROM C