SAP HANA SQL SUBSTR_REGEXPR Match Aggregation - sql

Question

I am using HANA and am trying to create a new column based on the following:
Regex Example 1: SUBSTR_REGEXPR('([PpSs][Tt][Ss]?\w?\d{2,6})' in "TEXT") as "Location"
How can I get this to return all results instead of just the first? Is it a string agg of this expression repeated? There would be at most 6 matches in each text field (per row).
Regex Example 1 Current Output:
Row Text Location(new column)
1 msdfmsfmdf PT2222, ST 43434 asdasdas PT2222
Regex Example 1 Desired Output:
Row Text Location(new column)
1 msdfmsfmdf PT2222, ST 43434 asdasdas PT2222, ST43434
I also have varying formats so I need to be able to use multiple variations of that regex to be able to capture all matches and put them into the new "Location" column as a delimited aggregation. Is this possible?
One of the other variations is where I would need to pull the numbers from this series:
"Locations 1, 2, 35 & 5 lkfaskjdlsaf .282 lkfdsklfjlkdsj 002"
So far I have:
Regex Example 2: "Locations (\d{1,2}.?){1,5}"
but I know that is not working. When I remove the "Locations" it picks up the numbers but also picks up the .282 and 002 which I do not want.
Regex Example 2 Current Output:
Row Text Location(new column)
1 msdfmsfmdf Locations 3,5,7 & 9" asdasdas Locations 3
Regex Example 2 Desired Output:
Row Text Location(new column)
1 msdfmsfmdf Locations 3,5,7 & 9" asdasdas 3,5,7,9
Sometimes the "Location" in the text field is in the format which would require Example 1s Regex and sometimes it is in the format requiring example 2s regex so I would need to have the regex searching for both possible formats.
Example 3 Regex in Select Statement:
Select "Primary Key",
"Text",
STRING_AGG(SUBSTR_REGEXPR('([PpSs][Tt][Ss]?\w?\d{2,6})' OR '(\d{1,2}.?){1,5})' in "Text" ),',') as "Location"
FROM Table
Needs to capture both example 1 and 2 location formats using some sort of OR condition in the create column SQL
Regex Example 3 Current Output:
Not working, no output
Regex Example 3 Desired Output:
Row Text Location(new column)
1 msdfmsfmdf Locations 3,5,7 & 9" asdasdas 3,5,7,9
2 msdfmsfmdf PT2222, ST 43434 asdasdas PT2222, ST43434
Other Tools I have access to are SAS and python. Any alternate recommendations to simplify the process are welcome. I did already try in Tableau but same problem with only returning the first match. Aggregating them makes the calculation super slow and very long.
Please help me figure this out. Any help is much appreciated.
Thanks.

Answer 1

For single input string values, following script can be used.
Use of SubStr_RegExpr with Series_Generate_Integer to split string using SQLScript in HANA can be descriptive to understand the use of series_generate function
declare pString nvarchar(5000);
pString := 'msdfmsfmdf PT2222, ST 43434 asdasdas';
select
STRING_AGG(SUBSTR_REGEXPR( '([PpSs][Tt][Ss]?\w?\d{2,6})' IN Replace(pString,' ','') OCCURRENCE NT.Element_Number GROUP 1),',') as "Location"
from
DUMMY as SplitString,
SERIES_GENERATE_INTEGER(1, 0, 10 ) as NT;
Output will return as PT2222,ST43434

Answer 2

Thanks for adding the necessary requirement examples. This makes it a lot easier to work through the problem.
In this case, your requirement is to match multiple strings against multiple patterns and to apply multiple formatting operations on the output.
This cannot be done in a single regular expression in SAP HANA.
Basically, SAP HANA SQL allows two kinds of regex operations:
Match against a pattern and return one occurrence
Match against a pattern and replace one or ALL occurrences of this match
That means for this transformation we basically can try to remove everything that does not match the pattern or loop over the input string and pick out everything that matches.
The problem with the remove-approach (e.g. using SUBSTR_REGEXPR()) is that the matching patterns are not guaranteed to not overlap. That means we could remove matches for other patterns in the process.
Instead, I would use the first approach and try and pick all matches against all pattern and return those.
For that a scalar user-defined function can be created like this:
drop function extract_locators;
create function extract_locators(IN input_text NVARCHAR(1000))
returns location_text NVARCHAR(1000)
as
begin
declare matchers NVARCHAR(100) ARRAY;
declare part_res NVARCHAR(100) := '';
declare full_res NVARCHAR (2000) := '';
declare occn integer;
declare curr_matcher integer;
-- setting up matchers
matchers[1] := '(PT\s*[[:digit:]]+)|(ST\s*[[:digit:]]+)'; -- matches PTxxxx, pt xxxx , St ... , STxxxx
matchers[2] := '(?>\s)[1-9][0-9]*'; -- matches 21, 1, 23, 34
curr_matcher :=0;
-- loop over all matchers
while (:curr_matcher < cardinality(:matchers)) do
curr_matcher := :curr_matcher + 1;
-- loop over all occurrences
occn := 1;
part_res := '';
while (:part_res IS NOT NULL) do
part_res := SUBSTR_REGEXPR(:matchers[:curr_matcher]
FLAG 'i'
IN :input_text
OCCURRENCE :occn);
if (:part_res IS NOT NULL) then
occn := :occn + 1;
full_res := :full_res
|| MAP(LENGTH(:full_res), 0, '', ',')
|| IFNULL(:part_res, '');
else
BREAK;
end if;
end while; -- occurrences
-- if current matcher matched, don't apply the others
if (:full_res !='') then
BREAK;
end if;
end while; -- matchers
-- remove spaces
location_text := replace (:full_res, ' ', '');
end;
With your test data in a table like the following:
drop table loc_data;
create column table loc_data ("CASE" integer primary key,
"INPUT_TEXT" NVARCHAR(2000));
-- PT and ST
insert into loc_data values (1, 'msdfmsfmdf PT2222, ST 43434 asdasdas');
-- Locations
insert into loc_data values (2, 'Locations 1, 2, 35 & 5 lkfaskjdlsaf .282 lkfdsklfjlkdsj 002');
You can now simply run
select
*
, extract_locators("INPUT_TEXT") as location_text
from
loc_data;
To get
1 | msdfmsfmdf PT2222, ST 43434 asdasdas | PT2222,ST43434
2 | Locations 1, 2, 35 & 5 lkfaskjdlsaf .282 lkfdsklfjlkdsj 002 | 1,2,35,5
This approach also allows for keeping the matching rules in a separate table and use a cursor (instead of the array) to loop over them. In addition to that, it keeps the single regular expressions rather small and relatively easy to understand, which is probably the biggest benefit here.
The runtime performance obviously can be an issue, therefore I would probably try and save the results of the operation and only run the function when the data changes.