Destruct column into multiple columns with Regex on Azure? - sql

Question

I've encountered a problem which I think can only be solved by regex functions only.
Sadly the support for regex based operations seems to be very poor on Microsoft side.
(Forgive me if I'm wrong, that's the first case when I've to use this platform)
What I've:
~400 million records in MS Azure SQL DB
encoded functionality what I've to decode into multiple columns (later I'll join metadata by these columns)
What I need:
a regex based function which can parse out the data (the output will be written into de1-6 columns)
The encoded column (what needs to be decoded) looks like this:
|encoded_val |
|-------------|
|PIT273OF_21 |
|PT273CT_21 |
|LT171CT2_31 |
|TV273JM_11 |
|TV273CND_13 |
|FIT865_11_CLC|
|AT865_104 |
|E865MFSP01 |
|LIT273CU_61 |
|E273_RH |
|E273CU_GTH |
|VSZ171JM_31 |
|E171CU_GTH |
|IT171RC_11 |
|WY171CU_61N |
|FV864_11 |
I need to decode this column with a regexp and create multiple columns with
| encoded | de1 | de2 | de3 | de4 | de5 | de6 |
|---------------|-----|-----|------|------|------|------|
| PIT273OF_21 | PIT | 273 | OF | NULL | 21 | NULL |
| PT273CT_21 | PT | 273 | CT | NULL | 21 | NULL |
| LT171CT2_31 | LT | 171 | CT | 2 | 31 | NULL |
| TV273JM_11 | TV | 273 | JM | NULL | 11 | NULL |
| TV273CND_13 | TV | 273 | CND | NULL | 13 | NULL |
| FIT865_11_CLC | FIT | 865 | NULL | NULL | 11 | CLC |
| AT865_104 | AT | 865 | NULL | NULL | 104 | NULL |
| E865MFSP01 | E | 865 | MFSP | 01 | NULL | NULL |
| LIT273CU_61 | LIT | 273 | CU | NULL | 61 | NULL |
| E273_RH | E | 273 | NULL | NULL | NULL | RH |
| E273CU_GTH | E | 273 | CU | NULL | NULL | GTH |
| VSZ171JM_31 | VSZ | 171 | JM | NULL | 31 | NULL |
| E171CU_GTH | E | 171 | CU | NULL | NULL | GTH |
| IT171RC_11 | IT | 171 | RC | NULL | 11 | NULL |
| WY171CU_61N | WY | 171 | CU | NULL | 61 | N |
| FV864_11 | FV | 864 | NULL | NULL | 11 | NULL |
Problems
the format is not fixed
the length of the blocks can vary
there can be missing values
... but let's say, I've some magic regex pattern, what can parse useful data from any string
What I've tried:
compute columns - they seem to only construct new columns from existing ones, not to destruct existing columns into new columns in clever way with pattern matching
user defined function - I did not figured out yet, how they could help, but they seem promising
overcomplicated functions what are impossible to understand/maintain, their execution time seems to be unacceptable
update [bd].[table]
set unitid = cast(LEFT(SUBSTRING([colname],PATINDEX('%[0-9]%',[colname]),100),PATINDEX('%[^0-9]%',SUBSTRING([colname],PATINDEX('%[0-9]%',[colname]),100) + '*') -1) as smallint);
Question:
What is the appropriate way to do this fast (on 400M records)?

Answer 1

CREATE TABLE [dbo].[Sample](
[id] [int] IDENTITY(1,1) NOT NULL,
[encoded] [varchar](60) NOT NULL,
CONSTRAINT [PK_Sample] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
insert into [Sample] values ('FIT865_11_CLC')
insert into [Sample] values ('PIT273OF_21')
declare #count int
declare #id int
declare #content varchar(50)
declare #table table
(
id int,
String varchar(256)
)
set #count = (select count(1) from sample)
set #id = 0
while(#count > 0)
begin
if (#id > 0)
select top(1) #id = id, #content = encoded from Sample where id > #id order by id
else
select top(1) #id = id, #content = encoded from Sample order by id
declare #len int = len(#content)
declare #position int = 1
declare #output varchar(256) = ''
declare #currentChar varchar(1)
declare #lastChar varchar(1)
while(#len > 0)
begin
set #currentChar = (select substring(#content, #position, 1))
if(#position = 1)
begin
set #lastChar = #currentChar
end
if(ISNUMERIC(#currentChar)<> ISNUMERIC(#lastChar)) --alpha
set #output = #output + ',' + #currentChar
else
set #output = #output + #currentChar
set #len = #len -1
set #position = #position + 1
set #lastChar = #currentChar
end
set #count = #count - 1
set #output = (select replace (#output,'_',','))
insert into #table values (#id, #output)
end
SELECT DISTINCT B.*
FROM #table A
CROSS APPLY (
SELECT
JSON_VALUE(J,'$[0]') AS de0
,JSON_VALUE(J,'$[1]') AS de1
,JSON_VALUE(J,'$[2]') AS de2
,JSON_VALUE(J,'$[3]') AS de3
,JSON_VALUE(J,'$[4]') AS de4
,JSON_VALUE(J,'$[5]') AS de5
,JSON_VALUE(J,'$[6]') AS de6
,JSON_VALUE(J,'$[7]') AS de7
FROM (VALUES ('["'+replace(replace(String,'"','\"'),',','","')+'"]')) A(J)
) B