Azure Data Warehouse Generate serial faster query - sql

Question

Environment is Azure DW
I have a raw table like below;
ID Start End Action date
1 10 15 Processed 25-10-2019
2 55 105 In-Progress 21-10-2019
.....
I need to expand/transform the Start and End column so that they become serial number;
SN Action date
10 Processed 25-10-2019
11 Processed 25-10-2019
12 Processed 25-10-2019
13 Processed 25-10-2019
14 Processed 25-10-2019
.....
Azure Data Warehouse doesn't support recursive CTE or Cursor. So, have tried a while loop,
create table #temp_output (SerialNumber int not null, startSerialNumber int not null, endSerialNumber int not null);
insert into #temp_output select startSerialNumber, startSerialNumber, endSerialNumber from dbo.raw
declare #rowcount int, #cnt int, #start int, #end int
set #cnt = 1
set #rowcount = (select count(*) from dbo.raw)
while #cnt <= #rowcount
begin
select top (#cnt) #start = startSerialNumber from dbo.raw
select top (#cnt) #end = endSerialNumber from dbo.raw
while #start <= #end
begin
insert #temp_output
select max(SerialNumber) + 1,
startSerialNumber,
endSerialNumber
from #temp_output group by startSerialNumber, endSerialNumber having max(SerialNumber) < endSerialNumber
set #start = #start + 1
end
set #cnt = #cnt + 1
end
select SerialNumber, startSerialNumber, endSerialNumber from #temp_output_delta order by SerialNumber
However this takes ages (6 hrs, when I cancelled the query) as the raw table has 50 million rows.
Need a better way to do this.
Updated information 31-10-2019
Distribution for the source table is hash. 500 DWu .
60 million row in source table.
Average difference between start and end 3000.
The start can be 2million as well.
No index on main table.
Column count 15
Clustered columnstore index on raw table.

Answer 1

Your sample is incomplete but you don't need a loop. You can join it to a tally table using BETWEEN
If you have a tally table (which is a table that simply has number from 1 to... 1 million in it)
SELECT T.TallyNumber As SN, E.Action, E.Date
FROM YourTable E
INNER JOIN TallyTable As T
ON T.TallyNumber BETWEEN E.Start AND E.End
Since you are loading this into a new table, you should use CTAS
CREATE TABLE [dbo].[NewTable]
WITH
(
DISTRIBUTION = HASH([Start])
,CLUSTERED COLUMNSTORE INDEX
)
AS
SELECT T.TallyNumber As SN, E.Action, E.Date
FROM YourTable E
INNER JOIN TallyTable As T
ON T.TallyNumber BETWEEN E.[Start] AND E.[End];
Note there is a whole lot of design around DISTRIBUTION. You need to get this right for performance. The above statement is just an example. You should probably use a different hash.
You need to get the distribution of the two source tables as well as the distribution of the target table right for good performance.