"Error" value on name and description creating strange functionality - sql

Question

I got a raw data file with its content looking like this:
MSN_Check,Text,25,MSN check
0,Text,1,(Result)
HWIMPL,Text,10,HWIMPL version reading
007F,Text,6,(Measure)
1,Text,1,(Result)
VHW,Text,10,FMT hardware version
494131383346,Text,10,(Measure)
0,Text,1,(Result)
TOTAL_VER,Text,25,Total version reading
313031303130,Text,6,(Measure)
1,Text,1,(Result)
CAL_MCU,Text,25,Total version reading
05,Text,6,(Measure)
Error,Text,25,Error
9.8499985089315E-07,Numeric,Float 3.3,(Measure)
CAL_EEPROM,Text,25,Total version reading
05,Numeric,Float 3.3,(Measure)
1,Text,1,(Result)
And I needed to extract and store in variables the name, example MSN_Check ,the description, example MSN check its result for example 0 and its measure , for example 007F but in some places I have results only or measures only so just spliting them wouldn't have helped.So my idea was:
First of all I created a template table named dbo.template that looks like this:
Name TestDescription Measure Result ID
----------------------------------------------
MSN_Check MSN check 0 1 1
HWIMPL HWIMPL version reading 1 1 2
VHW FMT hardware version 1 1 3
TOTAL_VER Total version reading 1 1 4
CAL_MCU Total version reading 1 0 5
Error Error 1 0 6
CAL_EEPROM Total version reading 1 1 7
In this table we have the name,description,if_measure(meaning 1 if we have a measure or 0 if we dont) and the if_result.And I made a query looking like this:
DECLARE #crlf AS CHAR(2) = CHAR(13) + CHAR(10)
declare #testname varchar(max),#testDescription varchar(max), #if_measure char(1), #if_result char(1), #row int = '1', #id int
set #LogEntry = (SELECT REPLACE(#LogEntry,#crlf,','))
declare #name varchar(max),#description varchar(MAX), #measure varchar(20), #result char(1)
declare #Output table(OutTestName varchar(max),OUTTestDescription varchar(max), OutMeasure varchar(50), OutResult varchar(50))
declare #maximum int = (select MAX(ID) from dbo.template_FMT)
declare #LogEntry1 as nvarchar(max)
declare #LogEntry2 as nvarchar(max)
while #row <= #maximum
BEGIN
set #name = null
set #description = null
set #measure = null
set #result = null
set #testname = (select Name from dbo.template_FMT where ID = #row)
set #testDescription = (select TestDescription from dbo.template_FMT where ID = #row)
set #if_measure = (select Measure from dbo.template_FMT where ID = #row)
set #if_result = (select Result from dbo.template_FMT where ID = #row)
set #id = (select ID from dbo.Split(#LogEntry, ',') where Data = #testname)
SELECT #LogEntry1 = Name FROM dbo.template_FMT where id = #row
set #name = #LogEntry1
SELECT #LogEntry2 = TestDescription FROM dbo.template_FMT where id = #row
set #description = #LogEntry2
if #if_measure > 0 and #if_result > 0
begin
set #measure = (select Data from dbo.Split(#LogEntry, ',') where ID = #id+4)
set #result = (select Data from dbo.Split(#LogEntry, ',') where ID = #id+8)
insert into #Output (OutTestName, OUTTestDescription, OutMeasure, OutResult) Values(#name,#description, #measure, #result)
end
if #if_measure > 0 and #if_result = 0
begin
set #measure = (select Data from dbo.Split(#LogEntry, ',') where ID = #id+4)
set #result = null
insert into #Output (OutTestName, OUTTestDescription, OutMeasure, OutResult) Values(#name,#description, #measure, #result)
end
if #if_measure = 0 and #if_result > 0
begin
set #measure = null
set #result = (select Data from dbo.Split(#LogEntry, ',') where ID = #id+4)
insert into #Output (OutTestName, OUTTestDescription, OutMeasure, OutResult) Values(#name,#description, #measure, #result)
end
set #row = #row + 1
END
select * from #Output
And it worked! but the only problem I have is where I have the row with the name Error with the description Error,it would return the last remembered value so instead of having
CAL_MCU Total version reading 05 NULL
Error Error 9.8499985089315E-07 NULL
CAL_EEPROM Total version reading 05 1
I get:
CAL_MCU Total version reading 05 NULL
Error Error 05 NULL
CAL_EEPROM Total version reading 05 1
And I would like to store the Error cant find Result with ID into variables if any of you have any suggestions :)
P.S. I think it has something to do with the fact that the name and description have the same name (Error)

Answer 1

I believe that your problem can be solved without the need for while loops and string splitting functions. I recommend using the OPENROWSET function to read your raw data file as a standard table. You can then use standard T-SQL query to format the result into the desired output.
The first step is to ensure that ad-hoc queries is enable on your server this can be accomplished by executing the following command.
sp_configure 'show advanced options', 1;
RECONFIGURE;
GO
sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
The next step is to define a format file for your text file. This will help SQL Server understand the text file structure when loading the raw data. Based on the supplied sample data your format file should look as follow:
10.0
4
1 SQLCHAR 0 100 "," 1 Col1 SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 0 100 "," 2 Col2 SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 0 100 "," 3 Col3 SQL_Latin1_General_CP1_CI_AS
4 SQLCHAR 0 100 "\r\n" 4 Col4 SQL_Latin1_General_CP1_CI_AS
I have uploaded the format file and example raw data file I have used to test the example at the following links:
http://www.filedropper.com/format
http://www.filedropper.com/rawdatafile
The final step is to run the OPENROWSET query to load the file data and transform the data to the desired output. If you are using SQL Server 2008 r2 then the following query should work:
-- 2008 R2 Version
WITH CTE_VariableRawData
AS
(
SELECT ROW_NUMBER() OVER(ORDER BY (SELECT 0)) AS ID
,[RawData].Col1 AS [VariableOrMeasure]
,(
CASE [RawData].Col4
WHEN '(Result)' THEN 0
WHEN '(Measure)' THEN 0
ELSE 1
END
) AS IsVariable
,(
CASE [RawData].Col4
WHEN '(Result)' THEN 1
ELSE 0
END
) AS IsResult
,(
CASE [RawData].Col4
WHEN '(Measure)' THEN 1
ELSE 0
END
) AS IsMeasure
,[RawData].Col4 AS [Description]
FROM OPENROWSET(BULK N'C:\temp\raw_data_file.txt', FORMATFILE = 'c:\temp\format.txt') AS [RawData]
)
,
CTE_RawDataByVariableID
AS
(
SELECT ID
,(
SELECT SUM([IsVariable])
FROM CTE_VariableRawData RunningTotal
WHERE RunningTotal.ID <= CTE_VariableRawData.ID
) AS VariableID
,[VariableOrMeasure]
,[IsVariable]
,[IsResult]
,[IsMeasure]
,[Description]
FROM CTE_VariableRawData
)
SELECT VariableID
,MAX(
CASE [IsVariable]
WHEN 1 THEN [VariableOrMeasure]
ELSE NULL
END
) AS [Variable]
,MAX(
CASE [IsVariable]
WHEN 1 THEN [Description]
ELSE NULL
END
) AS [Description]
,MAX(
CASE [IsMeasure]
WHEN 1 THEN [VariableOrMeasure]
ELSE NULL
END
) AS [Measure]
,MAX(
CASE [IsResult]
WHEN 1 THEN [VariableOrMeasure]
ELSE NULL
END
) AS [Result]
FROM CTE_RawDataByVariableID
GROUP BY VariableID
ORDER BY VariableID
If you are using SQL Server 2012 or later then the following query will be a bit more optimal:
WITH CTE_VariableRawData
AS
(
SELECT ROW_NUMBER() OVER(ORDER BY (SELECT 0)) AS ID
,[RawData].Col1 AS [VariableOrMeasure]
,(
CASE [RawData].Col4
WHEN '(Result)' THEN 0
WHEN '(Measure)' THEN 0
ELSE 1
END
) AS IsVariable
,(
CASE [RawData].Col4
WHEN '(Result)' THEN 1
ELSE 0
END
) AS IsResult
,(
CASE [RawData].Col4
WHEN '(Measure)' THEN 1
ELSE 0
END
) AS IsMeasure
,[RawData].Col4 AS [Description]
FROM OPENROWSET(BULK N'C:\temp\raw_data_file.txt', FORMATFILE = 'c:\temp\format.txt') AS [RawData]
)
,
CTE_RawDataByVariableID
AS
(
SELECT ID
,SUM([IsVariable]) OVER (ORDER BY ID) AS VariableID
,[VariableOrMeasure]
,[IsVariable]
,[IsResult]
,[IsMeasure]
,[Description]
FROM CTE_VariableRawData
)
SELECT VariableID
,MAX(
CASE [IsVariable]
WHEN 1 THEN [VariableOrMeasure]
ELSE NULL
END
) AS [Variable]
,MAX(
CASE [IsVariable]
WHEN 1 THEN [Description]
ELSE NULL
END
) AS [Description]
,MAX(
CASE [IsMeasure]
WHEN 1 THEN [VariableOrMeasure]
ELSE NULL
END
) AS [Measure]
,MAX(
CASE [IsResult]
WHEN 1 THEN [VariableOrMeasure]
ELSE NULL
END
) AS [Result]
FROM CTE_RawDataByVariableID
GROUP BY VariableID
ORDER BY VariableID;
Note that in both queries you will have to change the location of your raw data file and format file to the desired location within the OPENROWSET(BULK N'C:\temp\raw_data_file.txt', FORMATFILE = 'c:\temp\format.txt') call.