Reshape table wide to long - sql

I'm trying to reshape a wide data table to long. My current code works but I think it's inefficient and wonder if there's a better way to do this. The original table looks like this:
Location Date1 Date2 Date3 ..... Date 80
A 1-Jan-15 3-Mar-15 7-Apr-15 4-Apr-16
B 3-Jan-15 5-Mar-15 6-Apr-15 3-Apr-16
c 2-Jan-15 7-Mar-15 8-Apr-15 2-Apr-16
And I want to reshape it like this:
Location Date
A 1-Jan-15
A 3-Mar-15
A 7-Apr-15
.
.
A 4-Apr-16
B 3-Jan-15
...
This is the code I used but since there are 80 date variables, I found it inefficient to list all 80 values in the cross apply clause. Is there a better way to get the same result?
select t.Location, Date
from my_table t
cross apply
(
values (1, Date1),
(2, Date2),
(3, Date3),
...
(80, Date80)
) x (Location, Date);

Here is an option that will dynamically unpivot your data with using dynamic sql
Example
Select A.Location
,B.*
From YourTable A
Cross Apply (
Select [Key]
,Value
From OpenJson((Select A.* For JSON Path,Without_Array_Wrapper ))
Where [Key] not in ('Location')
) B
Returns

Related

Unique combination of multiple columns, order doesn't matter

Suppose a table with 3 columns. each row represents a unique combination of each value:
a a a
a a b
a b a
b b a
b b c
c c a
...
however, what I want is,
aab = baa = aba
cca = cac = acc
...
Finally, I want to get these values in a CSV format as a combination for each value like the image that I attached.
Thanks for your help!
Below is the query to generate my problem, please take a look!
--=======================================
--populate test data
--=======================================
drop table if exists #t0
;
with
cte_tally as
(
select row_number() over (order by (select 1)) as n
from sys.all_columns
)
select
char(n) as alpha
into #t0
from
cte_tally
where
(n > 64 and n < 91) or
(n > 96 and n < 123);
drop table if exists #t1
select distinct upper(alpha) alpha into #t1 from #t0
drop table if exists #t2
select
a.alpha c1
, b.alpha c2
, c.alpha c3
, row_number()over(order by (select 1)) row_num
into #t2
from #t1 a
join #t1 b on 1=1
join #t1 c on 1=1
drop table if exists #t3
select *
into #t3
from (
select *
from #t2
) p
unpivot
(cvalue for c in (c1,c2,c3)
) unpvt
select
row_num
, c
, cvalue
from #t3
order by 1,2
--=======================================
--these three rows should be treated equally
--=======================================
select *
from #t2
where concat(c1,c2,c3) in ('ABA','AAB', 'BAA')
--=======================================
--what i've tried...
--row count is actually correct, but the problem is that it ommits where there're any duplicate alphabet.
--=======================================
select
distinct
stuff((
select
distinct
'.' + cvalue
from #t3 a
where a.row_num = h.row_num
for xml path('')
),1,1,'') as comb
from #t3 h
As pointed out in the comments, you can unpivot the values, sort them in the right order and reaggregate them into a single row. Then you can group the original rows by those new values.
SELECT *
FROM #t2
CROSS APPLY (
SELECT a = MIN(val), b = MIN(CASE WHEN rn = 2 THEN val), c = MAX(val)
FROM (
SELECT *, rn = ROW_NUMBER() OVER (ORDER BY val)
FROM (VALUES (c1),(c2),(c3) ) v3(val)
) v2
) v
GROUP BY v.a, v.b, v.c;
Really, what you should perhaps do, is ensure that the values are in the correct order in the first place:
ALTER TABLE #t2
ADD CONSTRAINT t2_ValuesOrder
CHECK (c1 <= c2 AND c2 <= c3);
Would be curious why, sure you have a reason. Might suggest having a lookup table, holding all associated keys to a "Mapping Table". You might optimize some of this as you implement it. First create one table for holding the "Next/New Key" (this is where the 1, 2, 3...) come from. You get a new "New Key" after each batch of records you bulk insert into your "Mapping Table". The "Mapping Table" holds the combination of the key values, one row for each combinations along with your "New Key" Should get a table looking something like:
A, B, C, 1
A, C, B, 1
B, A, C, 1
...
X, Y, Z, 2
X, Z, Y, 2
If you can update your source table to hold a column for your "Mapping Key" (the 1,2,3) then you just look up from the mapping table where (c1=a, c2=a, c3=b) order for this look-up shouldn't matter. One suggestion would create a composite unique key using c1,c2,c3 on your mapping table. Then to get your records just look up the "mapping key value" from the mapping table and then query for records matching the mapping key value. Or, if you don't do a pre-lookup to get the mapping key you should be able to do a self-join using the mapping key value...
If you want them in a CSV format:
select distinct v.cs
from #t2 t2 cross apply
(select string_agg(c order by c desc, ',') as cs
from (values (t2.c1), (t2.c2), (t2.c3)
) v(c)
) v;
It seems to me that what you need is some form of masking*. Take this fiddle:
http://sqlfiddle.com/#!18/fc67f/8
where I have created a mapping table that contains all of the possible values and paired that with increasing orders of 10. Doing a cross join on that map table, concatenating the values, adding the masks and grouping on the total will yield you all the unique combinations.
Here is the code from the fiddle:
CREATE TABLE maps (
val varchar(1),
num int
);
INSERT INTO maps (val, num) VALUES ('a', 1), ('b', 10), ('c', 100);
SELECT mask, max(vals) as val
FROM (
SELECT concat(m1.val, m2.val, m3.val) as vals,
m1.num + m2.num + m3.num as mask
FROM maps m1
CROSS JOIN maps m2
CROSS JOIN maps m3
) q GROUP BY mask
Using these values of 10 will ensure that mask contains the count for each value, one for each place column in the resulting number, and then you can group on it to get the unique(ish) strings.
I don't know what your data looks like, and if you have more than 10 possible values then you will have to use some other base than 10, but the theory should still apply. I didn't write code to extract the columns from the value table into the mapping table, but I'm sure you can do that.
*actually, I think the term I was looking for was flag.

Is there a method to simply transpose a table in SQL. This table contains Numeric and Varchar values

I would like to know how to transpose very simply a table in SQL. There is no sum or calculations to do.
This table contains Numeric and Varchar values.
Meaning, I have a table of 2 rows x 195 columns. I would like to have the same table with 195 rows x 2 columns (maybe 3 columns)
time_index
legal_entity_code
cohort
...
...
0
AAA
50
...
...
1
BBB
55
...
...
TO
Element
time_index_0
time_index_1
legal_entity_code
AAA
BBB
cohort
50
55
...
...
...
...
...
...
I have created this piece of code for testing
SELECT time_index, ValueT, FieldName
FROM (select legal_entity_code, cohort, time_index from ifrs17.output_bba where id in (1349392,1349034)) as T
UNPIVOT
(
ValueT
FOR FieldName in ([legal_entity_code],[cohort])
) as P
but I receive this error message :
The type of column "cohort" conflicts with the type of other columns specified in the UNPIVOT list.
I would recommend using apply for this. I don't fully follow the specified results because the query and the sample data are inconsistent in their naming.
I'm pretty sure you want:
select o.time_index, v.*
from ifrs17.output_bba o cross apply
(values ('Name1', o.name1),
('Value1', convert(varchar(max), o.value1)),
('Name2', o.name2)
) v(name, value)
where o.id in (1349392,1349034);
Gordon's approach is correct and certainly more performant. +1
However, if you want to dynamically unpivot 195 columns without having to list them all, consider the following:
Note: if not 2016+ ... there is a similar XML approach.
Example or dbFiddle
Select Element = [Key]
,Time_Index_0 = max(case when time_index=0 then value end)
,Time_Index_1 = max(case when time_index=1 then value end)
From (
Select [time_index]
,B.*
From YourTable A
Cross Apply (
Select [Key]
,Value
From OpenJson( (Select A.* For JSON Path,Without_Array_Wrapper ) )
Where [Key] not in ('time_index')
) B
) A
Group By [Key]
Returns
Element Time_Index_0 Time_Index_1
cohort 50 55
legal_entity_code AAA BBB

UNPIVOT Holiday Hours

I have a table, that keeps track of store holiday hours:
LOCATION_ID DATE1 TIMES1 DATE2 TIMES2
123456 2020-12-12 10:00AM-09:00PM 2020-12-19 10:00AM-09:00PM
This is a highly oversimplified table. There's about 30 columns horzontially consisting of store operating hours by date - It continues (DATE3, TIMES3, DATE4, TIMES4, etc).
I need to unpivot the values vertically, ensuring the date and time values are on the same record.
(NOTE: Once I figure out to structure the UNPIVOT expression properly, I will use Dynamic SQL on my own to pivot the column names)
Desired Outcome:
LOCATION_ID DATE TIME
123456 2020-12-12 10:00AM-09:00PM
123456 2020-12-19 10:00AM-09:00PM
I tried using UNPIVOT, but I'm stuck. Any ideas?
SAMPLE DATA:
CREATE TABLE #HOURS (LOCATION_ID int, DATE1 varchar(255), TIMES1 varchar(255), DATE2
varchar(255), TIMES2 varchar(255));
INSERT INTO #HOURS VALUES ('123456', '2020-12-12', '10:00AM-09:00PM','2020-12-19','10:00AM-09:00PM' )
Code that I tried:
SELECT *
FROM (SELECT location_id,
[date1],
[times1],
[date2]
FROM #hours) AS cp
UNPIVOT ( pivotvalues
FOR pivvalues IN ([Date1],
[date2],
[times1]) ) AS up1
Gordon is 100% correct (+1).
However, if you are looking for a dynamic approach WITHOUT having to use Dynamic SQL, consider the following.
Example
Select Location_ID
,Date = max(case when [Item] like 'DATE%' then Value end)
,Time = max(case when [Item] like 'TIME%' then Value end)
From (
select A.Location_ID
,Grp = replace(replace([Item],'DATE',''),'TIMES','')
,B.*
from #hours A
Cross Apply [dbo].[tvf-XML-Unpivot-Row]( (Select A.* for XML RAW) ) B
Where [Item] not in ('LOCATION_ID')
) A
Group By Location_ID,Grp
Returns
Location_ID Date Time
123456 2020-12-12 10:00AM-09:00PM
123456 2020-12-19 10:00AM-09:00PM
The Table-Valued Function if Interested
CREATE FUNCTION [dbo].[tvf-XML-UnPivot-Row](#XML xml)
Returns Table
As
Return (
Select Item = xAttr.value('local-name(.)', 'varchar(100)')
,Value = xAttr.value('.','varchar(max)')
From #XML.nodes('//#*') xNode(xAttr)
)
Don't use unpivot. Use apply:
select h.location_id, v.date, v.time
from #hours h cross apply
(values (h.date1, h.times1), (h.date2, h.times2)
) v(date, time);
unpivot is non-standard syntax that does exactly one thing. APPLY is the SQL Server implementation of lateral joins. This is a very powerful join type -- using it for unpivoting is a good way to start learning the syntax.

Semivariance of variable

I'm new with sql and I struggle with such a problem. Let's suppose I have a table like this:
Date Value
2014-01-01 1248.56
2014-01-02 1247.24
2014-01-03 1245.82
2014-01-04 1252.07
...
All I want to do is count semivariance of variable 'Value'.
Semivariance only takes into account those records which are less than the average of the sample. So basically it is just a transformartion of simply variance.
Any help would be appreciated!
you can try something like this
SELECT COUNT(*)
FROM
Table1
WHERE Value < (SELECT AVG(Value) FROM Table1)
If you need avg value then you can use such code.
CREATE TABLE #test
(
date DATE,
value NUMERIC(10,2)
)
INSERT INTO #test VALUES ('2014-01-01' , 1248.56 ),
('2014-01-02' , 1247.24),
('2014-01-03' , 1245.82),
('2014-01-04' , 1252.07);
SELECT * FROM #test a
CROSS JOIN (SELECT AVG(value) avg_value FROM #test) b
WHERE a.value < b.avg_value

Unpivoting multiple columns

I have a table in SQL Server 2014 called anotes with the following data
and I want to add this data into another table named final as
ID Notes NoteDate
With text1, text2, text3, text4 going into the Notes column in the final table and Notedate1,notedate2,notedate3,notedate4 going into Notedate column.
I tried unpivoting the data with notes first as:
select createdid, temp
from (select createdid,text1,text2,text3,text4 from anotes) p
unpivot
(temp for note in(text1,text2,text3,text4)) as unpvt
order by createdid
Which gave me proper results:
and then for the dates part I used another unpivot query:
select createdid,temp2
from (select createdid,notedate1,notedate2,notedate3,notedate4 from anotes) p
unpivot (temp2 for notedate in(notedate1,notedate2,notedate3,notedate4)) as unpvt2
which also gives me proper results:
Now I want to add this data into my final table.
and I tried the following query and it results into a cross join :(
select a.createdid, a.temp, b.temp2
from (select createdid, temp
from (select createdid,text1,text2,text3,text4 from anotes) p
unpivot
(temp for note in(text1,text2,text3,text4)) as unpvt) a inner join (select createdid,temp2
from (select createdid,notedate1,notedate2,notedate3,notedate4 from anotes) p
unpivot (temp2 for notedate in(notedate1,notedate2,notedate3,notedate4)) as unpvt) b on a.createdid=b.createdid
The output is as follows:
Is there any way where I can unpivot both the columns at the same time?
Or use two select queries to add that data into my final table?
Thanks in advance!
I would say the most concise, and probably most efficient way to unpivot multiple columns is to use CROSS APPLY along with a table valued constructor:
SELECT t.CreatedID, upvt.Text, upvt.NoteDate
FROM anotes t
CROSS APPLY
(VALUES
(Text1, NoteDate1),
(Text2, NoteDate2),
(Text3, NoteDate3),
(Text4, NoteDate4),
(Text5, NoteDate5),
(Text6, NoteDate6),
(Text7, NoteDate7)
) upvt (Text, NoteDate);
Simplified Example on SQL Fiddle
ADDENDUM
I find the concept quite a hard one to explain, but I'll try. A table valued constuctor is simply a way of defining a table on the fly, so
SELECT *
FROM (VALUES (1, 1), (2, 2)) t (a, b);
Will Create a table with Alias t with data:
a b
------
1 1
2 2
So when you use it inside the APPLY you have access to all the outer columns, so it is just a matter of defining your constructed tables with the correct pairs of values (i.e. text1 with date1).
Used the link above mentioned by #AHiggins
Following is my final query!
select createdid,temp,temp2
from (select createdid,text1,text2,text3,text4,text5,text6,text7,notedate1,notedate2,notedate3,notedate4,notedate5,notedate6,notedate7 from anotes) main
unpivot
(temp for notes in(text1,text2,text3,text4,text5,text6,text7)) notes
unpivot (temp2 for notedate in(notedate1,notedate2,notedate3,notedate4,notedate5,notedate6,notedate7)) Dates
where RIGHT(notes,1)=RIGHT(notedate,1)
Treat each query as a table and join them together based on the createdid and the fieldid (the numeric part of the field name).
select x.createdid, x.textValue, y.dateValue
from
(
select createdid, substring(note, 5, len(note)) fieldId, textValue
from (select createdid,text1,text2,text3,text4 from anotes) p
unpivot
(textValue for note in(text1,text2,text3,text4)) as unpvt
)x
join
(
select createdid, substring(notedate, 9, len(notedate)) fieldId, dateValue
from (select createdid,notedate1,notedate2,notedate3,notedate4 from anotes) p
unpivot (dateValue for notedate in(notedate1,notedate2,notedate3,notedate4)) as unpvt2
) y on x.fieldId = y.fieldId and x.createdid = y.createdid
order by x.createdid, x.fieldId
The other answer given won't work if you have too many columns and the rightmost number of the field name is duplicated (e.g. text1 and text11).