Related
I'm trying to convert a piece of SQL code to HiveQL, and it's not working as expected.
Please find below the code snippet in SQL that I'm attempting to convert:
SQL Code:
UPDATE
C
SET
C.prod_l = P.prod_l, C.numprod = P.numprod, C.prod_cng = P.prod_cng
FROM
[cnc].dbo.[c_cnc_analysis] C
LEFT JOIN
(
SELECT
X.*,
Len(prod_l) - Len(Replace(prod_l, ' ~ ', ' ')) + 1 AS NumProd,
CASE
WHEN
Len(prod_l) - Len(Replace(prod_l, ' ~ ', ' ')) + 1 = 1
THEN
0
ELSE
1
END
AS PROD_CNG
FROM
(
SELECT DISTINCT
ST2.uitid,
Substring((
SELECT
' ~ ' + ST1.product_id AS [text()]
FROM
(
SELECT
[uitid],
[product_id]
FROM
dbo.[c_cnc_dedup_bse]
GROUP BY
[uitid],
[product_id]
)
ST1
WHERE
ST1.uitid = ST2.uitid
ORDER BY
ST1.uitid FOR xml path ('')), 4, 1000 ) [PROD_L]
FROM
(
SELECT
[uitid],
[product_id]
FROM
dbo.[c_cnc_dedup_bse]
GROUP BY
[uitid],
[product_id]
)
ST2
)
X
)
P
ON C.uitid = P.uitid;
Converted HIVE Query:
create
or replace view prd_temp as
SELECT
`UITID`,
`PRODUCT_ID`
FROM
`C_CNC_DEDUP_BSE`
GROUP BY
`UITID`,
`PRODUCT_ID`;
create
or replace view prd_temp2 as
SELECT
`UITID`,
`PRODUCT_ID`
FROM
`C_CNC_DEDUP_BSE`
GROUP BY
`UITID`,
`PRODUCT_ID`;
create
or replace view prd_temp3 as
SELECT
st1.`uitid`,
concat(' ~ ', st1.`PRODUCT_ID`) AS `text()`
FROM
prd_temp ST1
left join
prd_temp2 st2
on ST1.`UITID` = ST2.`UITID`
where
st1.`UITID` = st2.`UITID`
ORDER BY
ST1.`UITID`;
create
or replace view prd_temp4 as
SELECT
st1.`uitid`,
concat_ws('''', `text()`)
FROM
prd_temp3 ST1
ORDER BY
ST1.`UITID`;
create
or replace view st2 as
SELECT DISTINCT
`UITID`,
SUBSTRING(`_c1` , 4, 1000) as `PROD_L`
FROM
prd_temp4;
create
or replace view x as
SELECT
*,
LENGTH(PROD_L) - LENGTH(REPLACE(PROD_L, ' ~ ', ' ')) + 1 as NumProd,
CASE
WHEN
LENGTH(PROD_L) - LENGTH(REPLACE(PROD_L, ' ~ ', ' ')) + 1 = 1
then
0
ELSE
1
END
as PROD_CNG
from
ST2;
create table C_CNC_ANALYSIS1 as
select
c.*,
P.numprod as numprod,
p.prod_cng as prod_cng,
p.prod_l as prod_l
from
`C_CNC_ANALYSIS` C
LEFT JOIN
X P
ON C.UITID = P.UITID ;
SELECT
*
from
c_cnc_analysis1 limit 100;
Appreciate all the help with this. I think the code converted for the XML path is not working in HIVE, since I'm getting multiple UITIDs (key) and the information in separate rows rather than just one single record per UITID.
Thank You,
Viswanath Sitaraman
I have table structure like
select catalog_item_id,metal_type,metal_color
from catalog_item_castings
where catalog_Item_Id =465173
It returns output as:
And I want output as:
And I want to insert this data into new temp table in SQL Server.
Thanks in advance.
Conditional aggregation is an option:
SELECT
catalog_item_id,
MAX(CASE WHEN rn % 3 = 1 THEN CONCAT(metal_type, '/', metal_color) END) AS Casting_1,
MAX(CASE WHEN rn % 3 = 2 THEN CONCAT(metal_type, '/', metal_color) END) AS Casting_2,
MAX(CASE WHEN rn % 3 = 0 THEN CONCAT(metal_type, '/', metal_color) END) AS Casting_3
FROM (
SELECT
catalog_item_id, metal_type, metal_color, ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS rn
FROM (VALUES
(465173, 'na', 'METALCOLOR'),
(465173, 'na', 'METAL-001'),
(465173, 'na', 'na')
) catalog_item_castings (catalog_item_id, metal_type, metal_color)
WHERE catalog_Item_Id = 465173
) t
GROUP BY catalog_item_id
-- or if you have more than three rows per [catalog_item_id]
-- GROUP BY catalog_item_id, (rn - 1) / 3
Result:
catalog_item_id Casting_1 Casting_2 Casting_3
-------------------------------------------------
465173 na/METALCOLOR na/METAL-001 na/na
You can use Conditional Aggregation within a Dynamic Pivot Statement in order to include all distinct combinations of the columns [metal_type] and [metal_color], even different values for combinations are inserted in the future :
DECLARE #cols AS NVARCHAR(MAX), #query AS NVARCHAR(MAX)
SELECT #cols = (SELECT STRING_AGG(CONCAT('MAX(CASE WHEN [dr]=',dr,
' THEN CONCAT([metal_type],''/'',[metal_color]) END) AS [Casting_',dr,']'),',')
WITHIN GROUP ( ORDER BY dr )
FROM
(
SELECT DISTINCT
DENSE_RANK() OVER
(PARTITION BY [catalog_item_id]
ORDER BY CONCAT([metal_type],[metal_color])) AS dr
FROM [catalog_item_castings] ) c);
SET #query =
'SELECT [catalog_item_id],'+ #cols +
' FROM
(
SELECT *, DENSE_RANK() OVER
( PARTITION BY [catalog_item_id]
ORDER BY CONCAT([metal_type], [metal_color]) ) AS dr
FROM [catalog_item_castings]
) c
GROUP BY [catalog_item_id]';
EXEC sp_executesql #query;
Demo
I have a table where every ID has one or more places, and each place comes with a count. Places can be repeated within IDs. It is stored in rows like so:
ID ColumnName DataValue
1 place1 ABC
1 count1 5
2 place1 BEC
2 count1 12
2 place2 CDE
2 count2 6
2 place3 BEC
2 count3 9
3 place1 BBC
3 count1 5
3 place2 BBC
3 count2 4
Ultimately, I want a table where every possible place name is its own column, and the count per place per ID is summed up, like so:
ID ABC BEC CDE BBC
1 5 0 0 0
2 0 21 6 0
3 0 0 0 9
I don't know the best way to go about this. There are around 50 different possible place names, so specifically listing them out in a query isn't ideal. I know I ultimately have to pivot the data, but I don't know if I should do it before or after I sum up the counts. And whether it's before or after, I haven't been able to figure out how to go about summing it up.
Any ideas/help would be greatly appreciated. At this point, I'm having a hard time finding where to even start. I've seen a few posts with similar problems, but nothing quite as convoluted as this.
EDIT:
Right now I'm working with this to pivot the table, but this leaves me with columns named place1, place2, .... count1, count2,...
and I don't know how to appropriately sum up the counts and make new columns with the place names.
DECLARE #cols NVARCHAR(MAX), #query NVARCHAR(MAX);
SET #cols = STUFF(
(
SELECT DISTINCT
','+QUOTENAME(c.[ColumnName])
FROM #temp c FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)'), 1, 1, '');
SET #query = 'SELECT [ID], '+#cols+'from (SELECT [ID],
[DataValue] AS [amount],
[ColumnName] AS [category]
FROM #temp
)x pivot (max(amount) for category in ('+#cols+')) p';
EXECUTE (#query);
Your table structure is pretty bad. You'll need to normalize your data before you can attempt to pivot it. Try this:
;WITH IDs AS
(
SELECT DISTINCT
id
,ColId = RIGHT(ColumnName, LEN(ColumnName) - 5)
,Place = datavalue
FROM #temp
WHERE ISNUMERIC(datavalue) = 0
)
,Counts AS
(
SELECT DISTINCT
id
,ColId = RIGHT(ColumnName, LEN(ColumnName) - 5)
,Cnt = CAST(datavalue AS INT)
FROM #temp
WHERE ISNUMERIC(datavalue) = 1
)
SELECT
piv.id
,ABC = ISNULL(piv.ABC, 0)
,BEC = ISNULL(piv.BEC, 0)
,CDE = ISNULL(piv.CDE, 0)
,BBC = ISNULL(piv.BBC, 0)
FROM (SELECT i.id, i.Place, c.Cnt FROM IDs i JOIN Counts c ON c.id = i.id AND c.ColId = i.ColId) src
PIVOT ( SUM(Cnt)
FOR Place IN ([ABC], [BEC], [CDE], [BBC])
) piv;
Doing it with dynamic SQL would yield the following:
SET #query =
';WITH IDs AS
(
SELECT DISTINCT
id
,ColId = RIGHT(ColumnName, LEN(ColumnName) - 5)
,Place = datavalue
FROM #temp
WHERE ISNUMERIC(datavalue) = 0
)
,Counts AS
(
SELECT DISTINCT
id
,ColId = RIGHT(ColumnName, LEN(ColumnName) - 5)
,Cnt = CAST(datavalue AS INT)
FROM #temp
WHERE ISNUMERIC(datavalue) = 1
)
SELECT [ID], '+#cols+'
FROM
(
SELECT i.id, i.Place, c.Cnt
FROM IDs i
JOIN Counts c ON c.id = i.id AND c.ColId = i.ColId
) src
PIVOT
(SUM(Cnt) FOR Place IN ('+#cols+')) piv;';
EXECUTE (#query);
Try this out:
SELECT id,
COALESCE(ABC, 0) AS ABC,
COALESCE(BBC, 0) AS BBC,
COALESCE(BEC, 0) AS BEC,
COALESCE(CDE, 0) AS CDE
FROM
(SELECT id,
MIN(CASE WHEN columnname LIKE 'place%' THEN datavalue END) AS col,
CAST(MIN(CASE WHEN columnname LIKE 'count%' THEN datavalue END) AS INT) AS val
FROM t
GROUP BY id, RIGHT(columnname, 1)
) src
PIVOT
(SUM(val)
FOR col in ([ABC], [BBC], [BEC], [CDE])) pvt
Tested here: http://rextester.com/XUTJ68690
In the src query, you need to re-format your data, so that you have a unique id and place in each row. From there a pivot will work.
If the count is always immediately after the place, the following query will generate a data set for pivoting.
The result data set before pivoting has the following columns:
id, placename, count
select placeTable.id, placeTable.datavalue, countTable.datavalue
from
(select *, row_number() over (order by id, %%physloc%%) as rownum
from test
where isnumeric(datavalue) = 1
) as countTable
join
(select *, row_number() over (order by id, %%physloc%%) as rownum
from test
where isnumeric(datavalue) <> 1
) as placeTable
on countTable.id = placeTable.id and
countTable.rownum = placeTable.rownum
Tested on sqlfidde mssqlserver: http://sqlfiddle.com/#!6/701c91/18
Here is one other approach using PIVOT operator with dynamic style
declare #Col varchar(2000) = '',
#Query varchar(2000) = ''
set #Col = stuff(
(select ','+QUOTENAME(DataValue)
from table where isnumeric(DataValue) = 0
group by DataValue for xml path('')),1,1,'')
set #Query = 'select id, '+#Col+' from
(
select id, DataValue,
cast((case when isnumeric(DataValue) = 1 then DataValue else lead(DataValue) over (order by id) end) as int) Value
from table
) as a
PIVOT
(
sum(Value) for DataValue in ('+#Col+')
)pvt'
EXECUTE (#Query)
Note : I have used lead() function to access next rows data if i found character string values and replace with numeric data values
Result :
id ABC BBC BEC CDE
1 5 NULL NULL NULL
2 NULL NULL 21 6
3 NULL 9 NULL NULL
I'm struggling to even explain what I need to do so please be patient with me.
I have the following table and rows in it:
TBNAME: Distances
Track, Person, Date, Distance
TRACK1, P1, 1/1/2014, 15
TRACK2, P1, 13/1/2014, 12
TRACK1, P1, 20/2/2014, 10
TRACK2, P1, 15/1/2014, 9
TRACK1, P2, 2/1/2014, 11
TRACK2, P2, 14/1/2014, 13
TRACK1, P2, 21/2/2014, 8
TRACK2, P2, 16/1/2014, 6
What I would, ideally, like to see as a result is something like this:
P1, TRACK1, 20/2/2014, 10, TRACK2, 15/1/2014, 9
P2, TRACK1, 21/2/2014, 8, TRACK2, 16/1/2014, 6
Or, in other words, for each person, the most recent date and distance for that date for each track in one row.
Hope someone can understand this and offer a solution too :)
Cheers,
Pierre
Try this:
SELECT T1.Person, T1.Track, MAX(T1.Date), MIN(T1.Distance),
T2.Track, MAX(T2.Date), MIN(T2.Distance)
FROM Distances AS T1 INNER JOIN
Distances AS T2 ON T1.Person = T2.Person
WHERE T1.Track <> T2.Track AND T1.Track = 'Track1'
GROUP BY T1.Track, T1.Person, T2.Track
The output result of the query is showing exactly the same of your expected result.
Try combining the table by itself and connecting them with the common column.
In your case you want Person.
Select t1.Person,
t1.Tract,
t1.Date,
t1.Distance,
t2.Tract,
t2.date,
t2.Distance
From table_name t1, table_name t2
WHERE t1.Person = t2.Person;
Try this:
SELECT DISTINCT ON ("Person", "Track") *
FROM "Table"
ORDER BY "Person", "Date" DESC NULLS LAST;
Here is a query to get the records needed. First get the maximum date per track and person. Then join with the table to get the complete record.
If you know beforehand which tracks you will get, you can use a pivot query for this. As I've never done this, I ask you to look this up yourself. However, as mentioned in my comment to your request, I would use a programming language (C#, Java, PHP or whatever) to care about that.
select d.track, d.person, d.date, d.distance
from
(
select track, person, max(distances.date) as `date`
from distances
group by track, person
) lastd
inner join distances d on d.track = lastd.track and d.person = lastd.person and d.date = lastd.date
order by d.track, d.person;
BTW: date is a reserved keyword. I would not recommend to use it for a column name. Whenever you use it without a qualifier you will have to use those strange quotes.
Look for ROW_NUMBER() and OVER PARITION BY.
Idea is something like (I did not try to run this query):
;WITH
data AS
(
SELECT
*,
-- returns number for each pair of person and track starting from most recent date
--Code enhanced at here
row_number() over (partition BY person, track order by dte DESC) nr
FROM distances
)
SELECT
*
FROM
data
WHERE
nr = 1 -- we want just the most recent one
ORDER BY
person, -- "group by" person
track ;
It's still doesn't support showing one row for each person...
I don't think you can do it with SQL (because of unknown number of tracks).
There is PIVOT/UNPIVOT, but I don't think it fits here.
WITH CTE AS
(
Select P1.Track,P1.Person,ROW_NUMBER() OVER (Partition by Person,Track Order by Date
Desc) AS RN1
,Date,Distance
from Distances P1
)Select T.Person,T.Track1,T.T1Date
,T.T1Distance,T.Track2,T.T2Date,T.T2Distance
From (
Select C1.Person,C1.Track AS 'Track1',C1.Date AS 'T1Date',
C1.Distance 'T1Distance',
C2.Track AS 'Track2',C2.Date As 'T2Date',C2.Distance 'T2Distance',
ROW_NUMBER() OVER (Partition BY C1.Person Order by C1.Date Desc) RNX
from
CTE C1
JOIN
CTE C2 ON C1.RN1=1 AND C2.RN1=1
AND C1.Person=C2.Person
AND C1.Track<>C2.Track
)t Where t.RNX=1
you may also use dynamic query to achieve your expected result :)
DECLARE #nCount INT
DECLARE #nStart INT
DECLARE #Query NVARCHAR(MAX) =' '
DECLARE #sPerson NVARCHAR(MAX)
DECLARE #sTrack NVARCHAR(MAX)
SET #nCount = (SELECT COUNT(DISTINCT(person)) FROM DISTANCES)
SET #nStart = 1
WHILE #nStart <= #nCount
BEGIN
SET #sPerson = (SELECT PERSON FROM (
SELECT PERSON, ROW_NUMBER() OVER (ORDER BY PERSON) RN FROM (
SELECT DISTINCT(PERSON) FROM DISTANCES
) T1
) T2 WHERE RN = #nStart
)
SET #Query = #Query + '
SELECT '''+#sPerson+''' + '','' + STUFF( '','' +(
SELECT TRACK + '', '' + DATE + '', '' + DISTANCE FROM (
SELECT TRACK, DATE,DISTANCE,
ROW_NUMBER() OVER (PARTITION BY TRACK ORDER BY DATE DESC) RN FROM (
SELECT TRACK,date,DISTANCE FROM DISTANCES WHERE PERSON = '''+#sPerson+'''
) T1
) T2
WHERE RN = 1 FOR XML PATH('''')
),1,1,''''
)
'
IF(#nStart != #nCount)
SET #Query = #Query + ' UNION ALL '
SET #nStart = #nStart + 1
END
EXEC SP_EXECUTESQL #Query
To have a general query it need to be dynamic
DECLARE #query AS NVARCHAR(MAX)
DECLARE #pivotCols AS NVARCHAR(MAX)
DECLARE #cols AS NVARCHAR(MAX)
SELECT #pivotCols = STUFF((SELECT DISTINCT ',' + QUOTENAME([Track])
FROM Distances
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)') , 1, 1, '')
;WITH T AS (
SELECT Track
, _ID = CAST(Row_Number() OVER (ORDER BY Track) AS VARCHAR)
FROM Distances
GROUP BY Track
)
SELECT #Cols = STUFF((
SELECT ', Track_' + _ID + ' = ''' + Track + ''''
+ ', LastRun_' + _ID + ' = ' + QUOTENAME([Track])
+ ', Distance_' + _ID + '
= SUM(CASE WHEN d.Date = ' + QUOTENAME([Track]) + '
AND d.Track = ''' + Track + '''
THEN d.Distance ELSE NULL END)'
FROM T FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)') , 1, 1, '')
SELECT #query = '
With LR AS (
SELECT Person, ' + #pivotCols + '
FROM (SELECT Track, Person, [Date] FROM Distances) d
PIVOT (MAX([Date]) FOR Track IN (' + #pivotCols + ')) pvt
)
SELECT d.Person, ' + #Cols + '
FROM Distances d
INNER JOIN LR ON d.Person = LR.Person AND d.Date IN (' + #pivotCols + ')
GROUP BY d.Person, ' + #pivotCols + ''
execute(#query);
SQLFiddle demo
The first query generate the list of field for the PIVOT.
The second one generate the fields for the compound query.
The PIVOT is used to get, for every person, the last run on every track, that is than joined back to the base data to get the distance
ID State Name Department City
1 O George Sales Phoenix
1 N George Sales Denver
2 O Michael Order Process San diego
2 N Michael Marketing San jose
I got a situation that I need to convert the above tables values to the following format.(Consider the top row is column names)
ID Column OldValue New Value
1 Department Phoenix Denver
2 Department Order Process Marketing
2 City San diego San jose
I.e : I need to capture the changed column values for a table from its old and new records and record them in a different table.But the problem is we have many tables like that and the column names and no of columns are different for each table.
If anyone come with a solution that would be greatly appreciated..!
Thank you in advance.
Is this what you want?
ID Column OldValue New Value
1 City Phoenix Denver
2 Department Order Process Marketing
2 City San Diego San jose
Here is the dynamic code:
DECLARE #sqlStm varchar(max);
DECLARE #sqlSelect varchar(max);
DECLARE #tablename varchar(200);
SET #tablename = 'testtable';
-- Assume table has ID column and State column.
SET #sqlSelect = ''
SET #sqlStm = 'WITH old AS
(
SELECT *
FROM '+#tablename+'
WHERE State=''O''
), new AS
(
SELECT *
FROM '+#tablename+'
WHERE State=''N''
)';
DECLARE #aCol varchar(128)
DECLARE curCols CURSOR FOR
SELECT column_name
FROM information_schema.columns
WHERE table_name = #tablename
AND UPPER(column_name) NOT IN ('ID','STATE')
OPEN curCols
FETCH curCols INTO #aCol
WHILE (##FETCH_STATUS = 0)
BEGIN
SET #sqlStm = #sqlStm +
', changed'+#aCol+' AS
(
SELECT n.ID, '''+#aCol+''' AS [Column], o.['+#aCol+'] AS oldValue, n.['+#aCol+'] AS newValue
FROM new n
JOIN old o ON n.ID = o.ID AND n.['+#aCol+'] != o.['+#aCol+']
)'
IF LEN(#sqlSelect) > 0 SET #sqlSelect = #sqlSelect + ' UNION ALL '
SET #sqlSelect = #sqlSelect + '
SELECT * FROM changed'+#aCol
FETCH curCols INTO #aCol
END
CLOSE curCols
DEALLOCATE curCols
SET #sqlSelect = #sqlSelect + '
ORDER BY id, [Column]'
PRINT #sqlStm+#sqlSelect
EXEC (#sqlStm+#sqlSelect)
Which in my test output the following:
WITH old AS
(
SELECT *
FROM testtable
WHERE State='O'
), new AS
(
SELECT *
FROM testtable
WHERE State='N'
), changedName AS
(
SELECT n.ID, 'Name' AS [Column], o.[Name] AS oldValue, n.[Name] AS newValue
FROM new n
JOIN old o ON n.ID = o.ID AND n.[Name] != o.[Name]
), changedDepartment AS
(
SELECT n.ID, 'Department' AS [Column], o.[Department] AS oldValue, n.[Department] AS newValue
FROM new n
JOIN old o ON n.ID = o.ID AND n.[Department] != o.[Department]
), changedCity AS
(
SELECT n.ID, 'City' AS [Column], o.[City] AS oldValue, n.[City] AS newValue
FROM new n
JOIN old o ON n.ID = o.ID AND n.[City] != o.[City]
)
SELECT * FROM changedName UNION ALL
SELECT * FROM changedDepartment UNION ALL
SELECT * FROM changedCity
ORDER BY id, [Column]
Original answer below:
I would do it like this -- because I think it is clearer than other ways which might be faster:
with old as
(
Select ID, Name,Department,City
From table1
Where State='O'
), new as
(
Select ID, Name,Department,City
From table1
Where State='N'
), oldDepartment as
(
Select ID, 'Department' as Column, o.Department as oldValue, n.Department as newValue
From new
join old on new.ID = old.ID and new.Department != old.Department
), oldCity as
(
Select ID, 'City' as Column, o.City as oldValue, n.City as newValue
From new
join old on new.ID = old.ID and new.City != old.City
)
select * from oldDepartment
union all
select * from oldCity
Depending on many things (size of tables and indexes etc) it might actually be faster than using pivots or cases or grouping. It really depends on your data. If this is a one-off run I'd just go for the easiest to grok.
The cleanest approach is probably to unpivot the data and then use aggregation. This does require custom coding for each table, which you might be able to generalize by using some form a dynamic SQL.
For your particular example, here is an illustration of what to do:
select id, col,
max(case when OldNew = 'Old' then value end) as OldValue,
max(case when OldNew = 'New' then value end) as NewValue
from ((select ID, OldNew, 'Name' as col, Name as value
from t
) union all
(select ID, OldNew, 'Department' as col, Department as value
from t
) union all
(select ID, OldNew, 'City' as col, City as value
from t
)
) unpvt
group by id, col
having max(value) <> min(value) and max(value) is not null;
This is for illustration purposes. The unpivot can be done more efficiently than using union all, particularly when there are many scans. Here is a more efficient version, although the exact syntax depends on the database:
select id, col,
max(case when OldNew = 'Old' then value end) as OldValue,
max(case when OldNew = 'New' then value end) as NewValue
from (select ID, OldNew, cols.col,
(case when cols.col = 'Name' then Name
when cols.col = 'Department' then Department
when cols.col = 'City' then City
end) as value
from t cross join
(select 'Name' as col union all select 'Department' union all select 'City') cols
) unpvt
group by id, col
having max(value) <> min(value) and max(value) is not null;
This is more efficient because it will typically only scan your table once, rather than once for each column as in the union all version.
In either version, there is an implicit assumption that all the columns have the same character type. This is implicit in the format you are converting to, where all the values are in a single column.