Create a sort column in SQL for hierarchy order based on NextId

Create a sort column in SQL for hierarchy order based on NextId - sql

I have a hierarchy table in SQL containing NodeId, ParentId, ChildId, NextId, NodeLevel, and a validity period (DateFrom, DateTo) since the location of a node within the hierarchy can change based on the date. I am able to loop through the records recursively using the NodeId and ParentId with no problem. The piece that I am missing is how to order the nodes. The order of the nodes is based on ChildId, NextId, and validity period. I need a way to dynamically add a sort column based on the validity period. This should give me a column called SortOrder that just numbers the nodes sequentially as they would appear if the hierarchy was displayed expanded down to the lowest level. The code to recreate the table I am using is:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[NextIDTest](
[NodeName] [nvarchar](60) NOT NULL,
[NodeId] [int] NOT NULL,
[ParentId] [int] NOT NULL,
[ChildId] [int] NOT NULL,
[NextId] [int] NOT NULL,
[NodeLevel] [int] NOT NULL,
[DateFrom] [date] NOT NULL,
[DateTo] [date] NOT NULL
) ON [PRIMARY]
GO
INSERT [dbo].[NextIDTest] ([NodeName], [NodeId], [ParentId], [ChildId], [NextId], [NodeLevel], [DateFrom], [DateTo])
VALUES
(N'TOTAL CAPITAL EXPENDITURE', 1, 0, 2, 0, 1, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'UTILITY CAPITAL EXPENDITURES', 2, 1, 3, 28, 2, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'CATEGORY 1', 3, 2, 4, 24, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'NEW MAINS', 4, 3, 7, 9, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000713', 5, 4, 0, 8, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000712', 6, 4, 0, 5, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000711', 7, 4, 0, 6, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00007199', 8, 4, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'NEW SERVICES', 9, 3, 13, 14, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00007299', 10, 9, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000723', 11, 9, 0, 10, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000722', 12, 9, 0, 11, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000721', 13, 9, 0, 12, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000777', 14, 3, 0, 17, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000740', 15, 3, 0, 16, 4, CAST(N'1000-01-01' AS Date), CAST(N'2014-11-30' AS Date))
,(N'00000586', 16, 3, 0, 18, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000015', 17, 3, 0, 15, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'METERS', 18, 3, 20, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000023', 19, 18, 0, 21, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000021', 20, 18, 0, 19, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'IND AND COMM SOLUTIONS', 21, 18, 23, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000024', 22, 21, 0, 0, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000022', 23, 21, 0, 22, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'CATEGORY 2', 24, 2, 25, 29, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'BARE STEEL', 25, 24, 27, 40, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000319', 26, 25, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000119', 27, 25, 0, 26, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'NON-UTILITY CAPITAL EXPENDITURES', 28, 1, 35, 0, 2, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'CATEGORY 3', 29, 2, 52, 30, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'CATEGORY 4', 30, 2, 55, 31, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'CATEGORY 5', 31, 2, 57, 32, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'CATEGORY 6', 32, 2, 125, 129, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'NON-UTILITY', 33, 28, 122, 0, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'BUSINESS DEVELOPMENT', 34, 28, 120, 33, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'GAS STORAGE', 35, 28, 95, 34, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000320', 37, 24, 0, 39, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000120', 38, 24, 0, 37, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000112', 39, 24, 0, 127, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'LEAKAGE', 40, 24, 59, 128, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'GENERAL', 41, 29, 79, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'GAS SUPPLY', 42, 29, 71, 41, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'REGULATORS', 43, 29, 66, 42, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000336', 44, 29, 0, 46, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000318', 45, 29, 0, 43, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000317', 46, 29, 0, 45, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000140', 47, 29, 0, 50, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000136', 48, 29, 0, 44, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000115', 49, 29, 0, 47, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000014', 50, 29, 0, 48, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'RELOCATES/ABANDONMENTS', 51, 29, 64, 126, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'PUBLIC WORKS', 52, 29, 62, 51, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000585', 53, 30, 0, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'LAND AND STRUCTURES', 54, 30, 86, 53, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'INFORMATION TECHNOLOGY', 55, 30, 83, 54, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'TOTAL NON-UTIL INTERSTATE STOR', 56, 31, 93, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'TOTAL UTILITY STORAGE PROJECTS', 57, 31, 87, 56, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000313', 58, 40, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000113', 59, 40, 0, 58, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000314', 60, 52, 0, 61, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000117', 61, 52, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000114', 62, 52, 0, 60, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000316', 63, 51, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000116', 64, 51, 0, 63, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000019', 65, 43, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000013', 66, 43, 0, 65, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000529', 67, 42, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000524', 68, 42, 0, 67, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000018', 69, 42, 0, 68, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000012', 70, 42, 0, 69, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000011', 71, 42, 0, 70, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000571', 72, 41, 0, 73, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000509', 73, 41, 0, 74, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000044', 74, 41, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000043', 75, 41, 0, 76, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000040', 76, 41, 0, 80, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000035', 77, 41, 0, 78, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000033', 78, 41, 0, 75, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000032', 79, 41, 0, 77, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000031', 80, 41, 0, 72, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000039', 81, 55, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000038', 82, 55, 0, 81, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000017', 83, 55, 0, 82, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000511', 84, 54, 0, 85, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000036', 85, 54, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000034', 86, 54, 0, 84, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000569', 87, 57, 0, 88, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000562', 88, 57, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000581', 89, 56, 0, 90, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000563', 90, 56, 0, 124, 5, CAST(N'1000-01-01' AS Date), CAST(N'2014-11-30' AS Date))
,(N'00000514', 91, 56, 0, 92, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000513', 92, 56, 0, 89, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000512', 93, 56, 0, 91, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'GAS STORAGE LLC', 94, 35, 116, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'OTHER STORAGE', 95, 35, 96, 94, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'OTHER (COMPANY 1234)', 96, 95, 111, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000480', 97, 96, 0, 112, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000470', 98, 96, 0, 97, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000460', 99, 96, 0, 98, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000450', 100, 96, 0, 99, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000440', 101, 96, 0, 100, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000430', 102, 96, 0, 101, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000420', 104, 96, 0, 102, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000410', 111, 96, 0, 104, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000401', 112, 96, 0, 0, 6, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000491', 113, 94, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000489', 114, 94, 0, 117, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000486', 115, 94, 0, 113, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000481', 116, 94, 0, 114, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000413', 117, 94, 0, 115, 5, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000508', 118, 34, 0, 119, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000507', 119, 34, 0, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000506', 120, 34, 0, 121, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000505', 121, 34, 0, 118, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000587', 122, 33, 0, 123, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000510', 123, 33, 0, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000564', 124, 56, 0, 0, 5, CAST(N'1000-01-01' AS Date), CAST(N'2014-11-30' AS Date))
,(N'00000760', 125, 32, 0, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000752', 126, 29, 0, 49, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000751', 127, 24, 0, 0, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000750', 128, 24, 0, 38, 4, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'CATEGORY 7', 129, 2, 132, 0, 3, CAST(N'1000-01-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000740', 130, 129, 0, 0, 4, CAST(N'2014-12-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000564', 131, 129, 0, 130, 4, CAST(N'2014-12-01' AS Date), CAST(N'9999-12-31' AS Date))
,(N'00000563', 132, 129, 0, 131, 4, CAST(N'2014-12-01' AS Date), CAST(N'9999-12-31' AS Date))
GO
EDIT:
Expected output would look like this:
NodeName NodeId ParentId ChildId NextId NodeLevel SortOrder
------------------------------------------------------------ ----------- ----------- ----------- ----------- ----------- -----------
TOTAL CAPITAL EXPENDITURE 1 0 2 0 1 1
UTILITY CAPITAL EXPENDITURES 2 1 3 28 2 2
CATEGORY 1 3 2 4 24 3 3
NEW MAINS 4 3 7 9 4 4
00000713 5 4 0 8 5 7
00000712 6 4 0 5 5 6
00000711 7 4 0 6 5 5
00007199 8 4 0 0 5 8

I do not know if I really got this correctly, but this might be what you are looking for:
Btw: You do not need all these columns in your table (ChildId, NextId and NodeLevel will result out of the query). It is quite erronous actually to keep them as stored values...
Btw: Next time please try to reduce your example to a small amount of rows and add the expected output fitting to the given data. This makes it much easier to understand what you really need...
--A recursive CTE traverses down the hierarchy
WITH RecCTE AS
(
SELECT NodeName,NodeId,ParentId,1 AS NodeLevel,DateFrom,DateTo
FROM NextIDTest
WHERE ParentId=0
UNION ALL
SELECT y.NodeName,y.NodeId,y.ParentId,x.NodeLevel+1,y.DateFrom,y.DateTo
FROM RecCTE AS x
INNER JOIN NextIDTest AS y ON x.NodeId=y.ParentId
)
--the second CTE will take the ResultSet as it comes out of recCTE and number it
,Numbered AS --
(
SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS RunningNumber,*
FROM RecCTE
)
--the third CTE will get another numbering, don't know if this really is what you need
,InternalSort AS
(
SELECT ROW_NUMBER() OVER(PARTITION BY NodeLevel ORDER BY DateFrom DESC,RunningNumber) AS InternalSort,*
FROM Numbered
)
--And finally this is again numbered "as is"
SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS YourSortColumn,*
FROM InternalSort

Related

how to do list_agg with a character limit of 1440 characters in Snowflake

I have a table as below, I have 1775 ids and length of id column is 10 characters, I want to create multiple groups of list_agg of ids with a limit of not more than 1440 characters to distribute 1775 ids into groups
id
distributor_name
1234567890
Sample_name1
2345678901
Sample_name1
3456789012
Sample_name1
4567890123
Sample_name2
5678901234
Sample_name2
6789012345
Sample_name3
7890123456
Sample_name3
8901234567
Sample_name3
Required output is:
group
id_count
list_agg
1
120
1234567890,2345678901,3456789012...
2
122
7890123456,5678901234,8901234567...
Very much appreciate your help!

If your ID space is symmetrically distributed you can use WIDTH_BUCKET
with data1 as (
select
row_number() over (order by true)-1 as rn
from table(generator(rowcount=>100))
)
select
width_bucket(rn, 0, 100, 5) as group_id,
count(*) as c,
array_agg(rn) within group(order by rn) as bucket_values
from data1
group by 1
order by 1;
GROUP_ID
C
BUCKET_VALUES
1
20
[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ]
2
20
[ 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39 ]
3
20
[ 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59 ]
4
20
[ 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79 ]
5
20
[ 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99 ]
If your data is not symmetrical, you can allocate row numbers to each row, and then shave the yack again.
you can also be data driven:
width_bucket(rn, (select min(rn) from data1), (select max(rn) from data1)+1, 5) as group_id,

It's easier done with array_agg but if you must use listagg, here is a spin on Simeon's answer. The basic idea is to keep track of length of numbers when stitched together and also account for the number of commas so we don't go over 1440 char limit.
create or replace temporary table t as
select row_number() over (order by true)-1 as id,
uniform(1000000000, 1999999999, random()) as num
from table(generator(rowcount=>1775));
with cte as
(select *, ceil((sum(len(num)) over (order by id) + count(num) over (order by id) -1)/1440) as group_id
from t)
select group_id,
count(num) as id_count,
listagg(num,',') as id_list,
len(id_list) as len_check
from cte
group by group_id
order by group_id;

Outliers in data

I have a dataset like so -
15643, 14087, 12020, 8402, 7875, 3250, 2688, 2654, 2501, 2482, 1246, 1214, 1171, 1165, 1048, 897, 849, 579, 382, 285, 222, 168, 115, 92, 71, 57, 56, 51, 47, 43, 40, 31, 29, 29, 29, 29, 28, 22, 20, 19, 18, 18, 17, 15, 14, 14, 12, 12, 11, 11, 10, 9, 9, 8, 8, 8, 8, 7, 6, 5, 5, 5, 4, 4, 4, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
Based on domain knowledge, I know that larger values are the only ones we want to include in our analysis. How do I determine where to cut off our analysis? Should it be don't include 15 and lower or 50 and lower etc?

You can do a distribution check with quantile function. Then you can remove values below lowest 1 percentile or 2 percentile. Following is an example:
import numpy as np
data = np.array(data)
print(np.quantile(data, (.01, .02)))
Another method is calculating the inter quartile range (IQR) and setting lowest bar for analysis is Q1-1.5*IQR
Q1, Q3 = np.quantile(data, (0.25, 0.75))
data_floor = Q1 - 1.5 * (Q3 - Q1)

Error message More than one column has the same display name

I keep getting the error message
More than one column has the same display name
but I cannot find the route cause. Any help is greatly appreciated!
SELECT
gl_ap_details.ledger_name,
gl_ap_details.company_code,
gl_ap_details.location_code,
gl_ap_details.cost_center,
gl_ap_details.account_number,
gl_ap_details.account_name,
gl_ap_details.product_code,
gl_ap_details.channel_code,
gl_ap_details.journal_name,
gl_ap_details.line_description,
gl_ap_details.gl_posted_date,
gl_ap_details.currency,
gl_ap_details.je_source,
gl_ap_details.je_category,
gl_ap_details.effective_date,
gl_ap_details.created_by,
gl_ap_details.invoice_num,
gl_ap_details.invoice_id,
gl_ap_details.invoice_date,
gl_ap_details.vendor_name,
gl_ap_details.vendor_number,
gl_ap_details.invoice_image,
gl_ap_details.po_number,
gl_ap_details.po_requestor,
gl_ap_details.period_name,
gl_ap_details.amount,
gl_ap_details.gl_posted_date,
gl_ap_details.project_code
FROM
wbr_global.gl_ap_details
WHERE
wbr_global.gl_ap_details.ledger_name = 'Amazon.com, Inc.'
AND cost_center IN ('1172')
AND period_name = 'JUL-21'
AND wbr_global.gl_ap_details.account_number = '60820'
GROUP BY
1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
21, 22, 23, 24, 25, 26, 27, 28;

Null values at the end of rows after INSERT INTO

I am currently trying to INSERT INTO my SQL database a row of 144 columns.
The problem is that the last 10 values of the new row are NULL while they are supposed to be float and int.
That's an example of what I have in my DB after the INSERT INTO :
First column
Before last column
Last column
1
NULL
NULL
That's the SQL request I am using
INSERT INTO "historic_data2"
VALUES (28438, 163, 156, 1, 'FIST 2', 91, 81, 82, 84, 90, 6, '2 Pts Int M', 'Offensive', 0, '91_81_82_84_90', 86, 85, 0, 36, 62, 24, 0, 132, 86, 0, 83, 0, 0, 0, 0, 42, 77, 24, 0, 173, 107, 0, 204, 0, 0, 0, 0, 42, 77, 24, 0, 173, 107, 0, 204, 0, 0, 0, 81, 62, 34, 23, 19, 45, 32, 18, 9, 19, 0.5555555555555556, 0.5161290322580645, 0.5294117647058824, 0.391304347826087, 1.0, 82, 54, 34, 18, 28, 49, 27, 17, 8, 28, 0.5975609756097561, 0.5, 0.5, 0.4444444444444444, 1.0, 302, 233, 132, 89, 69, 168, 116, 69, 35, 69, 0.5562913907284768, 0.4978540772532189, 0.5227272727272727, 0.39325842696629215, 1.0, 214, 161, 84, 73, 53, 119, 79, 39, 36, 53, 0.5560747663551402, 0.4906832298136646, 0.4642857142857143, 0.4931506849315068, 1.0, 717, 544, 298, 233, 173, 416, 285, 175, 97, 173, 0.5801952580195258, 0.5238970588235294, 0.587248322147651, 0.41630901287553645, 1.0, 466, 315, 183, 128, 151, 357, 233, 138, 91, 151, 0.7660944206008584, 0.7396825396825397, 0.7540983606557377, 0.7109375,1.0,112)
I can't figure out how to solve this issue. My guess would be that there is a hard limit on how much column you can insert at once but I don't know how to solve that.
Thank you in advance for your help

Convert ECC PKCS#8 public and private keys to traditional format

I have ECC public and private generated with BouncyCastle:
Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());
ECNamedCurveParameterSpec ecSpec = ECNamedCurveTable
.getParameterSpec("secp192r1");
KeyPairGenerator g = KeyPairGenerator.getInstance("ECDSA", "BC");
g.initialize(ecSpec, new SecureRandom());
KeyPair pair = g.generateKeyPair();
System.out.println(Arrays.toString(pair.getPrivate().getEncoded()));
System.out.println(Arrays.toString(pair.getPublic().getEncoded()));
byte[] privateKey = new byte[]{48, 123, 2, 1, 0, 48, 19, 6, 7, 42, -122, 72, -50, 61, 2, 1, 6, 8, 42, -122, 72, -50, 61, 3, 1, 1, 4, 97, 48, 95, 2, 1, 1, 4, 24, 14, 117, 7, -120, 15, 109, -59, -35, 72, -91, 99, -2, 51, -120, 112, -47, -1, -115, 25, 48, -104, -93, 78, -7, -96, 10, 6, 8, 42, -122, 72, -50, 61, 3, 1, 1, -95, 52, 3, 50, 0, 4, 64, 48, -104, 32, 41, 13, 1, -75, -12, -51, -24, -13, 56, 75, 19, 74, -13, 75, -82, 35, 1, -50, -93, -115, -115, -34, -81, 119, -109, -50, -39, -57, -20, -67, 65, -50, 66, -122, 96, 84, 117, -49, -101, 54, -30, 77, -110, -122}
byte[] publicKey = new byte[]{48, 73, 48, 19, 6, 7, 42, -122, 72, -50, 61, 2, 1, 6, 8, 42, -122, 72, -50, 61, 3, 1, 1, 3, 50, 0, 4, 64, 48, -104, 32, 41, 13, 1, -75, -12, -51, -24, -13, 56, 75, 19, 74, -13, 75, -82, 35, 1, -50, -93, -115, -115, -34, -81, 119, -109, -50, -39, -57, -20, -67, 65, -50, 66, -122, 96, 84, 117, -49, -101, 54, -30, 77, -110, -122}
How to convert them into traditional format which can be reused later in https://github.com/kmackay/micro-ecc/blob/master/uECC.h? I need 24 bytes private and 48 public key while now it is 125 and 75.

Gives 24 and 48, sometimes when 0 is added at the beginning 25 or 49:
ECPrivateKey ecPrivateKey = (ECPrivateKey)privateKey;
System.out.println(ecPrivateKey.getS().toByteArray().length);
ECPublicKey ecPublicKey = (ECPublicKey)publicKey;
System.out.println(ecPublicKey.getW().getAffineX().toByteArray().length + ecPublicKey.getW().getAffineY().toByteArray().length);

We Keep Coding

sql objective-c vba vb.net react-native apache vue.js tensorflow api pandas

Create a sort column in SQL for hierarchy order based on NextId - sql

Related

how to do list_agg with a character limit of 1440 characters in Snowflake

Outliers in data

Error message More than one column has the same display name

Null values at the end of rows after INSERT INTO

Convert ECC PKCS#8 public and private keys to traditional format

Categories

Resources