Output multiple rows after joining and pivoting multiple tables - sql

Question

I have a couple of tables (in a SQL Server 2014 SP2 database) on which I have successfully pivoted (thanks to lptr in Creating SQL pivot where duplicate column names exist). The next problem is that each datasource has one or more datapoints, but I am only getting one of the datapoints, per datasource.
The tables looks like this:
dataSourceSnapshot
| snapshotId | snapshotTime |
-----------------------------------
| 1 | 2021-12-01 07:00 |
| 2 | 2021-12-02 07:00 |
...
datasource
| snapshotId | dsRecordId | dsId | dsName | dsTableDate |
---------------------------------------------------------------
| 1 | 1 | 1 | cpu | 2021-12-01 07:00 |
| 1 | 2 | 2 | mem | 2021-12-01 07:00 |
| 1 | 3 | 3 | cache | 2021-12-01 07:00 |
| 2 | 4 | 1 | cpu | 2021-12-01 07:00 |
| 2 | 5 | 2 | mem | 2021-12-01 07:00 |
| 2 | 6 | 3 | cache | 2021-12-01 07:00 |
...
datasourceProperty
This is a truncated list of datasource properties.
| dsRecordId | dsPropPropertyRecordId | dsPropPropertyName | dsPropPropertyValue |
----------------------------------------------------------------------------------
| 1 | 1 | id | 1 |
| 1 | 2 | description | cpu stats |
| 1 | 3 | name | cpu |
| 1 | 4 | collectInterval | 300 |
| 1 | 5 | group | |
| 1 | 6 | dataPoints | System.Object[] |
| 2 | 7 | id | 2 |
| 2 | 8 | description | memory stats |
| 2 | 9 | name | mem |
| 2 | 10 | collectInterval | 300 |
| 2 | 11 | group | |
| 2 | 12 | dataPoints | System.Object[] |
| 3 | 13 | id | 3 |
| 3 | 14 | description | |
| 3 | 15 | name | cache |
| 3 | 16 | collectInterval | 600 |
| 3 | 17 | group | |
| 3 | 18 | dataPoints | System.Object[] |
...
datapointProperty
This is a truncated list of datapoint properties.
| dsRecordId | dpRecordId | dpPropertyName | dpPropertyValue |
-----------------------------------------------------------------
| 1 | 1 | id | 1 |
| 1 | 2 | datasourceId | 1 |
| 1 | 3 | name | cpuState |
| 1 | 4 | description | |
| 1 | 5 | alertExpr | != 0 0 |
| 1 | 6 | type | 2 |
| 1 | 7 | id | 2 |
| 1 | 8 | datasourceId | 1 |
| 1 | 9 | name | freePercent |
| 1 | 10 | description | CPU utilization |
| 1 | 11 | alertExpr | >= 90 90 |
| 1 | 12 | type | 2 |
| 2 | 13 | id | 3 |
| 2 | 14 | datasourceId | 2 |
| 2 | 15 | name | freePercent |
| 2 | 16 | description | Memory utilization |
| 2 | 17 | alertExpr | >= 90 90 |
| 2 | 18 | type | 2 |
| 3 | 19 | id | 4 |
| 3 | 20 | datasourceId | 3 |
| 3 | 21 | name | state |
| 3 | 22 | description | |
| 3 | 23 | alertExpr | = 1 1 1 |
| 3 | 24 | type | 4 |
...
I am trying to get a row per datasource (but only the most recent instance of that datasource's entry) and per datapoint, which includes all of the datasource properties. Something like this:
| dsRecordId | id | description | name | collectInterval | group | dataPoints | dp_id | dp_datasourceId | dp_name | dp_description | dp_alertExpr | dp_type |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 1 | 1 | cpu stats | cpu | 300 | | System.Object[] | 1 | 1 | cpuState | | != 0 0 | 2 |
| 1 | 1 | cpu stats | cpu | 300 | | System.Object[] | 2 | 1 | freePercent | | >= 90 90 | 2 |
| 2 | 2 | memory stats | mem | 300 | | System.Object[] | 3 | 2 | freePercent | Memory utilization | >= 90 90 | 2 |
| 3 | 3 | | cache | 600 | | System.Object[] | 4 | 3 | state | | = 1 1 1 | 4 |
I have a bit of T-SQL that will pivot, but because datasources and datapoints both have id, name, and description properties, I have to exclude those properties in the pivot. This query:
DECLARE
#dsPropPropertyColumns NVARCHAR(MAX) = '',
#dpPropertyColumns NVARCHAR(MAX) = '',
#sql NVARCHAR(MAX) = '';
-- select the property names
--- properties in the datasourceProperty table
SELECT #dsPropPropertyColumns = (
SELECT DISTINCT '[' + [dsPropPropertyName] + ']' + ','
FROM dbo.dataSourceSnapshot dss
LEFT OUTER JOIN dbo.datasource ds ON ds.snapshotId = dss.snapshotId
LEFT OUTER JOIN dbo.datasourceProperty dsp ON dsp.dsRecordId = ds.dsRecordId
WHERE dss.snapshotTime = (
SELECT MAX(snapshotTime) FROM dbo.dataSourceSnapshot
)
FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)')
--- properties in the datapointProperty table
SELECT #dpPropertyColumns = (
SELECT DISTINCT '[' + [dpPropertyName] + ']' + ','
FROM dbo.dataSourceSnapshot dss
LEFT OUTER JOIN dbo.datasource ds ON ds.snapshotId = dss.snapshotId
LEFT OUTER JOIN dbo.datapointProperty dp ON dp.dsRecordId = ds.dsRecordId
WHERE dss.snapshotTime = (
SELECT MAX(snapshotTime) FROM dbo.dataSourceSnapshot
)
FOR XML PATH(''), TYPE
).value('.', 'nvarchar(max)')
-- remove the trailing comma
SET #dsPropPropertyColumns = LEFT(#dsPropPropertyColumns, LEN(#dsPropPropertyColumns) - 1);
SET #dpPropertyColumns = LEFT(#dpPropertyColumns, LEN(#dpPropertyColumns) - 1);
-- construct dynamic SQL
SET #sql ='
SELECT * FROM (
SELECT dsp.dsRecordId, dsp.dsPropPropertyName, dsp.dsPropPropertyValue, CONCAT(''dp_'', dp.dpPropertyName) as dpPropertyName, dp.dpPropertyValue
FROM dbo.snapshot dss
LEFT OUTER JOIN dbo.datasource ds ON ds.snapshotId = dss.snapshotId
LEFT OUTER JOIN dbo.datasourceProperty dsp ON dsp.dsRecordId = ds.dsRecordId
LEFT OUTER JOIN dbo.datapointProperty dp ON dp.dsRecordId = ds.dsRecordId
WHERE dss.snapshotTime = (
SELECT MAX(snapshotTime) FROM dbo.dataSourceSnapshot
)
) t1
PIVOT (
MAX(dsPropPropertyValue) FOR dsPropPropertyName IN ('+ #dsPropPropertyColumns +')
) AS pivot_table
PIVOT (
MAX(dpPropertyValue) FOR dpPropertyName IN ('+ #dpPropertyColumns +')
) AS pivot_table2
ORDER BY id
'
EXECUTE sp_executesql #sql;
Produces this output:
| dsRecordId | id | description | name | collectInterval | group | dataPoints | dp_id | dp_datasourceId | dp_name | dp_description | dp_alertExpr | dp_type |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 1 | 1 | cpu stats | cpu | 300 | | System.Object[] | 1 | 1 | cpuState | CPU utilization | >= 90 90 | 2 |
| 2 | 2 | memory stats | mem | 300 | | System.Object[] | 3 | 2 | freePercent | Memory utilization | >= 90 90 | 2 |
| 3 | 3 | | cache | 600 | | System.Object[] | 4 | 3 | state | | = 1 1 1 | 4 |
This output shows one row for the "cpu" datasource (which has two datapoints). Confusingly, it also shows the datapoint (dp_) description and alertExpr values for the CPU's freePercent datapoint, with the "cpuState" dp_name.
In Creating SQL pivot where duplicate column names exist, it was suggested to use:
ROW_NUMBER() OVER(partition by dp.dsRecordId, dp.dpPropertyName order by dp.dpRecordId) as dpPropertyName
With this T-SQL:
SELECT dsp.dsRecordId, dsp.dsPropPropertyName, dsp.dsPropPropertyValue, ROW_NUMBER() OVER(partition by dp.dsRecordId, dp.dpPropertyName order by dp.dpRecordId) as dpPropertyName, dp.dpPropertyValue
FROM logicmonitor.dataSourceSnapshot dss
LEFT OUTER JOIN logicmonitor.datasource ds ON ds.dataSourceSnapshotId = dss.dataSourceSnapshotId
LEFT OUTER JOIN logicmonitor.datasourceProperty dsp ON dsp.dsRecordId = ds.dsRecordId
LEFT OUTER JOIN logicmonitor.datapointProperty dp ON dp.dsRecordId = ds.dsRecordId
WHERE dss.dataSourceSnapshotTime = (
SELECT MAX(dataSourceSnapshotTime) FROM logicmonitor.dataSourceSnapshot
)
I get get a table that looks like this:
| dsRecordId | dsPropPropertyName | dsPropPropertyValue | dpPropertyName | dpPropertyValue |
--------------------------------------------------------------------------------------------------
| 1 | id | 1 | 1 | |
| 1 | description | cpu stats | 2 | |
| 1 | name | cpu | 3 | |
That does not look right, so I assume I am mis-using ROW_NUMBER() and OVER().
The question is, how do I get a pivoted row per datapoint, joined to (or joined from?) the datasourceProperty and datasourceSnapshot tables