How to handle multiple associated log messages? - sql

Question

I have a service that calculates a bunch of things for a project. A user can trigger this calculation multiple times a day. Every calculation generates a few interesting metrics (let's call them A, B, C).
I report these metrics to a log service with individual log messages. The log messages look like this:
date | calculationID1 | projectID1 | metricA | valueA
date | calculationID1 | projectID1 | metricB | valueB
date | calculationID1 | projectID1 | metricC | valueC
date | calculationID2 | projectID2 | metricA | valueA
date | calculationID2 | projectID2 | metricB | valueB
date | calculationID2 | projectID2 | metricC | valueC
date | calculationID3 | projectID1 | metricA | valueA
date | calculationID3 | projectID1 | metricB | valueB
date | calculationID3 | projectID1 | metricC | valueC
In this example the project with ID 1 was run two times on this particular day. In my analytics backend I have a Hive cluster to analyze this data and I want to generate a table with the last reported metrics for every project for a given day day:
date | calculationID3 | projectID1 | valueA | valueB | valueC
date | calculationID2 | projectID2 | valueA | valueB | valueC
Apparently this calculation is really costly as I do a lot of joins. My company has a strict logging format and that's why I created one value per log message. Should I create one log message containing all metrics instead to ease the reporting?
Can anyone point me to best practices for these kind of problems?

Answer 1

If we use DB, supporting PIVOT clause in SQL, then we can gather data from log report using the following query.
The same results can be fetched without PIVOT, but another way requires a lot of copy-paste and juggling, and since you are "pragmatic with implementation", I suppose we don't need to speak about those dirty things.
To see what's happening in the query you can do 3 steps:
run the query without PIVOT (just remove the PIVOT keyword and the rest of the code)
then run it as is
compare results of the first and the second step, recognising how the rows are transposing to the columns
WITH
data_table (stamp, calculation_ID, project_ID, metric_name, metric_value) as ( select
timestamp '2015-01-01 00:00:01', 'calc_ID_1', 'project_WHITE', 'metric_A', 11 from dual union all select
timestamp '2015-01-01 00:00:02', 'calc_ID_1', 'project_WHITE', 'metric_B', 21 from dual union all select
timestamp '2015-01-01 00:00:03', 'calc_ID_1', 'project_WHITE', 'metric_C', 31 from dual union all select
timestamp '2015-01-01 00:01:04', 'calc_ID_2', 'project_WHITE', 'metric_A', 12 from dual union all select
timestamp '2015-01-01 00:01:05', 'calc_ID_2', 'project_WHITE', 'metric_B', 22 from dual union all select
timestamp '2015-01-01 00:01:06', 'calc_ID_2', 'project_WHITE', 'metric_C', 32 from dual union all select
timestamp '2015-01-01 00:00:11', 'calc_ID_3', 'project_BLACK', 'metric_A', 41 from dual union all select
timestamp '2015-01-01 00:00:12', 'calc_ID_3', 'project_BLACK', 'metric_B', 51 from dual union all select
timestamp '2015-01-01 00:00:13', 'calc_ID_3', 'project_BLACK', 'metric_C', 61 from dual union all select
timestamp '2015-01-01 00:01:14', 'calc_ID_4', 'project_BLACK', 'metric_A', 42 from dual union all select
timestamp '2015-01-01 00:01:15', 'calc_ID_4', 'project_BLACK', 'metric_B', 52 from dual union all select
timestamp '2015-01-01 00:01:16', 'calc_ID_4', 'project_BLACK', 'metric_C', 62 from dual
)
SELECT *
FROM (
select trunc(stamp) AS day,
calculation_id,
project_id,
metric_name,
metric_value
from (
select t.*,
rank() OVER (PARTITION BY project_ID, metric_name, trunc(stamp) ORDER BY stamp DESC) calculation_rank
from data_table t
-- take only the last log row for (project_ID, metric_name) for every given day
) where calculation_rank = 1
)
PIVOT (
-- aggregate function is required here,
-- and SUM can be replaced with something more relevant to custom logic
SUM(metric_value)
FOR
metric_name IN ('metric_A' AS "Metric A",
'metric_B' AS "Metric B",
'metric_C' AS "Metric C")
);
Results:
DAY | CALCULATION_ID | PROJECT_ID | Metric A | Metric B | Metric C
------------------------------------------------------------------------------
2015-01-01 | calc_ID_4 | project_BLACK | 42 | 52 | 62
2015-01-01 | calc_ID_2 | project_WHITE | 12 | 22 | 32
In this query calculation_ID is redundant (I use it only to make example clearer for code reader). But you still can apply this info to check integrity of logging data format, exploring if equal calculation_ID corresponds to the metrics involved in the same group/time-period.