Most effective way to aggregate clob value in Oracle - sql

Question

I have a table like:
SOME_ID FIRST_CLOB ANOTHER_CLOB
0 1.5|3.6|0.3 5.5|9.6
1 0.2 4.0|7.2
2 3.1|0.7|1.2 9.2|8.8|6.3
Length of clob is not defined. It could (and in reality will be) much longer than in example.
I need to get something like that:
SOME_ID FIRST_CLOB_MEAN ANOTHER_CLOB_MEAN
0 1.8 7.55
1 0.2 5.6
2 1.66 8.1
It could be not only a mean, but a count for example. Now we counting it when processing data in pandas but it takes very long time to load a table with clob values.
I see only one way: create a trigger on input which would:
1.Split every CLOB field like that:
CLOB
1.5
3.6
0.3
2.Find mean value from it
I'm not sure is that best way to do it and will it work if I'm importing table from csv. Will be grateful for advices
P.S. I have another solution: just split CLOB without aggregation like that:
NEW_ID SOME_ID FIRST_CLOB ANOTHER_CLOB
0 0 1.5 5.5
1 0 3.6 9.6
2 0 0.3 NULL
3 1 0.2 4.0
4 1 NULL 7.2
5 2 3.1 9.2
6 2 0.7 8.8
7 2 1.2 6.3
It will load to pandas faster but the table will be increased enormously(one CLOB can contain 10,25,50,100 items) and its also undesirable

Answer 1

You can use:
WITH bounds ( some_id, first_clob, start1, end1, another_clob, start2, end2 ) AS (
SELECT some_id,
first_clob,
1,
INSTR( first_clob, '|', 1 ),
another_clob,
1,
INSTR( another_clob, '|', 1 )
FROM table_name
UNION ALL
SELECT some_id,
first_clob,
DECODE( end1, 0, 0, end1 + 1 ),
DECODE( end1, 0, 0, INSTR( first_clob, '|', end1 + 1 ) ),
another_clob,
DECODE( end2, 0, 0, end2 + 1 ),
DECODE( end2, 0, 0, INSTR( another_clob, '|', end2 + 1 ) )
FROM bounds
WHERE end1 > 0
OR end2 > 0
)
SELECT some_id,
AVG(
TO_NUMBER(
CASE
WHEN start1 = 0 THEN NULL
WHEN end1 = 0 THEN SUBSTR( first_clob, start1 )
ELSE SUBSTR( first_clob, start1, end1 - start1 )
END
)
) AS FIRST_CLOB_MEAN,
AVG(
TO_NUMBER(
CASE
WHEN start2 = 0 THEN NULL
WHEN end2 = 0 THEN SUBSTR( another_clob, start2 )
ELSE SUBSTR( another_clob, start2, end2 - start2 )
END
)
) AS ANOTHER_CLOB_MEAN
FROM bounds
GROUP BY some_id
ORDER BY some_id
or
SELECT some_id,
first_clob_mean,
another_clob_mean
FROM table_name t
CROSS APPLY (
SELECT AVG( TO_NUMBER(column_value) ) AS first_clob_mean
FROM XMLTABLE( ('"' || REPLACE( t.first_clob, '|', '","' ) || '"') )
)
CROSS APPLY (
SELECT AVG( TO_NUMBER(column_value) ) AS another_clob_mean
FROM XMLTABLE( ('"' || REPLACE( t.another_clob, '|', '","' ) || '"') )
)
Which, for the sample data:
INSERT INTO table_name ( some_id, first_clob, another_clob )
SELECT 0, '1.5|3.6|0.3', '5.5|9.6' FROM DUAL UNION ALL
SELECT 1, '0.2', '4.0|7.2' FROM DUAL UNION ALL
SELECT 2, '3.1|0.7|1.2', '9.2|8.8|6.3' FROM DUAL;
DECLARE
v_clob CLOB;
BEGIN
FOR i IN 1 .. 4000 LOOP
IF v_clob IS NOT NULL THEN
v_clob := v_clob || '|';
END IF;
v_clob := v_clob || i;
END LOOP;
INSERT INTO table_name VALUES ( 4, v_clob, NULL );
END;
/
Both output:
SOME_ID | FIRST_CLOB_MEAN | ANOTHER_CLOB_MEAN
------: | ---------------------------------------: | ----------------:
0 | 1.8 | 7.55
1 | .2 | 5.6
2 | 1.66666666666666666666666666666666666667 | 8.1
4 | 2000.5 | null
db<>fiddle here

Answer 2

A query like this one can be used to split the CLOBs by the pipe delimiter, then take the averages. Each CLOB needs to be split in its own sub-query so the count of numbers in the CLOB does not affect the count of numbers in another CLOB in the same row.
SELECT c1.some_id, c1.first_clob_mean, c2.another_clob_mean
FROM ( SELECT some_id,
AVG (TO_NUMBER (REGEXP_SUBSTR (first_clob,
'[^|]+',
1,
LEVEL))) AS first_clob_mean
FROM clobs
CONNECT BY LEVEL <= REGEXP_COUNT (first_clob, '\|') + 1
AND PRIOR SYS_GUID () IS NOT NULL
AND PRIOR some_id = some_id
GROUP BY some_id) c1,
( SELECT some_id,
AVG (TO_NUMBER (REGEXP_SUBSTR (another_clob,
'[^|]+',
1,
LEVEL))) AS another_clob_mean
FROM clobs
CONNECT BY LEVEL <= REGEXP_COUNT (another_clob, '\|') + 1
AND PRIOR SYS_GUID () IS NOT NULL
AND PRIOR some_id = some_id
GROUP BY some_id) c2
WHERE c1.some_id = c2.some_id
ORDER BY some_id;