Query Performance Netezza SQL - sql

Question

What is the best way to have the below logic to be in a single Netezza SQL. I implemented the logic in a for loop but for my data set it is taking a long time in netezza (say 47 mins to complete the loop) I have two tables, “TABLE - A” (Sector_ID | Value) and “TABLE B” holds which sector_id is intersected with other sector_id combination.
Now, the TABLE-A will be sorted descending on Value, and need to take the each highest sector_id from table A and eliminate all the corresponding intersected sector_id for point A in TABLE- B.
For Example,
TABLE – A (After Sorting)
SECTOR_ID VALUE(DESC) DELETED ROWS
6 150
1 140 DELETED
4 50
2 45 DELETED
3 15
TABLE – B
SECTOR_ID INTERSECTED_ID DELETED ROWS
6 6
6 1 DELETED
6 2 DELETED
1 1 DELETED
1 4 DELETED
1 2 DELETED
4 4
4 1 DELETED
2 6 DELETED
2 1 DELETED
2 2 DELETED
3 3
Now the remaining values in TABLE – A will be the desired output. Please suggest. The DB I am using is Netezza.

Answer 1

I'm going to attempt to restate your problem, so if it is not accurate let me know in the comments so that we can formulate it (and this answer).
You need to remove records in table_a when table_a.sector_id appears in the list of previous table_b.intersected_ids given that table_b is sorted by table_a.value.
Solution
Note that this solution is not relegated to Netezza-only, but rather relational algebra. Also, as far as I know, this will be faster than any cursor or loop-based approach for any RDBMS.
The biggest chore is setting up the list of sector_ids that need to be deleted from table_a. See in-line comments for descriptions.
create temporary table table_b_extended as
with tba as ( --Enhance table_a to include a row number.
select
row_number() over (order by sector_value desc) rwn
,*
from
table_a
), tab as ( --Join tables A and B together to attach the sorting key.
select
tba.rwn table_a_rwn
,row_number() over (order by tba.rwn) table_b_rwn
,tbb.*
,case
when tbb.sector_id = tbb.intersected_id
then 1
else 0
end sector_is_intersected
from
tba
join table_b tbb on
tba.sector_id = tbb.sector_id
)
select * from tab
distribute on (sector_id);
-- Find out the row where the intersected id first appears.
create temporary table table_b_first_appearance as
select
intersected_id sector_id
,min(table_b_rwn) first_appearance
from
table_b_extended
where
sector_id <> intersected_id
group by 1
distribute on random;
create temporary table table_a_deletes as
with pid as ( --Get all previous intersected_ids.
select distinct
tab.*
,case --See if this row is after the intersected_id's first appearance.
when app.first_appearance < tab.table_b_rwn then 1
else 0
end sector_in_previous
from
table_b_extended tab
left outer join table_b_first_appearance app using (sector_id)
), vld as ( --Select records that qualify to delete from table_a.
select distinct
intersected_id
from
pid
where
--If it hasn't been seen and isn't equal to the intersected_id, delete it.
sector_is_intersected + sector_in_previous = 0
)
select
*
from
vld
distribute on random;
Given your initial input for table_b:
+------------+----------------+
| sector_id | intersected_id |
+------------+----------------+
| 6 | 6 |
| 1 | 2 |
| 1 | 1 |
| 1 | 4 |
| 4 | 4 |
| 4 | 1 |
| 2 | 6 |
| 2 | 1 |
| 2 | 2 |
| 3 | 3 |
| 6 | 1 |
| 6 | 2 |
+------------+----------------+
This generates a table, table_a_deletes, with two values: 1 and 2. Then deleting from table_a is simple.
delete from table_a tbl
where tbl.sector_id in (select sector_id from table_a_deletes);
And I'm not sure if those DELETED flags in table_b need to be replicated or not, but if so:
delete from table_b tbl
where
tbl.sector_id in (select sector_id from table_a_deletes)
or tbl.sector_id <> tbl.intersected_id;
Performance
On an 8 SPU test system, including the delete steps:
Test 1
table_a size 14976
table_b size 179427095
Runtime 2:50
Test 2
table_a size 14976
table_b size 196240063
Runtime 3:16
Test 3
table_a size 19919
table_b size 317428924
Runtime 5:28
By far the longest time is creating the _extended table. So if you can use some other comparator rather than establishing a new id field, that would be best.
Extra Cases
Here are some different cases to show that it works in a variety of situations. In all cases, I have just modified table_b, since changing table_a is always the trivial case.
Case 1
table_b
+--+-------------+------------------+--+
| | sector_id | intersected_id | |
+--+-------------+------------------+--+
| | 6 | 6 | |
| | 1 | 1 | |
| | 1 | 2 | |
| | 4 | 4 | |
| | 4 | 1 | |
| | 2 | 6 | |
| | 2 | 1 | |
| | 2 | 2 | |
| | 3 | 3 | |
+--+-------------+------------------+--+
Deletes 1 and 2.
Case 2
table_b
+--+-------------+------------------+--+
| | sector_id | intersected_id | |
+--+-------------+------------------+--+
| | 6 | 6 | |
| | 6 | 1 | |
| | 1 | 1 | |
| | 1 | 2 | |
| | 4 | 4 | |
| | 4 | 1 | |
| | 2 | 6 | |
| | 2 | 1 | |
| | 2 | 2 | |
| | 3 | 3 | |
+--+-------------+------------------+--+
Deletes 1.
Case 3
table_b
+--+-------------+------------------+--+
| | sector_id | intersected_id | |
+--+-------------+------------------+--+
| | 6 | 6 | |
| | 1 | 1 | |
| | 1 | 4 | |
| | 4 | 4 | |
| | 4 | 1 | |
| | 2 | 6 | |
| | 2 | 1 | |
| | 2 | 2 | |
| | 3 | 3 | |
+--+-------------+------------------+--+
Deletes 1, 4, and 6.