I'm searching a way to do a recursive delete on a table.
The situation is that table have 3 foreign key 1 on itself and 2 others, I want to delete depending on the date of the occurrence.
Table1 --> Id1, dateOCC, ParentID
1, 13-12-26, null
2, 13-07-18, null
3, 14-12-31, 1
4, 13-06-26, 1
5, 14-07-23, null
6, 13-07-22, 2
Table2--> ID, stuff
Table3 --> ID, stuff
The ID of Table 2 and Table 3 are linked directly on ID of Table1.
The amount of data inside table 1 is approximately 20 000 000 row and the others table is approximately the same amount.
Here is on of the request I tried(its inside of a cursor who delete the data returned.
SELECT EO.ID,
EO.DATEOCC,
EO.PARENTID
FROM TABLE1 EO
WHERE EO.DATEOCC <= TO_DATE ('2013-12-31','YYYY-MM-DD')
AND NOT EXISTS(SELECT 1 FROM TABLE2 WHERE ID = EO.ID)
AND NOT EXISTS( SELECT 1 FROM TABLE3 WHERE ID = EO.ID)
START WITH EO.PARENTID IS NULL
CONNECT BY PRIOR EO.ID = EO.PARENTID;
This request is really really slow to output the data that I want.
And it seems that is not return the data that I need to delete.
Edit #1
Ok so heres an example of what I need to do(In this example I suppose that the table 2 and table 3 have no matching ID on Table 1)
Table1 --> Id1, dateOCC, ParentID
1, 13-12-26, null
2, 13-07-18, null
3, 14-12-31, 1
4, 13-06-26, 1
5, 14-07-23, null
6, 13-07-22, 2
After the delete sequence the table have to be like that if the >= date is 13-12-31
Table1 --> Id1, dateOCC, ParentID
1, 13-12-26, null
3, 14-12-31, 1
5, 14-07-23, null
So as you can see I delte the child that I can delete with his parent if possible. If I cant delete his parent because another child exist and I cant delete it I dont delete de parent(delete only the child that I can).
In a hierarchical query, the WHERE clause is applied after the START WITH and CONNECT BY are used to build the hierarchy. But syntactically it comes first, which makes it intuitively seem that it will be applied first.
If what you really want is to apply the WHERE clause first, then build the hierarchy, you can use a subquery like this:
SELECT EO.ID,
EO.DATEOCC,
EO.PARENTID
FROM (
SELECT * FROM TABLE1 EO
WHERE EO.DATEOCC <= TO_DATE ('2013-12-31','YYYY-MM-DD')
AND NOT EXISTS(SELECT 1 FROM TABLE2 WHERE ID = EO.ID)
AND NOT EXISTS( SELECT 1 FROM TABLE3 WHERE ID = EO.ID)
) EO
START WITH EO.PARENTID IS NULL
CONNECT BY PRIOR EO.ID = EO.PARENTID;
But it is not clear whether that is what you want. This would give you the top-level parents within the desired date range, and without children in the other tables, then build the entire hierarchy for those parents. It's possible that lower nodes in the hierarchy would have children in the other tables, which would cause the delete to fail.
If that's not what you want, I think you need to describe your requirements more clearly.
Related
Im struggling how to migrate datas using .sql script I'm quite new to SQL and trying to figure out how to migrate data's purely on .SQL. I want to add my old data to the new table as a new record with a different structure
Here's my case: I have old two tbl and i want to merge it to my new structured tbl with an additional columns. I'm kinda stuck here since I'm not used in using conditional on .SQL
Prefixes of the tables are schemas
Old table
old.groups
id
group_name
10
Apex
11
Pred
12
Tor
old.sub_groups
parent_id
sub_group
10
sub-apex
11
sub-pred
11
sub-sub-pred
New Table:
Expected Migrated Data
public.new_groups *id is auto incremented
Fresh New populated table
id
group_name
level
parent_id
0
Apex
1
10
1
Pred
1
11
2
Tor
null
null
3
sub-apex
2
10
4
sub-pred
2
11
5
sub-sub-pred
2
11
I want to merge it with conditions. but i can't keep up with SQL queries
Condition 1: If old.groups.id doesn't detect any match on old.sub_groups.parent_id it will be inserted to public.new_groups but the public.new_groups.level and public.new_groups.parent_id will be default to null.
Condition 2: If old.groups.id detects a match on old.sub_groups.parent_id it will be also inserted to public.new_groups then tag the level as 1 (1 means parent group in my structure) but with another new three inserted records which is the sub_groups it detected refer to tbl.new_groups id [3, 4, and 5] and tag the level as 2. and the parent_id will be the parent_id of the old.sub_groups or the id of the parent in old.groups
This is my unfinished Query im only able to call the data its missing out the conditional and the update but i think this is also wrong:
INSERT INTO public.new_groups(
SELECT *, b.sub_group as group_name, b.parent_id FROM old.groups as a
LEFT JOIN old.sub_groups as b ON a.id = b.parent_id....
)
When you created your table like this:
CREATE TABLE new (
id SERIAL PRIMARY KEY ,
group_name VARCHAR(20),
level INTEGER,
parent_id INTEGER
);
You can copy the tables with this statement:
INSERT INTO new(group_name, level, parent_id)
SELECT DISTINCT
group_name,
CASE WHEN subgroups.parent_id IS NULL THEN NULL ELSE 1 END as level,
subgroups.parent_id
FROM old
LEFT JOIN subgroups ON old.id = subgroups.parent_id
UNION ALL
SELECT
sub_group,
2,
parent_id
FROM subgroups;
see: DBFIDDLE
just my id starts with 1, and not with 0.
I have a table that looks essentially like this (the first row pk1=1 is the parent row)
pk1
event_id
parent_event_id
1
123
123
2
456
123
3
789
456
Given any particular row in the above table, I need a query that returns all the related rows (up and down the hierarchy). I was trying to do this via an initial CTE table that grabs all the parent rows. Then use that as my base table and join back into the above table using a recursive query to navigate down (this seems wildly inefficient and I assume there is a better way???).
However, trying even the first step (populating my CTE table) and using a query like below to navigate up returns the connect by LOOP error.
select event_id, level
from myTable
start with pk1 = 2
connect by prior parent_event_id = event_id
I assume this is due to the fact the parent row is self-referencing (event_id = parent_event_id)? If I add in the NOCYCLE statement, then the recursion stops at the row prior to the actual parent.
Two questions:
1.) Is there a better way to do this in one query?
2.) Any clue how to tweak the above to get the parent row returned?
Thanks
I'm not super clear on what you mean by "all the related rows (up and down the tree)", but it might be possible.
Here, I'm adding more logic to the connect clause to go up OR down the tree. This includes direct parents and descendants, but also includes siblings/cousins to the starting node. That might or might not be what you want.
with mytable as (select 1 as pk1, 123 as event_id, 123 as parent_event_id from dual
union select 2, 456, 123 from dual
union select 3, 789, 456 from dual
union select 4, 837, 123 from dual)
select pk1, event_id, level, SYS_CONNECT_BY_PATH(event_id, '/') as path
from myTable
start with pk1 = 2
connect by nocycle (prior parent_event_id = event_id and prior event_id <> event_id)
or (prior event_id = parent_event_id)
The tweak to get the root parent to show up is just and prior event_id <> event_id - ie, don't go further up the tree if the parent node = the current node.
I added an example row (pk1=4) to show a sibling row (not direct parent or descendant) being returned.
I have an Oracle table that represents parent-child relationships, and I want to improve the performance of a query that searches the hierarchy for an ancestor record. I'm testing with the small data set here, though the real table is much larger:
id name parent_id tagged
== ==== ========= ======
1 One null null
2 Two 1 1
3 Three 2 null
4 Four 3 null
5 Five null null
6 Six 5 1
7 Seven 6 null
8 Eight null null
9 Nine 8 null
parent_id refers back to id in this same table in a foreign key relationship.
I want to write a query that returns each leaf record (those records that have no descendants... id 4 and id 7 in this example) which has an ancestor record that has tagged = 1 (walking back through the parent_id relationship).
So, for the above source data, I want my query to return:
id name tagged_ancestor_id
== ==== ==================
4 Four 2
7 Seven 6
My current query to retrieve these records is:
select * from (
select id,
name,
connect_by_root id tagged_ancestor_id
from mytree
connect by prior id = parent_id
start with tagged is not null
) m1
where not exists (
select * from mytree m2 where m2.parent_id = m1.id
)
This query works fine on this simple little example table, but its performance is terrible on my real table which has about 11,000,000 records. The query takes over a minute to run.
There are indexes on both fields in the connect by clause.
The "tagged" field in the start with clause also has an index on it, and there are about 1,500,000 records in my table with non-null values in this field.
The where clause doesn't seem to be the problem, because when modify it to return a specific name (also indexed) with where name = 'somename' instead of where not exists ..., the query still takes about the same amount of time.
So, what are some strategies I can use to try to make these types queries on this hierarchy run faster?
Here is what I would check first:
Make sure your table has a primary key.
Make sure the statistics on the table are current. Use DBMS_STATS.GATHER_TABLE_STATS to collect the statistics. See this URL: (for ORACLE version 11.1):
http://docs.oracle.com/cd/B28359_01/appdev.111/b28419/d_stats.htm
Even if you have indexes on both fields individually, you still need
an index on the 2 fields combined; Create an index on the ID and PARENT_ID:
CREATE INDEX on TABLE_NAME(ID, PARENT_ID);
See this URL:
Optimizing Oracle CONNECT BY when used with WHERE clause
Make sure the underlying table does not have row chaining or other problems (E.G. corruption).
Make sure the table and all indexes are in the same tablespace.
I'm not sure if this is any faster without the volume of data to test with... but something to consider. I guess I'm hoping by starting with only those that are tagged, and only those that are leafs we are dealing with a smaller volume to process which may result in a performance gain. but the overhead for the string manipulation seems hackish.
with cte(id, name, parent_id, tagged) as (
SELECT 1, 'ONE', null, null from dual union all
SELECT 2, 'TWO', 1, 1 from dual union all
SELECT 3, 'THREE', 2, null from dual union all
SELECT 4, 'FOUR', 3, null from dual union all
select 5, 'FIVE', null, null from dual union all
select 6, 'SIX', 5, 1 from dual union all
select 7, 'SEVEN', 6, null from dual union all
select 8, 'EIGHT', null, null from dual union all
select 9, 'NINE', 8, null from dual),
Leafs(id, name) as (select id, Name
from cte
where connect_by_isleaf = 1
Start with parent_Id is null
connect by nocycle prior id =parent_id),
Tagged as (SELECT id, name, SYS_CONNECT_BY_PATH(ID, '/') Path, substr(SYS_CONNECT_BY_PATH(ID, '/'),2,instr(SYS_CONNECT_BY_PATH(ID, '/'),'/',2)-2) as Leaf
from cte
where tagged=1
start with id in (select id from leafs)
connect by nocycle prior parent_id = id)
select l.*, T.ID as Tagged_ancestor from leafs L
inner join tagged t
on l.id = t.leaf
In essence I created 3 cte's one for the data (Cte) one for the leafs(leafs) and one for the tagged records (tagged)
We traverse the hierarchy twice. Once to get all the leafs, once to get all the tagged. We then parse out the first leaf value from the tagged hierarchy and join it back to leafs to get the leafs related to tagged records.
As to if this is faster than what you're doing... Shrug I didn't want to spend the time testing since I don't have your indexes nor do I have your data volume
The problem:
I have a table that records data rows in foo. Each time the row is updated, a new row is inserted along with a revision number. The table looks like:
id rev field
1 1 test1
2 1 fsdfs
3 1 jfds
1 2 test2
Note: the last record is a newer version of the first row.
Is there an efficient way to query for the latest version of a record and for a specific version of a record?
For instance, a query for rev=2 would return the 2, 3 and 4th row (not the replaced 1st row though) while a query for rev=1 yields those rows with rev <= 1 and in case of duplicated ids, the one with the higher revision number is chosen (record: 1, 2, 3).
I would not prefer to return the result in an iterative way.
To get only latest revisions:
SELECT * from t t1
WHERE t1.rev =
(SELECT max(rev) FROM t t2 WHERE t2.id = t1.id)
To get a specific revision, in this case 1 (and if an item doesn't have the revision yet the next smallest revision):
SELECT * from foo t1
WHERE t1.rev =
(SELECT max(rev)
FROM foo t2
WHERE t2.id = t1.id
AND t2.rev <= 1)
It might not be the most efficient way to do this, but right now I cannot figure a better way to do this.
Here's an alternative solution that incurs an update cost but is much more efficient for reading the latest data rows as it avoids computing MAX(rev). It also works when you're doing bulk updates of subsets of the table. I needed this pattern to ensure I could efficiently switch to a new data set that was updated via a long running batch update without any windows of time where we had partially updated data visible.
Aging
Replace the rev column with an age column
Create a view of the current latest data with filter: age = 0
To create a new version of your data ...
INSERT: new rows with age = -1 - This was my slow long running batch process.
UPDATE: UPDATE table-name SET age = age + 1 for all rows in the subset. This switches the view to the new latest data (age = 0) and also ages older data in a single transaction.
DELETE: rows having age > N in the subset - Optionally purge old data
Indexing
Create a composite index with age and then id so the view will be nice and fast and can also be used to look up by id. Although this key is effectively unique, its temporarily non-unique when you're ageing the rows (during UPDATE SET age=age+1) so you'll need to make it non-unique and ideally the clustered index. If you need to find all versions of a given id ordered by age, you may need an additional non-unique index on id then age.
Rollback
Finally ... Lets say you're having a bad day and the batch processing breaks. You can quickly revert to a previous data set version by running:
UPDATE table-name SET age = age - 1 -- Roll back a version
DELETE table-name WHERE age < 0 -- Clean up bad stuff
Existing Table
Suppose you have an existing table that now needs to support aging. You can use this pattern by first renaming the existing table, then add the age column and indexing and then create the view that includes the age = 0 condition with the same name as the original table name.
This strategy may or may not work depending on the nature of technology layers that depended on the original table but in many cases swapping a view for a table should drop in just fine.
Notes
I recommend naming the age column to RowAge in order to indicate this pattern is being used, since it's clearer that its a database related value and it complements SQL Server's RowVersion naming convention. It also won't conflict with a column or view that needs to return a person's age.
Unlike other solutions, this pattern works for non SQL Server databases.
If the subsets you're updating are very large then this might not be a good solution as your final transaction will update not just the current records but all past version of the records in this subset (which could even be the entire table!) so you may end up locking the table.
This is how I would do it. ROW_NUMBER() requires SQL Server 2005 or later
Sample data:
DECLARE #foo TABLE (
id int,
rev int,
field nvarchar(10)
)
INSERT #foo VALUES
( 1, 1, 'test1' ),
( 2, 1, 'fdsfs' ),
( 3, 1, 'jfds' ),
( 1, 2, 'test2' )
The query:
DECLARE #desiredRev int
SET #desiredRev = 2
SELECT * FROM (
SELECT
id,
rev,
field,
ROW_NUMBER() OVER (PARTITION BY id ORDER BY rev DESC) rn
FROM #foo WHERE rev <= #desiredRev
) numbered
WHERE rn = 1
The inner SELECT returns all relevant records, and within each id group (that's the PARTITION BY), computes the row number when ordered by descending rev.
The outer SELECT just selects the first member (so, the one with highest rev) from each id group.
Output when #desiredRev = 2 :
id rev field rn
----------- ----------- ---------- --------------------
1 2 test2 1
2 1 fdsfs 1
3 1 jfds 1
Output when #desiredRev = 1 :
id rev field rn
----------- ----------- ---------- --------------------
1 1 test1 1
2 1 fdsfs 1
3 1 jfds 1
If you want all the latest revisions of each field, you can use
SELECT C.rev, C.fields FROM (
SELECT MAX(A.rev) AS rev, A.id
FROM yourtable A
GROUP BY A.id)
AS B
INNER JOIN yourtable C
ON B.id = C.id AND B.rev = C.rev
In the case of your example, that would return
rev field
1 fsdfs
1 jfds
2 test2
SELECT
MaxRevs.id,
revision.field
FROM
(SELECT
id,
MAX(rev) AS MaxRev
FROM revision
GROUP BY id
) MaxRevs
INNER JOIN revision
ON MaxRevs.id = revision.id AND MaxRevs.MaxRev = revision.rev
SELECT foo.* from foo
left join foo as later
on foo.id=later.id and later.rev>foo.rev
where later.id is null;
How about this?
select id, max(rev), field from foo group by id
For querying specific revision e.g. revision 1,
select id, max(rev), field from foo where rev <= 1 group by id
Take this table:
id name sub_id
---------------------------
1 A (null)
2 B (null)
3 A2 1
4 A3 1
The sub_id column is a relation to his own table, to column ID.
subid --- 0:1 --- id
Now I have the problem to make a correctly SELECT query to show that the child rows (which sub_id is not null) directly selected under his parent row. So this must be a correctly order:
1 A (null)
3 A2 1
4 A3 1
2 B (null)
A normal SELECT order the id. But how or which keyword help me to order this correctly?
JOIN isn't possible I think because I want to get all the rows separated. Because the rows will be displayed on a Gridview (ASP.Net) with EntityDataSource but the child rows must be displayed directly under his parent.
Thank you.
Look at Managing Hierarchical Data in MySQL.
Since recursion is an expensive operation because basicly you're firing multiple queries to your database you could consider using the Nested Set Model. In short you're assigning numbers to ranges in your table. It's a long article but it worth reading it. I've used it during my internship as a solution not to have 1000+ queries, But bring it down to 1 query.
Your handling 'overhead' now lies at the point of updating the table by adding, updating or deleting records. Since you then have to update all the records with a bigger 'right-value'. But when you're retrieving the data, it all goes with 1 query :)
select * from table1 order by name, sub_id will in this case return your desired result but only because the parents names and the child name are similar. If you're using SQL 2005 a recursive CTE will work:
WITH recurse (id, Name, childID, Depth)
AS
(
SELECT id, Name, ISNULL(childID, id) as id, 0 AS Depth
FROM table1 where childid is null
UNION ALL
SELECT table1.id, table1.Name, table1.childID, recurse.Depth + 1 AS Depth FROM table1
JOIN recurse ON table1.childid = recurse.id
)
SELECT * FROM recurse order by childid, depth
SELECT
*
FROM
table
ORDER BY
COALESCE(id,sub_id), id
btw, this will work only for one level.. any thing more than that requires recursive/cte function