I'm working in an oracle DB that has 20 tables with the same structure but divided by years. So, it started in ft_expenses_2002 and goes until ft_expenses_2021 (the year when I'm writing this). I need to put all these tables' columns together before doing some maths and my first approach was to use UNIAN ALL statements. It worked but I'm wondering if it's possible to do something more elegant, like using a FOR LOOP. It would not only make the query far more elegant but would avoid future maintenances because every year a new table with the "_new_year" suffix will be created.
Create a view for all the tables:
CREATE VIEW ft_expenses AS
SELECT * FROM ft_expenses_2002
UNION ALL SELECT * FROM ft_expenses_2003
UNION ALL SELECT * FROM ft_expenses_2004
UNION ALL SELECT * FROM ft_expenses_2005
UNION ALL SELECT * FROM ft_expenses_2006
UNION ALL SELECT * FROM ft_expenses_2007
UNION ALL SELECT * FROM ft_expenses_2008
UNION ALL SELECT * FROM ft_expenses_2009
UNION ALL SELECT * FROM ft_expenses_2010
UNION ALL SELECT * FROM ft_expenses_2011
-- ...
UNION ALL SELECT * FROM ft_expenses_2021
Then just do your query using the view.
Next year when you add a 2022 table then recreate the view with the extra table added to the view.
Alternatively, create a table from the originals so that everything is in one table that you can query directly:
CREATE TABLE ft_expenses (year, col1, col2, col3) AS
SELECT 2002, col1, col2, col3 FROM ft_expenses_2002
UNION ALL SELECT 2003, col1, col2, col3 FROM ft_expenses_2003
UNION ALL SELECT 2004, col1, col2, col3 FROM ft_expenses_2004
UNION ALL SELECT 2005, col1, col2, col3 FROM ft_expenses_2005
UNION ALL SELECT 2006, col1, col2, col3 FROM ft_expenses_2006
UNION ALL SELECT 2007, col1, col2, col3 FROM ft_expenses_2007
UNION ALL SELECT 2008, col1, col2, col3 FROM ft_expenses_2008
-- ...
UNION ALL SELECT 2021, col1, col2, col3 FROM ft_expenses_2021
Then drop the individual tables (make sure you backed everything up first) and create views if you still need to access then by the original names:
CREATE VIEW ft_expenses_2002 (col1, col2, col3) AS
SELECT col1, col2, col3 FROM ft_expenses WHERE year = 2002;
CREATE VIEW ft_expenses_2003 (col1, col2, col3) AS
SELECT col1, col2, col3 FROM ft_expenses WHERE year = 2003;
-- ...
CREATE VIEW ft_expenses_2021 (col1, col2, col3) AS
SELECT col1, col2, col3 FROM ft_expenses WHERE year = 2021;
I've found here a really good and short way to solve my problem, it was:
SELECT
GROUP_CONCAT(
CONCAT(
'SELECT * FROM `',
TABLE_NAME,
'`') SEPARATOR ' UNION ALL ')
FROM
`INFORMATION_SCHEMA`.`TABLES`
WHERE
`TABLE_NAME` LIKE 'ft_expenses_%'
INTO #sql;
PREPARE stmt FROM #sql;
EXECUTE stmt;
Depends on your definition of "elegant".
It's certainly possible to use dynamic SQL to look for every table whose name fits a particular pattern. My guess is that the simplest thing to do would be to create a view that does the union all, write your processing code against that view, and then have a bit of dynamic SQL that can rebuild the view based on what tables exist. You could then run that procedure when new tables are created (if you can hook in to that process) or based on a DDL trigger or just schedule it to run in the early morning hours every year/ month/ day depending on how likely it is that a new table is going to suddenly show up.
create or replace procedure build_view
as
l_sql_stmt varchar2(32000);
type typ_table_names is table of varchar2(256);
l_tables typ_table_names;
begin
l_sql_stmt := 'create or replace view my_view as ';
select table_name
bulk collect into l_tables
from user_tables
where table_name like 'ft_expenses%';
for i in 1 .. l_tables.count
loop
l_sql_stmt := l_sql_stmt ||
' select * from ' || l_tables(i);
if( i != 1_tables.count )
then
l_sql_stmt := l_sql_stmt ||
' union all ';
end if;
end loop;
dbms_output.put_line( l_sql_stmt );
execute immediate l_sql_stmt;
end;
You physical setup of the tables is state of the art of Oracle 6 (late 1980's).
You have two possibilities to upgrade. Either the DIY union all view as proposed in other answers, or simple follow the development the Oracle implemented since the above mentioned version.
Note, I'd not recommend to follow the advise to put all data in one table - why? You'd see no difference in the queries covering all data, but you'll spot a significant decrease in performance in queries on one or few years.
What was done in Oracle on this topic since the release 6?
In Oracle 7 (1990's) a partitioned views was introduced - which is similar idea to the proposal of the UNION ALL view.
Starting with Oracle 8 partitioning concept was introduced any improved in each following release.
So if you want to leverage the current Oracle features the partitioning should be applied:
managing the data in acceptable size
providing a flexibility in the access
Here an example how could you migrate in Oracle 19c
I assume that you table contains a column trans_dt containing a DATE with a year same as the tables year.
Start with the oldest table and change it to a partitioned table
alter table ft_expenses_2002 modify
partition by range(trans_dt) interval(NUMTOYMINTERVAL(1,'YEAR'))
( partition p_init values less than (DATE'2002-01-01')
) online
Rename the table eliminating the year
rename ft_expenses_2002 to ft_expenses;
Now the table in partitioned and contains two partitions, the initial one and the partition for the 2002 year.
select PARTITION_NAME from user_tab_partitions where table_name = 'FT_EXPENSES' order by PARTITION_POSITION;
PARTITION_NAME
----------------
P_INIT
SYS_P2340
For each following years perform the below steps
Add a new partition
alter table ft_expenses
exchange partition for( DATE'2003-01-01' ) with table ft_expenses_2003
Note that you use the for syntax to address the partition, so there is no need to know the partition name.
Additionally the recent version can create the partition in the exchange statement, so there is no need to lock table anymore.
Final Notes
You may include indexes in the reorganization.
As always backup all tables before start.
Test carefully to check possible limitations.
Related
I need to move some data from Environment A to Environment B. So far, so easy. But some of the columns have FK Constraints and unfortunately the lookup data is already on Environment B and has different PKs. Lucky me, there are other unique columns I could do a mapping on. Therefore I wonder if SQL Developer has an export feature which allows me to replace certain column values by subqueries. To make it clear, I'm looking for a SQL Developer Feature, Query or similar which generates INSERT Statements that look like this:
INSERT INTO table_name(col1, col2, fkcol)
VALUES('value', 'value', (SELECT id FROM lookup_table WHERE unique_value = 'unique'))
My best approach was to try to generate them by hand, like this:
SELECT
'INSERT INTO table_name(col1, col2, fkcol) '
|| 'VALUES( '
|| (SELECT LISTAGG(col1, col2,
'SELECT id FROM lookup_table
WHERE unique_value = ''' || lookup_table.uniquevalue || '''', ', ')
WITHIN GROUP(ORDER BY col1)
FROM table_name INNER JOIN lookup_table ON table_name.fkcol = lookup_table.id)
|| ' );'
FROM table_name;
Which is absolutely a pain. Do you know something better to achive this without approaching the other db?
Simple write a query that produces the required data (with the mapped key) using a join of both tables.
For example (see the sample data below) such query (mapping the unique_value to the id):
select
tab.col1, tab.col2, lookup_table.id fkcol
from tab
join lookup_table
on tab.fkcol = lookup_table.unique_value
COL1 COL2 FKCOL
---------- ------ ----------
1 value1 11
2 value2 12
Now you can use the normal SQL Developer export feature in the INSERT format, which would yield following script - if you want to transer it to other DB or insert it direct with INSERT ... SELECT.
Insert into TABLE_NAME (COL1,COL2,FKCOL) values ('1','value1','11');
Insert into TABLE_NAME (COL1,COL2,FKCOL) values ('2','value2','12');
Sample Data
select * from tab;
COL1 COL2 FKCOL
---------- ------ -------
1 value1 unique
2 value2 unique2
select * from lookup_table
ID UNIQUE_
---------- -------
11 unique
12 unique2
I am facing the following situation:
I import an Excel-Sheet, then some columns are modified (e.g. "comments")
After a while, I would receive an updated Excel-Sheet containing the records from the old Excel-sheet as well as new ones.
I do not want to import the records that already exist in the database.
Step-by-Step:
Initial Excel-sheet
col1 col2 comments
A A
A B
After import, some fields will get manipulated
col1 col2 comments
A A looks good
A B fine with me
Then I receive an excel sheet with updates
col1 col2 comments
A A
A B
A C
After this update-step, the database should look like
col1 col2 comments
A A looks good
A B fine with me
A C
I was planning to simply create a unique index on all fields that won't get manipulated, so only the new records will get imported. (sth like
ALTER TABLE tbl ADD CONSRAINT unique_key UNIQUE (col1,col2)
My problem now is that Access somehow only allows composite indices of max. 10 fields. My tables all have around 11-20 cols...
I could maybe import the updated xls to a temp. table, and do s.th like
INSERT INTO tbl_old SELECT col1,col2, "" FROM tbl_new WHERE (col1,col2) NOT IN (SELECT col1,col2 FROM tbl_old UNION SELECT col1,col2 FROM tbl_new)
But I'm wondering if there isn't a more straigt-forward way...
Any ideas how I can solve that?
Try the EXISTS condition:
INSERT INTO tbl_old (col1, col2, comments)
SELECT col1, col2, Null
FROM tbl_new
WHERE NOT EXISTS (SELECT col1, col2 FROM tbl_old WHERE tbl_old.col1 = tbl_new.col1 AND tbl_old.col2 = tbl_new.col2);
Considering you will use SQL approach:
INSERT INTO table_old (col1, col2)
SELECT col1, col2 FROM table_new
EXCEPT
SELECT col1, col2 FROM table_old
:)
It will insert null in comments column though. Use this:
INSERT INTO table_old
SELECT * FROM table_new
EXCEPT
SELECT * FROM table_old
to avoid null values. Also both tables have to have the same amount of columns. For Oracle go with minus instead of except. Equivalent SQL query would be made with LEFT OUTER JOIN.
INSERT INTO table_old (col1 , col2)
SELECT N.col1, N.col2
FROM table_new N
LEFT OUTER JOIN table_old O ON O.col2 = N.col2
WHERE O.col2 IS NULL
Which will also provide null values to comments column, as we are inserting only col1 and col2. All inserts tested on provided table examples.
I would just put PK ID column in those tables.
I have a 50 tables with similar structures with name TABLE_1, TABLE_2, TABLE_3 etc. I want to select some information like SELECT * FROM TABLE WHERE CRM_ID = 100 but I dont know which table consists this ID. I guess that I should make a union with this tables and make a query from this union but I am afraid that it is not the best solution. Could anybody help me?
To find the table containing the field CRM_ID, you can use:
select TABLE_NAME
from SYS.ALL_TAB_COLUMNS
where COLUMN_NAME = 'CRM_ID'
From here, you can query the relevant tables with your union
select 'table_1', count(*) from table_1 where CMR_ID = 100
union all
select 'table_2', count(*) from table_2 where CMR_ID = 100
[.....]
If you have 50 tables, you may want to use some advanced text editing so you don't have to type the same line 50 times.
I can't see anything different from a UNION to get what you need.
However, I would like to avoid repeating all the code every time you need to query your tables, for example by creating a view:
create view all_my_tables as
select * from table1
union
select * from table2
...
and then
select *
from all_my_tables
where crm_id = 100
I would
create a view table_all as
select * from table_1 union all
select * from table_2 ...
but I guess you have 50 tables due to some performance reason like creating partitioning without partitioned table. If that is the case you need some table that simulates index and you need to have data "sorted". Then create table that contains: table_name, min_val, max_val:
table_1, 1, 1000
table_2, 1001, 2000
and procedure selecting looks like:
procedure sel(crmid) is
a varchar2(10);
value table%rowtype;
begin
select table_name into a from lookup where crmid between min_val and max_val;
execute immediate 'select * from ' || a || ' where crm_id = ' || crmid into value;
--do logic with value
end;
but if data is not ordered you need to iterate selects in loop. In such case use view.
Try this:
Select tmp.tablename from (
select 'table1' as tablename from table1 where CRM_ID=100
union all
select 'table2' as tablename from table2 where CRM_ID=100
) as tmp
I have a table with data in the following format
col1 col2
a1 a2;a3;a4
b1 b2
c1 c2;c3
d1 null
...
I'm trying to split the strings, get unique combinations of col1/col2 and insert them into tableB. So the expected outcome should look like this:
a1 a2
a1 a3
a1 a4
b1 b2
c1 c2
c1 c3
...
I tried the following query:
INSERT INTO tableB (col1, col2)
SELECT col1, (regexp_substr(col2,'[^;]+', 1, LEVEL)) FROM tableA
CONNECT BY regexp_substr(col2, '[^;]+', 1, LEVEL) IS NOT NULL;
Not sure what's going wrong here but it keeps executing (it actually went on for more than an hour) and when I finally cancel the task, nothing's been inserted. The table is quite large (around 25000 rows) but I've done similar inserts with larger tables and they worked fine.
I also tried adding a where clause (although it seems redundant) with
WHERE col2 LIKE'%;%'
That didn't help either.
Any suggestions would be great.
Edit: I tried counting the max number of substrings in col2, to ballpark the number of rows to be inserted, and found the max to be 42 substrings. The whole table has 25814 rows, so worst case scenario, it's inserting 1084104 rows. If that has anything to do with it.
Don't use connect by to split string into rows.
Use a PL/SQL procedure that does varchar2 -> collection split.
For a ad-hoc kind of query, stick with xmltable as a simple way to split string into rows (it is a bit slower than PL/SQL).
The following kind of query is expected to take 3-4 seconds for each input 1000 rows.
select t.col1, c2.val
from (
select 'a1' col1, 'a2;a3;a4' col2 from dual union all
select 'b1', 'b2' from dual union all
select 'c1', 'c2;c3' from dual union all
select 'd1', null from dual
) t
, xmltable('$WTF' passing
xmlquery(('"'||replace(replace(t.col2,'"','""'),';','","')||'"')
returning sequence
) as wtf
columns val varchar2(4000) path '.'
)(+) c2
Fiddle: http://sqlfiddle.com/#!4/9eecb7d/5059
One thing you can try is to select all the distinct values.
INSERT INTO tableB (col1, col2)
SELECT distinct col1, (regexp_substr(col2,'[^;]+', 1, LEVEL))
FROM tableA
CONNECT BY regexp_substr(col2, '[^;]+', 1, LEVEL) IS NOT NULL;
commit;
You should also commit the transaction if you need the changes to be permanent.
While I still don't really understand what was wrong with the initial query, I found a work around. The column with the strings I'm trying to split is participantcountries and the p_id contains unique identifiers.
CREATE OR REPLACE PROCEDURE split_countries IS
CURSOR get_rows IS
SELECT p_id, participantcountries
FROM staging_projects;
row_rec get_rows%ROWTYPE;
nb_countries number:=0;
country VARCHAR2(4);
BEGIN
OPEN get_rows;
LOOP
FETCH get_rows INTO row_rec;
EXIT WHEN get_rows%NOTFOUND;
SELECT regexp_count(row_rec.participantcountries, ';') INTO nb_countries
FROM staging_projects
WHERE p_id=row_rec.p_id;
IF (row_rec.participantcountries IS NULL) THEN
nb_countries:=0; -- if the field is null set counter to zero
ELSE
nb_countries:=nb_countries+1; -- nb of delimiters so +1 --> number of items
END IF;
WHILE (nb_countries >0) LOOP -- loop and get single country at current counter position
SELECT (regexp_substr(PARTICIPANTCOUNTRIES,'[^;]+', 1, nb_countries)) INTO country
FROM (SELECT * FROM staging_projects WHERE p_id=row_rec.p_id)
WHERE participantcountries IS NOT NULL;
nb_countries:=nb_countries-1;
INSERT INTO project_countries(proj_id, country_code) VALUES(row_rec.p_id, country);
END LOOP;
END LOOP;
CLOSE get_rows;
END split_countries;
This works fine but it seems like an overly complicated solution for the problem at hand.
I have some tables with the same structure and I want to make a select in a group of them.
Rather than just making a loop to all of those tables, I would like to put a subquery after the FROM of the main query.
Is it possible or it will fail?
Thanks!
(Using Oracle)
Additional info: I don't have the name of the table right away! They're stored in another table. Is it possible to have a subquery that I could put after the FROM of my main query?
"I don't have the name of the table
right away! They're stored in another
table"
Oracle doesn't do this sort of thing in SQL. You'll need to use PL/SQL and assemble a dynamic query.
create or replace function get_dynamic_rows
return sys_refcursor
is
stmt varchar2(32767) := null;
return_value sys_refcursor;
begin
for r in ( select table_name from your_table )
loop
if stmt is not null then
stmt := stmt||' union all ';
end if;
stmt := stmt||'select * from '||r.table_name;
end loop;
open return_value for stmt;
return return_value;
end;
/
This will assemble a query like this
select * from table_1 union all select * from table_2
The UNION ALL is a set operator which combines the output of several queries in a single result set without removing duplicates. The columns in each query must match in number and datatype.
Because the generated statement will be executed automatically there's no real value in formatting it (unless the actual bits of the query are more complicated and you perhaps need to debug it).
Ref Cursors are PL/SQL contructs equivalent to JDBC or .Net ResultSets. Find out more.
Sure, just union them together:
select * from TableA
union all
select * from TableB
union all
select * from TableC
You can union in a subquery:
select *
from (
select * from TableA
union all
select * from TableB
) sub
where col1 = 'value1'
Use union if you're only interested in unique rows, and union all if you want all rows including duplicates.