When I run that query in GBQ I receive this error:
Error running query
Resources exceeded during query execution: Not enough resources for
query planning - too many subqueries or query is too complex.. at
[1:1]
The code below is not developed by me so I have a hard time rewriting it. So would hear if you could help me get it running again.
execute immediate (select concat('create or replace table `ssmdw-208309.temp_piwik_and_ga.005_visitor_union` as ',query) from (select string_agg(CONCAT(' select substr(generate_uuid(),25,12) as rowid, ',columns,' from `piwik-272210.piwik.',table_name,'`'),'\nUnion all\n') query FROM (
SELECT
table_column.table_name,
STRING_AGG(CASE
WHEN x.column_name IS NULL THEN CONCAT('null as ',table_column.column_name)
ELSE
table_column.column_name
END
,', '
ORDER BY
table_column.column_name) columns
FROM (
SELECT
tables.table_name,
columns.column_name
FROM (
SELECT
DISTINCT column_name
FROM
`piwik-272210.piwik.INFORMATION_SCHEMA.COLUMNS` a where table_name like 'visitor_%') columns
FULL OUTER JOIN (
SELECT
DISTINCT table_name
FROM
`piwik-272210.piwik.INFORMATION_SCHEMA.COLUMNS` b where table_name like 'visitor_%') tables
ON
(1=1)) table_column
LEFT JOIN
`piwik-272210.piwik.INFORMATION_SCHEMA.COLUMNS` x
ON
(x.table_name = table_column.table_name
AND x.column_name = table_column.column_name)
GROUP BY
1 )))
Avoid "order by" clause wherever it is possible. "Order by" tries to pull result in one node that in general causes this issue.
Try this. then call the procedure.
create or replace procedure `dataset_name'.'procedure_name'()
BEGIN
create temp table TABL1 as (
SELECT DISTINCT column_name FROM `piwik-272210.piwik.INFORMATION_SCHEMA.COLUMNS` a where table_name like 'visitor_%');
create temp table TABL2 as (
SELECT DISTINCT table_name FROM `piwik-272210.piwik.INFORMATION_SCHEMA.COLUMNS` b where table_name like 'visitor_%');
create temp table TABL3 as
SELECT tables.table_name, columns.column_name FROM (select column_name from TABL1 columns
FULL OUTER JOIN
select table_name from TABL2 tables ON (1=1));
create or replace table dataset.TABL4 as
SELECT table_column.table_name,
STRING_AGG(CASE WHEN x.column_name IS NULL THEN CONCAT('null as ',table_column.column_name) ELSE table_column.column_name END) as columns
FROM (
SELECT table_name, column_name FROM TABL3
) table_column
left join `piwik-272210.piwik.INFORMATION_SCHEMA.COLUMNS` x
ON (x.table_name = table_column.table_name AND x.column_name = table_column.column_name)
))
drop table TABL1 ;
drop table TABL2 ;
drop table TABL3;
END;
Related
I have a table in which one column represent a name of a table in my db.
TableA
| tableName |
-------------
| table 1 |
| table 2 |
| ....... |
| table n |
What I need to do is to count all the records that are inside each table listed in tableA.
What I managed to do so far is this:
select count(*)
from (
select tableName
from tableA
) tables
but this is wrong because it counts the number of rows in the tableA table, which is not what I need.
The list of table don't have any relationship pretty much so there are no join operations, I just need to add to a counter the number of all records in each table.
Can that be done with a plain sql query? I'm using postgresql but would like to come up with a solution that doesn't depend on any db vendor.
select sum(row_count) as totalRecords
from(
select table_schema,
table_name,
(xpath('/row/cnt/text()', xml_count))[1]::text::int as row_count
from (
select table_name, table_schema,
query_to_xml(format('select count(*) as cnt from %I.%I', table_schema, table_name), false, true, '') as xml_count
from information_schema.tables t
join tableA ta on ta.tablename = t.table_name
)t
) t
A pure SQL solution that works in about any DBMS? Well, as you know you cannot simply query the table names from your control table and use these in a SQL FROM clause, because the table names in the control table are data while the table names in the FROM cause are structure, so to say.
As you should know your database, however, you can use the table names known to you and compare them to the entries in your control table:
select 'TABLE_1', count(*) from table_1 having 'TABLE_1' in (select tablename from tablea)
union all
select 'TABLE_2', count(*) from table_2 having 'TABLE_2' in (select tablename from tablea)
union all
select 'TABLE_3', count(*) from table_3 having 'TABLE_3' in (select tablename from tablea)
union all
select 'TABLE_4', count(*) from table_4 having 'TABLE_4' in (select tablename from tablea)
union all
select 'TABLE_5', count(*) from table_5 having 'TABLE_5' in (select tablename from tablea)
This query must be altered when new tables get added to the database. As adding tables to a database is something that happens seldom if ever, this shouldn't be an issue.
Demo: https://dbfiddle.uk/Dfk9nIFo
I have a database and have 2 schema called DWFR and DWFR3 , what i need to do is compare 2 schmea and take the matching table name and also the table present in DWFR schema which is not available on DWFR3 and also with the below condition
SELECT schema_name, table_name
FROM tables
WHERE schema_name = 'DWFR'
AND table_name NOT LIKE 'RA%'
AND table_name NOT LIKE 'ZZ%'
AND table_name NOT LIKE 'DELETE%'
AND table_name NOT LIKE '%TEST%'
AND table_name NOT LIKE 'XX%'
AND table_name NOT LIKE 'ROB%'
AND table_name IN (
SELECT table_name
FROM tables
WHERE schema_name = 'DWFR3'
AND table_name NOT LIKE 'RA%'
AND table_name NOT LIKE 'ZZ%'
AND table_name NOT LIKE 'DELETE%'
AND table_name NOT LIKE '%TEST%'
AND table_name NOT LIKE 'XX%'
AND table_name NOT LIKE 'ROB%'
)
ORDER BY schema_name, table_name;
This will get the matched table names but i also want table name available in DWFR which is not their on DWFR3.
You can use a LEFT OUTER JOIN for selecting all data rows from a set and matchings from a second set.
I used a SQL CTE Common Table Expression in the script, you can think of it as a sub-select with additional uses
Here is how you can use for your case
with t2 as (
select table_name
from tables
where schema_name = 'DWFR3'
)
select t1.table_name, t2.table_name
from tables t1
left outer join t2
on t1.table_name = t2.table_name
where t1.schema_name = 'DWFR'
And you also need the append the additional requirements or WHERE clauses into above script
For the reason that I used CTE can be explained as follows.
In fact, the actual solution should be as simple as following.
Unfortunately, HANA seems to not supporting additional filtering criteria with ON clause on JOINs
select t1.table_name, t2.table_name
from tables t1
left outer join tables t2
on t1.table_name = t2.table_name and t2.schema_name = 'DWFR3
where t1.schema_name = 'DWFR'
Because of this reason, I had to filter the LEFT part of the JOIN earlier than applying the ON, join condition
To find matched table names in both schemas:
select
'DWFR' as schema_name,
table_name
from
(
select table_name from tables where schema_name = 'DWFR'
intersect
select table_name from tables where schema_name = 'DWFR3'
) x
where
table_name not like 'RA%' and
table_name not like 'ZZ%' and
table_name not like 'DELETE%' and
table_name not like '%TEST%' and
table_name not like 'XX%' and
table_name not like 'ROB%';
To find table names in DWFR but not in DWFR3:
select
'DWFR' as schema_name,
table_name
from
(
select table_name from tables where schema_name = 'DWFR'
except
select table_name from tables where schema_name = 'DWFR3'
) x
where
table_name not like 'RA%' and
table_name not like 'ZZ%' and
table_name not like 'DELETE%' and
table_name not like '%TEST%' and
table_name not like 'XX%' and
table_name not like 'ROB%';
I have 81 tables where I want to find matching columns and output a list like:
"columnName" found in 3 tables:
table1
table2
table3
"columnName2" found in 4 tables:
table1
table3
table4
table5
The table INFORMATION_SCHEMA.COLUMNS is a system table which contains all columns for all tables. You can look at it by selecting from it.
For your question, you should try this :
select I.Column_name,
table_name,
table_count
from INFORMATION_SCHEMA.COLUMNS I
inner join
(
Select Column_name,
count(*) as table_count
from INFORMATION_SCHEMA.COLUMNS
group by Column_name) as T on T.Column_name = I.Column_name
You will have a table with, for each column_name, the tables, and the count in the column table_count.
I ust don't know how to output a list. You should put it on excel then if you want to format it I guess ..
Tell me if you have problem !
Below Query should give you the desired result. Let me know if you face any issues. You can modify script if you want to be specific about the single database
This query provides you Column Name - Concatinated Table Names - Number of Tables
SELECT t.CNAME AS ColumnName, STUFF(
(SELECT ',' + S.TNAME
FROM
(
SELECT C.NAME AS CNAME , T.NAME AS TNAME
FROM SYS.OBJECTS AS T
JOIN SYS.COLUMNS AS C
ON T.OBJECT_ID=C.OBJECT_ID
WHERE T.TYPE_DESC='USER_TABLE'
)s
WHERE s.CNAME = t.CNAME
FOR XML PATH('')),1,1,'') AS TablesUsed,
COUNT(t.TNAME)
FROM
(
SELECT C.NAME AS CNAME , T.NAME AS TNAME
FROM SYS.OBJECTS AS T
JOIN SYS.COLUMNS AS C
ON T.OBJECT_ID=C.OBJECT_ID
WHERE T.TYPE_DESC='USER_TABLE'
)t
GROUP BY t.CNAME
HAVING COUNT(t.TNAME) > 1
ORDER BY COUNT(t.TNAME) DESC
I'm trying to get make a query to return a list of the column names from a query result in postgresql
e.g.
SELECT column_name
FROM (
SELECT table1.*
FROM table1
LEFT JOIN table2
ON table1.id = table2.tbl1_id
)
is this possible
I DO NOT WANT THE COLUMNS FROM A SINGULAR TABLE!!! so please dont tell me to do
SELECT column_name
FROM information_schema.columns
WHERE table_name = 'table1'
I don't know why you would ever need to do this, but it is possible using some of the JSON functions introduced in 9.3.
SELECT json_object_keys(row_to_json(t)) FROM
(SELECT * FROM table1
LEFT JOIN table2 ON table1.id = table2.tbl1_id LIMIT 1) t;
This will give you the name of every column returned for a single row. Without the LIMIT you would get the columns repeated for every row returned. If you wanted to see the values returned as well you can get more complex:
WITH t as
(SELECT * FROM table1
LEFT JOIN table2 ON table1.id = table2.tbl1_id LIMIT 1)
SELECT json_data.key, json_data.value
FROM t, json_each_text(row_to_json(t)) AS json_data;
Both these queries will return all the columns even if they are named the same. If all you want is a list of unique column names, you can utilize hstore:
CREATE EXTENSION hstore; --Create the extension if you need it.
SELECT akeys(hstore(t)) as array_of_columns
FROM
(SELECT * FROM table1
LEFT JOIN table2 ON table1.id = table2.tbl1id LIMIT 1) t;
Maybe there is a less hacky way, but I think this should work:
create table query_column_sample from (your query where 1=2);
SELECT column_name
FROM information_schema.columns
WHERE table_name = 'query_column_sample'
Querying tables returns data. Querying the information schema returns metadata. Column names are metadata.
SELECT 'table1', column_name
FROM information_schema.columns
WHERE table_catalog = 'database_name'
AND table_schema = 'schema_name'
AND table_name = 'table1'
UNION ALL
SELECT 'table2', column_name
FROM information_schema.columns
WHERE table_catalog = 'database_name'
AND table_schema = 'schema_name'
AND table_name = 'table2';
"schema_name" is probably public. But be warned that you can have multiple tables with the same name, as long as they are in different schemas. (You could have "public.table1" and "private.table1".) You can also have multiple tables with the same name in different databases. You need to specify all three values--database name, schema name, table name--to be sure of getting the right data.
If I had to do this in production I'd probably output more data. I'd also probably include the column "ordinal_position", and sort on it. Sorting by "ordinal_position" will order column names as they appear in the current table definition.
SELECT table_catalog
, table_schema
, table_name
, column_name
, ordinal_position
FROM information_schema.columns
WHERE table_catalog = 'sandbox'
AND table_schema = 'public'
AND table_name IN ('table1', 'table2')
ORDER BY table_catalog
, table_schema
, table_name
, ordinal_position;
The only reasonable approach I can come up with is to do something like this. Schema location plays into this, but I'm just posting the basic idea. SQL doesn't have reflection at run time unless you go and hit meta data tables, which is what you seem to be trying to achieve here.
SELECT table1.*
INTO x
FROM table1
LEFT JOIN table2
ON table1.id = table2.tbl1_id LIMIT 0;
select *
from information_schema.columns
where table_Name = 'x';
Also interesting enough while traveling the Internets over lunch I ran into this. Works with PostgreSQL 8.2.0 or higher.
https://github.com/theory/colnames/tree/master/test/sql
I have a table lets say table_names which contains some table names ( table1 , table2 ,table3 etc ). Now I want to fetch the data from table1 , table2 , table3 etc by going through the table table_names.
I am unable to achieve this task. Could anyone please help me out of this situation ?
I am using Oracle DB.
Thank you in advance.
If from TableA you have to get value from Single table then use this,
SELECT * FROM (SELECT table_name FROM tableA WHERE id=3)AS b
Otherwise from getting from multiple table, I prefer below PHP-
while($a=mysql_fetch_assoc(mysql_query((SELECT table_name FROM tableA)))
{ $tablename=$a[table_name];
while($b=mysql_fetch_assoc(mysql_query((SELECT * FROM $tablename))))
{
echo $b[column_name];
}
}
try this query:
DECLARE
str VARCHAR2(30);
num number(5);
BEGIN
SELECT count(*) into num FROM table_names;
FOR i in 1..num loop
SELECT name into str FROM
(SELECT ROWNUM r,t.name FROM table_names t) where r = i;
EXECUTE IMMEDIATE 'SELECT * FROM '||str;
str:='';
end loop;
END;
Select t1.,t2.,t3.* from table1 t1,table2 t2, table3 t3 where (select name from tables)