Storing WHERE clauses in a global table - sql

I'm running this query 50+ times, and I want to abstract the AND query statements below and store them in one/global table, so in the future I only have to edit one table (vs. 50) if I want to edit any of the AND statements. What's the most efficient way to store the AND statements in a separate table and then cal them in the below query?
SELECT
Field,
Field2,
Field3
into table1
FROM table2
WHERE (DESCRIPTION iLIKE '%ADVANCE%AUTO%Pa%')
AND is_duplicate!=1
AND amount >0
AND currency_id = 152
AND transaction_base_type = 'debit'
AND TRANSACTION_STATUS <> 'D'

You could create a view/materialized view:
CREATE VIEW my_view
AS
SELECT
Field,
Field2,
Field3,
DESCRIPTION
FROM table2
WHERE is_duplicate!=1
AND amount >0
AND currency_id = 152
AND transaction_base_type = 'debit'
AND TRANSACTION_STATUS <> 'D'
and then:
SELECT Field, Field2, Field3
FROM my_view
WHERE (DESCRIPTION iLIKE '%ADVANCE%AUTO%Pa%')
EDIT
I simply need to store WHERE clauses in one place so I can update them once and call them in 50 queries vs. including them in every query and updating them 50 times. Is it really that complicated?
As I wrote in comment you cannot simply parametrize table name(plus it may indicate that schema is flawed). SQL is powerful language so you could use dynamic SQL and functions.
CREATE OR REPLACE FUNCTION my_func(tab_name text)
RETURNS TABLE (
id INT, -- here goes common column list shared across all 50 tables
col1 INT,
col2 INT
)
LANGUAGE 'plpgsql'
AS $BODY$
BEGIN
RETURN QUERY
EXECUTE format('SELECT * from %I where col2 > 0',tab_name);
-- here goes shared conditions
END
$BODY$;
SELECT * FROM my_func('tab1');
SELECT * FROM my_func('tab2') WHERE col2 = 2;
-- condition that is not shared
db<>fiddle demo

Related

BigQuery using Dynamic sql as the input source of a query

How can I use a dynamic query as an input source for a larger query?
There is a query I'm getting the union of values in different datasets/tables scattered around and the list is growing so I'm thinking of using the dynamic query than to write queries for each tables like this
SET QUERY = "";
SET tables = ["table1", "table2"...];
SET tables_size = ARRAY_LENGTH(tables);
WHILE i < tables_size DO
IF (i = tables_size -1) THEN
BEGIN
SET query = CONCAT(query, " SELECT id, name FROM ", tables[OFFSET(i)]);
BREAK;
END;
ELSE
SET query = CONCAT(query, " SELECT id, name FROM ", tables[OFFSET(i)], ' UNION ALL ');
END IF;
SET i = i + 1;
END WHILE;
EXECUTE IMMEDIATE query;
My goal is to use the output of the executed query as a FROM clause for a larger query.
It will be something like
Select A, B, C, D ... From *EXECUTE IMMEDIATE query* LEFT JOIN ... ON..
Is there a way to inject an output of a dynamic query as a table for another query?
I don't see TABLE as a variable type for bigquery so that was not my option.
I'm getting a bit tired of copy pasting table names to the exact query every time a new table is introduced to this logic.
SELECT id, name FROM table1 UNION ALL
SELECT id, name FROM table1 UNION ALL
SELECT id, name FROM table3...
If there is a simple way to do this? or maybe a reason to not use dynamic queries for performance reasons?
Hope one of these are helpful:
1. Wildcard tables
If tables you want to union have a common prefix, you can consider to use a wildcard table like below. I think this is more concise form rather than union-all:
-- Sample Tables
CREATE TABLE IF NOT EXISTS testset.table1 AS SELECT 1 AS id, 'aaa' AS name;
CREATE TABLE IF NOT EXISTS testset.table2 AS SELECT 2 AS id, 'bbb' AS name;
CREATE TABLE IF NOT EXISTS testset.table3 AS SELECT 3 AS id, 'ccc' AS name;
--- Wildcard tables
SELECT * FROM `testset.table*` WHERE _TABLE_SUFFIX IN ('1', '2', '3');
2. Dynamic SQL & Temp Table
You can't inject a dynamic SQL directly into another query but you can use a temp table to emulate it.
2.1 Dynamic SQL
More concise dynamic query to union all tables:
DECLARE tables DEFAULT ["testset.table1", "testset.table2", "testset.table3"];
SELECT ARRAY_TO_STRING(ARRAY_AGG(FORMAT('SELECT id, name FROM %s', t)), ' UNION ALL\n')
FROM UNNEST(tables) t;
2.2 Using a temp table
I thinks you can modify your larger query to use a dynamically generated temp table.
DECLARE tables DEFAULT ["testset.table1", "testset.table2", "testset.table3"];
CREATE TABLE IF NOT EXISTS testset.table1 AS SELECT 1 AS id, 'aaa' AS name;
CREATE TABLE IF NOT EXISTS testset.table2 AS SELECT 2 AS id, 'bbb' AS name;
CREATE TABLE IF NOT EXISTS testset.table3 AS SELECT 3 AS id, 'ccc' AS name;
EXECUTE IMMEDIATE (
SELECT "CREATE TEMP TABLE IF NOT EXISTS union_tables AS \n"
|| ARRAY_TO_STRING(ARRAY_AGG(FORMAT('SELECT id, name FROM %s', t)), ' UNION ALL\n') FROM UNNEST(tables) t
);
-- your larger query using a temp table
SELECT * FROM union_tables;
output:

SQL Finding data from similar tables

I have a 50 tables with similar structures with name TABLE_1, TABLE_2, TABLE_3 etc. I want to select some information like SELECT * FROM TABLE WHERE CRM_ID = 100 but I dont know which table consists this ID. I guess that I should make a union with this tables and make a query from this union but I am afraid that it is not the best solution. Could anybody help me?
To find the table containing the field CRM_ID, you can use:
select TABLE_NAME
from SYS.ALL_TAB_COLUMNS
where COLUMN_NAME = 'CRM_ID'
From here, you can query the relevant tables with your union
select 'table_1', count(*) from table_1 where CMR_ID = 100
union all
select 'table_2', count(*) from table_2 where CMR_ID = 100
[.....]
If you have 50 tables, you may want to use some advanced text editing so you don't have to type the same line 50 times.
I can't see anything different from a UNION to get what you need.
However, I would like to avoid repeating all the code every time you need to query your tables, for example by creating a view:
create view all_my_tables as
select * from table1
union
select * from table2
...
and then
select *
from all_my_tables
where crm_id = 100
I would
create a view table_all as
select * from table_1 union all
select * from table_2 ...
but I guess you have 50 tables due to some performance reason like creating partitioning without partitioned table. If that is the case you need some table that simulates index and you need to have data "sorted". Then create table that contains: table_name, min_val, max_val:
table_1, 1, 1000
table_2, 1001, 2000
and procedure selecting looks like:
procedure sel(crmid) is
a varchar2(10);
value table%rowtype;
begin
select table_name into a from lookup where crmid between min_val and max_val;
execute immediate 'select * from ' || a || ' where crm_id = ' || crmid into value;
--do logic with value
end;
but if data is not ordered you need to iterate selects in loop. In such case use view.
Try this:
Select tmp.tablename from (
select 'table1' as tablename from table1 where CRM_ID=100
union all
select 'table2' as tablename from table2 where CRM_ID=100
) as tmp

Return id if a row exists, INSERT otherwise

I'm writing a function in node.js to query a PostgreSQL table.
If the row exists, I want to return the id column from the row.
If it doesn't exist, I want to insert it and return the id (insert into ... returning id).
I've been trying variations of case and if else statements and can't seem to get it to work.
A solution in a single SQL statement. Requires PostgreSQL 8.4 or later though.
Consider the following demo:
Test setup:
CREATE TEMP TABLE tbl (
id serial PRIMARY KEY
,txt text UNIQUE -- obviously there is unique column (or set of columns)
);
INSERT INTO tbl(txt) VALUES ('one'), ('two');
INSERT / SELECT command:
WITH v AS (SELECT 'three'::text AS txt)
,s AS (SELECT id FROM tbl JOIN v USING (txt))
,i AS (
INSERT INTO tbl (txt)
SELECT txt
FROM v
WHERE NOT EXISTS (SELECT * FROM s)
RETURNING id
)
SELECT id, 'i'::text AS src FROM i
UNION ALL
SELECT id, 's' FROM s;
The first CTE v is not strictly necessary, but achieves that you have to enter your values only once.
The second CTE s selects the id from tbl if the "row" exists.
The third CTE i inserts the "row" into tbl if (and only if) it does not exist, returning id.
The final SELECT returns the id. I added a column src indicating the "source" - whether the "row" pre-existed and id comes from a SELECT, or the "row" was new and so is the id.
This version should be as fast as possible as it does not need an additional SELECT from tbl and uses the CTEs instead.
To make this safe against possible race conditions in a multi-user environment:
Also for updated techniques using the new UPSERT in Postgres 9.5 or later:
Is SELECT or INSERT in a function prone to race conditions?
I would suggest doing the checking on the database side and just returning the id to nodejs.
Example:
CREATE OR REPLACE FUNCTION foo(p_param1 tableFoo.attr1%TYPE, p_param2 tableFoo.attr1%TYPE) RETURNS tableFoo.id%TYPE AS $$
DECLARE
v_id tableFoo.pk%TYPE;
BEGIN
SELECT id
INTO v_id
FROM tableFoo
WHERE attr1 = p_param1
AND attr2 = p_param2;
IF v_id IS NULL THEN
INSERT INTO tableFoo(id, attr1, attr2) VALUES (DEFAULT, p_param1, p_param2)
RETURNING id INTO v_id;
END IF;
RETURN v_id:
END;
$$ LANGUAGE plpgsql;
And than on the Node.js-side (i'm using node-postgres in this example):
var pg = require('pg');
pg.connect('someConnectionString', function(connErr, client){
//do some errorchecking here
client.query('SELECT id FROM foo($1, $2);', ['foo', 'bar'], function(queryErr, result){
//errorchecking
var id = result.rows[0].id;
};
});
Something like this, if you are on PostgreSQL 9.1
with test_insert as (
insert into foo (id, col1, col2)
select 42, 'Foo', 'Bar'
where not exists (select * from foo where id = 42)
returning foo.id, foo.col1, foo.col2
)
select id, col1, col2
from test_insert
union
select id, col1, col2
from foo
where id = 42;
It's a bit longish and you need to repeat the id to test for several times, but I can't think of a different solution that involves a single SQL statement.
If a row with id=42 exists, the writeable CTE will not insert anything and thus the existing row will be returned by the second union part.
When testing this I actually thought the new row would be returned twice (therefor a union not a union all) but it turns out that the result of the second select statement is actually evaluated before the whole statement is run and it does not see the newly inserted row. So in case a new row is inserted, it will be taken from the "returning" part.
create table t (
id serial primary key,
a integer
)
;
insert into t (a)
select 2
from (
select count(*) as s
from t
where a = 2
) s
where s.s = 0
;
select id
from t
where a = 2
;

sql query to artificially create rows

I have a piece of code that runs on a variety of databases. It simply runs a configurable sql query which returns a number of rows. From each row, i pull some text and a number to create a new object. Our latest client has decided to put all the text number combinations in a single row of the database i.e
text_1, num_1, text_2, num_2, text_3, num_3
Is there a clever way I can query this to return
text_1,num_1
text_2,num_2
text_3,num_3
so that I don't have to re-code the section for this client.
EDIT:
(Different databases means different RDBMS)
(the commas delimit different columns within a table)
SELECT
CASE row.id WHEN 1 THEN field1
WHEN 2 THEN field3
ELSE field5
END AS new_field_1,
CASE row.id WHEN 1 THEN field2
WHEN 2 THEN field4
ELSE field6
END AS new_field_2
FROM
myTable
CROSS JOIN
(SELECT 1 AS id UNION ALL SELECT 2 UNION ALL SELECT 3) AS row
This should work for most, though still need a little modification (Such as adding 'FROM dual' in the UNIONs for Oracle...)
Alternatively, just UNION three queries together...
SELECT field1, field2 FROM myTable
UNION ALL
SELECT field3, field4 FROM myTable
UNION ALL
SELECT field5, field6 FROM myTable
You can create a function/SP to return a ResultSet the way you need.

PostgreSQL: Select a single-row x amount of times

A single row in a table has a column with an integer value >= 1 and must be selected however many times the column says. So if the column had '2', I'd like the select query to return the single-row 2 times.
How can this be accomplished?
Don't know why you would want to do such a thing, but...
CREATE TABLE testy (a int,b text);
INSERT INTO testy VALUES (3,'test');
SELECT testy.*,generate_series(1,a) from testy; --returns 3 rows
You could make a table that is just full of numbers, like this:
CREATE TABLE numbers
(
num INT NOT NULL
, CONSTRAINT numbers_pk PRIMARY KEY (num)
);
and populate it with as many numbers as you need, starting from one:
INSERT INTO numbers VALUES(1);
INSERT INTO numbers VALUES(2);
INSERT INTO numbers VALUES(3);
...
Then, if you had the table "mydata" that han to repeat based on the column "repeat_count" you would query it like so:
SELECT mydata.*
FROM mydata
JOIN numbers
ON numbers.num <= mydata.repeat_count
WHERE ...
If course you need to know the maximum repeat count up front, and have your numbers table go that high.
No idea why you would want to do this thought. Care to share?
You can do it with a recursive query, check out the examples in
the postgresql docs.
something like
WITH RECURSIVE t(cnt, id, field2, field3) AS (
SELECT 1, id, field2, field3
FROM foo
UNION ALL
SELECT t.cnt+1, t.id, t.field2, t.field3
FROM t, foo f
WHERE t.id = f.id and t.cnt < f.repeat_cnt
)
SELECT id, field2, field3 FROM t;
The simplest way is making a simple select, like this:
SELECT generate_series(1,{xTimes}), a.field1, a.field2 FROM my_table a;