Building dymanic query with many laterail joins - sql

I am building a dynamic query based on user-submitted conditions on user-specified fields.
I have a two tables
Visitors(id, name, email...)
and a
Trackings(id, visitor_id, field, string_value, integer_value, boolean_value, date_value)
The conditions comes in the form of an array of SQL fragments built earlier. Default filters for attributes that are hardcoded on the visitors table, and custom filters, for the values that are user-submitted and stored in an EAV schema (trackings)
Example:
Default:
{"name ILIKE 'Jack'", "(last_seen < (current_date - (7 || ' days')::interval))"}
Custom:
{"field = 'number_of_orders' > 10", "is_pro_user = true"}
A visitor can have many trackings, each one recording some custom, user-submitted data field for that visitor. But each visitor also has some default data that lies on the table itself, such as email, name or last_seen etc.
Now, users should be to ask queries such as:
Give me all the visitors for which no custom field named number_of_orders has been recored (is unknown)
Give me all visitors with the default attribute name set to Jack and for who the custom attribute total_purchase_value is greater than 1000
My attempt at solving it was using a stored procedure that dynamically concatenated a series of conditions using AND (for default data on the visitor table), and OR statements (for the custom data in the trackings table) inside a WHERE-caluse
CREATE OR REPLACE FUNCTION find_matching_visitors(app_id text, default_filters text[], custom_filters text[])
RETURNS TABLE (
id varchar
) AS
$body$
DECLARE
default_filterstring text;
custom_filterstring text;
default_filter_length integer;
custom_filter_length integer;
sql VARCHAR;
BEGIN
default_filter_length := COALESCE(array_length(default_filters, 1), 0);
custom_filter_length := COALESCE(array_length(custom_filters, 1), 0);
default_filterstring := array_to_string(default_filters, ' AND ');
custom_filterstring := array_to_string(custom_filters, ' OR ');
IF custom_filterstring = '' or custom_filterstring is null THEN
custom_filterstring := '1=1';
END IF;
IF default_filterstring = '' or default_filterstring is null THEN
default_filterstring := '1=1';
END IF;
sql := format('
SELECT v.id FROM visitors v
LEFT JOIN trackings t on v.id = t.visitor_id
WHERE v.app_id = app_id and (%s) and (%s)
group by v.id
having case when %s > 0 then count(v.id) = %s else true end
', custom_filterstring, default_filterstring, custom_filter_length, custom_filter_length);
RETURN QUERY EXECUTE sql;
END;
$body$
LANGUAGE 'plpgsql';
This works great, but AFAIK there is not way to express the unknown filter for a custom attribute. In that it would require using a left outer join, or a not exists subquery with access to the outer scope.
I am now looking at alternative ways to accomplish the same as above, but also support this kind of query. I am thinking something like the below, using a series of lateral joins for each condition, but this seems like it won't perform very well as soon as there is more than 1-2 conditions/joins.
select v.id, v.name from visitors v
inner join lateral ( <-- custom fields
select * from trackings t
where field = 'admin'
) as t1 on t1.visitor_id = v.id
inner join lateral (
select * from trackings t
where field = 'users_created'
) as t2 on t2.visitor_id = v.id
inner join lateral (
select * from trackings t
where field = 'teams_created' and integer_value > 0
) as t3 on t3.visitor_id = v.id
where v.app_id = 'ASnYW1-RgCl0I' and (v.type = 'lead' or v.type = 'user')
and name ILIKE 'mads' and email is not null // <-- default fields
Any suggestions?

Related

Select query, insert into table, and return initial query in postgres

I have a rather complex plpgsql stored procedure and I need to select from multiple tables and insert as well.
This is part of what I currently have.
BEGIN
RETURN query
SELECT domains.id, webpages.id as page_id ...
FROM domains
LEFT JOIN domain_settings
ON domain_settings.domain_id = domains.id
RIGHT JOIN webpages
ON webpages.domain_id = domains.id
LEFT JOIN subscriptions
ON webpages.id = subscriptions.page_id
AND subscriptions.user_id = query_user_id
AND subscriptions.comment_id IS NULL
WHERE domains.domain_address = query_domain_url
IF NOT FOUND THEN ...
END;
$$ language plpgsql;
Now, I would like add an insert query into another table using certain values from the return query before the 'if not found then' statement:
INSERT INTO page_visits (domain_id, page_id)
SELECT id, page_id FROM ?? (return query statement)
And after the insert, I want to return the initial return query values. How do I go about doing this? I tried using WITH AS statements, but I can't seem to get it to work
A set-returning PL/pgSQL function builds the return stack while processing the function body. There is no way to access that return stack from within the same function. You could nest the function. Or use a temporary table.
But using a CTE is probably the simplest way for the cas at hand. Going out on a limb, you may be looking for something like this:
CREATE OR REPLACE FUNCTION demo(query_user_id int, query_domain_url text)
RETURNS TABLE (c1 int, c2 int)
LANGUAGE plpgsql AS
$func$
BEGIN
RETURN QUERY
WITH sel AS (
SELECT d.id, w.id as page_id ...
FROM webpages w
JOIN domains d ON d.id = w.domain_id
LEFT JOIN domain_settings ds ON ds.domain_id = d.id
LEFT JOIN subscriptions s ON s.page_id = w.id
AND s.user_id = query_user_id -- origin?
AND s.comment_id IS NULL
WHERE d.domain_address = query_domain_url -- origin?
)
, ins AS (
INSERT INTO tbl (col1, col2)
SELECT main.id, sel.page_id
FROM (SELECT 'foo') AS main(id)
LEFT JOIN sel USING (id) -- LEFT JOIN ?
)
TABLE sel;
IF NOT FOUND THEN
-- do something
END IF;
END
$func$;
Remember, if the transaction does not commit successfully, the INSERT is also rolled back.
The final TABLE sel is just short syntax for SELECT * FROM sel. See:
Is there a shortcut for SELECT * FROM?

Options for merging two separate complex postgres functions

I have two separate plpgsql functions for my project, that I would like to merge into one, or come up with an alternative solution.
The first function uses a domain address and path, to retrieve the domain_id and webpage_id, along other things (such as whether the domain uses a liking system or voting system or neither for its posts).
And the second function uses the webpage_id from the first function to retrieve appropriate posts for that page and whether to get post likes/votes/neither.
As of now, what I'm doing is to call the following function, first. What it does is that it checks for the domain_id and webpage_id based off the domain_url and slug.
If the values don't exist, it checks whether the domain is registered, and if it is, it makes an insert into the webpages table, else it returns an error.
create or replace function get_domain_details(query_domain_url text, query_slug text) returns table(id bigint, owner_id uuid page_id bigint, post_count int, is_likes boolean, is_votes boolean
)
as $$
DECLARE new_domain_id int;
new_page_id bigint;
BEGIN
RETURN query
WITH domain_row AS (
SELECT domains.id, domains.owner_id, webpages.id as page_id, webpages.post_count, domain_settings.is_likes, domain_settings.is_votes
FROM domains
JOIN domain_settings
ON domain_settings.domain_id = domains.id
RIGHT JOIN webpages
ON webpages.domain_id = domains.id AND webpages.slug = query_slug
WHERE domains.domain_address = query_domain_url),
abc as (
INSERT INTO page_visits (domain_id, page_id)
SELECT domain_row.id, domain_row.page_id FROM domain_row
)
TABLE domain_row;
IF NOT FOUND THEN
IF EXISTS (SELECT domains.id FROM domains WHERE domain_address = query_domain_url) THEN
WITH webpage_rows AS (INSERT INTO webpages (domain_id, slug)
VALUES ((SELECT domains.id FROM domains WHERE domain_address = query_domain_url), query_slug)
RETURNING *
)
SELECT webpage_rows.id, webpage_rows.domain_id
INTO new_page_id, new_domain_id
FROM webpage_rows;
RETURN query
WITH domain_row_2 AS (
SELECT domains.id, domains.owner_id, new_page_id as page_id, 0 as post_count, domain_settings.is_likes, domain_settings.is_votes
FROM domains
JOIN domain_settings
ON domain_settings.domain_id = domains.id
WHERE domains.id = new_domain_id),
abc_2 as (
INSERT INTO page_visits (domain_id, page_id)
SELECT domain_row_2.id, domain_row_2.page_id FROM domain_row_2
)
TABLE domain_row_2;
ELSE
RAISE EXCEPTION 'This is not a valid domain!';
END IF;
END IF;
RETURN;
END;
$$ language plpgsql;
And once I've received the page_id and liking/voting system info from the database, I use that to make another request to the database from my nodejs server to retrieve the posts and its details.
If is_votes is true, it joins with the votes table, else if is_likes is true, it joins with the likes table, else it doesn't join with either.
create or replace function get_page_posts (
query_page_id bigint, is_votes boolean, is_likes boolean
)
returns table (id bigint, full_name text, avatar_url text, author_id uuid, post_text text, created_at timestamptz, comment_count int, vote_count int, like_count int
)
as $$
BEGIN
IF is_votes = true THEN
RETURN query
SELECT posts.id, users.full_name, users.avatar_url, posts.author_id, posts.post_text, posts.created_at, post_data.comment_count, post_data.vote_count, post_data.like_count
FROM posts
LEFT JOIN users
ON users.user_id = posts.author_id
LEFT JOIN post_data
ON post_data.post_id = posts.id
LEFT JOIN votes v1
ON v1.post_id = posts.id
LEFT JOIN webpages
ON webpages.id = posts.page_id
WHERE webpages.id = query_page_id
AND NOT posts.status = 'deleted'
ORDER BY posts.id DESC;
ELSIF is_likes = true THEN
RETURN query
SELECT posts.id, users.full_name, users.avatar_url, posts.author_id, posts.post_text, posts.created_at, post_data.comment_count, post_data.vote_count, post_data.like_count
FROM posts
LEFT JOIN users
ON users.user_id = posts.author_id
LEFT JOIN post_data
ON post_data.post_id = posts.id
LEFT JOIN webpages
ON webpages.id = posts.page_id
LEFT JOIN likes
ON likes.post_id = posts.id
WHERE webpages.id = query_page_id
AND NOT posts.status = 'deleted'
ORDER BY posts.id DESC;
ELSE
RETURN query
SELECT posts.id, users.full_name, users.avatar_url, posts.author_id, posts.comment_text, posts.created_at, post_data.comment_count, post_data.vote_count, post_data.like_count
FROM posts
LEFT JOIN users
ON users.user_id = posts.author_id
LEFT JOIN post_data
ON post_data.post_id = posts.id
LEFT JOIN webpages
ON webpages.id = posts.page_id
WHERE posts.parent_id IS NULL
AND webpages.id = query_page_id
AND NOT posts.status = 'deleted'
ORDER BY posts.id DESC;
END IF;
END;
$$
language plpgsql
Now, the issue that I have is that this is slowing this entire loading process down, so I'm trying to merge both functions together so that I don't have to make two separate requests to the database.
I have currently combined both of them into a new function such that it looks like this.
However, I am unsure if this is a safe and viable approach.
... as $$
BEGIN
RETURN query
WITH abc AS (
SELECT * FROM get_domain_details(query_domain_url, query_slug)
),
def AS (
SELECT * FROM get_page_posts((SELECT abc.page_id FROM abc), (SELECT abc.is_votes FROM abc), (SELECT abc.is_likes FROM abc))
)
SELECT * FROM abc UNION ALL SELECT * FROM def;
END;
$$ language plpgsql;

PostgreSQL array of type table/setof

I need to code a function in postgresql that must 'pick' entire rows from a table depending on a condition and then insert them into an array, and finally return that array.
I'm actually querying the rows I need but there's an special column from the table that I must evaluate first to see if I can retrieve it or not (insert to the array), depending on the type of user is doing the query (customer or worker).
For example:
The special column is named "watchers" and has two variants: 'my_team' and 'only_me' and also has another column named "user_id".
If the column says 'my_team', the row can be inserted into the array with no problems, but if it says 'only_me' I must compare the id of the user calling the function and the user registered in that row, if they match I can insert it into the array, otherwise I can't.
Thing is that as I've read so far it seems that it can't be done with arrays, so I'd like to know if there's another way of doing this, maybe with an extra table or something?
Thanks for your help!
This is the code I have:
CREATE OR REPLACE FUNCTION public.calculates_permission_by_task(task_id integer)
RETURNS SETOF permission
LANGUAGE plpgsql
STABLE
AS $function$
DECLARE
num int := 5;
record permission;
ret_permissions permission [];
BEGIN
FOR record IN (select
p.id, p.slug
from task as t
join service_request_form as srf on t.service_request_form_id = srf.id
join service_group as sg on srf.service_group_id = sg.id
join worker_group as wg on wg.service_group_id = sg.id
join worker_group_member as wgm on wg.id = wgm.worker_group_id
join role as r on r.id = wgm.role_id
join permission_role as pr on r.id = pr.role_id
join permission as p on p.id = pr.permission_id
where t.id = task_id
and wgm.user_id = 'LFudaU6jzid4SKFlU8MgFAwezyP2'
and pr.value <> 'off'
and pr.value <> 'no')
LOOP
/* Here is where I pretend to do the comparison and insert data into the array */
ret_permission := array_append(ret_permissions, record.id);
END LOOP;
RETURN ret_permissions;
END
$function$
You could implement the logic in the query itself.
Assuming that the user_id column you are refering to corresponds to wgm.user_id, that is already in the query, the where clause of the query would become:
where
t.id = task_id
and pr.value not in ('off', 'no')
and (
?.watchers = 'my_team'
or (
?.watchers = 'only_me'
or wgm.user_id = 'LFudaU6jzid4SKFlU8MgFAwezyP2'
)
)
You don't tell which table watches belong to, so I used ?, which you need to replace that with the relevant table alias.
If there are just two possible values for watchers, then we can simplify the predicates:
where
t.id = task_id
and pr.value not in ('off', 'no')
and (?.watchers = 'my_team' or wgm.user_id = 'LFudaU6jzid4SKFlU8MgFAwezyP2')
You don't need an array
you don't need a cursor loop
You don't even need plpgsql
plain SQL wiill suffice:
CREATE TABLE permission (
id integer
, slug text
);
INSERT INTO permission(id,slug) VALUES (1,'WTF' );
CREATE TABLE task (
id integer
, service_request_form_id integer
);
INSERT INTO task(id, service_request_form_id) VALUES (1,1 );
CREATE TABLE service_request_form (
id integer
, service_group_id integer
);
INSERT INTO service_request_form(id, service_group_id) VALUES (1,1 );
CREATE TABLE service_group (
id integer
);
INSERT INTO service_group(id) VALUES (1);
CREATE TABLE worker_group (
id integer
, service_group_id integer
);
INSERT INTO worker_group(id, service_group_id) VALUES (1,1 );
CREATE TABLE worker_group_member (
worker_group_id integer
, role_id integer
, user_id text
);
INSERT INTO worker_group_member(worker_group_id, role_id,user_id) VALUES (1,1,'LFudaU6jzid4SKFlU8MgFAwezyP2' );
CREATE TABLE zrole (
id integer
);
INSERT INTO zrole(id) VALUES (1 );
CREATE TABLE permission_role (
role_id integer
, permission_id integer
, value text
);
INSERT INTO permission_role(role_id,permission_id,value) VALUES (1,1,'Yes' );
CREATE OR REPLACE FUNCTION public.calculates_permission_by_task(task_id integer)
RETURNS SETOF permission
LANGUAGE sql STABLE SECURITY INVOKER ROWS 30 -- COST 1
AS $func$
SELECT p.id, p.slug
FROM permission as p
WHERE EXISTS (
SELECT *
FROM task as t
JOIN service_request_form as srf
on t.service_request_form_id = srf.id
JOIN service_group as sg
on srf.service_group_id = sg.id
JOIN worker_group as wg
on wg.service_group_id = sg.id
JOIN worker_group_member as wgm
on wg.id = wgm.worker_group_id
JOIN zrole as zr
on zr.id = wgm.role_id
JOIN permission_role as pr
on zr.id = pr.role_id
WHERE p.id = pr.permission_id
AND t.id = task_id
and wgm.user_id = 'LFudaU6jzid4SKFlU8MgFAwezyP2'
and pr.value NOT IN( 'off' , 'no' )
-- -------------------------
-- Add the needed logic
-- (From #GMB) here
-- -------------------------
);
$func$
;
-- Usage:
-- --------------
SELECT * -- p.id, p.slug
FROM public.calculates_permission_by_task(42);

Error to postgresql

Pls Help!
count_rate :=
(
SELECT COUNT(trate.rid) AS count_rate
FROM tlot LEFT JOIN trate ON trate.ridlot = tlot.lid
GROUP BY tlot.lid
);
FULL:
CREATE FUNCTION editstatuswait()
RETURNS void
LANGUAGE plpgsql
AS $$
DECLARE count_rate INTEGER;
BEGIN
count_rate := (SELECT COUNT(trate.rid) AS count_rate FROM tlot LEFT JOIN trate ON trate.ridlot = tlot.lid GROUP BY tlot.lid);
IF (count_rate != 0) THEN
UPDATE tlot SET lstatus = 3
WHERE tlot.lexpirationdate < NOW()
AND tlot.lexpirationdate > NOW()-INTERVAL '24 hours' AND tlot.lstatus = 2;
ELSE
UPDATE tlot SET lstatus = 0
WHERE tlot.lexpirationdate < NOW()
AND tlot.lexpirationdate > NOW()-INTERVAL '24 hours' AND tlot.lstatus = 2;
END IF;
END;
$$;
ERROR: [21000] ERROR: more than one row returned by a subquery used as an expression Где: SQL statement
SELECT (SELECT COUNT(trate.rid) AS count_rate FROM tlot LEFT JOIN trate ON trate.ridlot = tlot.lid GROUP BY tlot.lid
I can not understand how to get rid of this error...
The solution will depend on what you are trying to achieve.
A variable can only contain a single value, so your attempt to store the result of a subselect that returns more than one row in count_rate is bound to fail.
You will have to come up with a subselect that returns at most one row (if it returns no row, NULL will be assigned to the variable).
If you are only interested in the first row (unlikely, since there is no ORDER BY), you could append LIMIT 1 to the query.
If you want only the count for a certain tlot.lid, you should use WHERE tlot.lid = ... instead of a GROUP BY.
If you want to process multiple results, you would use a construction like:
FOR count_rate IN SELECT ... LOOP
...
END LOOP;
Remove the GROUP BY:
count_rate := (SELECT COUNT(trate.rid) AS count_rate FROM tlot LEFT JOIN trate ON trate.ridlot = tlot.lid);
Of course, this may not do what you intend. It will at least fix the error.

How to add data from table to biggest object query in PostgreSQL

Postgres 9.1+ database contains different schema for every company named firma and company number, like firma1, firma5, firma99, firma12.
Every schema contains a table with company name:
-- this table contains always exactly one row:
create table firma5.company ( company char(50) not null );
Following query lists biggest objects:
select
(n.nspname||'.'||relname)::char(45) as tablename
, pg_size_pretty(pg_total_relation_size(c.oid))::char(10) as totalsize
, case
when c.relkind='i' then 'index'
when c.relkind='t' then 'toast'
when c.relkind='r' then 'table'
when c.relkind='v' then 'view'
when c.relkind='c' then 'composite type'
when c.relkind='S' then 'sequence'
else c.relkind::text
end ::char(14) as "type"
from
pg_class c
left join pg_namespace n on n.oid = c.relnamespace
left join pg_tablespace t on t.oid = c.reltablespace
where
(pg_total_relation_size(c.oid)>>20)>0 and c.relkind!='t'
order by
pg_total_relation_size(c.oid) desc
This query shows company schemas like firma1, firma5 etc.
How to show company names ( firman.company.company ) also in this query result? Query may return also tables from schemas other than firmaN. In this case company name column should be empty or null.
This is not possible with plain SQL because you cannot specify beforehand the table name to join on, so you need to run a dynamic query. You can, however, create a simple function that returns the company name from a dynamic query, if the schema has company tables:
CREATE FUNCTION company_name (sch text) RETURNS text AS $$
DECLARE
comp text := NULL;
BEGIN
IF strpos(sch, 'firma') = 1 THEN
EXECUTE 'SELECT company FROM ' || sch || '.company' INTO comp;
END IF;
RETURN comp;
END; $$ LANGUAGE plpgsql STRICT STABLE;
And then use that function in your query:
select
(n.nspname||'.'||c.relname)::char(45) as tablename
, pg_size_pretty(pg_total_relation_size(c.oid))::char(10) as totalsize
, case
when c.relkind='i' then 'index'
-- when c.relkind='t' then 'toast' FILTERED OUT IN WHERE CLAUSE
when c.relkind='r' then 'table'
when c.relkind='v' then 'view'
when c.relkind='c' then 'composite type'
when c.relkind='S' then 'sequence'
else c.relkind::text
end ::char(14) as "type"
, company_name(n.nspname) as company -- <<<<<<<<<<<<<<<<<<<<<<<
from pg_class c
left join pg_namespace n on n.oid = c.relnamespace
--left join pg_tablespace t on t.oid = c.reltablespace NOT USED
where (pg_total_relation_size(c.oid)>>20)>0 and c.relkind!='t'
order by pg_total_relation_size(c.oid) desc;