Guidance on using the WITH clause in SQL - sql

I understand how to use the WITH clause for recursive queries (!!), but I'm having problems understanding its general use / power.
For example the following query updates one record whose id is determined by using a subquery returning the id of the first record by timestamp:
update global.prospect psp
set status=status||'*'
where psp.psp_id=(
select p2.psp_id
from global.prospect p2
where p2.status='new' or p2.status='reset'
order by p2.request_ts
limit 1 )
returning psp.*;
Would this be a good candidate for using a WITH wrapper instead of the relatively ugly sub-query? If so, why?

If there can be concurrent write access to involved tables, there are race conditions in the following queries. Consider:
Postgres UPDATE … LIMIT 1
Your example can use a CTE (common table expression), but it will give you nothing a subquery couldn't do:
WITH x AS (
SELECT psp_id
FROM global.prospect
WHERE status IN ('new', 'reset')
ORDER BY request_ts
LIMIT 1
)
UPDATE global.prospect psp
SET status = status || '*'
FROM x
WHERE psp.psp_id = x.psp_id
RETURNING psp.*;
The returned row will be the updated version.
If you want to insert the returned row into another table, that's where a WITH clause becomes essential:
WITH x AS (
SELECT psp_id
FROM global.prospect
WHERE status IN ('new', 'reset')
ORDER BY request_ts
LIMIT 1
)
, y AS (
UPDATE global.prospect psp
SET status = status || '*'
FROM x
WHERE psp.psp_id = x.psp_id
RETURNING psp.*
)
INSERT INTO z
SELECT *
FROM y;
Data-modifying queries using CTEs were added with PostgreSQL 9.1.
The manual about WITH queries (CTEs).

WITH lets you define "temporary tables" for use in a SELECT query. For example, I recently wrote a query like this, to calculate changes between two sets:
-- Let o be the set of old things, and n be the set of new things.
WITH o AS (SELECT * FROM things(OLD)),
n AS (SELECT * FROM things(NEW))
-- Select both the set of things whose value changed,
-- and the set of things in the old set but not in the new set.
SELECT o.key, n.value
FROM o
LEFT JOIN n ON o.key = n.key
WHERE o.value IS DISTINCT FROM n.value
UNION ALL
-- Select the set of things in the new set but not in the old set.
SELECT n.key, n.value
FROM o
RIGHT JOIN n ON o.key = n.key
WHERE o.key IS NULL;
By defining the "tables" o and n at the top, I was able to avoid repeating the expressions things(OLD) and things(NEW).
Sure, we could probably eliminate the UNION ALL using a FULL JOIN, but I wasn't able to do that in my particular case.
If I understand your query correctly, it does this:
Find the oldest row in global.prospect whose status is 'new' or 'reset'.
Mark it by adding an asterisk to its status
Return the row (including our tweak to status).
I don't think WITH will simplify anything in your case. It may be slightly more elegant to use a FROM clause, though:
update global.prospect psp
set status = status || '*'
from ( select psp_id
from global.prospect
where status = 'new' or status = 'reset'
order by request_ts
limit 1
) p2
where psp.psp_id = p2.psp_id
returning psp.*;
Untested. Let me know if it works.
It's pretty much exactly what you have already, except:
This can be easily extended to update multiple rows. In your version, which uses a subquery expression, the query would fail if the subquery were changed to yield multiple rows.
I did not alias global.prospect in the subquery, so it's a bit easier to read. Since this uses a FROM clause, you'll get an error if you accidentally reference the table being updated.
In your version, the subquery expression is encountered for every single item. Although PostgreSQL should optimize this and only evaluate the expression once, this optimization will go away if you accidentally reference a column in psp or add a volatile expression.

Related

RETURNING causes error: missing FROM-clause entry for table

I am getting the users data from UUID WHERE empl_user_pub_uuid = 'e2bb39f1f28011eab66c63cb4d9c7a34'.
Since I don't want to make an additional query to fetch additional user data I'm trying to sneak them through the INSERT.
WITH _u AS (
SELECT
eu.empl_user_pvt_uuid,
ee.email,
ep.name_first
FROM employees.users eu
LEFT JOIN (
SELECT DISTINCT ON (ee.empl_user_pvt_uuid)
ee.empl_user_pvt_uuid,
ee.email
FROM employees.emails ee
ORDER BY ee.empl_user_pvt_uuid, ee.t DESC
) ee ON eu.empl_user_pvt_uuid = ee.empl_user_pvt_uuid
LEFT JOIN (
SELECT DISTINCT ON (ep.empl_user_pvt_uuid)
ep.empl_user_pvt_uuid,
ep.name_first
FROM employees.profiles ep
) ep ON eu.empl_user_pvt_uuid = ep.empl_user_pvt_uuid
WHERE empl_user_pub_uuid = 'e2bb39f1f28011eab66c63cb4d9c7a34'
)
INSERT INTO employees.password_resets (empl_pwd_reset_uuid, empl_user_pvt_uuid, t_valid, for_empl_user_pvt_uuid, token)
SELECT 'f70a0346-a077-11eb-bd1a-aaaaaaaaaaaa', '6efc2b7a-f27e-11ea-b66c-de1c405de048', '2021-04-18 19:57:47.111365', _u.empl_user_pvt_uuid, '19d65aea-7c4a-41bc-b580-9d047f1503e6'
FROM _u
RETURNING _u.empl_user_pvt_uuid, _u.email, _u.name_first;
However I get:
[42P01] ERROR: missing FROM-clause entry for table "_u"
Position: 994
What am I doing wrong?
It's true, as has been noted, that the RETURNING clause of an INSERT only sees the inserted row. More specifically, quoting the manual here:
The optional RETURNING clause causes INSERT to compute and return
value(s) based on each row actually inserted (or updated, if an ON CONFLICT DO UPDATE clause was used). This is primarily useful for
obtaining values that were supplied by defaults, such as a serial
sequence number. However, any expression using the table's columns
is allowed. The syntax of the RETURNING list is identical to that
of the output list of SELECT. Only rows that were successfully
inserted or updated will be returned. [...]
Bold emphasis mine.
So nothing keeps you from adding a correlated subquery to the RETURNING list:
INSERT INTO employees.password_resets AS ep
(empl_pwd_reset_uuid , empl_user_pvt_uuid , t_valid , for_empl_user_pvt_uuid, token)
SELECT 'f70a0346-a077-11eb-bd1a-aaaaaaaaaaaa', '6efc2b7a-f27e-11ea-b66c-de1c405de048', '2021-04-18 19:57:47.111365', eu.empl_user_pvt_uuid , '19d65aea-7c4a-41bc-b580-9d047f1503e6'
FROM employees.users eu
WHERE empl_user_pub_uuid = 'e2bb39f1f28011eab66c63cb4d9c7a34'
RETURNING for_empl_user_pvt_uuid AS empl_user_pvt_uuid -- alias to meet your org. query
, (SELECT email
FROM employees.emails
WHERE empl_user_pvt_uuid = ep.empl_user_pvt_uuid
ORDER BY t DESC -- NULLS LAST ?
LIMIT 1
) AS email
, (SELECT name_first
FROM employees.profiles
WHERE empl_user_pvt_uuid = ep.empl_user_pvt_uuid
-- ORDER BY ???
LIMIT 1
) AS name_first;
This is also much more efficient than the query you had (or what was proposed) for multiple reasons.
We don't run the subqueries ee and ep over all rows of the tables employees.emails and employees.profiles. That would be efficient if we needed major parts of those tables, but we only fetch a single row of interest from each. With appropriate indexes, a correlated subquery is much more efficient for this. See:
Efficient query to get greatest value per group from big table
Two SQL LEFT JOINS produce incorrect result
Select first row in each GROUP BY group?
Optimize GROUP BY query to retrieve latest row per user
We don't add the overhead of one or more CTEs.
We only fetch additional data after a successful INSERT, so no time is wasted if the insert didn't go through for any reason. (See quote at the top!)
Plus, possibly most important, this is correct. We use data from the row that has actually been inserted - after inserting it. (See quote at the top!) After possible default values, triggers or rules have been applied. We can be certain that what we see is what's actually in the database (currently).
You have no ORDER BY for profiles.name_first. That's not right. Either there is only one qualifying row, then we need no DISTINCT nor LIMIT 1. Or there can be multiple, then we also need a deterministic ORDER BY to get a deterministic result.
And if emails.t can be NULL, you'll want to add NULLS LAST in the ORDER BY clause. See:
Sort by column ASC, but NULL values first?
Indexes
Ideally, you have these multicolumn indexes (with columns in this order):
users (empl_user_pub_uuid, empl_user_pvt_uuid)
emails (empl_user_pvt_uuid, email)
profiles (empl_user_pvt_uuid, name_first)
Then, if the tables are vacuumed enough, you get three index-only scans and the whole operation is lightening fast.
Get pre-INSERT values?
If you really want that (which I don't think you do), consider:
Return pre-UPDATE column values using SQL only
According Postgres Docs about 6.4. Returning Data From Modified Rows :
In an INSERT, the data available to RETURNING is the row as it was
inserted.
But here you are trying to return columns from source table instead of destination. Returning will not be able to return columns form _u table rather only from employees.password_resets table's inserted row. But you can write nested cte for insertion and can select data from source table as well. Please try below approach.
WITH _u AS (
SELECT
eu.empl_user_pvt_uuid,
ee.email,
ep.name_first
FROM employees.users eu
LEFT JOIN (
SELECT DISTINCT ON (ee.empl_user_pvt_uuid)
ee.empl_user_pvt_uuid,
ee.email
FROM employees.emails ee
ORDER BY ee.empl_user_pvt_uuid, ee.t DESC
) ee ON eu.empl_user_pvt_uuid = ee.empl_user_pvt_uuid
LEFT JOIN (
SELECT DISTINCT ON (ep.empl_user_pvt_uuid)
ep.empl_user_pvt_uuid,
ep.name_first
FROM employees.profiles ep
) ep ON eu.empl_user_pvt_uuid = ep.empl_user_pvt_uuid
WHERE empl_user_pub_uuid = 'e2bb39f1f28011eab66c63cb4d9c7a34'
), I as
(
INSERT INTO employees.password_resets (empl_pwd_reset_uuid, empl_user_pvt_uuid, t_valid, for_empl_user_pvt_uuid, token)
SELECT 'f70a0346-a077-11eb-bd1a-aaaaaaaaaaaa', '6efc2b7a-f27e-11ea-b66c-de1c405de048', '2021-04-18 19:57:47.111365', _u.empl_user_pvt_uuid, '19d65aea-7c4a-41bc-b580-9d047f1503e6'
FROM _u
)
select _u.empl_user_pvt_uuid, _u.email, _u.name_first from _u

how to use listagg operator so that the query should fetch comma separated values

SELECT (SELECT STRING_VALUE
FROM EMP_NODE_PROPERTIES
WHERE NODE_ID=AN.ID ) containedWithin
FROM EMP_NODE AN
WHERE AN.STORE_ID = ALS.ID
AND an.TYPE_QNAME_ID=(SELECT ID
FROM EMP_QNAME
where LOCAL_NAME = 'document')
AND
AND AN.UUID='13456677';
from the above query I am getting below error.
ORA-01427: single-row subquery returns more than one row
so how to change the above query so that it should fetch comma separated values
This query won't return error you mentioned because
there are two ANDs and
there's no ALS table (or its alias).
I suggest you post something that is correctly written, then we can discuss other errors.
Basically, it is either select string_value ... or select id ... (or even both of them) that return more than a single value.
The most obvious "solution" is to use select DISTINCT
another one is to include where rownum = 1
or, use aggregate functions, e.g. select max(string_value) ...
while the most appropriate option would be to join all tables involved and decide which row (value) is correct and adjust query (i.e. its WHERE clause) to make sure that desired value will be returned.
You would seem to want something like this:
SELECT LISTAGG(NP.STRING_VALUE, ',') WITHIN GROUP(ORDER BY NP.STRING_VALUE)
as containedWithin
FROM EMP_NODE N
JOIN EMP_QNAME Q
ON N.TYPE_QNAME_ID = Q.ID
LEFT JOIN EMP_NODE_PROPERTIES NP
ON NP.NODE_ID = N.ID
WHERE Q.LOCAL_NAME = 'document'
AND AN.UUID = '13456677';
This is a bit speculative because your original query would not run for the reason explained by Littlefoot.

performing an update query with a select subquery returning either the same value for ALL of the records or ora-01427 error

I need to update a column in one table with the results from a select sub-query (and they should ultimately be different). But When I do this, I either get the exact same number for the hundreds of records, or I get the ORA-01427: single row sub-query returns more than one row query. error.
Can you please take a look and see what it is that I am overlooking? (I could just be overlooking something simple for all I know)
UPDATE WD_PRODUCT_CLASS
SET CURRENT_CASES = ( WITH STUFF_COUNT AS
(
SELECT sum(CURRENT_DETAIL.COMBINED_QTY) AS TOTAL_CASES
FROM CURRENT_DETAIL, SKU_MAJORS, WD_PRODUCT_CLASS
WHERE CURRENT_DETAIL.LOC_ID =
&PARM_LOC_ID
AND CURRENT_DETAIL.INVEN_ID = SKU_MAJORS.INVEN_ID
AND WD_PRODUCT_CLASS.CATEGORY = SKU_MAJORS.CONT_DESC
GROUP BY WD_PRODUCT_CLASS.CATEGORY
)
(
SELECT SUM(Z.TOTAL_CASES) FROM STUFF_COUNT Z
)
);
Maybe you need someting like this:
UPDATE WD_PRODUCT_CLASS wpc
SET wpc.CURRENT_CASES = (
SELECT sum(cd.COMBINED_QTY)
FROM CURRENT_DETAIL cd join SKU_MAJORS sm ON cd.INVEN_ID = sm.INVEN_ID
WHERE cd.LOC_ID = &PARM_LOC_ID
AND sm.CONT_DESC = wpc.CATEGORY
)
WHERE 1=1; -- if you don't set a condition all the rows will be updated
Your query updates the table with the same values because you're using a not correlated subquery in the SET clause. This subquery don't depends on the parent query, so it's calculated only once.
I suppose you need a correlated subquery so I changed your update + removed some extra parts.

What is the difference between the IN operator and = operator in SQL?

I am just learning SQL, and I'm wondering what the difference is between the following lines:
WHERE s.parent IN (SELECT l.parent .....)
versus
WHERE s.parent = (SELECT l.parent .....)
IN
will not generate an error if you have multiple results on the subquery. Allows to have more than one value in the result returned by the subquery.
=
will generate an error if you have more than one result on the subquery.
SQLFiddle Demo (IN vs =)
when you are using 'IN' it can compare multiple values....like
select * from tablename where student_name in('mari','sruthi','takudu')
but when you are using '=' you can't compare multiple values
select * from tablenamewhere student_name = 'sruthi'
i hope this is the right answer
The "IN" clause is also much much much much slower. If you have many results in the select portion of
IN (SELECT l.parent .....),
it will be extremely inefficient as it actually generates a separate select sql statement for each and every result within the select statement ... so if you return 'Cat', 'Dog', 'Cow'
it will essentially create a sql statement for each result... if you have 200 results... you get the full sql statement 200 times...takes forever... (This was as of a few years ago... maybe imporved by now... but it was horribly slow on big result sets.)
Much more efficient to do an inner join such as:
Select id, parent
from table1 as T
inner join (Select parent from table2) as T2 on T.parent = T2.parent
For future visitors.
Basically in case of equals (just remember that here we are talking like where a.name = b.name), each cell value from table 1 will be compared one by one to each cell value of all the rows from table 2, if it matches then that row will be selected (here that row will be selected means that row from table 1 and table 2) for the overall result set otherwise will not be selected.
Now, in case of IN, complete result set on the right side of the IN will be used for comparison, so its like each value from table 1 will be checked on whether this cell value is present in the complete result set of the IN, if it is present then that value will be shown for all the rows of the IN’s result set, so let say IN result set has 20 rows, so that cell value from table 1 will be present in overall result set 20 times (i.e. that particular cell value will have 20 rows).
For more clarity see below screen shot, notice below that how complete result set from the right of the IN (and NOT IN) is considered in the overall result set; whole emphasis is on the fact that in case comparison using =, matching row from second table is selected, while in case of IN complete result from the second table is selected.
In can match a value with more than one values, in other words it checks if a value is in the list of values so for e.g.
x in ('a', 'b', 'x') will return true result as x is in the the list of values
while = expects only one value, its as simple as
x = y returns false
and
x = x returns true
The general rule of thumb is:
The = expects a single value to compare with. Like this:
WHERE s.parent = 'father_name'
IN is extremely useful in scenarios where = cannot work i.e. scenarios where you need the comparison with multiple values.
WHERE s.parent IN ('father_name', 'mother_name', 'brother_name', 'sister_name')
Hope this is useful!!!
IN
This helps when a subquery returns more than one result.
=
This operator cannot handle more than one result.
Like in this example:
SQL>
Select LOC from dept where DEPTNO = (select DEPTNO from emp where
JOB='MANAGER');
Gives ERROR ORA-01427: single-row subquery returns more than one row
Instead use
SQL>
Select LOC from dept where DEPTNO in (select DEPTNO from emp
where JOB='MANAGER');
1) Sometimes = also used as comparison operator in case of joins which IN doesn't.
2) You can pass multiple values in the IN block which you can't do with =. For example,
SELECT * FROM [Products] where ProductID IN((select max(ProductID) from Products),
(select min(ProductID) from Products))
would work and provide you expected number of rows.However,
SELECT * FROM [Products] where ProductID = (select max(ProductID) from Products)
and ProductID =(select min(ProductID) from Products)
will provide you 'no result'. That means, in case subquery supposed to return multiple number of rows , in that case '=' isn't useful.

How to update a PostgreSQL table with a count of duplicate items

I found two bugs in a program that created a lot of duplicate values:
an 'index' was created instead of a 'unique index'
a duplication checks wasn't integrated in one of 4 twisted routines
So I need to go in and clean up my database.
Step one is to decorate the table with a count of all the duplicate values (next I'll look into finding the first value, and then migrating everything over )
The code below works, I just recall doing a similar "update from select count" on the same table years ago, and I did it in half as much code.
Is there a better way to write this?
UPDATE
shared_link
SET
is_duplicate_of_count = subquery.is_duplicate_of_count
FROM
(
SELECT
count(url) AS is_duplicate_of_count
, url
FROM
shared_link
WHERE
shared_link.url = url
GROUP BY
url
) AS subquery
WHERE
shared_link.url = subquery.url
;
You query is fine, generally, except for the pointless (but also harmless) WHERE clause in the subquery:
UPDATE shared_link
SET is_duplicate_of_count = subquery.is_duplicate_of_count
FROM (
SELECT url
, count(url) AS is_duplicate_of_count
FROM shared_link
-- WHERE shared_link.url = url
GROUP BY url
) AS subquery
WHERE shared_link.url = subquery.url;
The commented clause is the same as
WHERE shared_link.url = shared_link.url
and therefore only eliminating NULL values (because NULL = NULL is not TRUE), which is most probably neither intended nor needed in your setup.
Other than that you can only shorten your code further with aliases and shorter names:
UPDATE shared_link s
SET ct = u.ct
FROM (
SELECT url, count(url) AS ct
FROM shared_link
GROUP BY 1
) AS u
WHERE s.url = u.url;
In PostgreSQL 9.1 or later you might be able to do the whole operation (identify dupes, consolidate data, remove dupes) in one SQL statement with aggregate and window functions and data-modifying CTEs - thereby eliminating the need for an additional column to begin with.