I'm refactoring the data import procedure for an enterprise application and came across a snippet I'd like to find a better solution. When importing data we have to create a unique entity for each data set and there is a counter in a field to be used to assign this id sequentially. You read the field to get the next free id and increment it afterwards to prepare for the next time.
At the moment this is done in two steps in the original app, written in 'C':
SELECT idnext FROM mytable;
UPDATE mytable SET idnext = idnext + 1;
Obviously there is a race condition here, if multiple processes do the same thing.
Edit: Important corequisite: I can not touch the database/field definition, this rules out a sequence.
We are rewriting in perl, and I'd like to do the same thing, but better. An atomic solution would be nice. Unfortunately my SQL skills are limited, so I'm turning to collective wisdom :-)
In this particular case, a sequence is the right solution as mentioned. But if in some future situation you need to both update something and return a value in the same statement, you can use the RETURNING clause:
UPDATE atable SET foo = do_something_with(foo) RETURNING foo INTO ?
If the calling code is PL/SQL, replace the ? with a local PL/SQL variable; otherwise you can bind it as an output parameter in your program.
Edit: Since you mentioned Perl, something like this ought to work (untested):
my $sth = $dbh->prepare('UPDATE mytable SET idnext = idnext + 1 returning idnext into ?');
my $idnext;
$sth->bind_param_inout(1, \$idnext, 8);
$sth->execute; # now $idnext should contain the value
See DBI.
Why not use a sequence?
Create the sequence one time, using whatever START WITH value you want:
CREATE SEQUENCE mysequence
START WITH 1
MAXVALUE 999999999999999999999999999
MINVALUE 1
NOCYCLE
NOCACHE
NOORDER;
Then in your application code at runtime you can use this statement to get the next value:
SELECT mysequence.NEXTVAL
INTO idnext
FROM DUAL;
Update: Using a sequence would be the preferred method, but since you can't change the database then I agree that using RETURNING should work for your situation:
UPDATE mytable
SET idnext = idnext + 1
RETURNING idnext
INTO mylocalvariable;
Use SELECT FOR UPDATE statement. It guarantees mutually exclusive rights to the record :
"SELECT
FOR UPDATE;
A sequence will do the job, have a look at e.g. Oracle sequences
Related
I have some lines of SQL which will take a set of IDs from the same GROUP_ID that are not contiguous (ex. if some rows got deleted) and will make them contiguous again. I wanted to turn this into a function for reusability purposes. The lines work if executed individually but when I try to create the function I get the error
ERROR: relation "id_seq_temp" does not exist
LINE 10: UPDATE THINGS SET ID=nextval('id_se...
If I create a sequence outside of the function and use that sequence in the function instead then the function is created successfully (schema qualified or unqualified). However I felt like creating the temp sequence inside of the function rather than leaving it in the schema was a cleaner solution.
I have seen this question: Function shows error "relation my_table does not exist"
However, I'm using the public schema and schema qualifying the sequence with public. does not seem to help.
I've also seen this question: How to create a sql function using temp sequences and a SELECT on PostgreSQL8. I probably could use generate_series but this adds a lot of complexity that SERIES solves such as needing to know how big of a series to generate.
Here is my function, I anonymized some of the names - just in case there's a typo.
CREATE OR REPLACE FUNCTION reindex_ids(IN BIGINT) RETURNS VOID
LANGUAGE SQL
AS $$
CREATE TEMPORARY SEQUENCE id_seq_temp
MINVALUE 1
START WITH 1
INCREMENT BY 1;
ALTER SEQUENCE id_seq_temp RESTART;
UPDATE THINGS SET ID=ID+2000 WHERE GROUP_ID=$1;
UPDATE THINGS SET ID=nextval('id_seq_temp') WHERE GROUP_ID=$1;
$$;
Is it possible to use a sequence you create within a function later in the function?
Answer to question
The reason is that SQL functions (LANGUAGE sql) are parsed and planned as one. All objects used must exist before the function runs.
You can switch to PL/pgSQL, (LANGUAGE plpgsql) which plans each statement on demand. There you can create objects and use them in the next command.
See:
Why can PL/pgSQL functions have side effect, while SQL functions can't?
Since you are not returning anything, consider a PROCEDURE. (FUNCTION works, too.)
CREATE OR REPLACE PROCEDURE reindex_ids(IN bigint)
LANGUAGE plpgsql AS
$proc$
BEGIN
IF EXISTS ( SELECT FROM pg_catalog.pg_class
WHERE relname = 'id_seq_temp'
AND relnamespace = pg_my_temp_schema()
AND relkind = 'S') THEN
ALTER SEQUENCE id_seq_temp RESTART;
ELSE
CREATE TEMP SEQUENCE id_seq_temp;
END IF;
UPDATE things SET id = id + 2000 WHERE group_id = $1;
UPDATE things SET id = nextval('id_seq_temp') WHERE group_id = $1;
END
$proc$;
Call:
CALL reindex_ids(123);
This creates your temp sequence if it does not exist already.
If the sequence exists, it is reset. (Remember that temporary objects live for the duration of a session.)
In the unlikely event that some other object occupies the name, an exception is raised.
Alternative solutions
Solution 1
This usually works:
UPDATE things t
SET id = t1.new_id
FROM (
SELECT pk_id, row_number() OVER (ORDER BY id) AS new_id
FROM things
WHERE group_id = $1 -- your input here
) t1
WHERE t.pk_id = t1.pk_id;
And only updates each row once, so half the cost.
Replace pk_id with your PRIMARY KEY column, or any UNIQUE NOT NULL (combination of) column(s).
The trick is that the UPDATE typically processes rows according to the sort order of the subquery in the FROM clause. Updating in ascending order should never hit a duplicate key violation.
And the ORDER BY clause of the window function row_number() imposes that sort order on the resulting set. That's an undocumented implementation detail, so you might want to add an explicit ORDER BY to the subquery. But since the behavior of UPDATE is undocumented anyway, it still depends on an implementation detail.
You can wrap that into a plain SQL function.
Solution 2
Consider not doing what you are doing at all. Gaps in sequential numbers are typically expected and not a problem. Just live with it. See:
Serial numbers per group of rows for compound key
Not that familiar with SQL.
I have the follow SQL :
UPDATE my_Table
SET num = num + 1
WHERE id = 1
I would like limit that num column that it won't exceed a threshold, assuming 100. If num will reach 100, I would like it to stay 100 and will not increase.
I assume if statement should be included here somehow. but cant figure out how.
I'm seeing answers that would solve your problem from an update statement, all fine whatever way it is done.
Until someone creates a statement that doesn't adhere to your constraint, like a new co-worker, someone sleepy that forgets it or whatever reason. You should probably know about other options to force this rule.
You could create an SQL constraint that checks that num is never set to more than 100. That way any update or insert statement that makes this value more than 100 would result in an error rather than doing the insert or update.
Another way to force this rule "under the hood", not really beginner level, would be to create an update and insert trigger that would check for num being more than 100 and reset it to 100 in that case. That wouldn't result in an error when you run the insert or update statement. I don't have a good link for ANSI-SQL triggers, but each RDBMS has good documentation.
You can use a CASE expression:
UPDATE my_Table
SET num = CASE WHEN num+1 >= 100 THEN 100 ELSE num+1 END
WHERE id=1
try:
UPDATE my_Table SET num=num+1 WHERE id=1 AND num<=99
in that case you can use IIF for shorter version
UPDATE my_Table SET num=(IIF((num+1)>100,num,num+1)) WHERE id=1
I would like to know how to loop through each record in a database, and update them using a counter.
For instance, I have a large number of email fields. I would like to update them all using the pattern: 'hello+1#gmail.com', 'hello+2#gmail.com', 'hello+3#gmail.com', 'hello+4#gmail.com' ...
I tried to do this without SQL in my Rails console, but it would take too much time. Any suggestions on how to do this using SQL?
If you are using SQL server:
select <stuff>
,row_number() over (order by <something>) as row_number
from <things>
where <something>
Will do the trick. If not, let us know what you are using and we can go from there! Are you looking to put the '+Row_number' before the #? If so it should be a fairly simple matter of replacing the the '#' with '+4#' and so on.
In MySQL, you would use variables:
update table t
set email = concat('hello+', #rn := coalesce(#rn, 0) + 1, '#gmail.com');
If you have a particular ordering in mind, you can add an order by clause.
I'm looking for a possible solution to the following. I have data stored in a table to keep track of a special increment number the customer wants in the DB. This is a special number they use internally.
What I would like to do is automatically increment this number in the table when I select it. So I don't have the problem of another transaction, from someone else using the system, using the same ID number.
So I want to select the current number and increment it by one at once so I don't have duplicates. How would I go about doing this if it is even possible?
UPDATE the_table
SET the_column = the_column + 1
WHERE qualifier = X
RETURNING the_column;
This ought to do the trick, with the caveat that it will return the new id rather than the old one:
UPDATE foo
SET id=nextval('foo_sequence')
WHERE ...
RETURNING *
I'd like to get all the records from a huge table where any of the number columns countains a value greater than 0. What's the best way to do it?
E.g.:
/* table structure*/
create table sometable (id number,
somestring varchar2(12),
some_amount_1 number(17,3),
some_amount_2 number(17,3),
some_amount_3 number(17,3),
...
some_amount_xxx number(17,3));
/* "xxx" > 100, and yeah I did not designed that table structure... */
And I want any row where any of the some_amount_n > 0 (even better solution is to add a field in the first place to show which field(s) are greater than zero).
I know I can write this with a huge some_amount_1 > 0 OR some_amount_2 > 0 OR ... block (and the field names with some case when but is there should be some more elegant solution, isn't there?
Possible solutions:
Normalize the table. You said you are not allowed to. Try to convince those that forbid such a change by explaining the benefits (performance, ease of writing queries, etc).
Write the huge ugly OR query. You could also print it along with the version of the query for the normalized tables. Add performance tests (you are allowed to create another test table or database, I hope.)
Write a program (either in PL/SQL or in another procedural language) that produces the horrible OR query. (Again, print along with the elegant version)
Add a new column, say called Any_x_bigger_than_zero which is automatically filled with either 0 or 1 via a trigger (that uses a huge ugly OR). Then you just need to check: WHERE Test_x_bigger_than_zero = 1 to see if any of the rows is > 0
Similar to previous but even better, create a materialized view with such a column.
First, create a table to sort the data into something more easily read from...something simple like id,column_name,column_value. You'll have to bear with me, been a while since I've operated in oracle, so this is heavy pseudo code at best:
Quick dynamic sql blurb...you can set a variable to a sql statement and then execute that variable. There are some security risks and it's possible this feature is disabled in your environment...so confirm you can run this first. Declare a variable, set the variable to 'select 1' and then use 'execute immediate' to execute the sql stored in your variable.
set var = 'select id, ''some_amount_' || 1 || '', some_amount || 1 || ' from table where some_amount_' || 1 || ' <> 0'
Assuming I've got my oracle syntax right...( pipe is append right? I believe a 3 single quote as ''' should result in one ' when in a variable too, you may have to trial and error this line until you have the var set to):
select id, 'some_amount_1',some_amount_1
from table
where some_amount_1 <> 0
This should select the ID and the value in some_amount_1 for each id in your database. You can turn this into an insert statement pretty easily.
I'm assuming some_amount_xxx has an upper limit...next trick is to loop this giant statement. Once again, horrible pseudo code:
declare sql_string
declare i and set to 1
for i = 1 to xxx (whatever your xxx is)
set sql_string to the first set var statement we made, replacing the '1' with the i var here.
execute sql
increment i
loop
Hopefully it makes sense...it's one of the very few scenarios you would ever want to loop dynamic sql on. Now you have a relatively straight forward table to read from and this should be a relatively easy query from here