I have a table as follows
The table contains my application users and stores their clients. The column User Client ID refers to a foreign key linked to a different table that stores the clients details.
I need another column (User Client Counter) which is a just a counter of the clients of each user. I need it to start from 1 and goes up for each individual application user.
For the moment I'm populating this by counting the number of clients for each user + 1 before inserting a new row in the table :
select count(*) + 1 into MyVariable from Mytable where UserClientId = Something
Then I use MyVariable in the column User Client Counter
This methods works quite well, but in case the user is connected from two different sessions, the query may produce a wrong number of counts... in addition to that the performance may be bad in case of big tables...
Is there anyway better way to replace such process by using sequences ?
I've been looking to session sequences but there are reset after the end of each session.
(This column is a business need and cannot be replaced by something like rownumber in restitution queries. Since every client has to keep always the same identifier for the application user)
Thank you in advance.
Cheers,
I think you can just create a unique index on the app user and the running number:
create unique index idx on mytable (app_user_id, num);
And then insert with max + 1:
insert into mytable (app_user_id, client_id, num)
values
(
:app_user_id,
:client_id,
coalesce((select max(num) + 1 from mytable where app_user_id = :app_user_id), 1)
);
For this sort of requirement to be safe you will need to be able to lock rows at the right level so that you don't have two sessions that think the they are allowed to use the same value. The impact of this is that while one session is inserting a row for the 'Company X' user, another session will wait for the first user to commit if they're also trying to insert a row for 'Company X'.
This is super easy when you just have a table that stores information at the right level.
You can have a table of your users with a counter column which starts at 0.
MY_APPLICATION_USER CLIENT_COUNTER
-------------------------------------------------- --------------
Company X 1
Company Y 3
Company Z 1
As you insert rows into your main table, you update this table first setting the client_counter to be client_counter + 1 (you do this as one insert statement, no risky select then update!), then you return the updated value into your value for the client_id. This can all be done with a simple trigger.
create or replace trigger app_clients_ins
before insert
on app_clients
for each row
declare
begin
update app_users
set client_counter = client_counter + 1
where my_application_user = :new.my_application_user
return client_counter into :new.user_client_number;
end;
/
Of course, like any sequence if you delete a row it's not going to allow your next insert to fill that gap.
(db<>fiddle https://dbfiddle.uk/?rdbms=oracle_18&fiddle=7f1b4f4b6316d5983b921cae7b89317a )
if you want to have unique values to be inserted and there are chances that multiple users can insert rows into the same table at the same time then it is better to user Oracle Sequence.
CREATE SEQUENCE id_seq INCREMENT BY 1;
INSERT INTO Mytable(id) VALUES (id_seq.nextval);
In you case I think you want different sequence created for each Customer, How many different Customers you have, if you have in 100's then i don't think create sequence will work as you may have to create as many sequence .
Related
In our data model, which is derived from the Teradata industry models, we observe a common pattern, where the superclass and subclass relationships in the logical data model are transformed into one-to-one relationships between the parent and the child table.
I know you can roll-up or roll-down the attributes to end up with a single table but we are not using this option overall. At the end what we have is a model like this:
Where City Id references a Geographical Area Id.
I am struggling with a good strategy to load the records in these tables.
Option 1: I could select the max(Geographical Area Id) and calculate the next Ids for a batch insert and reuse them for the City Table.
Option 2: I could use an Identity column in the Geographical Area Table and retrieve it after I insert every record in order to use it for the City table.
Any other options?
I need to assess the solution in terms of performance, reliability and maintenance.
Any comment will be appreciated.
Kind regards,
Paul
When you say "load the records into these tables", are you talking about a one-time data migration or a function that creates records for new Geographical Area/City?
If you are looking for a surrogate key and are OK with gaps in your ID values, then use an IDENTITY column and specify the NO CYCLE clause, so it doesn't repeat any numbers. Then just pass NULL for the value and let TD handle it.
If you do need sequential IDs, then you can just maintain a separate "NextId" table and use that to generate ID values. This is the most flexible way and would make it easier for you to manage your BATCH operations. It requires more code/maintenance on your part, but is more efficient than doing a MAX() + 1 on your data table to get your next ID value. Here's the basic idea:
BEGIN TRANSACTION
Get the "next" ID from a lookup table
Use that value to generate new ID values for your next record(s)
Create your new records
Update the "next" ID value in the lookup table and increment it by the # rows newly inserted (you can capture this by storing the value in the ACTIVITY_COUNT value variable directly after executing your INSERT/MERGE statement)
Make sure to LOCK the lookup table at the beginning of your transaction so it can't be modified until your transaction completes
END TRANSACTION
Here is an example from Postgres, that you can adapt to TD:
CREATE TABLE NextId (
IDType VARCHAR(50) NOT NULL,
NextValue INTEGER NOT NULL,
PRIMARY KEY (IDType)
);
INSERT INTO Users(UserId, UserType)
SELECT
COALESCE(
src.UserId, -- Use UserId if provided (i.e. update existing user)
ROW_NUMBER() OVER(ORDER BY CASE WHEN src.UserId IS NULL THEN 0 ELSE 1 END ASC) +
(id.NextValue - 1) -- Use newly generated UserId (i.e. create new user)
)
AS UserIdFinal,
src.UserType
FROM (
-- Bulk Upsert (get source rows from JSON parameter)
SELECT src.FirstName, src.UserId, src.UserType
FROM JSONB_TO_RECORDSET(pUserDataJSON->'users') AS src(FirstName VARCHAR(100), UserId INTEGER, UserType CHAR(1))
) src
CROSS JOIN (
-- Get next ID value to use
SELECT NextValue
FROM NextId
WHERE IdType = 'User'
FOR UPDATE -- Use "Update" row-lock so it is not read by any other queries also using "Update" row-lock
) id
ON CONFLICT(UserId) DO UPDATE SET
UserType = EXCLUDED.UserType;
-- Increment UserId value
UPDATE NextId
SET NextValue = NextValue + COALESCE(NewUserCount,0)
WHERE IdType = 'User'
;
Just change the locking statement to Teradata syntax (LOCK TABLE NextId FOR WRITE) and add an ACTIVITY_COUNT variable after your INSERT/MERGE to capture the # rows affected. This assumes you're doing all this inside a stored procedure.
Let me know how it goes...
I'm did some research about SQL batch inserts - let's say I have 100k items to be inserted, and I set the batch size to 100.
If the ID column is not marked as Identity then that bulk insert will work.
But I found quite interesting (theoretical so far) problem, and I need some opinions:
The problem can be, if e.g. 5 users are making the bulk inserts in the same time, how then safely provide the the ID column value ? I can't just get the table rows count + 1, because in that way all of that 5 users will have the ID duplicates and the bulk insert operation will fail.
You can use SEQUENCE as an UNIQUE ID generator or try TRIGGER ON INSERT to get a unique ID.
EDIT
With mysql you can build trigger for every row
DELIMITER $$
CREATE TRIGGER adresse_trigger_insert_check
BEFORE INSERT ON adresse
FOR EACH ROW BEGIN
IF NEW.land IS NULL THEN
SET NEW.land := 'XY';
END IF;
END$$
DELIMITER ;
Should I Use IDENTITY or Not?
Sometimes Another Approach Works Better
I have the following problem, I want to have Composite Primary Key like:
PRIMARY KEY (`base`, `id`);
for which when I insert a base the id to be auto-incremented based on the previous id for the same base
Example:
base id
A 1
A 2
B 1
C 1
Is there a way when I say:
INSERT INTO table(base) VALUES ('A')
to insert a new record with id 3 because that is the next id for base 'A'?
The resulting table should be:
base id
A 1
A 2
B 1
C 1
A 3
Is it possible to do it on the DB exactly since if done programmatically it could cause racing conditions.
EDIT
The base currently represents a company, the id represents invoice number. There should be auto-incrementing invoice numbers for each company but there could be cases where two companies have invoices with the same number. Users logged with a company should be able to sort, filter and search by those invoice numbers.
Ever since someone posted a similar question, I've been pondering this. The first problem is that DBs don't provide "partitionable" sequences (that would restart/remember based on different keys). The second is that the SEQUENCE objects that are provided are geared around fast access, and can't be rolled back (ie, you will get gaps). This essentially this rules out using a built-in utility... meaning we have to roll our own.
The first thing we're going to need is a table to store our sequence numbers. This can be fairly simple:
CREATE TABLE Invoice_Sequence (base CHAR(1) PRIMARY KEY CLUSTERED,
invoiceNumber INTEGER);
In reality the base column should be a foreign-key reference to whatever table/id defines the business(es)/entities you're issuing invoices for. In this table, you want entries to be unique per issued-entity.
Next, you want a stored proc that will take a key (base) and spit out the next number in the sequence (invoiceNumber). The set of keys necessary will vary (ie, some invoice numbers must contain the year or full date of issue), but the base form for this situation is as follows:
CREATE PROCEDURE Next_Invoice_Number #baseKey CHAR(1),
#invoiceNumber INTEGER OUTPUT
AS MERGE INTO Invoice_Sequence Stored
USING (VALUES (#baseKey)) Incoming(base)
ON Incoming.base = Stored.base
WHEN MATCHED THEN UPDATE SET Stored.invoiceNumber = Stored.invoiceNumber + 1
WHEN NOT MATCHED BY TARGET THEN INSERT (base) VALUES(#baseKey)
OUTPUT INSERTED.invoiceNumber ;;
Note that:
You must run this in a serialized transaction
The transaction must be the same one that's inserting into the destination (invoice) table.
That's right, you'll still get blocking per-business when issuing invoice numbers. You can't avoid this if invoice numbers must be sequential, with no gaps - until the row is actually committed, it might be rolled back, meaning that the invoice number wouldn't have been issued.
Now, since you don't want to have to remember to call the procedure for the entry, wrap it up in a trigger:
CREATE TRIGGER Populate_Invoice_Number ON Invoice INSTEAD OF INSERT
AS
DECLARE #invoiceNumber INTEGER
BEGIN
EXEC Next_Invoice_Number Inserted.base, #invoiceNumber OUTPUT
INSERT INTO Invoice (base, invoiceNumber)
VALUES (Inserted.base, #invoiceNumber)
END
(obviously, you have more columns, including others that should be auto-populated - you'll need to fill them in)
...which you can then use by simply saying:
INSERT INTO Invoice (base) VALUES('A');
So what have we done? Mostly, all this work was about shrinking the number of rows locked by a transaction. Until this INSERT is committed, there are only two rows locked:
The row in Invoice_Sequence maintaining the sequence number
The row in Invoice for the new invoice.
All other rows for a particular base are free - they can be updated or queried at will (deleting information out of this kind of system tends to make accountants nervous). You probably need to decide what should happen when queries would normally include the pending invoice...
you can use the trigger for before insert and assign the next value by taking the max(id) with "base" filter which is "A" in this case.
That will give you the max(id) value as 2 and than increment it by max(id)+1. now push the new value to the "id" field. before insert.
I think this may help you
MSSQL Triggers: http://msdn.microsoft.com/en-in/library/ms189799.aspx
Test Table
CREATE TABLE MyTable
( base CHAR(1),
id INT
)
GO
Trigger Definition
CREATE TRIGGER dbo.tr_Populate_ID
ON dbo.MyTable
INSTEAD OF INSERT
AS
BEGIN
SET NOCOUNT ON;
INSERT INTO MyTable (base,id)
SELECT i.base, ISNULL(MAX(mt.id),0) +1 AS NextValue
FROM inserted i left join MyTable mt
on i.base = mt.base
GROUP BY i.base
END
Test
Execute the following statement multiple times and you will see the next values available in that group will be assigned to ID.
INSERT INTO MyTable VALUES
('A'),
('B'),
('C')
GO
SELECT * FROM MyTable
GO
Im writing a VB app that is scrubbing some data inside a DB2 database. In a few tables i want to update entire columns. For example an account number column. I am changing all account numbers to start at 1, and increment as I go down the list. Id like to be able to return both the old account number, and the new one so I can generate some kind of report I can reference so I dont lose the original values. Im updating columns as so:
DECLARE #accntnum INT
SET #accntnum = 0
UPDATE accounts
SET #accntnum = accntnum = #accntnum + 1
GO
Is there a way for me to return both the original accntnum and the new one in one table?
DB2 has a really nifty feature where you can select data from a "data change statement". This was tested on DB2 for Linux/Unix/Windows, but I think that it should also work on at least DB2 for z/OS.
For your numbering, you might considering creating a sequence, as well. Then your update would be something like:
CREATE SEQUENCE acct_seq
START WITH 1
INCREMENT BY 1
NO MAXVALUE
NO CYCLE
CACHE 24
;
SELECT accntnum AS new_acct, old_acct
FROM FINAL TABLE (
UPDATE accounts INCLUDE(old_acct INT)
SET accntnum = NEXT VALUE FOR acct_seq, old_acct = accntnum
)
ORDER BY old_acct;
The INCLUDE part creates a new column in the resulting table with the name and the data type specified, and then you can set the value in the update statement as you would any other field.
A possible solution is to add an additional column (let's call it oldaccntnum) and assign old values to that column as you do your update.
Then drop it when you no longer need it.
Here's what I'd do:
-- create a new table to track the changes.
- with columns identifying a unique key, old-vale, new-value, timestamp
-- create a trigger on the accounts table
to write the old and new values to the new table.
But, not knowing all the conditions, it may not be worth the trouble.
Is it possible in SQL (SQL Server) to retrieve the next ID (integer) from an identity column in a table before, and without actually, inserting a row? This is not necessarily the highest ID plus 1 if the most recent row was deleted.
I ask this because we occassionally have to update a live DB with new rows. The ID of the row is used in our code (e.g. Switch (ID){ Case ID: } and must be the same. If our development DB and live DB get out of sync, it would be nice to predict a row ID in advance before deployment.
I could of course SET IDENTITY OFF SET INSERT_IDENTITY ON or run a transaction (does this roll back the ID?) etc but wondered if there was a function that returned the next ID (without incrementing it).
try IDENT_CURRENT:
Select IDENT_CURRENT('yourtablename')
This works even if you haven't inserted any rows in the current session:
Returns the last identity value generated for a specified table or view. The last identity value generated can be for any session and any scope.
Edit:
After spending a number of hours comparing entire page dumps, I realised there is an easier way and I should of stayed on the DMVs.
The value survives a backup / restore, which is a clear indication that it is stored - I dumped all the pages in the DB and couldn't find the location / alteration for when
a record was added. Comparing 200k line dumps of pages isn't fun.
I had used the dedicated admin console I took a dump of every single internal table exposed inserted a row and then took a further dump of the system tables. Both of the dumps were identical, which indicates that whilst it survived, and therefore must be stored, it is not exposed even at that level.
So after going around in a circle I realised the DMV did have the answer.
create table foo (MyID int identity not null, MyField char(10))
insert into foo values ('test')
go 10
-- Inserted 10 rows
select Convert(varchar(8),increment_value) as IncrementValue,
Convert(varchar(8),last_value) as LastValue
from sys.identity_columns where name ='myid'
-- insert another row
insert into foo values ('test')
-- check the values again
select Convert(varchar(8),increment_value) as IncrementValue,
Convert(varchar(8),last_value) as LastValue
from sys.identity_columns where name ='myid'
-- delete the rows
delete from foo
-- check the DMV again
select Convert(varchar(8),increment_value) as IncrementValue,
Convert(varchar(8),last_value) as LastValue
from sys.identity_columns where name ='myid'
-- value is currently 11 and increment is 1, so the next insert gets 12
insert into foo values ('test')
select * from foo
Result:
MyID MyField
----------- ----------
12 test
(1 row(s) affected)
Just because the rows got removed, the last value was not reset, so the last value + increment should be the right answer.
Also going to write up the episode on my blog.
Oh, and the short cut to it all:
select ident_current('foo') + ident_incr('foo')
So it actually turns out to be easy - but this all assumes no one else has used your ID whilst you got it back. Fine for investigation, but I wouldn't want to use it in code.
This is a little bit strange but it will work:
If you want to know the next value, start by getting the greatest value plus one:
SELECT max(id) FROM yourtable
To make this work, you'll need to reset the identity on insert:
DECLARE #value INTEGER
SELECT #value = max(id) + 1 FROM yourtable
DBCC CHECKIDENT (yourtable, reseed, #value)
INSERT INTO yourtable ...
Not exactly an elegant solution but I haven't had my coffee yet ;-)
(This also assumes that there is nothing done to the table by your process or any other process between the first and second blocks of code).
You can pretty easily determine that the last value used is:
SELECT
last_value
FROM
sys.identity_columns
WHERE
object_id = OBJECT_ID('yourtablename')
Usually, the next ID will be last_value + 1 - but there's no guarantee for that.
Marc
Rather than using an IDENTITY column, you could use a UNIQUEIDENTIFIER (Guid) column as the unique row identifer and insert known values.
The other option (which I use) is SET IDENTITY_INSERT ON, where the row IDs are managed in a source controlled single 'document'.