We're using ORACLE 11.2.0.3.0, configured as 3 node RAC.
In our application we have hibernate over UCP and OJDBC with compatible version to RAC. Hibernate use some sequence to get ID for any record in database. I database we've got table with UNIQUE_CONSTRAINT (some_value) on it. It's used to synchronized many instance of application, every transaction in application requires unique row in this table. So application A tries to insert in this table (some_value="A"), if other application already inserted row with (some_value="A"), first instance get ORA-00001 unique constrain violated, and retry this with other value (some_value="B").
UNIQUE_CONSTRAINT fires very often. Like one in 8tx.
We run two tests:
service pinned to one node: response time avg 6ms
service on all 3 nodes: response time avg 800-1000ms
High level question is why? What is happening in 3 node RAC when UNIQUE_CONSTRAINT occurs, and why it's slowing down so much application. How can I diagnose this case?
Michal
Use service level scaling on RAC. Create a "LOADER" service the RAC side. Make this service active on one node only. And let hibernate use these service "LOADER" connections for loads.
The explanation is - very vague - each cluster node is mastering some subset of database's address space. When using unique constraint, each node must request data blocks of the unique index from it's mastering node. When a duplicit key is found and both duplicit keys were inserted via transactions which were not commited yet. Oracle has to enqueue one session and let it wait till the other session(belonging to another node) commits or rollbacks.
If you need to generate a unique value, you should let the database do it for you. You can create an object called a SEQUENCE. You then get the next value of a sequence simply by
my_seq.nextval
And the current value of the sequence is simply
my_seq.currval
So if you are inserting record...
insert into my_table( my_seq.nextval, 'xxx', yyy, 123, ... )
Related
I'm creating an application with Java Spring and Oracle DB.
In the app, I want to generate a primary key value that is unique as well as ordered and without gaps: 1,2,3,4,5 instead of 1,2,5,7,8,9.
I've at one point used max(id) + 1 to get the maximum value of the id, and the id of the next/current transaction. However I know it isn't perfect in the case of concurrency with multiple users or multiple sessions.
I've tried using sequences, but even with the ORDER tag it could still create gaps with the possibility of a failed transaction.
REATE SEQUENCE num_seq
START WITH 1
INCREMENT BY 1
ORDER NOCACHE NOCYCLE;
I need there to be gapless values as a requirement, however I'm unsure how it's possible in the case of multiple users/multiple sessions.
Don't do it.
The goal of primary keys is not to be displayed on the UI or to be exposed to the external world, but only to provide a unique identifier of the row.
In simple words, a primary key doesn't need to be sexy or good looking. It's an internal identifier.
If you are considering the idea of having serial identifier, that means you probably want to display it somewhere or you want to expose it to the external world. If that's the case, then create a secondary column (also unique) that serves this "public relations" goal. It can be automatically generated, or updated at leisure without affecting the integrity of the database.
It can also be generated by a secondary process that runs in a deferred way (e.g. every 10 minutes) that finds all the "unassigned" new rows, and gives them the new number. This has the advantage that is not vulnerable to concurrency.
Is it possible to get an
ORA-00001: unique constraint (XXX) violated
dueto an
ORA-12899: value too large for column (XXX)
in an oracle database using Hibernate (as is stated in this confluence page)?
(The columns for each error are in different tables but relatad to each other)
In that case, how is this possible?
* UPDATE *
I can confirm the causal relation between the exceptions. The given scenario is as follows:
The are a number processes that perform different operations to the database. This operations are stacked until Hibernate session flush. When you invoke the flush method, the queries are performed in the same transaction.
In my particular case I have the entities A and B that both have inside an entity C (the reference of the entity is the same, there is no copy for each father entity). When the program tries to save A (with a string field too large), first executes the C insert query, and then the insert to the entity itself that leads to a "ORA-12899: value too large for column". At this point C is in the database but not yet commited.
Then the next process tries to save B that contains a C entity and this leads to "ORA-00001: unique constraint violated" on C entity.
My questions are:
When the first process doesnt have errors (no column too large) the second one doesnt try to insert C again, only make the insert to entity B (probably detached state of the entity C?).
Why the execution is not aborted on the first error?
Both exceptions (unique constraint and value too large for column) are related. There are serveral processes executed in a single transaction. Those processes make calls to the methods save() or saveOrUpdate() stacking queries until the flush() of the Hibernate session or commit the transaction.
At some point a flush() of the session is invoked with the given scenario:
Entity A and B both contains the same reference of entity C. The first process tries to insert entity A, so first executes the insert of C without problem, later tries to insert A but fails due to a too large column exception. At this point C is in the database (not yet commited) but the hibernate session is in an incoherent state due to the previous fail and Hibernate doesnt know about C being inserted (a fail on session flush doesnt trigger a rollback, is responsability of the developer).
Then a second process is executed and tries to insert B into the database. If the previous process went ok, Hibernate only inserts the entity B because he knows C is already in the database. Due to the incoherent session state, Hibernate tries to save the C entity again in the database raising a unique constraint exception.
I'm trying Unique Index implemantation with Redis db (ServiceStack Client)
Normally
Check Unique Index Duplication
If Unique Index Exists RETURN WITH WARNING
WATCH for Unique Index (for race-condition)
Open Transaction
Insert new record, Insert new records unique index
Close Transaction
How can I get rid of 1st step?
WATCH for existence. I'm not related with changing of key. I'm related with creation or existance. (surely out of my transaction)
If you are trying to use redis just for checking duplicated then use hashset:
http://redis.io/commands#hash
how do you use the servicestack client? with native client? typed client? (then i can show you how to do that)
and use that command: http://redis.io/commands/hsetnx
Probably a trivial question, but I want to get the best possible solution.
Problem:
I have two or more workers that insert keys into one or more tables. The problem arises when two or more workers try to insert the same key into one of those key tables at the same time.
Typical problem.
Worker A reads the table if a key exists (SELECT). There is no key.
Worker B reads the table if a key exists (SELECT). There is no key.
Worker A inserts the key.
Worker B inserts the key.
Worker A commits.
Worker B commits. Exception is throws as unique constraint is violated
The key tables are simple pairs. First column is autoincrement integer and the second is varchar key.
What is the best solution to such a concurrency problem? I believe it is a common problem. One way for sure is to handle the exceptions thrown, but somehow I don't believe this is the best way to tackle this.
The database I use is Firebird 2.5
EDIT:
Some additional info to make things clear.
Client side synchronization is not a good approach, because the inserts come from different processes (workers). And I could have workers across different machines someday, so even mutexes are a no-go.
The primary key and the first columns of such a table is autoincrement field. No problem there. The varchar field is the problem as it is something that the client inserts.
Typical such table is a table of users. For instance:
1 2056
2 1044
3 1896
4 5966
...
Each worker check if user "xxxx" exists and if not inserts it.
EDIT 2:
Just for the reference if somebody will go the same route. IB/FB return pair of error codes (I am using InterBase Express components). Checking for duplicate value violation look like this:
except
on E: EIBInterBaseError do
begin
if (E.SQLCode = -803) and (E.IBErrorCode = 335544349) then
begin
FKeysConnection.IBT.Rollback;
EnteredKeys := False;
end;
end;
end;
With Firebird you can use the following statement:
UPDATE OR INSERT INTO MY_TABLE (MY_KEY) VALUES (:MY_KEY) MATCHING (MY_KEY) RETURNING MY_ID
assuming there is a BEFORE INSERT trigger which will generate the MY_ID if a NULL value is being inserted.
Here is the documentation.
Update: The above statement will avoid exceptions and cause every statement to succeed. However, in case of many duplicate key values it will also cause many unnecessary updates.
This can be avoided by another approach: just handle the unique constraint exception on the client and ignore it. The details depend on which Delphi library you're using to work with Firebird but it should be possible to examine the SQLCode returned by the server and ignore only the specific case of unique constraint violation.
I do not know if something like this is avalible in Firebird but in SQL Server you can check when inserting the key.
insert into Table1 (KeyValue)
select 'NewKey'
where not exists (select *
from Table1
where KeyValue = 'NewKey')
First option - don't do it.
Don't do it; Unless the WORKERS are doing extraordinary amounts of work (we're talking about computers, so requiring 1 second per record qualifies as "extraordinary amount of work"), just use a single thread; Even better, do all the work in a stored procedure, you'd be amazed by the speedup gained by not transporting data over whatever protocol into your app.
Second option - Use a Queue
Make sure your worker threads don't all work on the same ID. Set up a Queue, push all the ID's that need processing into that queue, have each working thread Dequeue an ID from that Queue. This way you're guaranteed no two workers work on the same record at the same time. This might be difficult to implement if your workers are not all part of the same process.
Last resort
Set up an DB-based "Reservation" system so an Worker Thread can mark a Key for "work in process" so no two workers would work on the same Key. I'd set up a table like this:
CREATE TABLE KEY_RESERVATIONS (
KEY INTEGER NOT NULL, /* This is the KEY you'd be reserving */
RESERVED_UNTIL TIMESTAMP NOT NULL /* We don't want to keep reservations for ever in case of failure */
);
Each of your workers would use short transactions to work on that table: Select a candidate Key, one that's not in the KEY_RESERVATIONS table. Try to INSERT. Failed? Try an other KEY. Periodically delete all reserved key with old RESERVED_UNTIL timestamps. Make sure the transactions for working with KEY_RESERVATIONS are as short as possible, so that two threads both trying to reserve the same key at the same time would fail quickly.
This is what you have to deal with in an optimistic (or no-) locking scheme.
One way to avoid it is to put a pessimistic lock on the table around the whole select, insert, commit sequence.
However, that means you will have to deal with not being able to access the table (handle table-locked exceptions).
If by workers you mean threads in the same application instance instead of different users (application instances), you will need thread synchronization like kubal5003 says around the select-insert-commit sequence.
A combination of the two is needed if you have multiple users/application instances each with multiple threads.
Synchronize your threads to make it impossible to insert the same value or use a db side key generation method (I don't know Firebird so I don't even know if it's there, eg. in MsSQL Server there is identity column or GUIDs also solve the problem because it's unlikely to generate two identical ones)
You should not rely the client to generate the unique key, if there's possibility for duplicates.
Use triggers and generators (maybe with help of stored procedure) to create always unique keys.
More information about proper autoinc implementation in Firebird here: http://www.firebirdfaq.org/faq29/
Earlier today I asked this question which arose from A- My poor planning and B- My complete disregard for the practice of normalizing databases. I spent the last 8 hours reading about normalizing databases and the finer points of JOIN and worked my way through the SQLZoo.com tutorials.
I am enlightened. I understand the purpose of database normalization and how it can suit me. Except that I'm not entirely sure how to execute that vision from a procedural standpoint.
Here's my old vision: 1 table called "files" that held, let's say, a file id and a file url and appropos grade levels for that file.
New vision!: 1 table for "files", 1 table for "grades", and a junction table to mediate.
But that's not my problem. This is a really basic Q that I'm sure has an obvious answer- When I create a record in "files", it gets assigned the incremented primary key automatically (file_id). However, from now on I'm going to need to write that file_id to the other tables as well. Because I don't assign that id manually, how do I know what it is?
If I upload text.doc and it gets file_id 123, how do I know it got 123 in order to write it to "grades" and the junction table? I can't do a max(file_id) because if you have concurrent users, you might nab a different id. I just don't know how to get the file_id value without having manually assigned it.
You may want to use LAST_INSERT_ID() as in the following example:
START TRANSACTION;
INSERT INTO files (file_id, url) VALUES (NULL, 'text.doc');
INSERT INTO grades (file_id, grade) VALUES (LAST_INSERT_ID(), 'some-grade');
COMMIT;
The transaction ensures that the operation remains atomic: This guarantees that either both inserts complete successfully or none at all. This is optional, but it is recommended in order to maintain the integrity of the data.
For LAST_INSERT_ID(), the most
recently generated ID is maintained in
the server on a per-connection basis.
It is not changed by another client.
It is not even changed if you update
another AUTO_INCREMENT column with a
nonmagic value (that is, a value that
is not NULL and not 0).
Using
LAST_INSERT_ID() and AUTO_INCREMENT
columns simultaneously from multiple
clients is perfectly valid. Each
client will receive the last inserted
ID for the last statement that client
executed.
Source and further reading:
MySQL Reference: How to Get the Unique ID for the Last Inserted Row
MySQL Reference: START TRANSACTION, COMMIT, and ROLLBACK Syntax
In PHP to get the automatically generated ID of a MySQL record, use mysqli->insert_id property of your mysqli object.
How are you going to find the entry tomorrow, after your program has forgotten the value of last_insert_id()?
Using a surrogate key is fine, but your table still represents an entity, and you should be able to answer the question: what measurable properties define this particular entity? The set of these properties are the natural key of your table, and even if you use surrogate keys, such a natural key should always exist and you should use it to retrieve information from the table. Use the surrogate key to enforce referential integrity, for indexing purpuses and to make joins easier on the eye. But don't let them escape from the database