I have a problem with JPA and SQlite.
I have created an Entity from Table. My generated Entity looks like:
#Entity
#Table(name="sqliteTestTable")
public class Test implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy=GenerationType.IDENTITY)
#Column(name="id")
private int id;
#Column(name="name")
private String name;
public Test() {
}
------
}
When i try to persist a few Test objects i get following error:
(I have executed the same code on mysql without problems)
Exception in thread "main" Local Exception Stack:
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.1.v20121003-ad44345): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: no such table: SEQUENCE
Error Code: 0
Call: UPDATE SEQUENCE SET SEQ_COUNT = SEQ_COUNT + ? WHERE SEQ_NAME = ?
bind => [50, SEQ_GEN_TABLE]
Query: DataModifyQuery(name="SEQ_GEN_TABLE" sql="UPDATE SEQUENCE SET SEQ_COUNT = SEQ_COUNT + ? WHERE SEQ_NAME = ?")
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:333)
How can i solve this problem?
IDENTITY will default to TABLE if IDENTITY is not supported in the database platform.
For TABLE you need to have a sequence table created. If you have ElipseLink create your tables it will automatically be created for you, otherwise you need to create it yourself.
If SQlite has IDENTITY support, then you can create your own DatabasePlatform subclass that adds the identity support.
In general I would never recommend using IDENTITY, it does not support pre-allocation and will lead to poor performance.
sqlite uses the table sqlite_sequence for AUTOINCREMENT field
replace:
#GeneratedValue(strategy=GenerationType.IDENTITY)
with
#GeneratedValue(generator="sqlite")
#TableGenerator(name="sqlite", table="sqlite_sequence",
pkColumnName="name", valueColumnName="seq",
pkColumnValue="sqliteTestTable")
If you are using multiple tables then the generators have to be unique names. With a new table, there is no sqlite_sequence record so you have to create yourself.
insert into sqlite_sequence (name,seq) values ("sqliteTestTable",1);
Related
Struggling with database queries - not a db expert by any means, any help would be appreciated.
When dynamically created databases and schemas, once in awhile I get this error:
Unable to apply database grants.
io.vertx.core.impl.NoStackTraceThrowable: Error granting permission.
io.vertx.pgclient.PgException:
ERROR: tuple concurrently updated (XX000)
The role names, database names and schema names are replaced in the query strings in a separate place, i modified the code to pass in the query string directly to the transaction for simplicity.
The permissions being granted are as follows:
private static final String ERR_PERMISSION_GRANT_ERROR_MESSAGE = "Error granting permission. ";
private static final String ADVISORY_LOCK = "SELECT pg_try_advisory_lock("
+ String.valueOf(BigInteger.valueOf(Double.valueOf(Math.random()).longValue())) + ")";
private static final String CREATE_USER = "CREATE ROLE <role-name> LOGIN PASSWORD <pwd>;";
private static final String GRANT_PERMISSION1 = "GRANT CREATE, CONNECT ON DATABASE <db-name> TO <role-name>;";
private static final String GRANT_PERMISSION2 = "GRANT USAGE ON SCHEMA <schema-name> TO <role-name>;";
private static final String GRANT_PERMISSION3 = "GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA <schema-name> TO <role-name>";
private static final String GRANT_PERMISSION5 = "ALTER DEFAULT PRIVILEGES IN SCHEMA <schema-name> GRANT ALL ON SEQUENCES TO <role-name>;";
private static Promise<Boolean> grantDatabase(PgPool pool, String databaseName, String userName, String schemaName,
Vertx vertx) {
Promise<Boolean> promise = Promise.promise();
pool.getConnection()
// Transaction must use a connection
.onSuccess(conn -> {
// Begin the transaction
conn.begin().compose(tx -> conn
// Various statements
.query(updateQueryString(ADVISORY_LOCK, databaseName, userName)).execute()
.compose(
res1 -> conn.query(
updateQueryString(GRANT_PERMISSION1 databaseName, userName))
.execute()
.compose(res2 -> conn.query(
updateQueryString(GRANT_PERMISSION2, schemaName, userName))
.execute()
.compose(res3 -> conn
.query(updateQueryString(
GRANT_PERMISSION3, schemaName, userName))
.execute()
.compose(res4 -> conn
.query(updateQueryString(GRANT_PERMISSION5,
schemaName, userName))
.execute()))))
// Commit the transaction
.compose(res5 -> tx.commit()))
// Return the connection to the pool
.eventually(v -> conn.close()).onSuccess(v -> promise.complete(Boolean.TRUE))
.onFailure(err -> promise
.fail(ERR_PERMISSION_GRANT_ERROR_MESSAGE
});
return promise;
}
How do I fix the tuple concurrently updated error in this case? I only have a single instance of my service running.
PostgreSQL v14.6 (Homebrew)
vertx-pg-client 4.3.8
You've probably already found this and established the error is caused by the non-zero chance of two of your threads trying to run those queries at the same time. They could be also competing with something else - missing commas and parentheses suggest the code you showed is not 1:1 what you're running, plus you could have more grant/revoke/alter elsewhere.
I think your plan to use advisory locks is better than the alternative of establishing a separate "grant queue" or trying to track and lock system tables.
Locking approach
private static final String ADVISORY_LOCK = "SELECT pg_try_advisory_lock("
/* ... */
.query(updateQueryString(ADVISORY_LOCK, databaseName, userName)).execute()
.compose( /* ... */
You might want to change your advisory lock function.
pg_advisory_lock() would make it wait for the lock if it's not available.
pg_try_advisory_lock() instead of making the client wait for lock to become available, returns false. I don't see the code in any way responding to the result of true if it got the lock or false if it didn't, which means that it just tries to acquire the lock and ignores the outcome, continuing regardless.
Both of the above obtain a session-level lock, so it won't be released unless you call pg_advisory_unlock() on the same ID. Lock obtained from pg_advisory_xact_lock() and pg_try_advisory_lock() would be released automatically at commit/rollback.
With a standalone connection, conn.close() should end the session which triggers the db to lift both session- and transaction-level locks it held. With a pool, it could live on after released, still holding the locks unless it happens to get cleaned up by a pool configured to do so.
ID used for locking
Your use of Math.random() seems to always result in a 0 because of narrowing primitive conversion in Double.longValue()
String.valueOf( //BigInt to String
BigInteger.valueOf( //Long to BigInt,
Double.valueOf( //Double to Double
Math.random() //returns between 0.0 and 1.0
).longValue() //Double to Long, basically flooring it to 0
)
)
Which means you're already always re-using a static ID.
But in case you tried to randomise the ID to make each thread use a different, unique lock id, they wouldn't be able to block each other. Threads need to use the same lock ID in reference to the same "action" that could interfere with the other threads if they attempted it at the same time.
private static final String ADVISORY_LOCK = "SELECT pg_try_advisory_lock("
+ String.valueOf(BigInteger.valueOf(Double.valueOf(Math.random()).longValue()))
+ ")";
--Random lock ID generated as 99:
/*1st agent:*/ SELECT pg_try_advisory_lock(99);
--lock acquired, 1st agent proceeds to run its queries
--In parallel, 2nd agent gets a random ID of 77:
/*2nd agent:*/ SELECT pg_try_advisory_lock(77);
--77 wasn't locked, so it immediately proceeds to attempt the same action
--as the 1st agent, disregarding the fact that it can make them compete
--and result in `ERROR: tuple concurrently updated`
Aside from swapping pg_try_advisory_lock() for a pg_advisory_xact_lock() I think replacing that Math.random() with a static, arbitrary number, will be enough:
private static final String ADVISORY_LOCK = "SELECT pg_advisory_xact_lock("
+ "123456789"
+ ")";
--now everyone trying to run those particular queries checks the same ID
/*1st agent:*/ SELECT pg_advisory_xact_lock(123456789);
--noone called dibs on that ID so far, so it's allowed to proceed
--In parallel, 2nd agent enters the same subroutine and asks about the same ID:
/*2nd agent:*/ SELECT pg_advisory_xact_lock(123456789);
--1st agent hasn't released the lock on that ID yet, so 2nd agent waits
If competing parts of your app were initialising their own Random() with the same, shared seed, or re-starting a shared Random(), they'd get the same ID - but that's only trading a predefined, static ID for a predefined seed.
Random, unique lock IDs could be useful to avoid accidental ID re-use for some unrelated action and to free you from having to keep track of what ID was used where. However, those IDs would have to be generated ahead of runtime or during each initialisation.
Using JpaRepository deleteAll() produce org.springframework.orm.jpa.JpaObjectRetrievalFailureException on Entities with many to one relation (where the "one" has a where clause)
I am using a #Where clause on an entity to perform soft deletion :
#Where(clause = "enabled = true")
#Table(name = "customer")
public class CustomerEntity {
...
}
There is a #ManyToOne relation to CustomerEntity on another entity :
#Entity
public abstract class CustomerEvent extends Event {
#ManyToOne
#JoinColumn(name = "customer_id")
private CustomerEntity customer;
...
}
With the following repository (BirthdayEvent extends CustomerEvent):
#Repository
public interface BirthdayEventRepository extends JpaRepository<BirthdayEvent, Integer> {
...
}
Now lets say customer with id '1' has enabled = false and there is a birthday event associated to that customer.
Performing birthdayEventRepository.deleteAll() will produce the following error :
org.springframework.orm.jpa.JpaObjectRetrievalFailureException: Unable to find org.example.app.entity.roles.customer.CustomerEntity with id 1; nested exception is javax.persistence.EntityNotFoundException: Unable to find org.example.app.entity.roles.customer.CustomerEntity with id 1
On the other hand, using deleteAllInBatch() works just fine.
Trying to examine the Hibernate SQL trace I found out that on the deleteAll() method, hibernate is performing the following binding for all customers (even if not enabled) :
2019-12-12 10:57:20.204 TRACE 2545551 --- [pool-4-thread-1] o.h.type.descriptor.sql.BasicBinder : binding parameter [1] as [BIGINT] - [1]
Using deleteAllInBatch() hibernate performs the same binding but only for 'enabled' customers.
Is that a bug or an expected behavior ? If this is expected can someone explains why it works this way ?
We had a similar situation with "deleted" instances of entities annotated with #Where(clause = "deleted = false"). If those were referenced by another entity, working with the latter led to the same mentioned javax.persistence.EntityNotFoundException.
Maybe I'm wrong, but you should see the same error when loading the event via BirthdayEventRepository and then accessing its CustomerEntity.
I can only guess from the docs that CrudRepository.deleteAll loads all events before deleting them whereas JpaRepository.deleteAllInBatch doesn't.
CrudRepository.deleteAll
void deleteAll()
Deletes all entities managed by the repository.
JpaRepository.deleteAllInBatch
void deleteAllInBatch()
Deletes all entities in a batch call.
Update 2019-12-13
I was curious, setup a test and recorded database accesses with a profiler.
1) deleteAll() starts by loading/selecting all records from database:
exec sp_executesql N'select user0_.id as id1_11_, ... from [user] user0_'
This was followed by some other reads to load/check for referenced objects. Then it started to delete one object by the other:
exec sp_executesql N'delete from [user] where id=#P0 ',N'#P0 bigint',1
exec sp_executesql N'delete from [user] where id=#P0 ',N'#P0 bigint',2
2) In contrast, deleteAllInBatch() tries to delete all instances directly:
exec sp_executesql N'delete from [user]'
Conclusion is that (2) goes faster since it performs no checks while (1) is more accurate, to put it simply.
When attempting to extract data from a BigQuery table requiring a partition filter, the extract job fails.
Here is a simple example creating a table and running an extract job.
package com.example;
import com.google.cloud.bigquery.*;
public class BigQueryExtractTest {
private static final String PROJECT_ID = "my-project-id";
private static final String DATASET_ID = "test_dataset";
private static final String GCS_LOCATION = "gs://my-bucket/path/to/files/part-*";
public static void main(String[] args) throws Exception {
// create BigQuery client
BigQuery bigQuery = BigQueryOptions.newBuilder().setProjectId(PROJECT_ID).build().getService();
// create dataset and table that requires partition filter
bigQuery.create(DatasetInfo.of(DATASET_ID));
bigQuery.query(QueryJobConfiguration.of(
String.format("CREATE TABLE %s.table1 (\n", DATASET_ID) +
"stringColumn STRING,\n" +
"timeColumn TIMESTAMP\n" +
") PARTITION BY DATE(timeColumn)\n" +
"OPTIONS(\n" +
"require_partition_filter=true\n" +
")"));
// extract table
Job job = bigQuery.getTable(TableId.of(DATASET_ID, "table1"))
.extract("NEWLINE_DELIMITED_JSON", GCS_LOCATION)
.waitFor();
// throw exception on error
if (job != null && job.getStatus().getError() != null) {
throw new Exception(job.getStatus().getError().toString());
}
}
}
The code snippet above produces the following error
Exception in thread "main" java.lang.Exception: BigQueryError{reason=invalidQuery, location=query, message=Cannot query over table 'my-project-id.test_dataset.table1' without a filter that can be used for partition elimination}
at com.example.BigQueryExtractTest.main(BigQueryExtractTest.java:34)
The google-cloud-bigquery maven dependency used for this example is shown below.
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-bigquery</artifactId>
<version>1.23.0</version>
</dependency>
The example also threw the exception using the dependency version 0.34.0-beta
How can a partition filter be specified when running an extract job?
This is a bug, and there is now a bug report tracking the problem. To work around the limitation, you can update the table to allow queries without a partition filter using the bq command-line tool, perform the export, and then update the table to require it again. For example, with an ingestion-time partitioned table:
bq update --time_partitioning_type=DAY --norequire_partition_filter \
<dataset_name>.<table name>
... (perform the export)
bq update --time_partitioning_type=DAY --require_partition_filter \
<dataset_name>.<table name>
... (resume querying the table)
We have a long running user operation that is handled by a pool of worker processes. Data input and output is from Azure SQL.
The master Azure SQL table structure columns are approximated to
[UserId, col1, col2, ... , col N, beingProcessed, lastTimeProcessed ]
beingProcessed is boolean and lastTimeProcessed is DateTime. The logic in every worker role is as shown below and with multiple workers processing (each with their own Entity Framework layer), in essence beingProcessed is being used a lock for MutEx purposes
Question: How can I deal with concurrency issues on the beingProcessed "lock" itself based on the above load? I think read-modify-write operation on the beingProcessed needs to be atomic but I'm open to other strategies. Open to other code refinements too.
[Update]: I wonder if TransactionScope is what's needed here ... http://msdn.microsoft.com/en-US/library/system.transactions.transactionscope(v=vs.110).aspx
Code:
public void WorkerRoleMain()
{
while(true)
{
try
{
dbContext db = new dbContext();
// Read
foreach (UserProfile user in db.UserProfile
.Where(u => DateTime.UtcNow.Subtract(u.lastTimeProcessed)
> TimeSpan.FromHours(24) &
u.beingProcessed == false))
{
user.beingProcessed = true; // Modify
db.SaveChanges(); // Write
// Do some long drawn processing here
...
...
...
user.lastTimeProcessed = DateTime.UtcNow;
user.beingProcessed = false;
db.SaveChanges();
}
}
catch(Exception ex)
{
LogException(ex);
Sleep(TimeSpan.FromMinutes(5));
}
} // while ()
}
What we usually do is this:
At the beginning of a long operation we start a transaction:
BEGIN TRANSACTION
Then we select a row from the table we would like to update/delete using these hints:
SELECT * FROM Table WITH (ROWLOCK, NOWAIT) Where ID = 123;
Then we check that we have the row. If the row is locked by another process there will be an SQL Error. In this case we rollback the transaction and advise the user.
If the record is locked we process the record, and do the required updates, using the same transaction object we used to lock the record:
UPDATE Table SET Col1='value' WHERE ID = 123;
Then we COMMIT the transaction.
COMMIT;
This is just the Pseudo-code of the process. You will have to implement it in your program.
One small note regarding the above process. When you lock the record in SQL Server (or Azure), use the primary key in your WHERE Clause, otherwise the SQL Server will decide to use a Page lock, or Table lock
I have a linq query running in a WCF Web Service that looks for a match and if one is not found then it creates one.
my code looks like
//ReadCommitted transaction
using (var ts = CreateTransactionScope(TransactionScopeOption.RequiresNew))
{
Contract contract = db.Contracts.SingleOrDefault(x => x.txtBlah == str);
if (contract == null)
{
contract = new Contract();
contract.txtBlah = str;
db.Contracts.InsertOnSubmit(contract);
db.SubmitChanges();
}
...
db.SubmitChanges();
}
The problem is that I am getting duplicates. I thought the transaction would have locked the database to ensure no duplicates would be found (and supply the ability to rollback). How can I ensure that there are no duplicates?
In a ReadCommited transaction the data can be changed before the end of the transaction. But you can use Serializable transaction which will do table locking.
db.Connection.Open();
using (db.Transaction = db.Connection.BeginTransaction(IsolationLevel.Serializable))
{
//your code here
db.Transaction.Commit();
}
Ok, if you don't want to use Serializable tran then you should write a sproc for atomic insert, the logic should look like this SQL in code:
db.ExecuteCommand("INSERT INTO Contract (txtBlah) SELECT {0} WHERE NOT EXISTS (SELECT 1 FROM Contract WITH (TABLOCK) WHERE txtBlah={0})", str);
note this will also lock the whole table during the insert.
Read more on how to create a sproc without a race condition at http://weblogs.sqlteam.com/dang/archive/2007/10/28/Conditional-INSERTUPDATE-Race-Condition.aspx.