spring batch insert using hibernateTemplate, JdbcTemplate - sql

I have a few questions related to batch insert in Spring.
When I do something like that:
public void save(Car car) {
String sql1 = "insert into Car values (1, 'toyota')";
String sql2 = "insert into Car values (2, 'chrysler')";
String sql3 = "insert into Car values (3, 'infinity')";
String[] tab = new String[2];
tab[0] = sql1;
tab[1] = sql2;
tab[2] = sql3;
getJdbcTemplate().update(sql1);
getJdbcTemplate().update(sql2);
getJdbcTemplate().update(sql3);
// getJdbcTemplate().batchUpdate(tab);
}
in mysql log file I see:
1 Query insert into Car values (1, 'toyota')
2 Query insert into Car values (2, 'chrysler')
3 Query insert into Car values (3, 'infinity')
so we have 3 insert statements (and 3 network calls).
When I use getJdbcTemplate().batchUpdate(tab) in the log file I can see:
1094 [main] DEBUG org.springframework.jdbc.core.JdbcTemplate - Executing SQL batch update of 3 statements
1110 [main] DEBUG org.springframework.jdbc.datasource.DataSourceUtils - Fetching JDBC Connection from DataSource
1110 [main] DEBUG org.springframework.jdbc.datasource.DriverManagerDataSource - Creating new JDBC DriverManager Connection to [jdbc:mysql://localhost:3306/test?useServerPrepStmts=true]
1610 [main] DEBUG org.springframework.jdbc.support.JdbcUtils - JDBC driver supports batch updates
and in mysql log:
1 Query insert into Car values (1, 'toyota')
1 Query insert into Car values (2, 'chrysler')
1 Query insert into Car values (3, 'infinity')
I understand that in background addBatch method is invoked on statement object and all these operations are performed simultaneously. Additional benefit is reduction of network calls. Is my reasoning correct?
I'm looking for something similar in HibernateTemplate. I can do it in this way:
getHibernateTemplate().saveOrUpdateAll(Arrays.asList(new Car(4, "infinity"), new Car(5, "ford")));
In that case, in the log file I can see:
3 Prepare select car_.id, car_.name as name0_ from Car car_ where car_.id=?
3 Prepare select car_.id, car_.name as name0_ from Car car_ where car_.id=?
3 Prepare insert into Car (name, id) values (?, ?)
So it seems that everything is done in one shot as it was done for getJdbcTemplate().updateBatch(...)
Please correct me if I'm wrong.

To produce result similar to Hibernate (with prepared statment), you should use JdbcTemplate.batchUpdate(String, BatchPreparedStatementSetter). Something like this:
final List<Car> cars = Arrays.asList(...);
getJdbcTemplate().batchUpdate("insert into Car (name, id) values (?, ?);",
new BatchPreparedStatementSetter() {
private int i = 0;
public int getBatchSize() { return cars.size(); }
public void setValues(PreparedStatement ps) {
ps.setString(1, cars.get(i).getName());
ps.setInt(2, cars.get(i).getId());
i++;
}
});

Related

Explain the SQL expression

I'm going through the spring-security samples on GitHub and found a class, which is a repository itself, where we use a #Query annotation.
Database used: HSQL
I don't clearly understand the contents of the query, particularly what does m mean and this part m.to_id = ?#{principal?.id } as well. This symbol is being underlined with red line and the following message is shown: Cannot resolve symbol 'm'
Code:
/**
* A repository that integrates with Spring Security for accessing {#link Message}s.
*/
#Repository
public interface SecurityMessageRepository extends MessageRepository {
#Query("SELECT m FROM Message WHERE m.to_id = ?#{principal?.id }")
List<Message> findAll();
}
Here's my data.sql file used to populate the initial data (taken from the example):
insert into user(id,email,password,firstName,lastName) values (0,'rob#example.com','password','Rob','Winch');
insert into user(id,email,password,firstName,lastName) values (1,'luke#example.com','password','Luke','Taylor');
insert into message(id,created,to_id,summary,text) values (100,'2023-01-05 10:00:00',0,'Hello Rob','This message is for Rob');
insert into message(id,created,to_id,summary,text) values (101,'2023-01-05 11:00:00',0,'How are you Rob?','This message is for Rob');
insert into message(id,created,to_id,summary,text) values (102,'2023-01-05 12:00:00',0,'Is this secure?','This message is for Rob');
insert into message(id,created,to_id,summary,text) values (110,'2023-01-05 10:00:00',1,'Hello Luke','This message is for Luke');
insert into message(id,created,to_id,summary,text) values (111,'2023-01-05 10:00:00',1,'Greetings Luke','This message is for Luke');
insert into message(id,created,to_id,summary,text) values (112,'2023-01-05 10:00:00',1,'Is this secure?','This message is for Luke');
Could you explain what does mean that SQL expression or provide a link to some article explaining this? (because I can't even figure out how to google it correctly to find the answer quickly).
this SQL will work, m is not a column in your table so you can't select it.
* will get all columns. And m is used as an alias for Message, so it will be resolved with the m.to_id column.
SELECT * FROM Message m WHERE m.to_id = ?#{principal?.id }

Postgres: on conflict, summing two vectrors(arrays)

I'm trying to handle an array of counters column in Postgres
for example, let's say I have this table
name
counters
Joe
[1,3,1,0]
and now I'm adding 2 values ("Ben", [1,3,1,0]) and ("Joe",[2,0,2,1])
I expect the query to sum between the 2 counters vectors on conflict ([1,3,1,0] + [2,0,2,1] = [3,3,3,1])
the expected result:
name
counters
Joe
[3,3,3,1]
Ben
[1,3,1,0]
I tried this query
insert into test (name, counters)
values ("Joe",[2,0,2,1])
on conflict (name)
do update set
counters = array_agg(unnest(test.counters) + unnest([2,0,2,1]))
but it didn't seem to work, what am I missing?
There are two problems with the expression:
array_agg(unnest(test.counters) + unnest([2,0,2,1]))
there is no + operator for arrays,
you cannot use set-valued expressions as an argument in an aggregate function.
You need to unnest both arrays in a single unnest() call placed in the from clause:
insert into test (name, counters)
values ('Joe', array[2,0,2,1])
on conflict (name) do
update set
counters = (
select array_agg(e1 + e2)
from unnest(test.counters, excluded.counters) as u(e1, e2)
)
Also pay attention to the correct data syntax in values and the use of a special record excluded (find the relevant information in the documentation.)
Test it in db<>fiddle.
Based on your reply to my comments that it will always be four elements in the array and the update is being done by a program of some type, I would suggest something like this:
insert into test (name, counters)
values (:NAME, :COUNTERS)
on conflict (name) do
update set
counters[1] = counters[1] + :COUNTERS[1],
counters[2] = counters[2] + :COUNTERS[2],
counters[3] = counters[3] + :COUNTERS[3],
counters[4] = counters[4] + :COUNTERS[4]

JPA 2.2 `EntityManager.merge` inserting instead of updating

I have only this simple Entity (no relationships at all, just some scalar fields):
#Entity
#Table
#Access(AccessType.FIELD)
public class MyEntity {
#Id
#Column
private String myPK;
/* ...other fields... */
public MyEntity(String myPK /* ... other fields ... */) {
this.myPK = myPK;
/* ... */
}
/* getters and setters and toString and hashcode et al. */
}
Then at the start of a test run I do this (inside arquillian, but it doesn't really matter):
EntityManager manager;
/* ... manager is injected actually ... */
manager.merge(new MyEntity("0" /* ... */));
It works. That is, the table line whose PK is "0" got updated with the remaining fields, so the test has a stable starting point.
Now I've changed the myPK field to be Long instead of String. Yes, only this change (and the new MyEntity(0L /* ... */)).
It doesn't work anymore.
The merge actually tries to insert a new row, where the previous one already exists. Then the database refused to insert a duplicate PK.
Why does it UPDATE the row when using String as PK, but then it INSERTS when using Long as PK?
I'm using OpenLiberty 21.0.0.4 (wlp-1.0.51.cl210420210407-0944).
It uses EclipseLink, version: Eclipse Persistence Services - 2.7.8.v20201217-ecdf3c32c4.
On transaction commit it spills:
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.7.8.v20201217-ecdf3c32c4): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (XPK) violate
Error Code: 1
Call: INSERT INTO MyEntity (myPK, ...) VALUES (?, ...)
bind => [0, ...]
Query: InsertObjectQuery(MyEntity(myPK=0, ...))
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:333)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:908)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:970)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:640)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:567)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2099)
at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:313)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:277)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:263)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.insertObject(DatasourceCallQueryMechanism.java:413)
at org.eclipse.persistence.internal.queries.StatementQueryMechanism.insertObject(StatementQueryMechanism.java:167)
at org.eclipse.persistence.internal.queries.StatementQueryMechanism.insertObject(StatementQueryMechanism.java:182)
at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.insertObjectForWrite(DatabaseQueryMechanism.java:504)
at org.eclipse.persistence.queries.InsertObjectQuery.executeCommit(InsertObjectQuery.java:82)
at org.eclipse.persistence.queries.InsertObjectQuery.executeCommitWithChangeSet(InsertObjectQuery.java:92)
at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.executeWriteWithChangeSet(DatabaseQueryMechanism.java:316)
at org.eclipse.persistence.queries.WriteObjectQuery.executeDatabaseQuery(WriteObjectQuery.java:60)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:911)
at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:810)
at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWorkObjectLevelModifyQuery(ObjectLevelModifyQuery.java:110)
at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWork(ObjectLevelModifyQuery.java:87)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2983)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1898)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1880)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1830)
at org.eclipse.persistence.internal.sessions.CommitManager.commitNewObjectsForClassWithChangeSet(CommitManager.java:229)
at org.eclipse.persistence.internal.sessions.CommitManager.commitAllObjectsForClassWithChangeSet(CommitManager.java:196)
at org.eclipse.persistence.internal.sessions.CommitManager.commitAllObjectsWithChangeSet(CommitManager.java:141)
at org.eclipse.persistence.internal.sessions.AbstractSession.writeAllObjectsWithChangeSet(AbstractSession.java:4398)

How to use cur.executemany() to store data from Twitter

I am trying to download tweets from a list of three different accounts and then store all the informations in a SQL3 database.
I have tried with the code below, but it seems to run forever. Am I missing something? Is this because I used .executemany() instead of .execute()?
step=0
a_list=["A","B","C"]
for s in a_list:
cursor = tweepy.Cursor(api1.user_timeline, id = s, tweet_mode='extended').items(3189)
for tweet in cursor:
tw_text.append(tweet.full_text)
created_at.append(tweet.created_at)
rtws.append(tweet.retweet_count)
favs.append(tweet.favorite_count)
for h in tweet.entities['hashtags']:
hashlist.append(h['text'])
for u in tweet.entities['urls']:
linklist.append(u['expanded_url'])
try:
medialist.append(media['media_url'] for media in tweet.entities['media'])
except:
pass
step+=1
print('step {} completed'.format(step))
#preparing all the data for .executemany()
g = [(s,tw,crea,rt,fv,ha,li,me) for s in ['GameOfThrones'] for tw in tw_text for crea in created_at for rt in rtws for fv in favs for ha in hashlist for li in linklist for me in medialist]
cur.executemany("INSERT INTO series_data VALUES (?,?,?,?,?,?,?,?)", (g))
con.commit()
print('db updated')
I expect the program to write table in SQL3 but I never receive the message 'db updated' (i.e. the very last print() line)
cur.executemany() takes a list of tuples. Each tuple will have as many elements as number of columns you want to insert value for.
For example, if you have a table with following structure
create table tbl_test(firstname varchar(20), lastname varchar(20));
and you want to insert 3 records in it using executemany(), your object and the call should be like following
list = [('Hans', 'Muster'), ('John', 'Doe'), ('Jane', 'Doe')]
cur.executemany('insert into tbl_test values(?, ?)', list)

How does hibernate use an empty string for an equality restriction?

I have a column that potentially has some bad data and I can't clean it up, so I need to check for either null or empty string. I'm doing a Hibernate Criteria query so I've got the following that returns incorrectly right now:
Session session = getSessionFactory().openSession();
Transaction tx = session.beginTransaction();
Criteria myCriteria = session.createCriteria(Object);
...
myCriteria.add(Restrictions.or(Restrictions.isNull("stringColumn"),
Restrictions.eq("stringColumn", "")));
List<Objects> list = myCriteria.list();
I can't get it to properly return the results I'd expect. So as an experiment I changed the second restriction to read:
Restrictions.eq("stringColumn", "''")
And it started returning the expected results, so is hibernate incorrectly translating my empty string (e.g. "") into a SQL empty string (e.g. ''), or am I just doing this wrong?
With HSQL (and Derby) and the following values:
insert into FOO values (1, 'foo');
insert into FOO values (2, 'bar');
insert into FOO values (3, NULL);
insert into FOO values (4, '');
You criteria query
Criteria crit = session.createCriteria(Foo.class);
crit.add(Restrictions.or(Restrictions.isNull("name"),
Restrictions.eq("name", "")));
crit.list();
returns:
Foo [id=3, name=null]
Foo [id=4, name=]
As expected.
What database are you using? Could it be Oracle?
It seems like you're doing it wrong. null in Java maps to NULL in SQL, and empty String ("") in Java maps to empty string in SQL ('')