My simple flow of batch process reads from a CSV file and write into a MySQL database (batch configuration is ok and works).
I'm using a custom implementation of JdbcBatchItemWriter in order to do the job and I'm actually making an Update in my writer constructor.
CsvReader.java
#Component
#StepScope
public class EducationCsvReader extends FlatFileItemReader {
public final static String CSV_FILE_NAME = "education.csv.file";
#Value("#{jobParameters['"+ CSV_FILE_NAME +"']}")
public void setResource(final String csvFileName) throws Exception {
setResource(
new FileSystemResource(csvFileName)
);
}
public EducationCsvReader() {
setLinesToSkip(1);
setEncoding("UTF-8");
setStrict(true);
setLineMapper((line, num) -> {
String[] values = line.split(";");
return new Education()
.setName(values[2].trim())
.setId(Integer.parseInt(values[0].trim()));
});
}
}
my custom JdbcBatchItemWriter : AbstractJdbcBatchItemWriter.java
public abstract class AbstractJdbcBatchItemWriter<T> extends JdbcBatchItemWriter<T>{
#Autowired
public AbstractJdbcBatchItemWriter(String SQL_QUERY) {
setSql(SQL_QUERY);
}
#Autowired
#Override
public void setItemSqlParameterSourceProvider(
#Qualifier("beanPropertyItemSqlParameterSourceProvider") ItemSqlParameterSourceProvider provider){
super.setItemSqlParameterSourceProvider(provider);
}
#Autowired
#Override
public void setDataSource(#Qualifier("mysqlDataSource") DataSource dataSource){
super.setDataSource(dataSource);
}
}
And here is my writer implementation : MySQLWriter.java
#Component
public class EducationMysqlWriter extends AbstractJdbcBatchItemWriter<Education> {
public EducationMysqlWriter(){
super("");
try {
setSql("UPDATE ecole SET nom=:name WHERE id=:id");
} catch (EmptyResultDataAccessException exception){
setSql("INSERT INTO ecole (nom, id) VALUES (:name, :id");
}
}
}
I need to update rows but if it fails (EmptyResultDataAccessException) I need to do an Insert.
But EmptyResultDataAccessException is shown on log console and kills the job but the exception catching is never reachable into MySQLWriter.java ...
JdbcBatchItemWriter#setSql doesn't throw an exception because it doesn't do anything but assign a string to an instance variable. The try block in the constructor doesn't have anything to do with the write method, it is executed when the itemwriter is first instantiated, while the write method is executed once for each chunk of items being processed. If you read the stacktrace I expect you'll see the JdbcBatchtemWriter is executing its write method and throwing the exception.
The ItemWriter is not going to get instantiated for each row so, assuming you will have some rows that need to be inserted and some that need to be updated, setting the sql string in the constructor does not seem like a good idea.
You could override the ItemWriter#write method, using a separate sql string for the insert, but it would be easier to use one string using the mysql upsert syntax:
INSERT INTO ecol (nom, id) VALUES (:name, :id)
ON DUPLICATE KEY UPDATE nom = :name;
Related
Hangfire DisableConcurrentExecution attribute not working as expected.
I have one method and that can be called with different Id. I want to prevent concurrent execution of method if same Id is passed.
string jobName= $"{Id} - Entry Job";
_recurringJobManager.AddOrUpdate<EntryJob>(jobName, j => j.RunAsync(Id, Null), "0 2 * * *");
My EntryJob interface having RunAsync method.
public class EntryJob: IJob
{
[DisableConcurrentExecution(3600)] <-- Tried here
public async Task RunAsync(int Id, SomeObj obj)
{
//Some coe
}
}
And interface look like this
[DisableConcurrentExecution(3600)] <-- Tried here
public interface IJob
{
[DisableConcurrentExecution(3600)] <-- Tried here
Task RunAsync(int Id, SomeObj obj);
}
Now I want to prevent RunAsync method to call multiple times if Id is same. I have tried to put DisableConcurrentExecution on top of the RunAsync method at both location inside interface declaration and also from where Interface is implemented.
But it seems like not working for me. Is there any way to prevent concurrency based on Id?
The existing implementation of DisableConcurrentExecution does not support this. It will prevent concurrent executions of the method with any args. It would be fairly simple to add support in. Note below is untested pseudo-code:
public class DisableConcurrentExecutionWithArgAttribute : JobFilterAttribute, IServerFilter
{
private readonly int _timeoutInSeconds;
private readonly int _argPos;
// add additional param to pass in which method arg you want to use for
// deduping jobs
public DisableConcurrentExecutionAttribute(int timeoutInSeconds, int argPos)
{
if (timeoutInSeconds < 0) throw new ArgumentException("Timeout argument value should be greater that zero.");
_timeoutInSeconds = timeoutInSeconds;
_argPos = argPos;
}
public void OnPerforming(PerformingContext filterContext)
{
var resource = GetResource(filterContext.BackgroundJob.Job);
var timeout = TimeSpan.FromSeconds(_timeoutInSeconds);
var distributedLock = filterContext.Connection.AcquireDistributedLock(resource, timeout);
filterContext.Items["DistributedLock"] = distributedLock;
}
public void OnPerformed(PerformedContext filterContext)
{
if (!filterContext.Items.ContainsKey("DistributedLock"))
{
throw new InvalidOperationException("Can not release a distributed lock: it was not acquired.");
}
var distributedLock = (IDisposable)filterContext.Items["DistributedLock"];
distributedLock.Dispose();
}
private static string GetResource(Job job)
{
// adjust locked resource to include the argument to make it unique
// for a given ID
return $"{job.Type.ToGenericTypeString()}.{job.Method.Name}.{job.Args[_argPos].ToString()}";
}
}
1. Is it possible to put non-POJO class instances as the value of a cache?
For example, I have a QueryThread class which is a subclass of java.lang.Thread and I am trying to put this instance in a cache. It looks like the put operation is failing because this cache is always empty.
Consider the following class:
public class QueryThread extends Thread {
private IgniteCache<?, ?> cache;
private String queryId;
private String query;
private long timeIntervalinMillis;
private volatile boolean running = false;
public QueryThread(IgniteCache<?, ?> dataCache, String queryId, String query, long timeIntervalinMillis) {
this.queryId = queryId;
this.cache = dataCache;
this.query = query;
this.timeIntervalinMillis = timeIntervalinMillis;
}
public void exec() throws Throwable {
SqlFieldsQuery qry = new SqlFieldsQuery(query, false);
while (running) {
List<List<?>> queryResult = cache.query(qry).getAll();
for (List<?> list : queryResult) {
System.out.println("result : "+list);
}
System.out.println("..... ");
Thread.sleep(timeIntervalinMillis);
}
}
}
This class is not a POJO. How do I store an instance of this class in the cache?
I tried implementing Serializable (didn't help).
I need to be able to do this:
queryCache.put(queryId, queryThread);
Next I tried broadcasting the class using the IgniteCallable interface. But my class takes multiple arguments in the constructor. I feel PeerClassLoading is easy if the class takes a no-arg constructor:
IgniteCompute compute = ignite.compute(ignite.cluster().forServers());
compute.broadcast(new IgniteCallable<MyServiceImpl>() {
#Override
public MyServiceImpl call() throws Exception {
MyServiceImpl myService = new MyServiceImpl();
return myService;
}
});
2. How do I do PeerClassLoading in the case of a class with multi-arg constructor?
It's restricted to put Thread instances to the cache, Thread instance cannot be serialized due to call to Native Methods. Thats why you always get empty value.
PeerClassLoading is a special distributed ClassLoader in Ignite for inter-node byte-code exchange. So, it's only about sharing classes between nodes. It doesn't make sense how many arguments in constructor class have.
But, on the other hand, object, that you created, will be serialised and sent to other nodes and for deserialisation it will need a default(non-arg) constructor.
I am trying to read a hive conf variable in initialize method, but not works, any suggestion plz?
My UDF Class:
public class MyUDF extends GenericUDTF {
MapredContext _mapredContext;
#Override
public void configure(MapredContext mapredContext) {
_mapredContext = mapredContext;
super.configure(mapredContext);
}
#Override
public StructObjectInspector initialize(ObjectInspector[] args) throws UDFArgumentException {
Configuration conf = _mapredContext.getJobConf();
// i am getting conf as null
}
}
Probably its too late to answer this question, but for others below is the answer inside a GenericUDF evaluate() method:
#Override
public Object evaluate(DeferredObject[] args) throws HiveException {
String myconf;
SessionState ss = SessionState.get();
if (ss != null) {
HiveConf conf = ss.getConf();
myconf= conf.get("my.hive.conf");
System.out.println("sysout.myconf:"+ myconf);
}
}
The code is tested on hive 1.2
You should also override configure method to support MapReduce
#Override
public void configure(MapredContext context) {
...................
........................
JobConf conf = context.getJobConf();
if (conf != null) {
String myhiveConf = conf.get("temp_var");
}
}
}
To test the code:
Build UDF Jar
On hive CLI, execute the below commands:
SET hive.root.logger=INFO,console;
SET my.hive.conf=test;
ADD JAR /path/to/the/udf/jar;
CREATE TEMPORARY FUNCTION test_udf AS com.example.my.udf.class.qualified.classname';
I was also running into this issue with a custom UDTF. It seems that the configure() method is not called on the user defined function until the MapredContext.get() method returns a non-null result (see UDTFOperator line 82 for example). MapredContext.get() likely returns a null result because the hive job has yet to spin up the mappers/reducers (you can see that MapredContext.get() will return null up until the MapredContext.init() method has been called; the init() method takes boolean isMap as a param, so this method doesn't get called until MR/Tez runtime - the comment associated with the GenericUDTF.configure() method confirms this).
TLDR the UDF/UDTF initialize() method will be called during job setup, and the configure() will be called at MR runtime, hence the null result in your example code.
I am using Spring JDBC and some nice Java 8 lambda-syntax to execute queries with the JDBCTemplate.
The reason for choosing Springs JDBCTemplate, is the implicit resource-handling that Spring-jdbc offers (I do NOT want a ORM framework for my simple usecase's).
My problem is that I want to debug the whole SQL statements with their parameters. Spring prints the SQL by default but not the parameters. Therefor I have subclassed the JDBCTemplate and overridden a query-method.
An example usage of the JDBCTemplate:
public List<Product> getProductsByModel(String modelName) {
List<Product> productList = jdbcTemplate.query(
"select * from product p, productmodel m " +
"where p.modelId = m.id " +
"and m.name = ?",
(rs, rowNum) -> new Product(
rs.getInt("id"),
rs.getString("stc_number"),
rs.getString("version"),
getModelById(rs.getInt("modelId")), // method not shown
rs.getString("displayName"),
rs.getString("imageUrl")
),
modelName);
return productList;
}
To get hold of the parameters I have, as mentioned, overridden the JDBCTemplate class. By doing a cast and using reflection I get the Object[] field with the parameters from an instance of ArgumentPreparedStatementSetter.
I suspect this implementation could potentially be dangerous, as the actual implementation of the PreparedStatementSetter may not always be ArgumentPreparedStatementSetter (Yes I should do an instanceOf check). Also, the reflection code may not be as elegant, but that is besides the point now though :).
Here's my custom implementation:
public class CustomJdbcTemplate extends JdbcTemplate {
private static final Logger log = LoggerFactory.getLogger(CustomJdbcTemplate.class);
public CustomJdbcTemplate(DataSource dataSource) {
super(dataSource);
}
public <T> T query(PreparedStatementCreator psc, final PreparedStatementSetter pss, final ResultSetExtractor<T> rse)
throws DataAccessException {
if(log.isDebugEnabled()) {
ArgumentPreparedStatementSetter aps = (ArgumentPreparedStatementSetter) pss;
try {
Field args = aps.getClass().getDeclaredField("args");
args.setAccessible(true);
Object[] parameters = (Object[]) args.get(aps);
log.debug("Parameters for SQL query: " + Arrays.toString(parameters));
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new GenericException(e.toString(), e);
}
}
return super.query(psc, pss, rse);
}
}
So, when I execute the log.debug(...) statement I would also like to have the original SQL query logged (same line). Has anyone done something similar or are there any better suggestions as to how this can be achieved?
I do quite a few queries using this CustomJDBCTemplate and all my tests run, so I think it may be an acceptable solution of for most debug purposes.
Kind regards,
Thomas
I found a way to get the SQL-statement, so I will answer my own question :)
The PreparedStatementCreator has the following implementation:
private static class SimplePreparedStatementCreator implements PreparedStatementCreator, SqlProvider
So the SqlProvider has a getSql() method which does exactly what I need.
Posting the "improved" CustomJdbcTemplate class if anyone ever should need to do the same :)
public class CustomJdbcTemplate extends JdbcTemplate {
private static final Logger log = LoggerFactory.getLogger(CustomJdbcTemplate.class);
public CustomJdbcTemplate(DataSource dataSource) {
super(dataSource);
}
public <T> T query(PreparedStatementCreator psc, final PreparedStatementSetter pss, final ResultSetExtractor<T> rse)
throws DataAccessException {
if(log.isDebugEnabled()) {
if(pss instanceof ArgumentPreparedStatementSetter) {
ArgumentPreparedStatementSetter aps = (ArgumentPreparedStatementSetter) pss;
try {
Field args = aps.getClass().getDeclaredField("args");
args.setAccessible(true);
Object[] parameters = (Object[]) args.get(aps);
log.debug("SQL query: [{}]\tParams: {} ", getSql(psc), Arrays.toString(parameters));
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new GenericException(e.toString(), e);
}
}
}
return super.query(psc, pss, rse);
}
private static String getSql(Object sqlProvider) { // this code is also found in the JDBCTemplate class
if (sqlProvider instanceof SqlProvider) {
return ((SqlProvider) sqlProvider).getSql();
}
else {
return null;
}
}
}
I am working in Scout and need SmartField. For this I need to set up lookup for suggestions.
I see the example with creating Lookup Call and than implement in Lookup Service getConfiguredSqlSelect
but I use Hibernate to work with classes, so my question is how to connect Smart field with Hibernate object filled service?
create a new lookup call according to [1] with the following differences:
don't select AbstractSqlLookupService as a lookup servic super type, but AbstractLookupService
in the associated lookup service you now need to implement getDataByAll, getDataByKey, and getDataByText
to illustrate the following snippet should help:
public class TeamLookupService extends AbstractLookupService<String> implements ITeamLookupService {
private List<ILookupRow<String>> m_values = new ArrayList<>();
public TeamLookupService() {
m_values.add(new LookupRow<String>("CRC", "Costa Rica"));
m_values.add(new LookupRow<String>("HON", "Honduras"));
m_values.add(new LookupRow<String>("MEX", "Mexico"));
m_values.add(new LookupRow<String>("USA", "USA"));
}
#Override
public List<? extends ILookupRow<String>> getDataByAll(ILookupCall<String> call) throws ProcessingException {
return m_values;
}
#Override
public List<? extends ILookupRow<String>> getDataByKey(ILookupCall<String> call) throws ProcessingException {
List<ILookupRow<String>> result = new ArrayList<>();
for (ILookupRow<String> row : m_values) {
if (row.getKey().equals(call.getKey())) {
result.add(row);
}
}
return result;
}
...
[1] https://wiki.eclipse.org/Scout/Tutorial/4.0/Minicrm/Lookup_Calls_and_Lookup_Services#Create_Company_Lookup_Call