spring batch : Read flat file which is getting changed continously - dynamic

Question

I have a requirement, where I have to read a flat text file which is continuously changing. Let's assume I have a file with 100 lines which I read using FlatFileReader in a batch and process those lines. Again when that step gets called let's say after 30 sec, then there are 110 lines. In that case that batch should read from line 101.
I know there is 'linesToSkip' parameter in Reader but I can define it at the start of a batch only not dynamically. Also the file I defined in batch configuration should be reloaded again on call of that Step(Step would be continuous process).
Any idea about this?
Thanks
Niraj

Answer 1

I would suggest the following approach:
Wrap your reader step with a listener and use before and after hooks.
Make sure that both step and FlatFileItemReader bean are defined at the step scope.
In the before step read the last line processed count from some persistence (file/db/etc) and place it on the stepExecutionContext.
Use the value placed on the stepExecutionContext to set the linesToSkip in the FlatFileItemReader using spel
In the after step take the current WriteCount and SkipCount from the execution context sum it with the value from the before step. Persist this value for then next execution
Your listener will look similar to the one below
#Component
public class LineCursorListener implements StepListener {
#BeforeStep
public ExitStatus beforeStep(StepExecution stepExecution){
int curser = 0;//read from persistence
stepExecution.getExecutionContext().put("linesToSkip", curser);
return stepExecution.getExitStatus();
}
#AfterStep
public ExitStatus afterStep(StepExecution stepExecution){
int nextCurser= stepExecution.getWriteCount() + stepExecution.getSkipCount();
nextCurser =nextCurser + stepExecution.getExecutionContext().getInt("linesToSkip");
// persistence the nextCurser
return stepExecution.getExitStatus();
}
}
Your Job xml will be similar to
<batch:job>
...
<batch:step id="processCsv">
<batch:tasklet transaction-manager="transactionManager">
<batch:chunk reader="someFileReader"
writer="writter" commit-interval="10" />
</batch:tasklet>
<batch:listeners>
<batch:listener ref="lineCurserListener" />
</batch:listeners>
</batch:job>
<bean id="someFileReader" scope="step"
class="org.springframework.batch.item.file.FlatFileItemReader" >
...
<property name="linesToSkip" value="#{stepExecutionContext['linesToSkip']}" />
<property name="lineMapper">
...
</property>
</bean>
I am only presenting the spring batch point of view for this issue, I guess that you need to watch out for concurrency issue related to file read/write