Spring Batch - Unable to deserialize the execution context - OffsetDateTime - cannot deserialize - serialization

I'm trying to create a spring batch job with multiples steps and passing object from step to step.
To do this I use ExecutionContext that i promoted from step to job context.
At first run, no problem data goes right from step to step
At next runs, I get the error :
"Unable to deserialize the execution context" Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of java.time.OffsetDateTime (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
I write context in a ItemWriter like so :
#Override
public void write(List<? extends Employee> items) throws Exception {
ExecutionContext stepContext = this.stepExecution.getExecutionContext();
List<Employee> e = new ArrayList<Employee>();
e.addAll(items);
stepContext.put("someKey", e);
}
And read it back in a ItemReader (from another step) with :
#BeforeStep
public void retrieveInterstepData(StepExecution stepExecution) {
JobExecution jobExecution = stepExecution.getJobExecution();
ExecutionContext jobContext = jobExecution.getExecutionContext();
this.someObject = (List<Employee>) jobContext.get("someKey");
}
I check spring database context and my dates (LocalDate, OffsetDateTime, ...) are store like :
"LocalDate": {
"year": 2019,
"month": "OCTOBER",
"dayOfMonth": 30,
"monthValue": 10,
"era": ["java.time.chrono.IsoEra", "CE"],
"dayOfWeek": "WEDNESDAY",
"dayOfYear": 303,
"leapYear": false,
"chronology": {
"id": "ISO",
"calendarType": "iso8601"
}
}
"OffsetDateTime": {
"offset": {
"totalSeconds": 0,
"id": "Z",
"rules": {
"fixedOffset": true,
"transitionRules": ["java.util.Collections$UnmodifiableRandomAccessList", []],
"transitions": ["java.util.Collections$UnmodifiableRandomAccessList", []]
}
},
"month": "OCTOBER",
"year": 2019,
"dayOfMonth": 28,
"hour": 13,
"minute": 42,
"monthValue": 10,
"nano": 511651000,
"second": 36,
"dayOfWeek": "MONDAY",
"dayOfYear": 301
}
I guess it's jackson's choice to store it like that (I custom nothing)
But it seems that jackson can't read it's own format at next run ?!
My stubs are generated with from swagger with "swagger-codegen-maven-plugin" and configOptions/dateLibrary=java8 so I can't change them.
I tried to add
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
</dependency>
And
#PostConstruct
public void init() {
objectMapper.registerModule(new JavaTimeModule());
objectMapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
}
In the #SpringBootApplication
no change
Any ideas ? Either to store dates more simply like "2019-11-04" or make jackson read it's own format ?

Your object mapper should be set on the Jackson2ExecutionContextStringSerializer used by the job repository. You can extend DefaultBatchConfigurer and override createJobRepository:
#Bean
public JobRepository createJobRepository() throws Exception {
ObjectMapper objectMapper = new ObjectMapper().registerModule(new JavaTimeModule());
objectMapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
Jackson2ExecutionContextStringSerializer defaultSerializer = new Jackson2ExecutionContextStringSerializer();
defaultSerializer.setObjectMapper(objectMapper);
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(dataSource);
factory.setTransactionManager(transactionManager);
factory.setSerializer(defaultSerializer);
factory.afterPropertiesSet();
return factory.getObject();
}

EDIT :
My bad I just saw that I have a
#Bean
public BatchConfigurer batchConfigurer(#Qualifier("batchDataSource") DataSource dataSource) {
return new DefaultBatchConfigurer(dataSource);
}
That provide 2 batchConfigurer to spring.
Thanks !
ORIGINAL :
Thanks it seems promising.
But I dont find where to extends and use it, on which class.
I have a Batch Class configuration :
#Configuration
#EnableConfigurationProperties(BatchProperties.class)
public class BatchDatabaseConfiguration {
#Value("${spring.datasource.driver-class-name}")
private String driverClassName;
#Value("${spring.datasource.url}")
private String dbURL;
#Bean("batchDataSource")
public DataSource batchDataSource() {
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driverClassName);
dataSource.setUrl(dbURL);
return dataSource;
}
#Bean
public BatchConfigurer batchConfigurer(#Qualifier("batchDataSource") DataSource dataSource) {
return new DefaultBatchConfigurer(dataSource);
}
#Bean(name = "batchTransactionManager")
public PlatformTransactionManager batchTransactionManager(#Qualifier("batchDataSource") DataSource dataSource) {
DataSourceTransactionManager tm = new DataSourceTransactionManager();
tm.setDataSource(dataSource);
return tm;
}
}
And a Class with Job's definition :
#Configuration
#EnableBatchProcessing
public class ExtractionJobConfiguration {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Bean
public Job creationJob() {
...
}
[...]
}
And the main :
#EntityScan(basePackages = { "..." })
#SpringBootApplication
#EnableAsync
public class App {
public static void main(String[] args) {
ApplicationContext ctx = SpringApplication.run(App.class, args);
}
What do you think ?
I also read that Spring Batch 4.2.0+ allow for customisation of ObjectMapper in Jackson2ExecutionContextStringSerializer (https://jira.spring.io/browse/BATCH-2828)
Is that what you propose ? (I don't find other information)

Related

JUnit 5 Parameterized test #ArgumentsSource parameters not loading

I have created below JUnit5 parameterized test with ArgumentsSource for loading arguments for the test:
public class DemoModelValidationTest {
public ParamsProvider paramsProvider;
public DemoModelValidationTest () {
try {
paramsProvider = new ParamsProvider();
}
catch (Exception iaex) {
}
}
#ParameterizedTest
#ArgumentsSource(ParamsProvider.class)
void testAllConfigurations(int configIndex, String a) throws Exception {
paramsProvider.executeSimulation(configIndex);
}
}
and the ParamsProvider class looks like below:
public class ParamsProvider implements ArgumentsProvider {
public static final String modelPath = System.getProperty("user.dir") + File.separator + "demoModels";
YAMLDeserializer deserializedYAML;
MetaModelToValidationModel converter;
ValidationRunner runner;
List<Configuration> configurationList;
List<Arguments> listOfArguments;
public ParamsProvider() throws Exception {
configurationList = new ArrayList<>();
listOfArguments = new LinkedList<>();
deserializedYAML = new YAMLDeserializer(modelPath);
deserializedYAML.load();
converter = new MetaModelToValidationModel(deserializedYAML);
runner = converter.convert();
configurationList = runner.getConfigurations();
for (int i = 0; i < configurationList.size(); i++) {
listOfArguments.add(Arguments.of(i, configurationList.get(i).getName()));
}
}
public void executeSimulation(int configListIndex) throws Exception {
final Configuration config = runner.getConfigurations().get(configListIndex);
runner.run(config);
runner.getReporter().consolePrintReport();
}
#Override
public Stream<? extends Arguments> provideArguments(ExtensionContext context) {
return listOfArguments.stream().map(Arguments::of);
// return Stream.of(Arguments.of(0, "Actuator Power"), Arguments.of(1, "Error Logging"));
}}
In the provideArguments() method, the commented out code is working fine, but the first line of code
listOfArguments.stream().map(Arguments::of)
is returning the following error:
org.junit.platform.commons.PreconditionViolationException: Configuration error: You must configure at least one set of arguments for this #ParameterizedTest
I am not sure whether I am having a casting problem for the stream in provideArguments() method, but I guess it somehow cannot map the elements of listOfArguments to the stream, which can finally take the form like below:
Stream.of(Arguments.of(0, "Actuator Power"), Arguments.of(1, "Error Logging"))
Am I missing a proper stream mapping of listOfArguments?
provideArguments(…) is called before your test is invoked.
Your ParamsProvider class is instantiated by JUnit. Whatever you’re doing in desiralizeAndCreateValidationRunnerInstance should be done in the ParamsProvider constructor.
Also you’re already wrapping the values fro deserialised configurations to Arguments and you’re double wrapping them in providesArguments.
Do this:
#Override
public Stream<? extends Arguments> provideArguments(ExtensionContext context) {
return listOfArguments.stream();
}}

Spring Cloud Sleuth: Initialise baggage item

I already have this Java Configuration:
#Configuration
public class FAPIAutoConfiguration {
private static final String INTERACTION_ID = "x-fapi-interaction-id";
private final BaggageField fapiBaggageField = BaggageField.create(INTERACTION_ID);
#Bean
BaggagePropagationCustomizer baggagePropagationCustomizer() {
return builder -> builder.add(SingleBaggageField.
remote(fapiBaggageField));
}
#Bean
CorrelationScopeCustomizer correlationScopeCustomizer() {
return builder -> builder.add(SingleCorrelationField.create(fapiBaggageField));
}
}
And the propagation in a Webflux application works, but I would like to know what is the best way to initialize the baggage if it is not present in the request headers. I mean, if the header is missing, generate a value and propagate this one.
I ended up adding a TracingCustomizer to the above configuration to fill the value when is missing in that context.
#Bean
TracingCustomizer tracingCustomizer(UniqueIdGenerator generator) {
return builder -> builder.addSpanHandler(new SpanHandler() {
#Override
public boolean begin(TraceContext context, MutableSpan span, TraceContext parent) {
var value = fapiBaggageField.getValue(context);
if (value == null) {
fapiBaggageField.updateValue(context, generator.next());
}
return super.begin(context, span, parent);
}
});
}
I do not know if this is the best option yet

WebTestClient used multiple times returns empty body sometimes

not sure, why this could be an issue, but I can't stabilize my unit-tests.
Here some snippets from my testclass:
#SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT, properties = { "spring.main.web-application-type=reactive" })
#RunWith(SpringRunner.class)
#TestPropertySource(locations = "classpath:application-test.properties")
public class SolrControllerV1Test {
#Inject
ApplicationContext context;
#LocalServerPort
int port;
private WebTestClient client;
#TestConfiguration
static class TestConfig {
#Bean
public TestingAuthenticationProvider testAuthentiationManager() {
return new TestingAuthenticationProvider();
}
#Bean
public SecurityWebFilterChain securityConfig(ServerHttpSecurity http, ReactiveAuthenticationManager authenticationManager) {
AuthenticationWebFilter webFilter = new AuthenticationWebFilter(authenticationManager);
return http.addFilterAt(webFilter, SecurityWebFiltersOrder.AUTHENTICATION)
.authorizeExchange()
.anyExchange()
.authenticated()
.and()
.build();
}
}
#Before
public void setUp() {
this.client = WebTestClient.bindToApplicationContext(context).configureClient().responseTimeout(Duration.ofDays(1L)).baseUrl("http://localhost:" + port).build();
}
private void defaultCheck(ResponseSpec spec) {
spec.expectStatus().isOk().expectBody().jsonPath("$.response.numFound").hasJsonPath();
}
#Test
#WithMockUser(roles = { "ADMIN" })
public void simpleUsrSelect() throws Exception {
ResponseSpec spec = this.client.get().uri("/" + serviceVersion + "/usr/select?q=*:*&fq=*:*&fl=USRTYP,USRKEY,USRCID&rows=1&start=10&sort=last_update desc").exchange();
defaultCheck(spec);
}
#Test
#WithMockUser(roles = { "ADMIN" })
public void simpleCvdSelect() throws Exception {
ResponseSpec spec = this.client.get().uri("/" + serviceVersion + "/cvd/select?q=*:*&rows=10000").exchange();
defaultCheck(spec);
}
.
.
.
}
There are some more unit-tests there, some of which are long running (>1sec). If I have enough unit-tests in the class (~5-8), of which 1 or 2 are taking a bit longer, the unit-tests start to break. This looks like a thread safety issue, but I don't know, what I'm doing wrong. Any ideas?
EDIT
Here the Server Part that made trouble:
#PreAuthorize("hasAnyRole('ADMIN','TENANT')")
public Mono<ServerResponse> select(ServerRequest request) {
return request.principal().flatMap((principal) -> {
return client.get().uri(f -> {
URI u = f.path(request.pathVariable("collection")).path("/select/").queryParams(
queryModifier.modify(principal, request.pathVariable("collection"), request.queryParams())
.onErrorMap(NoSuchFieldException.class, t -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Collection not found"))
.block()).build();
return u;
})
.exchange()
.flatMap((ClientResponse mapper) -> {
return ServerResponse.status(mapper.statusCode())
.headers(c -> mapper.headers().asHttpHeaders().forEach((name, value) -> c.put(name, value)))
.body(mapper.bodyToFlux(DataBuffer.class), DataBuffer.class);
})
.doOnError(t -> handleAuthxErrors(t, principal, request.uri()));
});
}
If I add a publishOn(Schedulers.elastic) right after the .exchange() part, it seems to be working. Since this is trial&error, and I don't really understand why the publishOn fixes the problem, does anybody else know? I'm not even sure, whether using springs reactive Webclient is blocking in this case, or not...
Thanks, Henning

SerializationException of Avro Date Object (Date LogicalType)

I have a publisher that accepts a GenericRecord class.
#Override
public Future<RecordMetadata> publish(GenericRecord genericRecord) {
Future<RecordMetadata> recordMetadataFuture =
getPublisher().send(new ProducerRecord<>(producerConfiguration.getProperties()
.getProperty(ProducerConfiguration.PROPERTY_NAME_TOPIC), "sample.key",genericRecord));
return recordMetadataFuture;
}
private KafkaProducer<String, GenericRecord> getPublisher() {
return new KafkaProducer<>(producerConfiguration.getProperties());
}
And I have the following avro schema:
{
"type" : "record",
"name" : "SampleDate",
"namespace": "com.sample.data.generated.avro",
"doc" : "sample date",
"fields" : [
{
"name" : "sampleDate",
"type" : {
"type" : "int",
"logicalType" : "date"
}
}
]
}
I have built my own serializer:
Date Serializer:
#Component
public class SampleDateSerializer implements Serializer<GenericRecord> {
private AvroGenericSerializer serializer;
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
serializer = new AvroGenericSerializer(SampleDate.SCHEMA$);
}
#Override
public byte[] serialize(String topic, GenericRecord data) {
return serializer.serialize(data);
}
#Override
public void close() {
}
Generic Serializer:
public class AvroGenericSerializer {
private EncoderFactory avroEncoderFactory;
private DecoderFactory avroDecoderFactory;
private GenericDatumWriter<GenericRecord> avroWriter;
private GenericDatumReader<GenericRecord> avroReader;
public AvroGenericSerializer(Schema schema) {
avroEncoderFactory = EncoderFactory.get();
avroDecoderFactory = DecoderFactory.get();
avroWriter = new GenericDatumWriter<>(schema);
avroReader = new GenericDatumReader<>(schema);
}
public byte[] serialize(GenericRecord data) {
final ByteArrayOutputStream stream = new ByteArrayOutputStream();
final BinaryEncoder binaryEncoder = avroEncoderFactory.binaryEncoder(stream, null);
try {
avroWriter.write(data, binaryEncoder);
binaryEncoder.flush();
stream.close();
return stream.toByteArray();
} catch (IOException e) {
throw new RuntimeException("Can't serialize Avro object", e);
}
}
public GenericRecord deserialize(byte[] bytes) {
try {
return avroReader.read(null, avroDecoderFactory.binaryDecoder(bytes, null));
} catch (IOException e) {
throw new RuntimeException("Can't deserialize Avro object", e);
}
}
}
However, when testing my publisher class, I am encountering the following error:
org.apache.kafka.common.errors.SerializationException: Can't convert value of class com.sample.data.generated.avro.SampleDate to class com.sample.message.serialize.SampleDateSerializer specified in value.serializer
Debugging the code, I have found out that the
GenericDatumWriter.write()...
method is returning null when calling the
Conversion conversion = this.getData().getConversionByClass(datum.getClass(), logicalType);
which is called from
org.apache.avro.generic.GenericData
public <T> Conversion<T> getConversionByClass(Class<T> datumClass, LogicalType logicalType) {
Map conversions = (Map)this.conversionsByClass.get(datumClass);
return conversions != null?(Conversion)conversions.get(logicalType.getName()):null;
}
In this regard, is there a way for me to populate the
GenericData.conversionsByClass
Map, so that it can return the correct converter to use for the given
date logicalType?
I have solved it by passing the GenericData object in my GenericDatumWriter.
My Generic Serializer now looks like this:
public AvroGenericSerializer(Schema schema) {
avroEncoderFactory = EncoderFactory.get();
avroDecoderFactory = DecoderFactory.get();
final GenericData genericData = new GenericData();
genericData.addLogicalTypeConversion(new TimeConversions.DateConversion());
avroWriter = new GenericDatumWriter<>(schema, genericData);
avroReader = new GenericDatumReader<>(schema);
}

Test EntityManager using JUnit Mockito

I am using Junit with Mockito. I want to test EntityManager, i am getting java.lang.NullPointerException
The below is what i have tried,
main class method is,
#Override
public ReplicationPerspective buildReplicationPerspective(final String replicationDomain)
throws ReplicationStateException {
try {
System.out.println("Test");
final ReplicationPerspective localPerspective =
this.replicationPerspectiveQuery.findReplicationPerspective(replicationDomain);
List<String> ncdKeys = new ArrayList<String>();
for (NodeChangeDelta ncd : this.nodeChangeDeltaQuery.findByChangeStatus(
replicationDomain, ChangeStatus.PENDING)) {
ncdKeys.add(ncd.getKey());
}
localPerspective.setPendingNodeChangeDeltaKeys(ncdKeys);
LOGGER.debug("Local perspective is {} ", localPerspective);
return localPerspective;
}
catch (Throwable t) {
LOGGER.error("Failed to build replication perspective", t);
throw new ReplicationStateException(t);
}
}
replicationPerspectiveQuery Bean file method is,
#PersistenceContext
private EntityManager em;
#Override
public ReplicationPerspective findReplicationPerspective(final String replicationDomain) {
Validate.notBlank(replicationDomain);
ReplicationPerspective perspective =
this.em.find(ReplicationPerspective.class, replicationDomain);
if (perspective == null) {
this.replicationPerspectiveInitializer
.initializeReplicationPerspective(replicationDomain);
perspective = this.em.find(ReplicationPerspective.class, replicationDomain);
}
return perspective;
}
And my test case method is,
#Test
public void testBuildReplicationPerspective() throws ReplicationStateException {
this.replicationStateServiceBean =
new ReplicationStateServiceBean(null, null, null, null,
new ReplicationPerspectiveQueryBean(), null, null);
this.em = Mockito.mock(EntityManager.class);
Mockito.when(this.em.find(ReplicationPerspective.class, REPLICATION_DOMAIN))
.thenReturn(null);
this.replicationStateServiceBean.buildReplicationPerspective(REPLICATION_DOMAIN);
}
I am getting NPE error in replicationPerspectiveQuery Bean file at the below line
ReplicationPerspective perspective =
this.em.find(ReplicationPerspective.class, replicationDomain);
How to test entity manager, help me to solve.
I have also tried to mock like below but didn't work,
Mockito.when(this.replicationPerspectiveQuery.findReplicationPerspective(REPLICATION_DOMAIN)).thenReturn(null);
You are lacking the instructions to have Mockito do the actual injection. Right now you have the EntityManager mocked, but it is not used anywhere.
You can declare your bean as a member of the testclass and annotate it with #InjectMocks to have Mockito do the wiring for you.
See also the documentation for more info and examples.