I use kafka send quote message, qps is up to 50w, the consumer can handle it, but just only deserialize message, the CPU utilization up to 70!
Anybody knows how to optimize it?
Here is the deserialize code:
public class QuoteMessageDeserializer implements Deserializer<RawQuote> {
private final ObjectMapper objectMapper;
public QuoteMessageDeserializer() {
objectMapper = new ObjectMapper();
objectMapper.configure(DeserializationFeature.FAIL_ON_IGNORED_PROPERTIES, false);
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
}
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
}
#Override
public RawQuote deserialize(String topic, byte[] data) {
try {
QuoteMessage msg = objectMapper.readValue(data, QuoteMessage.class);
return msg.getData();
} catch (Exception e) {
……
}
}
#Override
public void close() {
}
}
I try to use AfterBurner,but it doesn't work to lower cpu utilization
public QuoteMessageDeserializer() {
objectMapper = new ObjectMapper();
objectMapper.registerModule(new AfterburnerModule());
objectMapper.configure(DeserializationFeature.FAIL_ON_IGNORED_PROPERTIES, false);
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
}
thanks.
Related
I am trying to add logging to my application using ClientHttpRequestInterceptor.My interceptor is not being called.
Not sure what is going wrong here -
Here is my code -
#Component
#Slf4j
public final class RestTemplateInterceptor implements ClientHttpRequestInterceptor {
protected static final LoggingAspect aspect = new LoggingAspect();
private final RequestContext requestContext;
private boolean logResponseBody = true;
public RestTemplateInterceptor(RequestContext requestContext) {
this.requestContext = requestContext;
}
#Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution clientHttpRequestExecution) throws IOException {
populateHeader(request);
traceRequest(request, body);
ClientHttpResponse response = clientHttpRequestExecution.execute(request,body);
traceResponse(response);
return response;
}
private void populateHeader(HttpRequest request) {
final HttpHeaders headers = request.getHeaders();
// Propagate TAM headers
headers.add("iv-user", requestContext.getUser());
headers.add("MessageId", requestContext.getMessageId());
headers.add("CorrelationId", requestContext.getConversationId());
headers.add("BusinessId", requestContext.getBusinessId());
headers.add("ApplicationName", requestContext.getSourceSystem());
headers.add("iv-groups", requestContext.getGroups());
headers.add("MessageDateTime", requestContext.getSourceTimestamp());
}
...................
Here is my config file
#Configuration
public class RestTemplateConfig {
/**
* initialise restTemplate
*
* #param restTemplateInterceptor autowired in RestTemplateInterceptor
* #return
*/
#Bean
public RestTemplate restTemplate(ClientHttpRequestInterceptor restTemplateInterceptor, ObjectMapper objectMapper) {
RestTemplate restTemplate = new RestTemplate(new BufferingClientHttpRequestFactory(new SimpleClientHttpRequestFactory()));
List<ClientHttpRequestInterceptor> interceptors = restTemplate.getInterceptors();
if (CollectionUtils.isEmpty(interceptors)) {
interceptors = new ArrayList<>();
}
interceptors.add(restTemplateInterceptor);
restTemplate.setInterceptors(interceptors);
return restTemplate;
}
}
Here is my WebMVC file
#Configuration
public class WebMvcConfig implements WebMvcConfigurer {
#Bean
public WebMvcConfigurer webAuthentication() {
return new WebMvcConfigurer() {
#Override
public void addInterceptors(InterceptorRegistry registry) {
//registry.addInterceptor(myInterceptor());
registry.addInterceptor(new MVCLoggingInterceptor()).addPathPatterns("/api/**");
registry.addInterceptor(new WebAuthentication()).addPathPatterns("/api/**/");
}
};
}
}
Here is my application file
#EnableAsync
#EnableScheduling
#SpringBootApplication(exclude = { SecurityAutoConfiguration.class })
public class XManagementApplication {
public static void main(String[] args) {
SpringApplication.run(XManagementApplication.class, args);
}
}
Can anybody tell why my interceptor class is not called when I try to call any API
Any help would be appreciate?
I don't really understand why you want to instantiate your RestTemplateInterceptor as a Bean. Why not simply instantiate your interceptor inside the method RestTemplateConfig.restTemplate() ?
#Configuration
public class RestTemplateConfig {
#Bean
public RestTemplate restTemplate() {
RestTemplate restTemplate = new RestTemplate(new BufferingClientHttpRequestFactory(new SimpleClientHttpRequestFactory()));
List<ClientHttpRequestInterceptor> interceptors = restTemplate.getInterceptors();
if (CollectionUtils.isEmpty(interceptors)) {
interceptors = new ArrayList<>();
}
interceptors.add(new RestTemplateInterceptor());
restTemplate.setInterceptors(interceptors);
return restTemplate;
}
}
Btw, why do you need to pass RequestContext to the constructor of your interceptor ?
I am trying to have a RPC queue where messages from the Producer/client are prioritized to the consumer. I want all priority 2 messages processed before any priority one messages. I also want each consumer to be able to handle 10 messages at a time. I am able to get each consumer to process 10 messages at a time. I am NOT able to get the prioritization of messages to work. Below is my setup:
Config Files:
#Configuration
public class QueueConfig {
public static final String QUEUE_NAME = "requests";
private int maxPriority = 2;
#Autowired
private ConnectionFactory connectionFactory;
#Bean
public Queue requests() {
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-priority", maxPriority);
return new Queue(QUEUE_NAME,true,false,false, args);
}
#Bean
public Queue replies() {
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-max-priority", maxPriority);
return new Queue("replies",true,false,false, args);
}
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setQueueNames(replies().getName());
return container;
}
#Bean
public RabbitTemplate template() {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setRoutingKey(requests().getName());
return rabbitTemplate;
}
#Bean
public AsyncRabbitTemplate asyncRabbitTemplate(RabbitTemplate rabbitTemplate, SimpleMessageListenerContainer container) {
AsyncRabbitTemplate asyncRabbitTemplate = new AsyncRabbitTemplate(rabbitTemplate, container);
asyncRabbitTemplate.setReceiveTimeout(90000);
return asyncRabbitTemplate;
}
}
Client/Producer:
#Component
public class Client {
#Autowired
private AsyncRabbitTemplate template;
public void sendHigh(String name) {
MessagePostProcessor messageProcessor = new MessagePostProcessor() {
#Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setPriority(2);
return message;
}
};
ListenableFuture<String> response = template.convertSendAndReceive(QueueConfig.QUEUE_NAME, (Object) name,(MessagePostProcessor) messageProcessor);
try {
response.get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
public void sendLow(String name) {
MessagePostProcessor messageProcessor = new MessagePostProcessor() {
#Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setPriority(1);
return message;
}
};
ListenableFuture<String> response = template.convertSendAndReceive(QueueConfig.QUEUE_NAME, (Object) name,(MessagePostProcessor) messageProcessor);
try {
response.get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
config file 2:
#Configuration
#EnableAsync
public class ServiceConfig implements AsyncConfigurer {
#Override
#Bean
public Executor getAsyncExecutor() {
return new SimpleAsyncTaskExecutor();
}
}
Consumer:
#Component
public class Consumer {
#RabbitListener(queues = QueueConfig.QUEUE_NAME)
public String consume(#Payload String name) {
System.out.println("Request Consumer " + name);
String result = name;
if(result.equals("john") || result.equals("john1")) {
try {
Thread.sleep(22000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
return result;
}
Application.properties:
spring.rabbitmq.dynamic=true
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.rabbitmq.host=localhost
spring.rabbitmq.listener.simple.concurrency=10
spring.rabbitmq.listener.direct.acknowledge-mode=MANUAL
Junit:
I would expect all john and john1's to be received by the consumer before jeff, but that is not the behavior that I am seeing. Basically im looking at this println and expecting all john and john1's to print before jeff System.out.println("Request Consumer " + name);
#RunWith(SpringRunner.class)
#ComponentScan(basePackages = "com.test.test")
#EnableAutoConfiguration
#SpringBootTest
public class ApplicationTests {
#Autowired AsyncClass asyncClass;
#Test
public void contextLoads() throws InterruptedException, ExecutionException {
List<Future<String>> futures = new ArrayList<>();
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
futures.add(asyncClass.runAsyncHigh("john"));
Thread.sleep(1000);
futures.add(asyncClass.runAsyncLow("jeff"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
futures.add(asyncClass.runAsyncHigh("john1"));
for(Future<String> future : futures) {
future.get();
}
}
#Component
public class AsyncClass {
#Autowired Client client;
#Async
public Future<String> runAsyncHigh(String name){
client.sendHigh(name);
return new AsyncResult<String>(name);
}
#Async
public Future<String> runAsyncLow(String name){
client.sendLow(name);
return new AsyncResult<String>(name);
}
}
Thanks,
Brian
I been working on implementing a PUB/SUB service using spring-data-Redis.
I have been researching and following the web and got something to work fine.
my problem is that I need absolute reliability when a message is not processed ( either an Exception is thrown or a logic error occurs ).
In which case I need the message to return to the topic for a retry ( by another subscriber or even the same ).
I have looked at several questions, particularly the following:
Redis Pub/Sub with Reliability
and
How to implement Redis Multi-Exec by using Spring-data-Redis
I have understood that I should use multi, exec for managing a transaction, but I couldn't get it to work.
Here is a simplified version of my code
#Configuration
#PropertySource(value = { "classpath:application.properties" })
public class RedisConfig {
#Autowired
Environment env;
#Bean
public MessageListenerAdapter messageListener() {
MyMessageListenerAdapter messageListeneradapter = new MyMessageListenerAdapter(new RedisMessageSubscriber());
messageListeneradapter.afterPropertiesSet();
return messageListeneradapter;
}
#Bean(name="RedisMessagePublisherBean")
public RedisMessagePublisher messagePublisher() {
return new RedisMessagePublisher();
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String , Object> template = new RedisTemplate<>();
template.setValueSerializer(new GenericToStringSerializer<Object>(Object.class));
template.setEnableTransactionSupport(true);
template.setConnectionFactory(lettuceConnectionFactory());
return template;
}
#Bean
public RedisMessageListenerContainer redisContainer() {
RedisMessageListenerContainer container
= new RedisMessageListenerContainer();
container.setConnectionFactory(lettuceConnectionFactory());
container.addMessageListener(messageListener(), topic());
return container;
}
#Bean
public LettuceConnectionFactory lettuceConnectionFactory() {
LettuceConnectionFactory factory = new LettuceConnectionFactory();
factory.setValidateConnection(true);
factory.setDatabase(1);
factory.afterPropertiesSet();
return factory;
}
#Bean
public ChannelTopic topic() {
return new ChannelTopic("MQ_TOPIC");
}
public class MyMessageListenerAdapter extends MessageListenerAdapter{
public MyMessageListenerAdapter(RedisMessageSubscriber redisMessageSubscriber) {
super(redisMessageSubscriber);
}
#Override
public void onMessage(Message message, byte[] pattern) {
RedisTemplate<?, ?> template = redisTemplate();
template.execute(new SessionCallback<String>() {
#Override
public <K, V> String execute(RedisOperations<K, V> operations) throws DataAccessException {
operations.multi();
System.out.println("got message");
String result = doSomeLogic(message);
if (result == null)
operations.discard();
else
operations.exec();
return null;
}
}) ;
}
}
}
My requirements are that if a message failed to process ( I can leave without runtime exceptions etc.. strictly logical error would suffice for now ), It will return to the topic.
Any help is appreciated, Thanks!
I've seen some related questions here, but none worked for me, the rabbit will not serialize my message coming from another application.
Caused by: org.springframework.amqp.AmqpException: No method found for class [B
Below my configuration class to receive the messages.
#Configuration
public class RabbitConfiguration implements RabbitListenerConfigurer{
public final static String EXCHANGE_NAME = "wallet-accounts";
public final static String QUEUE_PAYMENT = "wallet-accounts.payment";
public final static String QUEUE_RECHARGE = "wallet-accounts.recharge";
#Bean
public List<Declarable> ds() {
return queues(QUEUE_PAYMENT, QUEUE_RECHARGE);
}
#Autowired
private ConnectionFactory rabbitConnectionFactory;
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(rabbitConnectionFactory);
}
#Bean
public TopicExchange exchange() {
return new TopicExchange(EXCHANGE_NAME);
}
private List<Declarable> queues(String ... names){
List<Declarable> result = new ArrayList<>();
for (int i = 0; i < names.length; i++) {
result.add(makeQueue(names[i]));
result.add(makeBinding(names[i]));
}
return result;
}
private static Binding makeBinding(String queueName){
return new Binding(queueName, DestinationType.QUEUE, EXCHANGE_NAME, queueName, null);
}
private static Queue makeQueue(String name){
return new Queue(name);
}
#Bean
public MappingJackson2MessageConverter jackson2Converter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
return converter;
}
#Bean
public DefaultMessageHandlerMethodFactory myHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setMessageConverter(jackson2Converter());
return factory;
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(myHandlerMethodFactory());
}
}
Using this other configuration, the error is almost the same:
Caused by: org.springframework.amqp.support.converter.MessageConversionException: failed to resolve class name. Class not found [br.com.beblue.wallet.payment.application.accounts.PaymentEntryCommand]
Configuration:
#Configuration
public class RabbitConfiguration {
public final static String EXCHANGE_NAME = "wallet-accounts";
public final static String QUEUE_PAYMENT = "wallet-accounts.payment";
public final static String QUEUE_RECHARGE = "wallet-accounts.recharge";
#Bean
public List<Declarable> ds() {
return queues(QUEUE_PAYMENT, QUEUE_RECHARGE);
}
#Autowired
private ConnectionFactory rabbitConnectionFactory;
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(rabbitConnectionFactory);
}
#Bean
public TopicExchange exchange() {
return new TopicExchange(EXCHANGE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
private List<Declarable> queues(String ... names){
List<Declarable> result = new ArrayList<>();
for (int i = 0; i < names.length; i++) {
result.add(makeQueue(names[i]));
result.add(makeBinding(names[i]));
}
return result;
}
private static Binding makeBinding(String queueName){
return new Binding(queueName, DestinationType.QUEUE, EXCHANGE_NAME, queueName, null);
}
private static Queue makeQueue(String name){
return new Queue(name);
}
}
Can anyone tell me what's wrong with these settings, or what's missing?
No method found for class [B
Means there is a default SimpleMessageConverter which can't convert your incoming application/json. It is just not aware of that content-type and just falls back to the byte[] to return.
Class not found [br.com.beblue.wallet.payment.application.accounts.PaymentEntryCommand]
Means that Jackson2JsonMessageConverter can't convert your application/json because the incoming __TypeId__ header, representing class of the content, cannot be found in the local classpath.
Well, definitely your configuration for the DefaultMessageHandlerMethodFactory does not make sense for the AMQP conversion. You should consider to use SimpleRabbitListenerContainerFactory bean definition and its setMessageConverter. And yes, consider to inject the proper org.springframework.amqp.support.converter.MessageConverter implementation.
https://docs.spring.io/spring-amqp/docs/1.7.3.RELEASE/reference/html/_reference.html#async-annotation-conversion
From the Spring Boot perspective there is SimpleRabbitListenerContainerFactoryConfigurer to configure on the matter:
https://docs.spring.io/spring-boot/docs/1.5.6.RELEASE/reference/htmlsingle/#boot-features-using-amqp-receiving
In this official example for Apache HttpClient, there's no mention of releasing request or response objects. Are they released as part of httpclient.close() or releaseResources method needs to be overridden with something?
final CountDownLatch latch2 = new CountDownLatch(1);
final HttpGet request3 = new HttpGet("http://www.apache.org/");
HttpAsyncRequestProducer producer3 = HttpAsyncMethods.create(request3);
AsyncCharConsumer<HttpResponse> consumer3 = new AsyncCharConsumer<HttpResponse>() {
HttpResponse response;
#Override
protected void onResponseReceived(final HttpResponse response) {
this.response = response;
}
#Override
protected void onCharReceived(final CharBuffer buf, final IOControl ioctrl) throws IOException {
// Do something useful
}
#Override
protected void releaseResources() {
}
#Override
protected HttpResponse buildResult(final HttpContext context) {
return this.response;
}
};
httpclient.execute(producer3, consumer3, new FutureCallback<HttpResponse>() {
public void completed(final HttpResponse response3) {
latch2.countDown();
System.out.println(request2.getRequestLine() + "->" + response3.getStatusLine());
}
public void failed(final Exception ex) {
latch2.countDown();
System.out.println(request2.getRequestLine() + "->" + ex);
}
public void cancelled() {
latch2.countDown();
System.out.println(request2.getRequestLine() + " cancelled");
}
});
latch2.await();
} finally {
httpclient.close();
}
One needs to override #releaseResources() only if the consumer makes use of system resources such as files, pipes, etc. If response content is always held in memory it gets GCed the normal way.