My code like this:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.enableCheckpointing(500);
DataStream<String> stream = env.addSource(getConsumer(TOPIC_1));
Jedis jedis = new Jedis("master1");
stream.map(new RichMapFunction<String, String>() {
#Override
public String map(String value) throws Exception {
String result = jedis.hget("rtc", value);
return result;
}
});
I want to get some data from Redis in map(), but it cannot run,because Jedis.class is not serializable.
How to use not serializable class in map(),such as ZkClient,Jedis?
All rich functions like the RichMapFunction have an open(Configuration) and close call which you can override. These lifecycle methods are called once the function has been deployed to a TaskManager where it is executed.
class MyMapFunction extends RichMapFunction<String, String> {
private transient Jedis jedis;
#Override
public void open(Configuration parameters) {
// open connection to Redis, for example
jedis = new Jedis("master1");
}
#Override
public void close() {
// close connection to Redis
jedis.close();
}
}
Related
ExpiredEvent seems to be lost..... if we use redis cluster.
It works perfect if we use a stand-alone redis server.
my redis.config
notify-keyspace-events "Egx"
my project code
implementation 'org.springframework.session:spring-session-data-redis:2.7.0'
implementation 'org.springframework.data:spring-data-redis:2.7.3'
implementation 'io.lettuce:lettuce-core:6.1.8.RELEASE'
#configuration
#EnableRedisHttpSession(maxInactiveIntervalInSeconds = 60 * 1)
#EnableRedisRepositories
public class RedisConfig {
#bean
public LettuceConnectionFactory lettuceConnectionFactory() {
RedisClusterConfiguration redisClusterConfiguration
= new RedisClusterConfiguration(List.of("ip:port", "ip:port", "ip:port"));
redisClusterConfiguration.setPassword("password");
return new LettuceConnectionFactory(redisClusterConfiguration);
}
}
#component
public class RedisSessionListener {
#eventlistener
public void onCreate(org.springframework.session.events.SessionCreatedEvent event) {
System.out.println("create!!");
System.out.println(event);
}
#EventListener
public void onDestroy(org.springframework.session.events.SessionDestroyedEvent event) {
System.out.println("destory!!");
System.out.println(event);
}
#EventListener
public void sessionExired(SessionExpiredEvent event) {
System.out.println("exired!!");
System.out.println(event);
}
}
I repeated log in and log out
test result..
As it's seem like the expired event seems to be lost.
Is there a solution?
I have the below Spring boot code to receive values whenever a Redis stream is appended with new record. The problem is receiver never receives any message, also, the subscriber, when checked with subscriber.isActive(), is always inactive. Whats wrong in this code? What did I miss? Doc for reference.
On spring boot start, initialize the necessary redis resources
Lettuce connection factory
#Bean
public RedisConnectionFactory redisConnectionFactory() {
return new LettuceConnectionFactory("127.0.0.1", 6379);
}
RedisTemplate from the connection factory
#Bean
public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory connectionFactory) {
RedisTemplate<String, String> redisTemplate = new RedisTemplate<>();
redisTemplate.setConnectionFactory(connectionFactory);
return redisTemplate;
}
Rest controller to append data to redis stream
#PutMapping("/{name}")
public String post(#PathVariable String name) {
return redisTemplate.opsForStream().add(StreamRecords.newRecord().in("streamx").ofObject(name)).getValue();
}
JMS style imperative message listener
#Component
public class MyStreamListener implements StreamListener<String, MapRecord<String, String, String>> {
#Override
public void onMessage(MapRecord<String, String, String> message) {
System.out.println("message received: " + message.getValue());
}
}
Initialize the listener
#Bean
public Subscription listener(MyStreamListener streamListener, RedisConnectionFactory redisConnectionFactory) throws InterruptedException {
StreamMessageListenerContainer<String, MapRecord<String, String, String>> container = StreamMessageListenerContainer
.create(redisConnectionFactory);
Subscription subscription = container.receive(Consumer.from("my-group-1", "consumer-1"),
StreamOffset.create("streamx", ReadOffset.latest())), streamListener);
System.out.println(subscription.isActive()); // always false
return subscription;
}
Though, I am able to append to the stream through api.
The important step is, start the StreamMessageListenerContainer after the subscription is done.
container.start();
I'm using Azure Redis Cache for development and wanted to verify the way I'm handling the exceptions. According to the best practices, it's possible to face RedisConnectionExceptions and to resolve this, we have to dispose the old ConnectionMultiplexer and create a new one. If abortConnect is set to false, then the multiplexer will silently retry connecting without throwing the error. So if the exception is thrown, it will only be after some attempts to reconect and still failing. Is my understanding of this correct?
This is my connection string -
cachename.redis.cache.windows.net:6380,password=Password,ssl=True,abortConnect=False
I believe the connection exception will only occus when you try to call GetConnection() on the multiplexer. Find my Code below -
static Lazy<ConnectionMultiplexer> multiplexer = CreateMultiplexer();
public static ConnectionMultiplexer GetConnection() => multiplexer.Value;
private static Lazy<ConnectionMultiplexer> CreateMultiplexer()
{
return new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(connectionString));
}
private static void CloseMultiplexer(Lazy<ConnectionMultiplexer> oldMultiplexer)
{
if (oldMultiplexer != null)
{
oldMultiplexer.Value.Close();
}
}
public static void Reconnect()
{
var oldMultiplexer = multiplexer;
CloseMultiplexer(multiplexer);
multiplexer = CreateMultiplexer();
}
And I'm Consuming this below in another class -
public class RedisCacheManager
{
private static IDatabase _cache;
private TimeSpan expiry = new TimeSpan(hours: 6, minutes: 0, seconds: 0);
public RedisCacheManager()
{
try
{
_cache = RedisCacheHelper.GetConnection().GetDatabase();
}
catch(RedisConnectionException)
{
RedisCacheHelper.Reconnect();
new RedisCacheManager();
}
}
public async Task<RedisValue[]> GetFromCacheAsync(List<string> keys)
{
var cacheValues = await _cache.StringGetAsync(keys.Select(k => (RedisKey)k).ToArray());
return cacheValues;
}
public async Task SaveInCacheAsync<TValue>(Dictionary<string, TValue> kvps)
{
var tasks = new List<Task>();
foreach(var kvp in kvps)
{
tasks.Add(_cache.StringSetAsync(kvp.Key, JsonConvert.SerializeObject(kvp), expiry));
}
await Task.WhenAll(tasks);
}
}
I'm not sure id calling the constructor in the catch block is a good practice. And are there any other exceptions that I would need to handle while calling StringGetAsync and StringSetAsync?
The CacheManager can look like this:
using Newtonsoft.Json;
using StackExchange.Redis;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
public sealed class RedisCacheManager : IDisposable
{
private readonly TimeSpan _expiry;
private readonly Lazy<ConnectionMultiplexer> _lazyConnection;
private ConnectionMultiplexer Connection { get => _lazyConnection.Value; }
public RedisCacheManager(string connectionString, TimeSpan expiry)
{
_expiry = expiry;
_lazyConnection = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(connectionString));
}
public async Task<RedisValue[]> GetFromCacheAsync(IEnumerable<string> keys)
{
var cacheValues = await Connection.GetDatabase()
.StringGetAsync(keys.Select(key => (RedisKey)key).ToArray()).ConfigureAwait(false);
return cacheValues;
}
public async Task SaveInCacheAsync<TValue>(Dictionary<string, TValue> kvps)
{
var tasks = kvps
.Select(kvp => Connection.GetDatabase().StringSetAsync(kvp.Key, JsonConvert.SerializeObject(kvp), _expiry))
.ToArray();
await Task.WhenAll(tasks).ConfigureAwait(false);
}
public void Dispose()
{
if (_lazyConnection.IsValueCreated)
{
_lazyConnection.Value.Dispose();
}
}
}
Using:
public readonly static RedisCacheManager RedisCacheManager = new RedisCacheManager("connection string", TimeSpan.FromHours(6));
Remarks:
it is intended that abortConnect=false (which means that the call succeeds even if a connection to the Azure Cache for Redis is not established) and from constructor shouldn't be thrown any Redis-exceptions
The object returned from GetDatabase is a cheap pass-thru object, and does not need to be stored.
GetFromCacheAsync / SaveInCacheAsync-methods can throw an exception to outside and it is OK. You can apply Retry-policy to resolve transient faults.
If you have any IoC-container then it should create RedisCacheManager with a single instance scope (for example, Autofac registration)
I been working on implementing a PUB/SUB service using spring-data-Redis.
I have been researching and following the web and got something to work fine.
my problem is that I need absolute reliability when a message is not processed ( either an Exception is thrown or a logic error occurs ).
In which case I need the message to return to the topic for a retry ( by another subscriber or even the same ).
I have looked at several questions, particularly the following:
Redis Pub/Sub with Reliability
and
How to implement Redis Multi-Exec by using Spring-data-Redis
I have understood that I should use multi, exec for managing a transaction, but I couldn't get it to work.
Here is a simplified version of my code
#Configuration
#PropertySource(value = { "classpath:application.properties" })
public class RedisConfig {
#Autowired
Environment env;
#Bean
public MessageListenerAdapter messageListener() {
MyMessageListenerAdapter messageListeneradapter = new MyMessageListenerAdapter(new RedisMessageSubscriber());
messageListeneradapter.afterPropertiesSet();
return messageListeneradapter;
}
#Bean(name="RedisMessagePublisherBean")
public RedisMessagePublisher messagePublisher() {
return new RedisMessagePublisher();
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String , Object> template = new RedisTemplate<>();
template.setValueSerializer(new GenericToStringSerializer<Object>(Object.class));
template.setEnableTransactionSupport(true);
template.setConnectionFactory(lettuceConnectionFactory());
return template;
}
#Bean
public RedisMessageListenerContainer redisContainer() {
RedisMessageListenerContainer container
= new RedisMessageListenerContainer();
container.setConnectionFactory(lettuceConnectionFactory());
container.addMessageListener(messageListener(), topic());
return container;
}
#Bean
public LettuceConnectionFactory lettuceConnectionFactory() {
LettuceConnectionFactory factory = new LettuceConnectionFactory();
factory.setValidateConnection(true);
factory.setDatabase(1);
factory.afterPropertiesSet();
return factory;
}
#Bean
public ChannelTopic topic() {
return new ChannelTopic("MQ_TOPIC");
}
public class MyMessageListenerAdapter extends MessageListenerAdapter{
public MyMessageListenerAdapter(RedisMessageSubscriber redisMessageSubscriber) {
super(redisMessageSubscriber);
}
#Override
public void onMessage(Message message, byte[] pattern) {
RedisTemplate<?, ?> template = redisTemplate();
template.execute(new SessionCallback<String>() {
#Override
public <K, V> String execute(RedisOperations<K, V> operations) throws DataAccessException {
operations.multi();
System.out.println("got message");
String result = doSomeLogic(message);
if (result == null)
operations.discard();
else
operations.exec();
return null;
}
}) ;
}
}
}
My requirements are that if a message failed to process ( I can leave without runtime exceptions etc.. strictly logical error would suffice for now ), It will return to the topic.
Any help is appreciated, Thanks!
I would like to create a Gemfire region in a Spring Boot application. Following this sample, it works well wihout adding database support. If I add database, it will shows error like " Error creating bean with name 'dataSource'". However, default gemfire cache bean works well with datasource integration.
#EnableAutoConfiguration
// Sprint Boot Auto Configuration
#ComponentScan(basePackages = "napo.demo")
#EnableCaching
#SuppressWarnings("unused")
public class Application extends SpringBootServletInitializer {
private static final Class<Application> applicationClass = Application.class;
private static final Logger log = LoggerFactory.getLogger(applicationClass);
public static void main(String[] args) {
SpringApplication.run(applicationClass, args);
}
/* **The commented code works well with database.**
#Bean
CacheFactoryBean cacheFactoryBean() {
return new CacheFactoryBean();
}
#Bean
ReplicatedRegionFactoryBean<Integer, Integer> replicatedRegionFactoryBean(final Cache cache) {
ReplicatedRegionFactoryBean<Integer, Integer> region= new ReplicatedRegionFactoryBean<Integer, Integer>() {{
setCache(cache);
setName("demo");
}};
return region;
} */
// This configuration will cause issue as beow
//
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceAutoConfiguration$NonEmbeddedConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method 'dataSource' threw exception; nested exception is java.lang.NullPointerException
#Bean
GemfireCacheManager cacheManager(final Cache gemfireCache) {
return new GemfireCacheManager() {
{
setCache(gemfireCache);
}
};
}
// NOTE ideally, "placeholder" properties used by Spring's PropertyPlaceholderConfigurer would be externalized
// in order to avoid re-compilation on property value changes (so... this is just an example)!
#Bean
public Properties placeholderProperties() {
Properties placeholders = new Properties();
placeholders.setProperty("app.gemfire.region.eviction.action", "LOCAL_DESTROY");
placeholders.setProperty("app.gemfire.region.eviction.policy-type", "MEMORY_SIZE");
placeholders.setProperty("app.gemfire.region.eviction.threshold", "4096");
placeholders.setProperty("app.gemfire.region.expiration.entry.tti.action", "INVALIDATE");
placeholders.setProperty("app.gemfire.region.expiration.entry.tti.timeout", "300");
placeholders.setProperty("app.gemfire.region.expiration.entry.ttl.action", "DESTROY");
placeholders.setProperty("app.gemfire.region.expiration.entry.ttl.timeout", "60");
placeholders.setProperty("app.gemfire.region.partition.local-max-memory", "16384");
placeholders.setProperty("app.gemfire.region.partition.redundant-copies", "1");
placeholders.setProperty("app.gemfire.region.partition.total-max-memory", "32768");
return placeholders;
}
#Bean
public PropertyPlaceholderConfigurer propertyPlaceholderConfigurer(
#Qualifier("placeholderProperties") Properties placeholders) {
PropertyPlaceholderConfigurer propertyPlaceholderConfigurer = new PropertyPlaceholderConfigurer();
propertyPlaceholderConfigurer.setProperties(placeholders);
return propertyPlaceholderConfigurer;
}
#Bean
public Properties gemfireProperties() {
Properties gemfireProperties = new Properties();
gemfireProperties.setProperty("name", "SpringGemFireJavaConfigTest");
gemfireProperties.setProperty("mcast-port", "0");
gemfireProperties.setProperty("log-level", "config");
return gemfireProperties;
}
#Bean
#Autowired
public CacheFactoryBean gemfireCache(#Qualifier("gemfireProperties") Properties gemfireProperties) throws Exception {
CacheFactoryBean cacheFactory = new CacheFactoryBean();
cacheFactory.setProperties(gemfireProperties);
return cacheFactory;
}
#Bean(name = "ExamplePartition")
#Autowired
public ReplicatedRegionFactoryBean<Object, Object> examplePartitionRegion(Cache gemfireCache,
#Qualifier("partitionRegionAttributes") RegionAttributes<Object, Object> regionAttributes) throws Exception {
ReplicatedRegionFactoryBean<Object, Object> examplePartitionRegion =
new ReplicatedRegionFactoryBean<Object, Object>();
examplePartitionRegion.setAttributes(regionAttributes);
examplePartitionRegion.setCache(gemfireCache);
examplePartitionRegion.setName("demo");
return examplePartitionRegion;
}
#Bean
#Autowired
public RegionAttributesFactoryBean partitionRegionAttributes(
EvictionAttributes evictionAttributes,
#Qualifier("entryTtiExpirationAttributes") ExpirationAttributes entryTti,
#Qualifier("entryTtlExpirationAttributes") ExpirationAttributes entryTtl) {
RegionAttributesFactoryBean regionAttributes = new RegionAttributesFactoryBean();
regionAttributes.setEvictionAttributes(evictionAttributes);
regionAttributes.setEntryIdleTimeout(entryTti);
regionAttributes.setEntryTimeToLive(entryTtl);
return regionAttributes;
}
#Bean
public EvictionAttributesFactoryBean defaultEvictionAttributes(
#Value("${app.gemfire.region.eviction.action}") String action,
#Value("${app.gemfire.region.eviction.policy-type}") String policyType,
#Value("${app.gemfire.region.eviction.threshold}") int threshold) {
EvictionAttributesFactoryBean evictionAttributes = new EvictionAttributesFactoryBean();
evictionAttributes.setAction(EvictionActionType.valueOfIgnoreCase(action).getEvictionAction());
evictionAttributes.setThreshold(threshold);
evictionAttributes.setType(EvictionPolicyType.valueOfIgnoreCase(policyType));
return evictionAttributes;
}
#Bean
public ExpirationAttributesFactoryBean entryTtiExpirationAttributes(
#Value("${app.gemfire.region.expiration.entry.tti.action}") String action,
#Value("${app.gemfire.region.expiration.entry.tti.timeout}") int timeout) {
ExpirationAttributesFactoryBean expirationAttributes = new ExpirationAttributesFactoryBean();
expirationAttributes.setAction(ExpirationActionType.valueOfIgnoreCase(action).getExpirationAction());
expirationAttributes.setTimeout(timeout);
return expirationAttributes;
}
#Bean
public ExpirationAttributesFactoryBean entryTtlExpirationAttributes(
#Value("${app.gemfire.region.expiration.entry.ttl.action}") String action,
#Value("${app.gemfire.region.expiration.entry.ttl.timeout}") int timeout) {
ExpirationAttributesFactoryBean expirationAttributes = new ExpirationAttributesFactoryBean();
expirationAttributes.setAction(ExpirationActionType.valueOfIgnoreCase(action).getExpirationAction());
expirationAttributes.setTimeout(timeout);
return expirationAttributes;
}
#Bean
public PartitionAttributesFactoryBean defaultPartitionAttributes(
#Value("${app.gemfire.region.partition.local-max-memory}") int localMaxMemory,
#Value("${app.gemfire.region.partition.redundant-copies}") int redundantCopies,
#Value("${app.gemfire.region.partition.total-max-memory}") int totalMaxMemory) {
PartitionAttributesFactoryBean partitionAttributes = new PartitionAttributesFactoryBean();
partitionAttributes.setLocalMaxMemory(localMaxMemory);
partitionAttributes.setRedundantCopies(redundantCopies);
partitionAttributes.setTotalMaxMemory(totalMaxMemory);
return partitionAttributes;
}
#Override
protected SpringApplicationBuilder configure(
SpringApplicationBuilder application) {
return application.sources(applicationClass);
}}
demoService java code:
#Service
public class demoService {
#Autowired
private demoMapper demoMapper;
#Cacheable("demo")
public Fund getDemo(String code) {
Demo demo= demoMapper.getDemo(Code);
return demo;
}
Here is an example of setting entry-ttl among other attributes: https://github.com/spring-projects/spring-gemfire-examples/blob/master/basic/java-config/src/main/java/org/springframework/data/gemfire/example/SpringJavaBasedContainerGemFireConfiguration.java