I am developing some app using Gemfire and it would be great to be able to provide some fake data while in Dev environment.
So instead of doing it in the code like I do today, I was thinking about using spring application-context.xml do pre-load some dummy data in the region I am currently working on. Something close to what DBUnit does but for DEV not Test scope.
Later I could just switch envs on Spring and that data would not be loaded.
Is it possible to add data using SpringData Gemfire to a local data grid?
Thanks!
There is no direct support in Spring Data GemFire to load data into a GemFire cluster. However, there are several options afforded to a SDG/GemFire developer to load data.
The most common approach is to define a GemFire CacheLoader attached to the Region. However, this approach is "lazy" and only loads data from a (potentially) external data source on a cache miss. Of course, you could program the logic in the CacheLoader to "prefetch" a number of entries in a somewhat "predictive" manner based on data access patterns. See GemFire's User Guide for more details.
Still, we can do better than this since it is more likely that you want to "preload" a particular data set for development purposes.
Another, more effective technique, is to use a Spring BeanPostProcessor registered in your Spring ApplicationContext that post processes your "Region" bean after initialization. For instance...
Where the RegionPutAllBeanPostProcessor is implemented as...
package example;
public class RegionPutAllBeanPostProcessor implements BeanPostProcessor {
private Map regionData;
private String targetRegionBeanName;
protected Map getRegionData() {
return (regionData != null ? regionData : Collections.emptyMap());
}
public void setRegionData(final Map regionData) {
this.regionData = regionData;
}
protected String getTargetRegionBeanName() {
Assert.state(StringUtils.hasText(targetRegionBeanName), "The target Region bean name was not properly specified!");
return targetBeanName;
}
public void setTargetRegionBeanName(final String targetRegionBeanName) {
Assert.hasText(targetRegionBeanName, "The target Region bean name must be specified!");
this.targetRegionBeanName = targetRegionBeanName;
}
#Override
public Object postProcessBeforeInitialization(final Object bean, final String beanName) throws BeansException {
return bean;
}
#Override
#SuppressWarnings("unchecked")
public Object postProcessAfterInitialization(final Object bean, final String beanName) throws BeansException {
if (beanName.equals(getTargetRegionBeanName()) && bean instanceof Region) {
((Region) bean).putAll(getRegionData());
}
return bean;
}
}
It is not too difficult to imagine that you could inject a DataSource of some type to pre-populate the Region. The RegionPutAllBeanPostProcessor was designed to accept a specific Region (based on the Region beans ID) to populate. So you could defined multiple instances each taking a different Region and different DataSource (perhaps) to populate the Region(s) of choice. This BeanPostProcess just take a Map as the data source, but of course, it could be any Spring managed bean.
Finally, it is a simple matter to ensure that this, or multiple instances of the RegionPutAllBeanPostProcessor is only used in your DEV environment by taking advantage of Spring bean profiles...
<beans>
...
<beans profile="DEV">
<bean class="example.RegionPutAllBeanPostProcessor">
...
</bean>
...
</beans>
</beans>
Usually, loading pre-defined data sets is very application-specific in terms of the "source" of the pre-defined data. As my example illustrates, the source could be as simple as another Map. However, it would be a JDBC DataSource, or perhaps a Properties file or well, anything for that matter. It is usually up to the developers preference.
Though, one thing that might be useful to add to Spring Data GemFire would be to load data from a GemFire Cache Region Snapshot. I.e. data that may have been dumped from a QA or UAT environment, or perhaps even scrubbed from PROD for testing purposes. See GemFire Snapshot Service for more details.
Also see the JIRA ticket (SGF-408) I just filed to add this support.
Hopefully this gives you enough information and/or ideas to get going. Later, I will add first-class support into SDG's XML namespace for preloading data sets.
Regards,
John
Related
I am using spring webflux to call one service from another via Schedulers.elastic()
Mono<Integer> anaNotificationCountObservable = wrapWithRetryForFlux(wrapWithTimeoutForFlux(
notificationServiceMediatorFlux.getANANotificationCountForUser(userId).subscribeOn(reactor.core.scheduler.Schedulers.elastic())
)).onErrorReturn(0);
In main thread i am setting one InhertitableThreadLocal variable and in the child thread I am trying to access it and it is working fine.
This is my class for storing threadlocal
#Component
public class RequestCorrelation {
public static final String CORRELATION_ID = "correlation-id";
private InheritableThreadLocal<String> id = new InheritableThreadLocal<>();
public String getId() {
return id.get();
}
public void setId(final String correlationId) {
id.set(correlationId);
}
public void removeCorrelationId() {
id.remove();
}
}
Now the issue is first time its working fine meaning the value i am setting in threadlocal is passed to other services.
But second time also, it is using old id(generated in last request).
I tried using Schedulers.newSingle() instead of elastic(), then its working fine.
So think since elastic() is re-using threads, thats why it is not able to clear / or it is re-using.
How should i resolve issue.
I am setting thread local in my filter and clearing the same in myfiler
requestCorrelation.setId(UUID.randomUUID().toString());
chain.doFilter(req,res)
requestCorrelation.removeCorrelationId();
You should never tie resources or information to a particular thread when leveraging a reactor pipeline. Reactor is itself scheduling agnostic; developers using your library can choose to schedule work on another scheduler - if you decide to force a scheduling model you might lose performance benefits.
Instead you can store data inside the reactor context. This is a map-like structure that’s tied to the subscriber and independent of the scheduling arrangement.
This is how projects like spring security and micrometer store information that usually belongs in a threadlocal.
I'm working my way through http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html to integrate Spring Cache into our AWS applications.
Can someone please explain to me how to correctly encapsulate different #Cacheable classes using the guide?
As far as I can tell, when you use #EnableElastiCache you need to specify the name of the the Elasticache cluster that create in AWS in the annotation:
#EnableElastiCache( #CacheClusterConfig( name = "myAwsCluster", expiration = 300 ) )
Then you have to use the same cluster name as the cache name in your #Cacheable class:
#Cacheable( "myAwsCluster" )
public String expensiveMethod()
Unless I've missed something, this completely breaks encapsulation as you have to tie the value on the annotation to the physical resource you create in AWS. Am I missing something, or is this the way that spring cloud expects you to work?
Furthermore, it means that you need to launch a seperate AWS ElastiCache cluster for each spring Cache class that you want to use, making it hugely expensive and prohibiting resource sharing.
#CacheConfig( "myAwsCluster" )
public class Class1
{
#Cacheable
public void something()
{
...
}
}
#CacheConfig( "mySecondAwsCluster" )
public class Class2
{
#Cacheable
public void somethingElse()
{
...
}
}
#EnableElastiCache encourages physical separation of caches, which may not what one needs all the time.
Instead of using #EnableElastiCache, use #EnableCaching for cache configuration. It can point to an Elasticache. Sample configuration for Elasticache with Redis cluster mode enabled.
spring.redis.cluster.nodes=<Elasticache Configuration endpoint>
Now you can use #Cacheable with the logical cache names.
I am developing a very simple dashboard to clear Gemfire regions for testing purposes. I am mainly doing this to get testers a tool for doing this by themselves.
I would like to dynamically fetch the current available Regions names to clear.
I am searching spring-data-gemfire documentation but I couldn't find a way to get all region names.
The best hint I have so far is <gfe:auto-region-lookup/>, but I guess I would still need to have a cache.xml with all region names and also I am not sure how to dynamically displaying their names and how to remove all data from those regions.
Thanks
<gfe:auto-region-lookup> is meant to automatically create beans in the Spring ApplicationContext for all GemFire Regions that have been explicitly created outside the Spring context (i.e. cache.xml or using GemFire's relatively new Cluster-based Configuration Service). However, a developer must use and/or enable those mechanisms to employ the auto-region-lookup functionality.
To get a list of all Region names in the GemFire "cluster", you need something equivalent to Gfsh's 'list region' command, which employs a Function to gather up all the Regions defined in the GemFire (Cache) cluster.
Note that members can define different Regions, i.e. all members participating in the cluster do not necessarily have to define the same Regions. In most cases they do since it is beneficial for replication and HA purposes. Still some members may define local Regions only that member will use.
To go on to clear the Regions from the list, you would again need to employ a GemFire Function to "clear" the other Regions in the cluster that the inquiring, acting member does not currently define.
Of course, this problem is real simple if you only want to clear Regions defined on the member itself...
#Autowired
private Cache gemfireCache;
...
public void clearRegions() {
for (Region rootRegion : gemfireCache.rootRegions()) {
for (Region subRegion : rootRegion.subregions(true)) {
subRegion.clear();
}
rootRegion.clear());
}
}
See rootRegions() and subregions(recursive:boolean) for more details.
Note, GemFire's Cache interface implements the RegionService interface.
Hope this helps.
Cheers!
I want to use Redis as a cache storage for multiple applications on the same physical machine.
I know at least two ways of doing it:
by running several Redis instances on different ports;
by using different Redis databases for different applications.
But I don't know which one is better for me.
What are advantages and disadvantages of these methods?
Is there any better way of doing it?
Generally, you should prefer the 1st approach, i.e. dedicated Redis servers. Shared databases are managed by the same Redis process and can therefore block each other. Additionally, shared databases share the same configuration (although in your case this may not be an issue since all databases are intended for caching). Lastly, shared databases are not supported by Redis Cluster.
For more information refer to this blog post: https://redislabs.com/blog/benchmark-shared-vs-dedicated-redis-instances
We solved this problem by namespacing the keys. Intially we tried using databases where each database ID would be used a specific applications. However, that idea was not scalable since there is a limited number of databases, plus in Premium offerings (like Azure Cache for Redis Premium instances with Sharding enabled), the concept of database is not used.
The solution we used is attaching a unique prefix for all keys. Each application would be annotated with a unique moniker which would be prefixed infront of each key.
To reduce churn, we have built a framework (URP). If you are using StackExchange.Redis then yuo will be able to use the URP SDK directly. If it helps, I have added some of the references.
Source Code and Documentation - https://github.com/microsoft/UnifiedRedisPlatform.Core/wiki/Management-Console
Blog Post (idea) - https://www.devcompost.com/post/__urp
You can use different cache manager for each application will also work same way I am using.
like :
#Bean(name = "myCacheManager")
public CacheManager cacheManager(RedisTemplate<String, Object> redisTemplate) {
RedisCacheManager cacheManager = new RedisCacheManager(redisTemplate);
return cacheManager;
}
#Bean(name ="customKeyGenerator")
public KeyGenerator keyGenerator() {
return new KeyGenerator() {
#Override
public Object generate(Object o, Method method, Object... objects) {
// This will generate a unique key of the class name, the method name,
// and all method parameters appended.
StringBuilder sb = new StringBuilder();
sb.append(o.getClass().getName());
sb.append(method.getName());
for (Object obj : objects) {
sb.append(obj.toString());
}
return sb.toString();
}
};
}
Is it possible to do Javaconfig i.e annotations in spring instead of xml to create client regions in Spring gemfire?
I need to plug in cache loader and cache writer also to the regions created...how is that possible to do?
I want to perform the client pool configuration as well..How is that possible?
There is a good example of this in the spring.io guides. However, GemFire APIs are factories, wrapped by Spring FactoryBeans in Spring Data Gemfire, so I find XML actually more straightforward for configuring Cache and Regions.
Regarding... "how can I create a client region in a distributed environment?"
In the same way the Spring IO guides demonstrate Regions defined in a peer cache on a GemFire Server, something similar to...
#Bean
public ClientRegionFactoryBean<Long, Customer> clientRegion(ClientCache clientCache) {
return new ClientRegionFactoryBean() {
{
setCache(clientCache);
setName("Customers");
setShortcut(ClientRegionShortcut.CACHING_PROXY); // Or just PROXY if the client is not required to store data, or perhaps another shortcut type.
...
}
}
}
Disclaimer, I did not test this code snippet, so it may need minor tweaking along with additional configuration as necessary by the application.
You of course, will defined a along with a Pool in your Spring config, or use the element abstraction on the client-side.