I am using Pivotal cloud cache and have configured regions as Partitioned Regions.I am trying to use #Cache Evict from Spring data gemfire to evict the region data.It was working fine when the region type was Replicated and not working now.
Is there a way I can remove the data (key + values) from the Partitioned region using my client java application
No matter which type of region you're using, Eviction should work. There are some caveats for Partitioned Regions, however, so you might want to have a look at How Eviction Works.
Regarding your second question, you can directly remove data from your java client application using several methods from the Region class, like remove and delete.
Hope this helps.
Cheers.
Related
I am trying to build a ETL process that extracts data out to GemFire and load in Teradata. However, I am not finding a good mechanism to export data out. The only thing I have found so far is rest api that gets all entries from a region. However, is this good for bulk export ? It will give data back in json which has to be parsed before loading in table, which I assume won't be very performant for large volume of data. Is there any other solution to this ? Like exporting data as csv from GemFire? Or ODBC/JDBC connection to GemFire ? I found both the bulk export and ODBC/JDBC in Gemfire XD documentation but not in core GemFire? So are they not supported in core GemFire? What is the difference between core GemFire and the XD version ?
These are two different products designed with different things in mind. GemFire XD provides a low-latency SQL interface to in-memory table data, so it's generally used as an in-memory RDBMS. GemFire, on the other hand, is an in-memory data grid with "no restrictions" regarding the data you insert into the regions, you basically deal with custom java objects, not with tables. Also, I know GemFire XD was built on top of GemFire in the past, not sure what's the current status of that (you might want to have a look at Snappy Data for more details).
That said, and strictly speaking of GemFire, you can export snapshots of your regions and import them afterwards into another cluster. Even better, you can read a snapshot entry by entry for further processing or transformation into other formats, which I believe is exactly what you're looking for. Please have a look at Cache and Region Snapshots to get the details.
Hope this helps. Cheers.
I have two questions regarding the reddison client:
Does redisson support automatic synchronization of local cache with remote redis cache (when remote cache data change or invalidate)?
I understand that redisson supports data partitioning only in pro edition but isn't that feature already supported OOTB by redis cluster mode? Am I missing something here?
Answering to your questions:
RLocalCachedMap has two synchronization strategies:
INVALIDATE - Used by default. Invalidate cache entry across all RLocalCachedMap instances on map entry change.
UPDATE - Update cache entry across all LocalCachedMap instances on map entry change.
Right, all Redisson objects works also in cluster mode. Each object tied to some Redis node and its content always remain only on the same Redis node and not distributed. If your object couldn't fit in single Redis node then you need to use data partitioning feature. This feature evenly distributes content of object across multiple Redis nodes in cluster.
Re: "local cache truely local" -- I think you can just use a java Map, initially populate it with a RMap contents then from then on just serve your requests from the 'truely local' map in memory.
I am new to GemFire.
Currently we are using an MySQL DB and would like to move to GemFire.
How to move the existing data stored in MySQL over to GemFire? I.e., is there any way to to import existing MySQL data into GemFire?
There are many different options available for you to migrate data from 1 data store (e.g. an RDBMS like MySQL) to an IMDG (e.g. Pivotal GemFire). Pivotal GemFire does not provide any tools for this purpose OOTB.
However, you could...
A) Write a Spring Batch application to migrate all your data from MySQL to Pivotal GemFire in 1 large swoop. This is typical for most large-scale conversion processes, converting from 1 data store to another, either as part of an upgrade or a migration.
The advantage of using Pivotal GemFire as your target data store is that it stores Java Objects. So, if you are, say, using an ORM tool (e.g. Hibernate) to map the data stored in your MySQL database tables back to your application domain objects, you can then immediately and simply turnaround and store those same Objects directly into a corresponding Region in Pivotal GemFire. There is no additional mapping required to store an Object into GemFire.
Although, if you need something less immediate, then you can also...
B) Take advantage of Pivotal GemFire's CacheLoader, and maybe even the CacheWriter mechanisms. The CacheLoader and CacheWriter are implementations of the "Read-Through" and "Write-Through" design patterns.
More details of this approach can be found here.
In a nutshell, you implement a CacheLoader to load data from some external data source on Cache miss. You attach, or register the CacheLoader with a GemFire Region when the Region is created. When a Key (which can correspond to your MySQL Table Primary Key) is requested (Region.get(key)) and an entry does not exist, then GemFire will consult the CacheLoader to resolve the value, providing you actually registered a CacheLoader with the Region.
In this way, you slowly build up Pivotal GemFire from the MySQL RDBMS based on need.
Clearly, it is quite likely Pivotal GemFire will not be able to store all the data from your RDBMS in "memory". So, you can enable both Persistence and Overflow [to Disk] capabilities. By enabling Persistence, GemFire will load the data from it's own DiskStores the next time the nodes come online, assuming you brought them down prior.
The CacheWriter mechanism is nice if you want to run both Pivotal GemFire and MySQL in parallel for while, until you can shift enough of the responsibilities of MySQL over to GemFire, for instance. The CacheWriter will write back to your underlying MySQL DB each time an entry is written or updated in the GemFire Region. You can even do this asynchronously (i.e. "Write-Behind") using GemFire's AsyncEventQueues and Listeners; see here.
Obviously, you many options at your disposal. You need to carefully way your options and choose an approach that best meets your application requirements and needs.
If you have additional questions, let me know.
We need to load millions of key/values into Apache Geode and we'd like to know what are some the options available. Our values happen to be in the 256kb range.
There are several options depending on your application requirements/SLAs or whether you need to perform conversion or other transformations, etc.
Out-of-the-box, Apache Geode provides the Cache & Region Snapshot Service. This is useful when you want to migrate data from 1 existing Apache Geode cluster to another, for instance. Not so useful if your data is coming from an external source, like a RDBMS.
Another option is to lazily load the data based on need. This can be accomplished by implementing the CacheLoader interface and registering the CacheLoader with a Region. Obviously, you could create a CacheLoader implementation that intelligently loads a block of data based on some rules/criteria in addition to loading and returning the single value of interests based on the current requests.
A lot of times, users create an external, custom Conversion process or tool to extract, transform and bulk load (ETL) a bunch of data into Apache Geode. This is typical in complex Use Cases or requirements. However, it is highly advisable to use perhaps a framework/tool like...
Spring XD (now Spring Cloud Data Flow on Pivotal's Cloud Foundry (PCF)) is great ETL tool and pipeline for creating stream-based applications. Spring XD / SCDF provides many different options for "sources" and "sinks" (e.g. GemFire Server). In addition to sources & sinks, you can even "tap" the stream to process the data with "Processors". So whether you are doing real-time stream or batch-oriented data operations (e.g. bulk loads), Spring XD is a great option.
I am sure Google might provide other answers on how to perform ETL with a KeyValue store like Apache Geode.
Hope this helps get you going.
Cheers,
John
We have very limited options to load Gemfire regions .
1) Spring batch:
Create Gemfire writer for load data and remove data
Create batch configuration and lod it
2) Apache Spark
https://www.linkedin.com/pulse/fast-data-access-using-gemfire-apache-spark-part-vaquar-khan-/
I want to back up entire Ignite cluster so that back up clutser will be used if the original(active) cluster is down. Is there any approach for this?
If you need two separate clusters with replication across data center, it would be better to look at GridGain solutions that supports Datacenter Replication.
Unfortunately, Ignite does not support DR.
With Apache Ignite you can logically divide you cluster to two zones to have guarantee that every zone contains full copy of data. However, there is no way to choose primary node for partitions manually. See, AffinityFunction and affinityBackupFilter() method of standard implementations.
As answered above, ready made solution is only available in paid version. Open source Apache ignite provides ability to take cluster wide absolute snapshot. You can add a cron job in your ignite cluster to take this snapshot and add another job to copy snapshot data to object storage like S3.
On the other side, you download this data node wise to work directories of respective nodes as per manual restore procedure and start the cluster. It should automatically activate when all baseline nodes are started successfully and your cluster is ready to use.