I have a requirement for a graph database that needs to be backed-up and potentially accessed at a lower level of abstraction. It is also must be distributed for the sake of load balancing, (single master replication will do).
I know that it is possible to implement a graph database using a self-referencing key-value store. The Git object database is an example of this pattern. One of the frustrating things I find about most graph databases is that they do not "disclose" their underlying persistence layer in public api.
Do any replicated graph databases exist that allow for an underlying key-value stores to be "plugged-in" or accessed directly?
I addition to Gremlin/Tinkerpop, mentionned by #amirouche above, I'm aware of two solutions:
Redis, completed by its Graph Module, matches your description.
Cayley could also be a solution as it provides graph features over various SQL and NoSQL backends, some of them supporting distributed mode (Postgresql, MySQL, MongoDB, CockroachDB)
Related
I need to migrate my Spring Boot app to Redis instead of some useless relational DBMS (i need some scalability among other reasons). I made some research but couldn't make clear some questions before the start:
Should i use Redisson as Hibernate cache
Should i somehow modify my model/entities to use 'em in NoSQL Redis? The relations in my model are not primitive and i wonder should i perform any steps to port the model to run under Redis?
Thanks in advance for any assist.
You should think around following:
Access Patterns: These help decide which NoSQL to chose. Redis is a Key-Value store. All the access patterns are key based. It doesn't have a query language.
Architecture: Redis is completely in-memory and therefore amazingly fast. It is one of the most popular cache solutions. However, if you use it as Database then work out the pricing as Memory is costly than disk space. Moreover, there is a minor chance to lose data if the cluster crashes and data has not yet been replicated to disk.
If you are able to tick all the boxes then I would suggest to keep it simple and use a Java Client like Jedis rather than Redisson.
You will have to write a new DAO layer to adapt to key based access patterns.
Is this configuration in Moqui possible? Everything I've seen on the subject of multiple instances (e.g. this question and the framework doc pages) involves per-instance databases, rather than a common shared data set.
We need the same data available in each application instance (and a consistent cache) so that we can load balance end-users across multiple instances. We will be supporting users world-wide, so we may potentially need to create application instances closer to the user's actual location in order to reduce latency; we also want to ensure we can make best use of elastic horizontal scaling in cloud-based deployments.
Multi-tenant and the newer multi-instance variation on that are the opposite of what you're looking for. They are for large numbers of small instances, not a single large distributed instance with multiple application server instances running against the same database.
For clustering support by default Moqui uses Hazelcast, though that is done through a series of interfaces that can be implemented with other distributed computing tools. Here is the component needed to run a multi-server cluster with Hazelcast:
https://github.com/moqui/moqui-hazelcast
The most important aspects of clustering are cache invalidation for the entity (database) caches and web session replication. It also supports other tools for distributing workload and data as mentioned in the readme.
For distribution across multiple data centers or geographical regions there are much bigger issues. Moqui Framework is primarily for transactional applications like accounting, inventory management, etc that need strict transactional consistency. Big data or NoSQL style eventual consistency and other similar approaches do not do well with ERP and other transactional applications, there is no way to use locks and such in the database to protect against double spend of funds, double reservation or issuance of inventory, etc.
Consider the challenge of distributed relational transactional databases, ie multi-master database clusters. With multi-master setups a transaction must propagate to and commit on all master nodes before it can be considered committed. This has performance impacts even if all master nodes are on the same local network, and unreasonable performance impact if the master nodes are in different data centers or geographical regions.
The main solution to this is geographical sharding at the application level, usually mirroring the structure of a large business with geographic divisions. Moqui has some tool level support for this sort of thing using Entity Sync or other tools to feed data from geographic regions to a central server (or cluster) where reporting, etc can be done. There is no OOTB Entity Sync or other configuration for this sort of deployment, it's not something there has been demand for yet. This only makes sense for extremely large global corporations, not a market where Moqui has any use to my knowledge.
If you're looking at doing something like ecommerce and need the ecommerce sites distributed more widely the problem is easier than coordinating inventory or accounting across multiple global entities. For that just have separate ecommerce instances in different data centers feeding order/etc data to the Moqui ERP instance, very much like any typical external ecommerce application.
We are soon going to start something with GEODE regarding reference data. I would like to get some guide lines for the same.
As you know in financial reference data world there exists complex relationships between various reference data entities like Instrument, Account, Client etc. which might be available in database as 3NF.
If my queries are mostly read intensive which requires joins across
tables (2-5 tables), what's the best way to deal with the same with in
memory grid?
Case 1:
Separate regions for all tables in your database and then do a similar join using OQL as you do in database?
Even if you do so, you will have to design it with solid care that related entities are always co-located within same partition.
Modeling 1-to-many and many-many relationship using object graph?
Case 2:
If you know how your join queries look like, create a view model per join query having equi join characteristics.
Confusion:
(1) I have 1 join query requiring Employee,Department using emp.deptId = dept.deptId [OK fantastic 1 region with such view model exists]
(2) I have another join query requiring, Employee, Department, Salary, Address joins to address different requirement
So again I have to create a view model to address (2) which will contain similar Employee and Department data as (1). This may soon reach to memory threshold.
Changes in database can still be managed by event listeners, but what's the recommendations for that?
Thanks,
Dharam
I think your general question is pretty broad and there isn't just one recommended approach to cover all UCs (primarily all your analytical views/models of your data as required by your application(s)).
Such questions involve many factors, such as the size of individual data elements, the volume of data, the frequency of access or access patterns originating from the application or applications, the timely delivery of information, how accurate the data needs to be, the size of your cluster, the physical resources of each (virtual) machine, and so on. Thus, any given approach will undoubtedly require application tuning, tuning GemFire accordingly and JVM tuning regardless of your data model. Still, a carefully crafted data model can determine the extent of such tuning.
In GemFire specifically, such tuning will involve different configuration such as, but not limited to: data management policies, eviction (Overflow) and expiration (LRU, or perhaps custom) settings along with different eviction/expiration thresholds, maybe storing data in Off-Heap memory, employing different partition strategies (PartitionResolver), and so on and so forth.
For example, if your Address information is relatively static, unchanging (i.e. actual "reference" data) then you might consider storing Address data in a REPLICATE Region. Data that is written to frequently (typically "transactional" data) is better off in a PARTITION Region.
Of course, as you know, any PARTITION data (managed in separate Regions) you "join" in a query (using OQL) must be collocated. GemFire/Geode does not currently support distributed joins.
Additionally, certain nodes could host certain Regions, thus dividing your cluster into "transactional" vs. "analytical" nodes, where the analytical-based nodes are updated from CacheListeners on Regions in transactional nodes (be careful of this), or perhaps better yet, asynchronously using an AEQ with AsyncEventListeners. AEQs can be separately made highly available and durable as well. This transactional vs analytical approach is the basis for CQRS.
The size of your data is also impacted by the form in which it is stored, i.e. serialized vs. not serialized, and GemFire's proprietary serialization format (PDX) is quite optimal compared with Java Serialization. It all depends on how "portable" your data needs to be and whether you can keep your data in serialized form.
Also, you might consider how expensive it is to join the data on-the-fly. Meaning, if your are able to aggregate, transform and enrich data at runtime relatively cheaply (compute vs. memory/storage), then you might consider using GemFire's Function Execution service, bringing your logic to the data rather than the data to your logic (the fundamental basis of MapReduce).
You should know, and I am sure you are aware, GemFire is a Key-Value store, therefore mapping a complex object graph into separate Regions is not a trivial problem. Dividing objects up by references (especially many-to-many) and knowing exactly when to eagerly vs. lazily load them is an overloaded problem, especially in a distributed, replicated data store such as GemFire where consistency and availability tradeoffs exist.
There are different APIs and frameworks to simplify persistence and querying with GemFire. One of the more notable approaches is Spring Data GemFire's extension of Spring Data Commons Repository abstraction.
It also might be a matter of using the right data model for the job. If you have very complex data relationships, then perhaps creating analytical models using a graph database (such as Neo4j) would be a simpler option. Spring also provides great support for Neo4j, led by the Neo4j team.
No doubt any design choice you make will undoubtedly involve a hybrid approach. Often times the path is not clear since it really "depends" (i.e. depends on the application and data access patterns, load, all that).
But one thing is for certain, make sure you have a good cursory knowledge and understanding of the underlying data store and it' data management capabilities, particularly as it pertains to consistency and availability, beginning with this.
Note, there is also a GemFire slack channel as well as a Apache DEV mailing list you can use to reach out to the GemFire experts and community of (advanced) GemFire/Geode users if you have more specific problems as you proceed down this architectural design path.
My understanding was that RavenDb was designed so that if one shard goes down, the other shards can operate without problems.
But recently I was implementing ShardingResolutionStrategy and found out the MetadataShardIdFor method. It is the method where for each document type we can specify what shard to use for storage.
So if I get it right, if the shard where Hilo for specific document type is stored is down, we can not create new documents of this type at other shards (at least autogenerated ids will not work). Or may be I am wrong and Hilo is replicated between shards in some magical way?
Sharding is designed to be independent, but in order to create consistent ids, we need to be able to create them from a consistent store.
Because of that, we separate the notion of splitting data to multiple nodes and HA.
The typical scenario is that the metadata shard is independent, and is running with replicated database that is shared on all sharded nodes. In this fashion, if you lose the metadata shard, you just switch over.
This take advantage on the fact that RavenDB sharding & replication are orthogonal
I am asking this in the context of NoSQL - which achieves scalability and performance without being expensive.
So, if I needed to achieve massively parallel distributed computing across databases ...
What are the various methodologies available today (within the RDBMS paradigm) to achieve distributed computing with high-scalability?
Does database clustering & mirroring contribute in any way towards distributed computing?
I guess you are asking about scalability of RDBMS databases. Talking about NoSQL databases based on ( amazon dynamo, BigTable ) are a whole another topic. I am talking about HBase, Cassandra etc. There are also commerical products like Oracle Coherence thats more like a distributed cache and key value store , to put it crudely.
going back to rdbms,
Sharding
to scale RDBMS one can do cusstom sharding. Sharding is a technique where you have multiple table is possibly multiple hosts. And then you decide in a certain fashion to assign certain rows to certain tables. For example you can say that rows 1-1M goes to table1, 1M-2M goes to table2 etc. But, this is a difficult process from an administration point of view. A lot of large scale websites scale by relying on sharding. Other techniques worth mentioning are partioning and mysql federation and mysql cluster.
MPP databases
Then there are databases are there very RDBMS which does distribution and scaling for you. Terradata is the most successful of these companies. I believe they used postgres core code at some point. A significant number of fortune 500 companies and a lot of the airlines use Terradata. But, its ridiculously expensive. There are newer companies like greenplum, vertica, netezza.
Unless you're a very big company with extreme scalability requirements, you can horizontally and ACID scale up your DB by building a cluster of identical RDBMS instances and synchronizing them with JTA transactions.
Take a look to this Java/JDBC based article the JEPLayer framework is used but you can use straight JDBC and JTA code.
Within the RDBMS paradigm: Sharding.
Outside the RDBMS paradigm: Key-value stores.
My pick: (I come from an RDBMS background) Key-value stores of the tabluar type - HBase.
Within the RDBMS paradigm, sharding will not get you far.
Use the RDBMS paradigm to design your model, to get your project up and running.
Use tabular key-value stores to SCALE OUT.
Sharding:
A good way to think about sharding is to see it as user-account-oriented
DB design.
The all schema entities touched by a user-account are kept on one host.
The assignment of user to host happens when the user creates an account.
The least loaded host gets that user.
When that user signs on after account creation, he gets connected
to the host that has his data.
Each host has a set of user accounts.
The problem with this approach is that if the host gets hosed,
a fraction of users will be blacked out.
The solution to this is have a replicated standby host that
becomes the primary when the primary host encounters problems.
Also, it's a fairly rigid setup for processes where the design does
not change dramatically.
From the user standpoint, I've noticed that web sites
with a sharded DB backend are not as quick to "turn on a dime"
to create different business models on their platform.
Contrast this with web sites that have truly distributed
key-value stores. These businesses can host any range of
services. Their platform is just that - a platform.
It's not relational and it does have an API interface,
but it just seems to work.