I have a database topology as in the picture attached below and postgresql9.6 is installed in all the machines. It is master-slave architecture with sharding. I have successfully configured master-slave replication and automatic failover using repmgr/repmgrd. But i am confused on how to achieve sharding in this scenario. I have tried citus extension, but for the table structure i have citus does not support sharding, here is the link. Can anyone suggest me how can i achieve sharding in this scenario.
Database topology:
Related
I've read https://github.com/redisson/redisson
And I found out that there are several
Redis Replicated setup (including support of AWS ElastiCache and Azure Redis Cache)
Redis Cluster setup (including support of AWS ElastiCache Cluster and Azure Redis Cache)
Redis Sentinel setup
Redis with Master with Slave only
I am not a big expert in clusters and I don't understand the difference between these setups.
Could you beiefly explain the differences ?
Disclaimer I am an AWS employee.
I do not know how Redis Replicated Setup is different from Redis in Master-Slave mode. Maybe they mean cross-region replication?
In any case, I can try and explain setups I know about:
Redis with Master with Slave only - is a single shard setup where you create a primary replica together with one or more secondary (slave) replicas (let's hope PC police won't arrest me). This setup is used to improve the durability of your in-memory store. It's not advised to use your secondaries for reads because such setup has eventual consistency guarantees and your replica reads may be stale (depending on the replication lag).
Redis Cluster setup - the setup supported by cloud provides such as AWS Elasticache. In this setup your workload can be spread horizontally across multiple shards and each shard may have its own secondary replicas. Your client library must support this setup since it requires maintaining multiple connections to several nodes at a client level. Moreover, there are some locality rules you need to follow in order to use cluster mode efficiently:
Keys with foo{<shard>}bar notation will be routed to their shard according to what is stored inside curly brackets.
You can not use mset, mget and other multi-key commands across shards. You can still use these commands if their keys contain the same {shard} part.
There are additional cluster mode admin commands that are exposed by Redis but they are usually hijacked and hidden from users by cloud providers since cloud provides use them in order to manage redis cluster themselves.
Redis cluster have an ability to migrate part of your workload between shards. However, it still obliged to preserve correctness with respect to {shard} notation. Since your client library is responsible to fetch data from specific shard it must handle "moved" response when a shard might redirect it to another node.
Redis Sentinel setup - using an additional server that provides service discovery functionality for Redis clusters. Not strictly required and I believe is less popular across users. It serves as a single source of truth regarding each node's health and state. It provides monitoring, management, and service discovery functions for managing your Redis cluster. Many Redis client libraries provide the option of connecting to Redis sentinel nodes in order to achieve automatic service discovery and seamless failover flow. One of the reasons why this setup is less popular is because cloud companies like AWS Elasticache provide this service out of the box.
I'm using Hortonworks schema registry with NIFI and things are working fine. I have installed Hortonworks schema registry on a single node and I'm afraid if that machine goes down what will happen to my NIFI flows. I have seen in Hortonworks schema registry architecture that we can use mysql, PostgreSql and In-Memory storage for storing schema. AFAIK none of them are distributed system. Is there any way to achieve cluster mode for high availability?
Sure, you can do active-active or active-passive replication for MySQL and Postgres, but that is left up to you to implement, as Hortonworks will likely forward you to the respective documentation on each tool, and that is the reason why the documentation for these tools doesn't guide you towards these design decisions in itself, as you should be aware of the drawbacks when having a SPoF
The Schema Registry itself is just a web-app, so you could put it behing your favorite reverse proxy, or within a container orchestrator, such as Docker support in HDP 3.x
We have checked both Redis installed in Azure VM and Azure Redis Cache both are working same I can't see a difference in the performance Have anyone used both in large scale application if so can anyone share the performance and durability of both ?
Have analysed the following
Monitoring
In-zone replication
Multi-zone replication
Auto fail-over
Data persistence
Backup
Pricing
SSL Authentication & Encryption
All the above Azure redis have the upper hand
Still I want make sure which one is the best
Does using VM has any bottlenecks ?
I would go for Azure Redis Cache. Mainly because its fully managed. At the end of the day you do have nodes under the hood. But why should you care for maintaining a VM? Hotfixes? Patches, Seucirty Updates ..etc ..etc.
I would ask the question the other way around. Why should you use VMs at all?
MG
Newbie to redis here so please bear with me.
I am looking for a method to have dual datacenter in active-active configuration. My need here is that:
if a datacenter goes down the other datacenter should not need any intervention to carry on working
Networking issues, if they occur, should not prevent one or the other datacenter to fail on set/get redis calls.
Restarting failed datacenter should have a means mirror the working redis' data back before going active
I have been reading up on replication abilities of redis, what I understand is there is
master-slave(s) replication
A cluster can have masters that are sharded
But what I haven't seen is a cluster replicating to another cluster.
Question:
Is there a architecture design where all redis in one dc replicate with the other? (Looks like couchbase appears to have this)
I do see a keyspace notifier which can be used in a pub/sub, what i want to know is whether I can use it to pub/sub redis to redis from one dc to the other to act as replication.
Cassandra offers controlled consistency like "write to 2 nodes and tell me it's done".
Two "master" nodes and some slaves makes system good failover.
MongoDB offers replication pairs - simmilar failover force like cassandra?
Is there any other database with this form-box functionality?
Cassandra is a fully distributed system, so there is no need for explicit failover. If the machine you are sending requests to dies, you just reconnect to another (RRDNS, haproxy, any method is fine). Even losing an entire datacenter is handled by Cassandra without your app having to care.