Here is a use case:
I have version 1 of a web app deployed.
It uses couple Ignite-powered distributed (configured for replication) Maps, Sets and other data structures.
I'm going to deploy v2 of this application and once data is replicated I'm going to shutdown v1 of this app and re-route users (using nginx) to new instance (v2).
I can see that Ignite on v1 and v2 can discover each other and automatically perform replication of data structures.
My intention: I don't want to shutdown 1st instance (v1) before all data is replicated to 2nd instance (v2).
Question is: how do I know if initial replication is completed? Is there any event that is fired in such cases, or maybe some other way to accomplish this task?
If you configure you caches to use synchronous rebalancing [1], second node will not complete start process before rebalancing is completed. This way you will guarantee that all the data is replicated to the second node (of course, assuming that you're using fully replicated caches).
[1] https://apacheignite.readme.io/docs/rebalancing#section-rebalance-modes
Related
I would like to know what each strategy means and how they work behind the scenes (i.e., Highlander, Red/Black, Rolling Push). It would be very useful to have this information on the official website.
Thanks
There is useful information out there that can help you with your question, I'll do my best to summarize it below.
Type and Strategies of Deployments Introduction
"There are a variety of techniques to deploy new applications to
production, so choosing the right strategy is an important decision,
weighing the options in terms of the impact of change on the system,
and on the endusers."
Recreate: (also known as Highlander) Version A is terminated then version B is rolled out.
Ramped (also known as Rolling-Update or Incremental): Version B is slowly rolled out and replacing version A.
Blue/Green (also known as Red/Black): Version B is released alongside version A, then the traffic is switched to version B.
Canary: Version B is released to a subset of users, then proceed to a full rollout.
A/B Testing: Version B is released to a subset of users under specific condition.
Shadow: Version B receives real-world traffic alongside version A and doesn’t impact the response.
Type and Strategies of Deployments Summary Table
Ref link 1: https://thenewstack.io/deployment-strategies/
Spinnaker Deployment Strategies
Spinnaker treats cloud-native deployment strategies as first class constructs, handling the underlying orchestration such as verifying health checks, disabling old server groups and enabling new server groups.
Spinnaker supported deployment strategies (in active development):
Highlander
Red/Black (a.k.a. Blue/Green)
Rolling Red/Black
Canary
Illustrated in the Figure below as follows:
Highlander: This deployment strategy is aptly named after the film Highlander because of the famous line, "there can be only one." With this strategy, there is a load balancer fronting a single cluster. Highlander destroys the previous cluster after the deployment is completed. This is the simplest strategy, and it works well when rollback speed is unimportant or infrastructure costs need to be kept down.
Red/Black: This deployment strategy is also referred to as Blue/Green. The Red/Black strategy uses a load balancer and two target clusters / server groups (known as red/black or blue/green). The load balancer routes traffic to the active (enabled) cluster / server group. Then, a new deployment replaces servers (w/ K8s provider -> Replica Sets & Pods) in the disabled cluster / server group. When the newly enabled cluster / server group is ready, the load balancer routes traffic to this cluster and the previous cluster becomes disabled. The currently disabled cluster / server group (previously enabled cluster / server groups) is kept around by spinnaker in case a rollback is needed for the next X deployments (which is a configurable parameter).
Rolling Red/Black: is a slower red/black with more possible verification points. The process is the same as red/black, but difference is in how traffic switches over. The above image illustrates this difference. Blue is the enabled cluster. Blue instances are gradually replaced by new instances in the green cluster until all enabled instances are running the newest version. The rollout may occur in 20% increments, so it can be 80/20, 60/40, 40/60, 20/80, or 100%. Both blue/green clusters receive traffic until the rollout is complete.
Canary: deployments is a process in which a change is partially deployed, then tested against baseline metrics before continuing. This process reduces the risk that a change will cause problems when it has been completely rolled out by limiting your blast radius to a small percentage of your user-base. The baseline metrics are set when configuring the canary. Metrics may be error count or latency. Higher-than-baseline error counts or latency spikes kill the canary, and thus stop the pipeline.
Ref link 2: https://www.spinnaker.io/concepts/#deployment-strategies
Ref link 3: https://blog.armory.io/advanced-deployment-strategies-with-armory-spinnaker/
Ref link 4: https://www.weave.works/blog/kubernetes-deployment-strategies
As I understand it:
Highlander: when the new Auto Scaling group (ASG) is up and healthy, all old ASGs are destroyed automatically.
Red/Black: A new ASG is launched, some manual (or more complicated than in Highlander) verification steps are done, and only after those steps are completed is the old ASG manually deleted. Netflix blog post here: http://techblog.netflix.com/2013/08/deploying-netflix-api.html
Rolling push: "Old instances get gracefully deleted and replaced by new instances one or two at a time until all the instances in the ASG have been replaced." Netflix blog post here: http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html
At my company we only use Highlander and Red/Black on a regular basis.
When adding a new node to an Aerospike cluster, a rebalance happens for the new node. For large data sets this takes time and some requests to the new node fail until rebalance is complete. The only solution I could figure out is retry the request until it gets the data.
Is there a better way?
I don't think it is possible to keep the node out of cluster for requests until it's done replicating because it is also master for one of the partitions.
If you are performing batch-reads, there is an improvement in 3.6.0. While the cluster is in-flux, if the client directs the read transaction to Node_A, but the partition containing the record has been moved to Node_B, Node_A proxies the request to Node_B.
Is that what you are doing?
You should not be in a position where the client cannot connect to the cluster, or it cannot complete a transaction.
I know that SO frowns on this, but can you provide more detail about the failures? What kinds of transactions are you performing? What versions are you using?
I hope this helps,
-DM
Requests shouldn't be failing, the new node will proxy to the node that currently has the data.
Prior to Aerospike 3.6.0 batch read requests were the exception. I suspect this is your problem.
Looking through the Infinispan getting started guide it states [When in replication mode]
Infinispan only replicates data to nodes which are already in the
cluster. If a node is added to the cluster after an entry is added, it
won’t be replicated there.
Which I read as any cluster member will always be ignorant of any data that existed in the cluster before it became a cluster member.
Is there a way to force Infinispan to replicate all existing data to a new cluster member?
I see two options currently but I'm hoping I can just get Infinispan to do the work.
Use a distributed cache and live with the increase in access times inherent in the model, but this at least leaves Infinispan to handle its own state.
Create a Listener to listen for a new cache member joining and iterate through the existing data, pushing it into the new member. Unfortunately this would in effect cause every entry to replicate out to the existing cluster members again. I don't think this option will fly.
This information sounds as misleading/outdated. When the node joins a cluster, a rebalance process is initiated and when you query for these data during the rebalance prior to delivering these data to the node, the entry is fetched by remote RPC.
My understanding could be amiss here. As I understand it, Couchbase uses a smart client to automatically select which node to write to or read from in a cluster. What I DON'T understand is, when this data is written/read, is it also immediately written to all other nodes? If so, in the event of a node failure, how does Couchbase know to use a different node from the one that was 'marked as the master' for the current operation/key? Do you lose data in the event that one of your nodes fails?
This sentence from the Couchbase Server Manual gives me the impression that you do lose data (which would make Couchbase unsuitable for high availability requirements):
With fewer larger nodes, in case of a node failure the impact to the
application will be greater
Thank you in advance for your time :)
By default when data is written into couchbase client returns success just after that data is written to one node's memory. After that couchbase save it to disk and does replication.
If you want to ensure that data is persisted to disk in most client libs there is functions that allow you to do that. With help of those functions you can also enshure that data is replicated to another node. This function is called observe.
When one node goes down, it should be failovered. Couchbase server could do that automatically when Auto failover timeout is set in server settings. I.e. if you have 3 nodes cluster and stored data has 2 replicas and one node goes down, you'll not lose data. If the second node fails you'll also not lose all data - it will be available on last node.
If one node that was Master goes down and failover - other alive node becames Master. In your client you point to all servers in cluster, so if it unable to retreive data from one node, it tries to get it from another.
Also if you have 2 nodes in your disposal you can install 2 separate couchbase servers and configure XDCR (cross datacenter replication) and manually check servers availability with HA proxies or something else. In that way you'll get only one ip to connect (proxy's ip) which will automatically get data from alive server.
Hopefully Couchbase is a good system for HA systems.
Let me explain in few sentence how it works, suppose you have a 5 nodes cluster. The applications, using the Client API/SDK, is always aware of the topology of the cluster (and any change in the topology).
When you set/get a document in the cluster the Client API uses the same algorithm than the server, to chose on which node it should be written. So the client select using a CRC32 hash the node, write on this node. Then asynchronously the cluster will copy 1 or more replicas to the other nodes (depending of your configuration).
Couchbase has only 1 active copy of a document at the time. So it is easy to be consistent. So the applications get and set from this active document.
In case of failure, the server has some work to do, once the failure is discovered (automatically or by a monitoring system), a "fail over" occurs. This means that the replicas are promoted as active and it is know possible to work like before. Usually you do a rebalance of the node to balance the cluster properly.
The sentence you are commenting is simply to say that the less number of node you have, the bigger will be the impact in case of failure/rebalance, since you will have to route the same number of request to a smaller number of nodes. Hopefully you do not lose data ;)
You can find some very detailed information about this way of working on Couchbase CTO blog:
http://damienkatz.net/2013/05/dynamo_sure_works_hard.html
Note: I am working as developer evangelist at Couchbase
Consider the following scenario.
There are 2 Hazelcast nodes. One is stopped, another is running under quite heavy load.
Now, the second node comes up. The application starts up and its Hazelcast instance hooks up to the first. Hazelcast starts data repartitioning. For 2 nodes, it essentially means
that each entry in IMap gets copied to the new node and two nodes are assigned to be master/backup arbitrarily.
PROBLEM:
If the first node is brought down during this process, and the replication is not done completely, part of the IMap contents and ITopic subscriptions may be lost.
QUESTION:
How to ensure that the repartitioning process has finished, and it is safe to turn off the first node?
(The whole setup is made to enable software updates without downtime, while preserving current application state).
I tried using getPartitionService().addMigrationListener(...) but the listener does not seem to be hooked up to the complete migration process. Instead, I get tens to hundreds calls migrationStarted()/migrationCompleted() for each chunk of the replication.
1- When you gracefully shutdown first node, shutdown process should wait (block) until data is safely backed up.
hazelcastInstance.getLifecycleService().shutdown();
2- If you use Hazelcast Management Center, it shows ongoing migration/repartitioning operation count in home screen.