Is there a way to do A/B or canary deployments with Lagom Persistent Entities?
I'm trying to plan out how we can build and deploy reactive microservices and I like the look of Lagom, but I can't see how we'd roll out new versions as canary or a/b testing releases. It seems like the stateful nature of the persistent entities would make that impossible?
Assuming it's possible, I'd also like to figure out how to use Istio on Kubernetes to deploy and manage these new releases, so any pointers on that would be great.
In my view, the Lagom framework and its associated tooling obviates many benefits of service meshes like Istio. Orthogonal to meshes is deployment; that said, there is documentation for deploying Lagom services using Kubernetes using Lightbend Orchestration.
The docs show an explicit example of Kubernetes canary deployments.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 months ago.
Improve this question
Can somebody explain the difference in tooling between Crossplane and Cluster API from the perspective of a managed kubernetes platform provision ?
https://github.com/kubernetes-sigs/cluster-api
I was using crossplane for sometime to create k8s clusters and recently got to know that Cluster API is also having the same capability to provision k8s clusters.
Cluster API
Cluster API is one of the projects in CNCF that you can use the create and manage the Kubernetes clusters. it's more like Kubeadm, kind, k3s, minikube(not for prod usecase) with YAML config & CLI.
You can create the YAML file and provision the Kubernetes cluster and manage it with that. There are different providers(AWS, AKS, GCP) available so you can use a provider-set service account or Access key secret and once you apply YAML to cluster API it will create a K8s cluster based on config and provider.
So with this, you can manage multiple Kubernetes clusters.
With cluster API YAML config you can create/manage K8s cluster on AWS, GCP, On-prem etc.
Crossplane
Consider crossplane as a deployment or service you are running on Kubernetes first.
You pass YAML config to cross-plane service based on rules it will create/manage the resources outside on cloud providers. It will also create/Manage the RDS, CloudSQL instances, Kubernetes clusters and other resources that the cloud provider supports.
It has also concept of provider(AWS, GCP, AKS)
Crossplane uses Kubernetes as the underlying platform to expose these features and workflows to manage cloud infrastructure outside of Kubernetes, such as data centers, the cloud or the edge. It installs a set of components called providers by extending Kubernetes to manage resources from cloud providers
Cluster API is a Kubernetes project to bring declarative, Kubernetes-style APIs to cluster creation, configuration, and management.
Kubernetes container orchestration technology has swept the enterprise world, with 75% of organizations running containers in production soon (most orchestrated by Kubernetes). Kubernetes delivers a cloud-native platform for application delivery, runtime, and scaling. While these are huge benefits for scale out applications, providing all these capabilities comes at the expense of the complexity of the platform itself. These complexities can be categorized as
Complex and heavyweight lifecycle management: significant time and resources needed to provision, upgrade, and maintain the control planes of clusters. Each cloud has a different set of IaaS primitives that need to be cobbled together to make a “cluster”.
Intricate and involved integrations for basic components such as storage, networking, security, and others.
Blockquote
Difficult and not always consistent multi-cluster management and operations.
Cluster API
Provisioning Kubernetes clusters is never an easy task. When there are 1000+ clusters, definitely want to have a standardized approach to ease tasks. Some of might know tools like kOps, Kubespray. Imagine Cluster API as their alternative solution, but more powerful!
According to the official page, "Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters."
Here are some highlighted points of Cluster API:
Pure YAML-based. Kubernetes style. Super handy.
Support any mainstream infrastructure provider. Provision your Kubernetes clusters in cloud/on-premise environments in the same place.
Managed Kubernetes services support. AWS EKS, Azure AKS, GCP GKE all are supported.
Bring your own infrastructure. Reuse existing infrastructures. Focus on provisioning Kubernetes clusters.
Crossplane
Crossplane uses Kubernetes as the underlying platform to expose these features and workflows to manage cloud infrastructure outside of Kubernetes, such as data centers, the cloud, or the edge.
It installs a set of components called providers by extending Kubernetes to manage resources from cloud providers. Providers build on the Kubernetes API, along with the custom resources and controllers, and expose declarative APIs to manage cloud provider resources.
Crossplane is geared toward platform teams and operators looking to build abstractions and easier-to-consume infrastructure components based on an organization's needs. It has a few concepts, such as CompositeResourceDefinition and Composition, that enable teams to extend and build on top of the default resources exposed for a cloud provider.
First of all lets understand what is Crossplane and what is Cluster API
Cluster API
Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.
Provide declarative APIs
Give the tools to simplify provisioning, upgrading and operating multiple clusters
Helps to manage the lifecycle (create, scale, upgrade, destroy) of Kubernetes-conformant cluster (uses a declarative API)
Give you the ability to work on different environments (on-premisses and cloud)
Can find this and more about kubernets cluster api at https://cluster-api.sigs.k8s.io/
Crossplane
Crossplane is a framework for building cloud native control planes without needing to write code. It has a highly extensible backend that enables you to build a control plane that can orchestrate applications and infrastructure no matter where they run, and a highly configurable frontend that puts you in control of the schema of the declarative API it offers.
I've found a few interesting posts about Crossplane vs Kubernetes API's
https://blog.crossplane.io/crossplane-vs-cloud-infrastructure-addons/
https://www.infracloud.io/blogs/cluster-provisioning-using-crossplane/
Its not much but hope this helps
I am new to canary deployments. We are going to start doing canary deployments via Istio.
I was assuming this would just be a deployment mechanism, probably with some Istio routing testing in a pre-prod env but in earlier test envs we'd ring fence to a version being tested as we do today.
It's been suggested the canary concept is applied to all test environments so we effectively run all versions we expect to canary test in prod in the Route To Live.
Wondring what approach others are taking?
Mirroring
As mentioned here
Using Istio, you can use traffic mirroring to duplicate traffic to another service. You can incorporate a traffic mirroring rule as part of a canary deployment pipeline, allowing you to analyze a service's behavior before sending live traffic to it.
If you're looking for best practices I would recommend to start with this tutorial on medium, because it is explained very well here.
How Traffic Mirroring Works
Traffic mirroring works using the steps below:
You deploy a new version of the application and switch on traffic
mirroring.
The old version responds to requests like before but also sends an asynchronous copy to the new version.
The new version processes the traffic but does not respond to the user.
The operations team monitor the new version and report any issues to the development team.
As the application processes live traffic, it helps the team uncover issues that they would typically not find in a pre-production environment. You can use monitoring tools, such as Prometheus and Grafana, for recording and monitoring your test results.
Additionally there is an example with nginx that perfectly shows how it should work.
Canary deployment
As mentioned here
One of the benefits of the Istio project is that it provides the control needed to deploy canary services. The idea behind canary deployment (or rollout) is to introduce a new version of a service by first testing it using a small percentage of user traffic, and then if all goes well, increase, possibly gradually in increments, the percentage while simultaneously phasing out the old version. If anything goes wrong along the way, we abort and rollback to the previous version. In its simplest form, the traffic sent to the canary version is a randomly selected percentage of requests, but in more sophisticated schemes it can be based on the region, user, or other properties of the request.
Depending on your level of expertise in this area, you may wonder why Istio’s support for canary deployment is even needed, given that platforms like Kubernetes already provide a way to do version rollout and canary deployment. Problem solved, right? Well, not exactly. Although doing a rollout this way works in simple cases, it’s very limited, especially in large scale cloud environments receiving lots of (and especially varying amounts of) traffic, where autoscaling is needed.
There are the differences between k8s canary deployment and istio canary deployment.
k8s
As an example, let’s say we have a deployed service, helloworld version v1, for which we would like to test (or simply rollout) a new version, v2. Using Kubernetes, you can rollout a new version of the helloworld service by simply updating the image in the service’s corresponding Deployment and letting the rollout happen automatically. If we take particular care to ensure that there are enough v1 replicas running when we start and pause the rollout after only one or two v2 replicas have been started, we can keep the canary’s effect on the system very small. We can then observe the effect before deciding to proceed or, if necessary, rollback. Best of all, we can even attach a horizontal pod autoscaler to the Deployment and it will keep the replica ratios consistent if, during the rollout process, it also needs to scale replicas up or down to handle traffic load.
Although fine for what it does, this approach is only useful when we have a properly tested version that we want to deploy, i.e., more of a blue/green, a.k.a. red/black, kind of upgrade than a “dip your feet in the water” kind of canary deployment. In fact, for the latter (for example, testing a canary version that may not even be ready or intended for wider exposure), the canary deployment in Kubernetes would be done using two Deployments with common pod labels. In this case, we can’t use autoscaling anymore because it’s now being done by two independent autoscalers, one for each Deployment, so the replica ratios (percentages) may vary from the desired ratio, depending purely on load.
Whether we use one deployment or two, canary management using deployment features of container orchestration platforms like Docker, Mesos/Marathon, or Kubernetes has a fundamental problem: the use of instance scaling to manage the traffic; traffic version distribution and replica deployment are not independent in these systems. All replica pods, regardless of version, are treated the same in the kube-proxy round-robin pool, so the only way to manage the amount of traffic that a particular version receives is by controlling the replica ratio. Maintaining canary traffic at small percentages requires many replicas (e.g., 1% would require a minimum of 100 replicas). Even if we ignore this problem, the deployment approach is still very limited in that it only supports the simple (random percentage) canary approach. If, instead, we wanted to limit the visibility of the canary to requests based on some specific criteria, we still need another solution.
istio
With Istio, traffic routing and replica deployment are two completely independent functions. The number of pods implementing services are free to scale up and down based on traffic load, completely orthogonal to the control of version traffic routing. This makes managing a canary version in the presence of autoscaling a much simpler problem. Autoscalers may, in fact, respond to load variations resulting from traffic routing changes, but they are nevertheless functioning independently and no differently than when loads change for other reasons.
Istio’s routing rules also provide other important advantages; you can easily control fine-grained traffic percentages (e.g., route 1% of traffic without requiring 100 pods) and you can control traffic using other criteria (e.g., route traffic for specific users to the canary version). To illustrate, let’s look at deploying the helloworld service and see how simple the problem becomes.
There is an example.
There are additional resources you may want to check about traffic mirroring in istio:
https://istio.io/latest/docs/tasks/traffic-management/mirroring/
https://itnext.io/use-istio-traffic-mirroring-for-quicker-debugging-a341d95d63f8
https://dev.to/peterj/mirroring-traffic-with-istio-service-mesh-2cm4
https://livebook.manning.com/book/istio-in-action/chapter-5/v-7/130
https://istio.io/latest/docs/tasks/traffic-management/traffic-shifting/#apply-weight-based-routing
I have some questions according to https://docs.konghq.com/2.0.x/clustering
I’ll really appreciated if someone help me.
1)according to Clustering Reference I need a load balancer , could you please introduce me a free one which I can use in front of my Kong nodes?
2)I still don’t know is it better to implement kong nodes in different VMs or in Docker using docker-compose file for a fully production environment ?
Best Regards,
I think both your questions are highly dependent of your tech stack / architecture.
Regarding the load balancing question, I can think of several options for different options:
DNS Load Balancing, which depends on client side load balancing
Services in an Kubernetes/OpenShift environment, which provide load balancing across a bunch of pods
AWS Load Balancers, if you deploy Kong directly on EC2 machines. (I am sure other cloud providers have simar concepts)
Whether you deploy Kong on a VM or as a Docker Container is quite hard to answer. It depends on your tech stack you already have in place and on your requirements (see https://docs.konghq.com/2.0.x/sizing-guidelines/). However, I would not recommend to use docker-compose for this use case. If you decide for a Docker-based solution you should take a look at container management solutions such as Kubernetes or OpenShift. There you have solved the management of your Kong containers (such as how many replicas are running and what happens if one replica is failing) and you have solved the load balancing issue (by using Kubernetes/OpenShift services objects).
I am in a situation where I can use Service Fabric (locally) but cannot leverage Azure Service Bus (or anything "cloud"). What would be the corollary for queuing/pub-sub? Service Fabric is allowed since it is able to run in a local container, and is "free". Other 3rd party messaging infrastructure, like RabbitMQ, are also off the table (at the moment).
I've built systems using a locally grown bus, built on MSMQ and WCF, but I don't see how to accomplish the same thing in SF. I suspect I can have SF services use a custom ICommunicationListener that exposes msmq, but that would only be available inside the cluster (the way I understand it). I can build an HTTPBridge (in SF) in front of those to make them available outside the cluster, but then I'd lose the lifetime decoupling (client being able to call a service, using queues, even if that service isn't online at the time) since the bridge itself wouldn't benefit from any of the aspects of queuing.
I have a few possibilities but all suffer from some malady that only exists because of SF, locally. Also, the same code needs to easily deploy to full Azure SF (where I can use ASB and this issue disappears) so I don't want to build two separate systems just because of where I am hosting it in some instances.
Thanks for any tips.
You can build this yourself, for example like this. This uses a BrokerService that will distribute message-data to subscribed services and actors.
You can also run a containerized queuing platform like RabbitMQ with volumes.
By running the queue system inside the cluster you won't introduce an external dependency.
The problem is not SF, The main issue with your design is that you are coupling architectural requirements to implementations. SF runs on top of VirtualMachines, in the end, the only difference is that SF put the services in those machines, using another solution you would have an Agent Deploying these services in there or doing a Manual deployment. The challenges are the same.
It is clear from the description that the requirement in your design is a need for a message queue, the concept of queues are the same does not matter if it is Service Bus, RabbitMQ or MSMQ. Each of then will have the basic foundations of queues with specifics of each implementation, some might add transactions, some might implement multiple patterns, and so on.
If you design based on specific implementation, you will couple your solution to the implementation and make your solution hard to maintain and face challenges like you described.
Solutions like NServiceBus and Masstransit reduce a lot of these coupling from your code, and if you think these are not enough, you can create your own abstraction. Then you use configurations to tied your business logic to implementations.
Despite the above advice, I would not recommend you using different
solutions per environment, because as said previously, each solution
has it's own implementations and they might not assimilate to each other, as example, you might face issues in
production because you developed against MSMQ on DEV and TEST
environments, and when deployed to Production you use ServiceBus, they
have different limitations, like message size, retention period and son
on.
If you are willing to use MSMQ, you can add MSMQ to the VMs running your cluster and connect from your services without any issue. Take a look into this SO first: How can I use MSMQ in Azure Service Fabric
A newbie question : If I am not using Sagas or pub/sub model, do I need any persistence layer with NServiceBus?
The default persistence is RavenDB, I was wondering if I just use NSB without RavenDB or any persistence layer?
Side note:
Currently I have to run RavenDBServer.exe before I can even play around with NserviceBus, so if I do start using NServiceBus, I don't want to have a dependency on another app running in production. This could be a issue with my install.
NServiceBus only uses persistence for storing subscriptions for pub/sub and for saga state. If you aren't using those then you don't need RavenDB or any other implementation. I believe the NServiceBus.Core binary by default comes bundled with RavenDB DLLs, but it won't be needed in production in your case.
If I understand correctly, you could also use the NServiceBus.Lite profile which I believe uses in memory persistence. This is a great way to go as you are developing and trying to learn NServiceBus. However, I'm not sure you'd want to do so in production. It would be a better practice to have some persistence either in a database or file system to ensure reliability of your application in the case of errors or application restarts.