Azure Datafactory tutorial(1) states the following:
In CI/CD scenarios, the integration runtime (IR) type in different environments must be the same. For example, if you have a self-hosted IR in the development environment, the same IR must also be of type self-hosted in other environments, such as test and production. Similarly, if you're sharing integration runtimes across multiple stages, you have to configure the integration runtimes as linked self-hosted in all environments, such as development, test, and production.
If I use dev/test/prod environments like described in the tutorial with self-hosted integration runtime(SHIR) do I need to create an extra Azure Datafactory which is serving SHIR for CI/CD managed environments as a linked service?
(1) https://learn.microsoft.com/en-us/learn/modules/operationalize-azure-data-factory-pipelines/4-continuous-integration-deployment
Yes, a separate data factory is needed whose role is to host the SHIR and share it to other data factories. This is from a Microsoft support ticket I had with the same question.
In my case I had two ADFs provisioned, prod and non-prod, and wanted to share a multi-node SHIR with both. I was unable share the one created on the non-prod ADF with the prod ADF like I was expecting. MS said I needed a third to host the SHIR and share it to prod & non-prod ADFs.
The same tutorial has the following extra best practices
Integration runtimes and sharing. Integration runtimes don't change often and are similar across all stages in your CI/CD. So Data Factory expects you to have the same name and type of integration runtime across all stages of CI/CD. If you want to share integration runtimes across all stages, consider using a ternary factory just to contain the shared integration runtimes. You can use this shared factory in all of your environments as a linked integration runtime type.
Other integration runtime related notes
Default Integration runtime
It should be also noted that on adf_publish only some of the Microsoft.DataFactory/factories/integrationRuntimes resources are visible.
E.g. AutoResolveIntegrationRuntime when deployed without VNet support is not visible but Self-Hosted Integration runtimes are.
Integration Runtime with Vnet support
Current portal driven deployment template is using vNetEnabled to control if
Microsoft.DataFactory/factories/managedVirtualNetworks
Microsoft.DataFactory/factories/integrationRuntimes with name AutoResolveIntegrationRuntime and managedVirtualNetwork config is created
VNet enriched integrationRuntimes is present in the adf_publish/ARM export
Related
I want to get the deployment method using Powershell for Self-Hosted Integration Runtime.
The requirement is to track whether the SHIR is installed using "Powershell" commands Or its done through the Azure Portal by a human user by manual steps.
Is there any Flag in Azure which indicates the method of the resource deployment? i.e. if its a PowerShell/ARM template/Portal deployment?
Self-Hosted Integration runtime is a service installed on a windows operating machine (the only OS supported now) either on cloud(VM) or on-prem. Closest you can get to its details is under Applications and Services Logs on system Event Viewer, here you can Filter Logs for timestamp between Start and End of your ADF Pipeline.
Additionally, if you feel this is not helpful, you can share your Feedback so the product team can look into this idea. ✌
We are creating a server-side agent which periodically fetches data from nodes and maps this data to cumulocity measurements, events.
What is an elegant approach for hosting and/or packaging such a server-side agent?
We are hosting our own instance of the Cumulocity platform.
It's preferable to keep this server-side agent as 'close' to the core platform as possible, e.g. share some core agent framework dependencies.
We'd like to limit the amount of setting up additional environments or containers (e.g. Tomcat).
Cumulocity uses Karaf, would it make any sense to deploy the server-side agent into Karaf as a bundle?
Is there any recommended approach for hosting server-side agents? Does the cumulocity platform offer an alternative to deploying the agent to some "own environment"?
The Cumulocity examples repository contains the "tracker-agent" server-side agent example, which is an embedded tomcat Java application. There is little information about the intended deployment location.
I don't recommend deploying agents/microservices directly into the core Karaf server, since that endangers the resources available to the core APIs and is not supported. (I.e., will likely be overwritten with the next upgrade...)
Typically, people just provision an additional VM or docker next to Cumulocity to place their agents/microservices in. On top of that, we, for example, often use Spring Boot, so the effort is pretty low (java -jar ...).
We do have a hosting system for agents/microservices and will make that generally available also for others to use in Q1/2018. Follow the announcement channel at https://support.cumulocity.com to stay posted...
Is it possible to have multiple versions of service(s) deployed in production at the same time. From my assumption, this should be pretty common pattern for microservice/api based projects or mobile projects. I want to know how do you do it and what are common pattern in industry for this kind of problems. It would be helpful if your answers around AWS environment or Kubernetes environment.
Thanks in Advance.
Is it possible to have multiple versions of service(s) deployed in production at the same time
Yes, it is possible. The idea is to keep all used microservices in production (v1, v2 ...) at the same time and to bring down the versions that are not used anymore. For this, you should somehow know when a version is not used anymore.
AFAIK, you have to options:
For every new version you make a new endpoint (like /v2/someApiCall) that is connected to the same (now upgraded) microservice and gradually instruct clients to use the new endpoind; when the old endpoint is not used anymore you deleted it; this is the preferred way.
For every new version you make a new microservice that share the same persistence with the old microservice; you should avoid the use of this solution; Netflix uses this strategy in rare occasions when the cost of changing old consumers is too high.
You can read more at page 62 from Building microservices by Sam Newman.
With AWS API Gateway you could deploy multiple versions of your code and switch between them from the mapping templates, as explained here. You might also want to look into stage variables.
Assuming your are exposing services over An HTTP REST API, the general standard is to always base line your service urls with a version.
Eg,
/v1/account/getUserInfo
If you need to release a new version, expose it over:
/v2/account/getUserInfo
Where v2 can run over a different branch of the codebase.
I have blogged about this: Multi-version Service Discovery using Spring Cloud Netflix Eureka and Ribbon, focussed on Spring Cloud Netflix components / libraries though.
But the idea is to deploy a new version of the artifact / binary in a new host / VPS / Container and have the service register with a registry server (Eureka, Consul, ....) and include metadata about the API versions it supports (v1, v2, ...). Client apps would discover which host / container / ... serves the API version needed.
As part of our pipelines, we currently use a deployment tool that has connectivity to our various instances and we can upload revisions/versions of our app to a central repository, archive them, and redeploy them at any time. Is Spinnaker intended to replace an existing deployment automation tool (there are many on market today) or is more meant for us to create pipelines that call the API of our other tool(s) when actually deploying our code to different servers?
Spinnaker has native support for deployment to supported cloud platforms (AWS, Google, CloudFoundry, and soon Azure).
In those environments, the Spinnaker model is an immutable infrastructure style deployment where new VMs are created to push new software versions.
If that fits your needs, then Spinnaker could replace an existing deployment automation tool.
If that doesn't fit your model, then Spinnaker also supports calling out to an external execution environment as a pipeline stage (currently Jenkins is well supported) where you could implement custom behaviors to integrate to an existing deployment tool.
I have a question about whether cloud vendors have an inter-operable mechanism. For example, I am developing a WCF service and hosting in Azure successfully. After a pro-long time using Azure, can I use the same code for deploying it in AWS? Will it be possible? Does the API of both matches the same for deploying? If not, what are all the extra care needed for hosting the same service when switching over other Cloud Vendors like Salesforce.com, OpenStack, etc.,
In general, you can't just take what you develop for one Cloud platform and put it on another: they have different functionality sets and expose different APIs. However, the more low-level you make your code, the more likely it is that you'll find another vendor with a very similar API, since virtualizing infrastructure is simpler (and closer to standardized) than virtualizing a CMS application.
If you're using just IaaS, you can probably port fairly rapidly but you have to do more work to make your application. If you're using PaaS (or SaaS!) then you're more locked-in but you get more support for developing rapidly: it's that support platform which is both the value-add and the lock-in, and you won't get one without the other.
If you're using an Azure web role for hosting your WCF service then from deployment point of view you will not have many problems with AWS. You'll simply use facilities offered by AWS SDK for .NET (aka Publish to AWS CloudFormation). For sure you'll have to change the logging part if you've used Azure Diagnostic and alla Azure services with related AWS services. We did this multiple times in the last year and it works.
For worker role it's not so simple because in Azure they are easily deployed like web role, but in AWS you haven't direct deployment from Visual Studio so you have to do some manual work using Windows Services or something else