Application information:
Spring Cloud Data Flow Server Cloudfoundry 1.0.0.RELEASE (DIY built with Spring Cloud Config Server dependencies)
Spring Cloud Config Server
PCF Elastic Runtime 1.7.x
I'm curious about the extent to which applications and the deployer depend on the Git repo and Maven artifact repository I'm binding my SCDF instance and my Spring Cloud Config Server instance to in PCF.
My suspicion is that the Maven repo is only used at deployment time, when an artifact needs to be downloaded for installation and deployment in the PCF space. Also, I'm thinking the Git repo is probably cloned by the Config Server whenever an application initialization, or refresh event occurs that would require the need to re-read the configuration information stored in Git.
Is this true, or are there ongoing dependencies that would require high availability for these external resources? My question is related to disaster recover planning activities, and how quickly these specific resources need to be recovered for Spring Cloud Data Flow and its deployed streams to continue working under adverse conditions.
My suspicion is that the Maven repo is only used at deployment time, when an artifact needs to be downloaded for installation and deployment in the PCF space.
Yes - The applications are resolved and downloaded upon stream deployment request and the resolved apps are cached and reused upon redeployments.
I'm thinking the Git repo is probably cloned by the Config Server whenever an application initialization
True - For a given URI of a configuration source, the server will clone the repository and make its configurations available to all the client applications bound to it.
These two capabilities are driven by application bootstrap event. As for the config-server, if you're running it as a service in Cloud Foundry, it's up to the platform to reliably serve the properties to the bound applications.
Related
I'm looking for the advice of how to manually (i.e. without using Runtime Manager - RM) deploy a mule application package on the on-premises Mule cluster. The official documentation suggests using the RM for the purpose either via the gui or cli or api. However, the RM is not available on our environment.
I can manually deploy the package on a single node by copying it to the /apps folder. But this way the application is only deployed on a single node, not on the cluster.
I've tried using the AMC agent rest API for the purpose with the same result - it only deploys on a single node.
So, what's the correct way of manually deploying a mule application on the Mule servers cluster without using Anypoint RM?
We are on Mule 4.4 EE.
Copy the application jar file into the apps directory of every node. Mule clusters do not transfer applications between nodes.
Alternatively ou can use the Runtime Manager Agent instead however it also works in a per node basis. You need to send the same request to each node to deploy.
Each connector may or may not be cluster aware. Read each connector documentation to understand how they behave. In particular the documentation of the VM connector states:
When running in cluster mode, persistent queues are instead backed by the memory grid. This means that when a Mule flow uses VM Connector to publish content to a queue, Mule runtime engine (Mule) decides whether to process that message in the same origin node or to send it out to the cluster to be picked up and processed by another node.
You can register the multiple nodes through AMC agent on the cloudhub control plane and create a server group and deploy code through control plain runtime manager it does the job of deployment to same app in n nodes
I have 2 apache nifi servers that are development and production hosted on AWS, currently the migration between development and production is done manually. I would like to know if it is possible to automate this process and ensure that people do not develop in production?
I thought about uploading the entire nifi in github and having it deploy the new nifi on the production server, but I don't know if that would be correct to do.
One option is to use NiFi registry, store the flows in the registry and share the registry between Development and Production environments. You can then promote the latest version of the flow from dev to prod.
As you say, another option is to potentially use Git to share the flow.xml.gz between environments and using a deploy script. The flow.xml.gz stores the data flow configuration/canvas. You can use parameterized flows (https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters) to point NiFi at different external dev/prod services (eg. NiFi dev processor uses a dev database URL, NiFi prod points to prod database URL).
One more option is to export all or part of the NiFi flow as a template, and upload the template to your production NiFi, however registry is probably a better way of handling this. More info on templates here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates.
I believe the original design plan behind NiFi was not necessarily to have different environments, and to allow live changes in production. I guess you would build your initial data flow using some test data in production and then once it's ready start the live data flow. But I think it's reasonable to want to have separate environments.
Does google cloud or aws provide manage Apache tomcat which just take war file and do auto-scaling based on load increase and decrease ? not compute engine. I dont want to create VM. this should be manage by manage service.
Google App Engine can directly take and run a WAR file - just use the appcfg deployment method.
You will have more options if you package with docker, as this then provides an image type that can be run in many places (Multilpe GCP, AWS and Azure options, on-prem Kubernetes, etc). This can even be as simple as building a dockerfile that just copies the WAR into a jetty image:
FROM jetty:latest
COPY YOUR_WAR.war /var/lib/jetty/webapps
It might be better to explode the war though - see discussion in this question
AWS provide ** AWS Elastic Beanstalk **
The AWS Elastic Beanstalk Tomcat platform is a set of environment configurations for Java web applications that can run in a Tomcat web container. Each configuration corresponds to a major version of Tomcat, like Java 8 with Tomcat 8.
Platform-specific configuration options are available in the AWS Management Console for modifying the configuration of a running environment. To avoid losing your environment's configuration when you terminate it, you can use saved configurations to save your settings and later apply them to another environment.
To save settings in your source code, you can include configuration files. Settings in configuration files are applied every time you create an environment or deploy your application. You can also use configuration files to install packages, run scripts, and perform other instance customization operations during deployments.
It also provide autoscaling
The Auto Scaling group in your Elastic Beanstalk environment uses two Amazon CloudWatch alarms to trigger scaling operations. The default triggers scale when the average outbound network traffic from each instance is higher than 6 MB or lower than 2 MB over a period of five minutes. To use Amazon EC2 Auto Scaling effectively, configure triggers that are appropriate for your application, instance type, and service requirements. You can scale based on several statistics including latency, disk I/O, CPU utilization, and request count.
I am trying to write scripts(using java) to deploy my mule application on top of the cluster. So that, application get deployed on the Mule ESB servers under cluster.
Already I have written a code to deploy my mule application on Mule ESB server using MMC Rest API(http://www.mulesoft.org/documentation/display/current/MMC+REST+API)
Now my next target is to deploy application on MMC cluster.
Can any one please suggest me a way to deploy mule application on cluster from java code(using API).
Thanks in Advance.
The MMC REST API allows to deploy to a cluster the same way as you deploy to a standalone server:
http://www.mulesoft.org/documentation/display/current/Deployments
Instead of Java code ... why don't you try Maven ... Maven Script directly create application zip and deploy to mmc cluster ... All you need to write the script in .pom file instead of java class
There is a maven plug in that you can use to deploy via MMC:
https://github.com/NicholasAStuart/Maven-Mule-REST-Plugin
mule-mmc-rest-plugin:deploy
This will:
delete an existing mule application archive from the MMC Repository if version contains "SNAPSHOT"
upload the mule application archive to the MMC Repository
delete an existing deployment having the same application name
create a new deployment this the uploaded archive, with target the
given serverGroup
perform a deploy request to make MMC deploy into target server group
I used it and it works (but you may need to make it some customizations)
When running jboss 7.1 as a windows service (or not), it occasionally takes more than one try to successfully deploy a war file. This is not a problem when starting jboss manually since restarts are easy. However, when jboss runs as a windows service and it is restarted automatically (due to a windows patch), jboss itself may launch, but the war may not.
Is there any way to cause jboss to retry deploying the war after it fails the first time - for example, by changing a setting in standalone.xml?
There are to ways to fix your problem.
1) go to standalone.xml (or whatever configuration you are running), find deployment-scanner and add/modify attribute deployment-timeout in seconds
2) Deploy your application as managed deployment, you can do that if you deploy trough admin console or via cli with deploy command. This way deployment will then be "managed" and will always be deployed and wont be using deployment scanner and its timeouts.
I recommend you to use deploy as managed deployment as deployment scanner is not really recommend to be used in production environments as it adds additional IO load on filesystem.
It is great for development / testing scenarios but should be avoided in production if possible.