Referring to https://github.com/newrelic/newrelic_api for the New Relic API, I was wondering what was the difference between hosts and instances.
Basically, I get what an application is and what a server is (obviously). I would assume instances are instances of the application, i.e. if my app were running on Heroku, each instance would correspond to a dyno running my app. But then what is a host? And what's the difference between host and instance?
Thanks,
-Billy
UPDATE
Thanks for the answer!
So if I got this right, in the general case, the mapping between applications and instances is 1-to-n, i.e. each app can have 1 or more instances. Also, the mapping between instances and hosts is n-to-m, i.e. each instance can be running on at most one host (at any given time), but instances are distributed among available hosts. Similarly, hosts are distributed among servers (say, m-to-s). Is that it? (Apologies if this sound like I'm saying very obvious stuff, but I'm unfamiliar with the terminology they are using over at New Relic)
If the above is correct, how can I get the instances - hosts and hosts - servers mappings from the API? I can see how to get the applications - instances and applications - hosts, but what about the other two?
Thanks again for your help!
A host (server) can run many instances of an application. Each process that responds to requests (e.g., a Unicorn worker) is an instance from the New Relic perspective. The host/instance distinction is roughly equivalent to the difference between an IP address and a port.
If you're using Heroku, New Relic treats the entire dyno grid as a single host/server, and each dyno as an instance.
Re: the updated question
A host is a machine or VM that applications run on, and each one can run N instances of the application.
A "server", for the purposes of the NR API, is an OS+hardware that's monitored by New Relic Server Monitoring. The NR application monitoring agent can also be running on a server monitored by the Server Monitoring agent. In that case, both the host and the server should report the same name to New Relic ("server01.example.com").
There isn't a way to get the instance-host or host-server mappings explicitly from the New Relic API. But in the case of server-host, the mapping is that they share the same name. You can probably infer the instance-host mapping from the instance names, too, since they will almost always contain the host name (and possibly also the port number).
Related
I am trying to setup a distributed system based on current spring-cloud release (meaning mostly Netflix OSS) using the following components
1 or more cloud config servers
1 or more Eureka servers
1 or more services using Eureka and Config Server clients
The setup above is easy enough to get going however once you start looking into setting up so that configuration changes in the cloud Config servers automatically trigger changes in the values of the actual clients, things start getting more complicated.
It is my understanding that for such a feature to work one should introduce spring-cloud-bus clients to the services which in turn will use, currently the only supported implementation, rabbitmq servers (the actual rabbitmq binaries and not some spring-boot app like eureka or Config servers) to allow change events in the Config server to be propagated to the clients automatically.
It sounds counterintuitive to setup such a system and have to hardcode addresses to rabbitmq servers in the clients (even if one will be keeping the amount of rabbitmq servers more or less static).
How is one supposed to register rabbitmq server instances in the Eureka service discovery server(s) to allow for clients to find them without having to have any knowledge about their location prior to startup?
I cannot seem to find any documentation on how this is done given that rabbitmq is not a spring-cloud component. In fact very little documentation seems to exist regarding on how the rabbitmq + eureka + spring-cloud-bus should be setup together.
I know that I am on a VERY old question, even though I think it worth a comment for people who read this in the future.
Most of the cloud services, lets take AWS as an example, have an Elastic IP solution - so you can configure IPs for RabbitMQ servers, and the IPs always belong to the RabbitMQ, no matter whether the instances change. You can re-attach the Elastic IP to different instances.
It works nearly the same with Elastic Load Balancer, which keeps its IP, so you could configure your microservices to a specific IP using Spring Cloud Config Server - and scale the RabbitMQ instances without a need to worry about configuration change.
The Rails application I'm currently working on is hosted at Amazon EC2 servers. It's using Resque for running background jobs, and there are 2 such instances (would-be production and a stage). Also I've mounted Resque monitoring web app to the /resque route (on stage only).
Here is my question:
Why there are workers from multiple hosts registered within my stage system and how can I avoid this?
Some additional details:
I see workers from apparently 3 different machines, but only 2 of them I managed to identify - the stage(obviously) and the production. The third has another address format(starts with domU) and haven't any clue what it could be.
It looks like you're sharing a single Redis server across multiple resque server environments.
The best way to do this safely is to use separate Redis servers or separate Redis databases or namespaces. The Redis-namespace gem can be used with Resque to isolate each environments Resque queues and worker data.
I can't really help you with what the unknown one is, but I had something similar happen when moving hosts and having dns names change. The only way I found to clear out the old ones was to stop all workers on the machine, fire up IRB, require 'resque' and look at Resque.workers. This will list all the workers resque knows about, which in your case will include about 20 bogus ones. You can then do:
Resque.workers.each do {|worker| worker.unregister_worker}
This should prune all the not-really-there workers and get you back to a proper display of the real workers.
I am looking to build an Akka-based application in the cloud, for a garage startup that I'm bootstrapping; by the nature of the app, it's semi-stateful, with as much as possible cached in RAM for performance. (It'll be tolerant of being shut down and restarted periodically, but we want to mostly operate via cached information inside the Actors.)
The architecture is designed for a cluster of servers, communicating between them as necessary so that a user session on node A can query a middleware Actor on node B when appropriate. So my question is, how hard is that in CloudBees?
My understand from this page is that there is no automatic directory service to manage this sort of intra-cluster communication yet, but I can probably live with that -- worse comes to worst, I should be able to manage discovery via the DB, with each node registering itself when it comes up and opening up many-to-many communications with the others.
What I want to check, though, is that this communication is straightforward. Does each node have a reliable local IP that it can advertise for others to contact it on, that is at least stable during this run of the application? Or is there another/better way for a node to advertise its address to the rest of the nodes running this app?
(I assume that the nodes of an app all share the same DB instance.)
Any guidance here would be greatly appreciated. I'd like to choose a hosting provider soon, and keep returning to CloudBees as the most promising-looking of the options...
There are no limitations currently on instances communicating with each other - the trick is in discovering membership. There is an api that will be shortly be released that will allow you to track membership - but for now, the following may work:
To get the port, look at the file names in $PWD/.genapp/ports (as applications can have multiple ports) - (eg System.getenv("PWD") + ".genapp/ports" - list the files in that directory - generally will be just 1 - the file name is the port). There are other ways - for example the "sun.java.command" system property on JVM apps too.
The hostname can be obtained via the usual means (eg InetAddress.getLocalHost().getHostName()): this host
name will be the private name - ie it will resolve to a private IP -
good for node to node communication.
Public IP/hostname: perform a HTTP get (from the server) to the following URL:
http://instance-data/latest/meta-data/public-hostname (will only
return the public IP on the server side of course).
(see http://developer-blog.cloudbees.com/2012/11/finding-port-or-address-of-your.html)
You can then, as you say, on startup, register the appropriate port/private hostname with a DB, and then read that on each node to "seed" the cluster (akka doesn't have to know about all members - just enough seeds) I would think a 2 phase startup: 1: register host/port, 2, look for other members, add them as seed members to the local Akka configuration (may need to periodically do the same for a while, as other nodes startup - to ensure it is seeded enough)
From my reading of Akka setup here: http://doc.akka.io/docs/akka/snapshot/scala/remoting.html
It looks like you can specify the port - so if possible, I would set that to be the app_port environment variable - that means each node can communicate via the private hostname with that port. However, http traffic will also be routed to it - can akka handle this as well - or does it need to have a discrete port for akka and another for any http interface?
I have a Procfile like so:
web: bundle exec rails server -p $PORT
em: script/eventmachine
The em process fires up an eventmachine with start_server (port ENV['PORT']) and my web process occasionally needs to communicate with it.
My question is how does the web process know what port to communicate with it on? If I understand heroku correctly it assigns you a random port when the process starts up (and it can change if the ps is killed or restarted). Thanks!
According to Heroku documentation,
Two processes running on the same dyno can communicate over TCP/IP using whatever ports they want.
Two processes running on different dynos cannot communicate over TCP/IP at all. They need to use memcached, or the database, or one of the Heroku plugins, to communicate.
Processes are isolated and cannot communicate directly with each other.
http://www.12factor.net/processes
There are, however, a few other ways. One is to use a backing service such as Redis, or Postgres to act as an intermediary - another is to use FIFO to communicate.
http://en.wikipedia.org/wiki/FIFO
It is a good thing that your processes are isolated and share-nothing, but you do need to architecture your application slightly differently to accommodate this.
I'm reading this while on my commute to work. So I haven't tried anything with it (sorry) but this looks relevant and potentially awesome.
https://blog.heroku.com/archives/2013/5/2/new_dyno_networking_model
Can someone explain to me how high-availability ("HA") works for a web application ... because I assume HA means that there exist no single-point-of-failure.
However, even if a load balancer is used- isn't that the single point of failure?
I have found this article on the subject:
http://www.tenereillo.com/GSLBPageOfShame.htm
Basically if you do not require long lasting sticky sessions you can configure your DNS servers to return multiple A records (IP addresses) for your website.
Web browsers are smart enough to try all the addresses until they find one that works.
In simple words high availability can be defined as running a system 24*7 without a downtime even if there are hardware and software failures. In other way a fault tolerance application. This helps ensure uninterrupted use of the application for it’s intended users.
Read more on High Availability Deployment Architecture
It works the following way that you setup two HA Proxy servers with heartbeat, so when one fails (stops responding to queries), it's being removed from the cluster.
Requests from HA Proxy can be forwarded to web servers in round robin fashion, and if one web server fails, HA Proxy servers do not try to contact it until it's alive.
Web servers are storing all dynamic information in database, which is replicated across two MySQL instances.
As you can see, HA Proxy and Cluster MySQL (or simply MySQL replication) as well IP Clustering here is the key.
Sure it is when operated alone. Usual highly available setup includes 2 or more load balancers running in cluster in either active/active or active/passive configuration. To further increase the availability you can have 2 different Internet Service Providers (or geo distributed datacenters) each running a pair of clustered load balancers. Then you configure DNS A record resolving to 2 distinct public IP addresses which guarantees round-robin processing splitting DNS requests evenly (CloudFlare is very fast and reliable at this). There's also possibility to return IP address of datacenter closest to your originating geo location by using something like PowerDNS dnsdist
This is what big players do to make their services highly available.
Please read https://docs.oracle.com/cd/E23824_01/html/821-1453/gkkky.html for more clearity. Actually both load balancer uses same vip(Virtual IP Address. https://techterms.com/definition/vip).
HA architecture is a entire field and multiple books were written on it, so it is hard to answer in a short paragraph.
To sum up the ideal situation, you would be using multiple servers, interconnected to a layer of multiple load balancers. The nodes and LB will be located in a few different data centers, and connected to different network backbone. Ideally the data centers will be located all over the world.
In short, all component will have redundancy, including the load balancers.
For a starting point, see Wikipedia for High Availability Cluster