Here is my situation. Recently, my production environment has been burned by a few Windows updates that caused some production servers to stop responding. While we have since resolved the issue of both of the servers (which are in a load balancing configuration) getting updates on the same day, the question arouse, how do we check that the application running on each server is still working? If we call the load balancing IP, we may or may not hit a server that is working. So if the update takes out the application on one server, how do we know that this has happened
The only idea I have for this is to purchase 2 more SSL certificates and allocate 2 ip addresses and assign one to each server. This way I would be guaranteed that I would know each server is up (we have a 3rd party service pinging our servers). But I have to believe that there is a better way to do this?
Please note that I am a .Net developer by trade with only an extremely small smattering of networking and IIS experience, but I'm what my small company has. So please assume I don't know where a lot of stuff is and dumb down the answer.
Load balancer maintains live status of the servers ( based on timeouts or http health checks ). It uses this status to route the traffic only to active servers.
Generally, LBs have a dashboard through which you can check this status. If not, you can check it's logs.
Related
I got a mail from amazon s3 webservices stating below details
"We are writing to you today to let you know about changes which impact your use of the Amazon Simple Storage Service (S3). In efforts to best serve our customers, we have improved the systems powering the Amazon S3 API and are in the process of shutting down legacy application server capacity. We have detected access on the legacy capacity for Amazon S3 buckets that you own. The legacy capacity is no longer in service, as the DNS entry for the S3 endpoint no longer includes the IP addresses associated with it. We will be shutting down the legacy capacity and retiring the set of IP addresses fronting this capacity after April 1, 2020."
I want to find out which legacy system I am using, and how to prevent from affecting my services.
Imagine you had a web site, www.example.com.
In DNS, that name was pointed to your web server at 203.0.113.100.
You decide to buy a new web server, and you give it a new IP address, let's say 203.0.113.222.
You update the DNS for example.com to point to 203.0.113.222. Within seconds, traffic starts arriving at the new server. Over the coming minutes, more and more traffic arrives at the new server, and less and less arrives at the old server.
Yet, for some strange reason, a few of your site's prior visitors are still hitting that old server. You check the DNS and it's correct. Days go by, then weeks, and somehow a few visitors who used your old server before the cutover are still hitting it.
How is that possible?
That's the gist of the communication here from AWS. They see your traffic arriving on unexpected S3 server IP addresses, for no reason that they can explain.
You're trying to connect to the right endpoint -- that's not the issue -- the problem is that for some reason you have somehow "cached" (using the term in a very imprecise sense) an old DNS lookup and are accessing a bucket by hitting a wrong, old S3 IP address.
If you have a Java backend service accessing S3, those can notorious for holding on to DNS lookups forever. You might need to restart that service, and look into how to resolve that issue and enable correct behavior which is -- as I understand it -- not how Java behaves by default. (Not claiming to be a Java expert but I've encountered this sort of DNS behavior many times.)
If you have an HAProxy or Nginx server that's front-ending for an S3 bucket and has been up for a while, those might need a restart and you should look into how to correctly configure them not to resolve DNS only at startup. I ran into exactly this issue once, years ago, except my HAProxy was forwarding requests to Amazon CloudFront on only 1 of the several IP addresses it could have been using. They took that CloudFront edge server offline, or it failed, or whatever, and the DNS was updated... but my proxy was not able to re-query DNS so it just kept trying and failing until I restarted it. Then I fixed it so that it periodically repeated the DNS lookup so it always had a current address.
If you have your own DNS resolver servers, you might want to verify that they aren't somehow misbehaving, and you might want to ensure that you don't for some reason have any /etc/hosts (or equivalent) static host entried for anything related to S3.
There could be any number of causes but I'm confident at least in my interpretation of what they say is happening.
A Weblogic server got hacked and the problem is now removed.
I am looking through the infected VM's now in a sandbox and want to see what if any data was accessed on the application servers.
the app servers were getting hammered with ssh requests and so we identified the infected VM's as the web logic VMS, we did not have http logging on. Is there any way to Identify if any PII was Compromised?
Looked through secure logs on weblogic as well as looked through the PIA logs
I am not sure how to identify what if any data was accessed
I would like to find out what went out of our network and info or data
what should I be looking for
is there anything I can learn from looking at the weblogic servers running on red hat?
I would want to believe that SSH was not the only service being hammered, and that was a large attempt to make eyes be on Auth logging whilst an attempt on other services is made.
Do you have a Time frame that you are working with?
Have the OS logs been checked for that time frame?
.bash_history been checked? env variables? /etc/pass* for added users? aliases? reverse shells open on the network connections? New users created on services running on that particular host?
Was WebLogic the only service running on this publicly available host?
What other services and ports were available?
Was this due to an older version of Weblogic or another service, application, plugin?
Create yourself an excel spreadsheet and start a timeline.
Look at all the OS level logging possible and start to make note of anything that looks suspicious, to then follow that breadcrumb to exhaustion.
what's the best way to achieve high availability for a dynamic website? If I create a second copy on another server and do not wish to use a load balancer since it will mess up user sessions, what are the best alternatives?
You can store session data in a database instead, which gets around that problem, then you can round-robin the requests to the application servers.
(Good) Load Balancers can be configured to be "sticky" which means they send requests from the same IP to the same server each time.
Even if you have a load balancer sitting infront of two backend webservers, you just move the single point of failure onto the load balancer instead of the webserver. So your application would still not be highly available.
I highly recommend using a load balancer and at least a pair of web servers. At work, we use HA Proxy, which is fully capable of ensuring sessions are 'sticky', and are sent to the same web server unless it goes down, where it will fail over.
To make your load balancer highly available, you can set up two load balancing servers which are a mirror image of each other. Assign a single virtual IP to both of your load balancers. Write a script that will poll the other server to check if it's down; if it's down, have that script pick up that virtual IP address. The script should be running on both servers.
This link describes one way of managing a virtual IP address. Similar articles have been written for a large number of linux distros, but they are all based on the same method.
Loadbalancers. They should be configured in such a way that they can handle the sessions. Maybe by sending the same ip to the same backend every time. Or store them inside a database, or some shared memory if it needs to be really fast for some reason i haven't thought of.
Perhaps I have the wrong product in mind for our needs -- but I want to know if I can use Application Request Routing (ARR) in IIS7 to load balance requests for our application tier.
We have a farm of web servers. Each will be running our MVC web application. We load balance these servers through our web application firewall and load balancing appliances. In turn, they will be make WCF calls to our application servers. It's these calls that I want to use ARR to manage.
However, after looking at ARR, it seems like it's all about rewriting URLs coming from the client. But that's not how our situation works. If a user browses to www.myapp.com/home/index, we will in turn be making WCF calls to services configured in the web.config to say myappservice.foo.local/home/GetInfo.
How do I configure for this scenario, or am I looking at the wrong product?
I am not really sure to understand your scenario, but if i understand correctly, I think you would be able to call your WCF sevice. If you dont need to keep the session on your call, just uncheck the client affinity checkbox in server affinity configuration.
Configure your load balance to Round Robin, or Least response time in the load balance interface and your request will suppose to be load balance.
If you got more that one ARR server, i suggest you to disable Shared configuration on your ARR, we get some problem with this features on the ARR server.
I agree with #Cedric, not sure if the question has been phrased as well as it could be?
What type of load balancing/distribution are you looking to achieve?
Are you looking to balance load? Distribute load to specific server farms based on request content? Some other function? A little more info here might help get a better answer.
Will ARR work with WCF?
Yes, but only with the HTTP bindings as far as I know (wsHttpBinding and basicHttpBinding).
I still imagine there being a hardware loadbalancer (or some other method) in front of your ARR servers.
Your web servers are going to act as clients of the application servers servicing your WCF requests. However, it appears as if your already going to a DNS name of myappservice.foo.local/home/GetInfo - if it resolves to a virtual ip your already getting loadbalancing of some kind?
Why not use your existing "load balancing appliances" to do the load balancing?
I could definitely imagine a hardware loadbalancer servicing requests for mayappservice.foo.local, which then resolves to a virtual ip backed by your ARR servers. Then conceivably your ARR servers could then further refine who services the request, maybe by some content of the request? Maybe map all /home requests to one group of servers and all /foo to another?
Pretty sure I muddied the waters a bit more! :) But I'm very curious for reasons for looking at ARR.
From what I understand, if you have multiple web servers, then you need some kind of load balancer that will split the traffic amongst your web servers.
Does this mean that the load balancer is the main connecting point on the network? ie. the load balancer has the IP address of the domain name?
If this is the case, it makes it really easy to add new hardware since you don't have to wait for any dns propogation right?
There are several solutions to this "problem".
You could round-robin at the DNS-level. I.e. have www.yourdomain.com point to several IP-addresses (well all your servers).
This doesn't give you any intelligence in the load balancing, but the load will be more or less randomly distributed, but you wouldn't be resilient to hardware failures as they would still require changes to DNS.
On the other hand you could use a proxy or a loadbalancing proxy that has a single IP but then distributes the traffic to several back-end boxes. This gives you a single point of failure (the proxy, you could of course have several proxies to defeat that problem) and would also give you the added bonus of being able to use some metric to divide the load more evenly and intelligently than with just round-robin dns.
This setup can also handle hardware failure in the back-end pretty seamlessly. The end user never sees the back-end, just the front-end.
There are other issues to think about as well, if your page uses sessions or other smart logic, you can run into synchronisation problems when your user (potentially) hits different servers on every access.
It does (in general). It depends on what server OS and software you are using, but in general, you'll hit the load balancer for each request, and the load balancer will then farm out the work according to the scheme you have in place (round robin, least busy, session controlled, application controlled, etc...)
andy has part of the answer, but for true load balancing and high availability you would want to use a pair of hardware load balancers like F5 bigips in an active passive configuration.
Yes your domain IP would be hosted on these devices and traffic would connect firstly to those devices. Bigips offer a lot of added functionality including multiple ways of load balancing and some great url rewriting, ssl acceleration, etc. It also allows you to run your web servers on a seperate non routable address scheme and even run multiple sites on different ports with the F5's handling the translations.
Once you introduce load balancing you may have some other considerations to take into account for your application(s) like sticky sessions and session state but that is a different subject