DNS Round-Robin on SSL - ssl

We're adding a second web server for redundancy and load sharing purposes. All connections are mandated to be SSL, and adding a dedicated appliance is not possible at this moment.
I'd like to use round robin DNS, where both servers answer to the same domain using different IPs (we have a wildcard SSL certificate, so that's OK). I can get the DNS to return in random/round robin order no problem.
Is this a bad setup when using SSL?
Our user pattern is consistent -- users will consistently be utilizing the web app for 8-10 hours. We want each page view to be as fast as possible, and my concern is users could constantly flip between the servers, potentially negating any SSL handshake caching/keep alive.
Thanks!

Firstly, SSL has the ability to resume an earlier session, so flipping between servers will cost you a few hundred ms per request (longer if several clients are accessing the site simultaneously, since this is CPU time we're talking about).
Whether the clients will actually flip depends, though - DNS "load balancing" is fiddly business:
if many of your users are using the same recursive nameservers, they'll get the same "first IP" hence no load balancing
if the DNS record has a high TTL (several hours), caching nameservers will store a particular permutation of IP addresses until they expire (good so long as your users aren't all using the same recursive nameservers)
if your users have multiple recursive nameservers configured, they may flip if each nameserver has a different "first IP" (bad)
if you have no mechanism for removing "bad" records, and a low TTL, then if one server goes down 50% of your clients will get the "bad" server and have to wait for a timeout before they can see your site
As you can see there are various tradeoffs depending on whether you're more concerned about redundancy/failover or load balancing; DNS isn't really the best tool here - you really need the servers to share an IP using either a reverse proxy, or something like Heartbeat (assuming you're Linux-based).
An aside: if both servers are answering to the same domain then you don't need a wildcard cert, although CAs often charge more if you intend to use a cert on more than one server.

TLDR: You will be fine. The SSL renegotiations shouldn't happen frequently enough to be noticeable by your end user.
Rant starts here:
Load distribution using DNS is a commonly misunderstood topic that leads into a lot of anecdotal evidence and straw-man arguments. I've been in too many of these meetings.
Here's how I usually settle these arguments:
"Wow yeah that sounds really exoteric [long dramatic pause] but it really can't be that bad since google uses it"
$host encrypted.google.com
encrypted.google.com is an alias for www3.l.google.com.
www3.l.google.com has address 74.125.224.195
www3.l.google.com has address 74.125.224.202
www3.l.google.com has address 74.125.224.193
www3.l.google.com has address 74.125.224.197
www3.l.google.com has address 74.125.224.207
www3.l.google.com has address 74.125.224.206
www3.l.google.com has address 74.125.224.203
www3.l.google.com has address 74.125.224.204
www3.l.google.com has address 74.125.224.196
www3.l.google.com has address 74.125.224.199
www3.l.google.com has address 74.125.224.201
www3.l.google.com has address 74.125.224.194
www3.l.google.com has address 74.125.224.192
www3.l.google.com has address 74.125.224.200
www3.l.google.com has address 74.125.224.205
www3.l.google.com has address 74.125.224.198
Updates:
Is this setup redundant?
It is not inherently redundant in the engineering sense since if one of those ips were to fail it would continue to be served to the customer until a DNS zone change is performed and all downstream caches expire. With that said, most browsers are smart enough to try another ip under these circumstance - reference.
Moreover, a system could easily be devise that instead of requiring a DNS zone change to remove the failed node would, instead, route the ip of the failed instance to a servicing device by simple ip takeover.
Is this setup resilient?
Yes, resilience is achieved by minimizing your failure domain. Back to our example failure of a single ip (and remember these ips may represent load balancers backed by hundred of servers or even an entire data center) the likelihood of a customer hitting that ip is 1/16, or ~6%, (using the google example above). This is inherently more resilient than a system with a single A address, wich would impact 100% of the users, or a system with 2 A records in which the user has an even 50/50 change of hitting a failed resource.

Don't worry about it. There are multiple levels of DNS caches so user is not going to flip between 2 IPs on every request. The IP will stay the same for hours for each client.
We have an opposite problem. When server goes down, the user still has the bad IP. We set the TTL to 1 minute but very few browsers honor it. Due to this issue, VIP is a much better option than DNS for load-balancing on the same network.

DNS round robin does not provide redundancy.
Without substantial additional help it only provides dumb load sharing (nb: not load "balancing", which implies dynamic load distribution based on server load).
Having the same cert on two IPs should be no problem, though.

Related

Is it better for a realtime multiplayer game server to send game statistics outbound via a socket or wait for an inbound socket connection?

I would like to report the number of players in each of my game server nodes to a central authority that will be used to load balance connections to those nodes. This is a real time multiplayer web game. This is a game design question for a game that could go in any load balance direction and at this point any solution is possible. At a high level if you were designing this would it be better to open an outbound tcp connection to the authority and report game stats in the game loop OR would it be better to wait for the authority to ask the game server for stats over an incoming tcp connection? Which solution would be better if you had 1000 nodes?
If this were outbound from game server:
If the game server reports that this is my ip address, my port and my number of players the authority could catalog that data and update the number of players if the same ip/port combo exists. In order to prevent security issues with rogue agents reporting themselves as game servers you would then need to try and secure the connection with both encryption and some type of authorization mechanism.
If this were inbound to the game server
If the authority already knows the address of each node it needs to connect too then you may only need to worry about man in the middle servers pretending to be the authority. Given the data is only game stats though and nothing sensitive, encrypting it shouldn't matter in the grand scheme for a connection inbound to a game server. I could have for example a config file or database table with all the known game servers to connect too. A mitm would get game stats or maybe just block the connection. To me this seems like the better option to just connect out to your game server unencrypted and get the stats. The downside is anyone else could connect to your game server as well to ask for game stats. Then in your game server you'd have to figure out either which connection is the real authority or just broadcast the game stats to anyone asking for game stats. The game server from a security standpoint though could get overloaded with to many connections wanting to get game stats and you don't have a good way to know which authority is the one that matters for your load balancing vs adhoc agents. I FEEL LIKE THIS IS THE WAY TO GO.
I agree that inbound to the game servers are the way to go, for the reasons you stated.
Here are a few other considerations.
I think practically that it is a lot harder for the servers to connect to users' hosts because of ip changes and firewalls at the user-local level.
You should always encrypt your customer's over the wire. And speaking of MITM, you should have some way of preventing one user from spoofing another user. No-one wants their game play taken over by a hacker.
You could have, as part of your client-server protocol, a way for the server to dictate a minimum wait time before the client sends their next update.
You could have the client updates put into a queue (rabbit, kafka, etc.) and have the servers update at their own capabilities. Queues are a lot more forgiving of elastic traffic, if you can handle them.

TURN servers: always or never needed for a given network, or needed unpredictably?

I am currently just using a STUN server and am wondering whether TURN is necessary for an MVP. The number of users accessing the website from a workplace with super secure firewalls should be near-zero.
Let's say 4 people are testing WebRTC connection reliability. Sometimes they all successfully connect and can see/hear one another, but other times they cannot see/hear someone and refresh the page to try again.
Does the fact that they can sometimes all see/hear each other rule out whether a TURN server would make a difference?
In other words, is it possible for a STUN server to identify my IP so I can connect one second, but fail if I try again a few seconds later? Or is it just network-based, so if I a STUN doesn't work for me on my network now, it 100% will not work in 5 minutes either?
If the latter (and a TURN is either always or never needed for a given network), I guess that tells me the problem in my code is elsewhere...

UDP Broadcast, Multicast, or Unicast for a "Toy Application"

I'm looking to write a toy application for my own personal use (and possibly to share with friends) for peer-to-peer shared status on a local network. For instance, let's say I wanted to implement it for the name of the current building you're in (let's pretend the network topology is weird, and multiple buildings occupy the same LAN). The idea is if you run the application, you can set what building you're in, and you can see the buildings of every other user running the application on the local network.
The question is, what's the best transport/network layer technology to use to implement this?
My initial inclination was to use UDP Multicast, but the more research I do about it, the more I'm scared off by it: while the technology is great and seems easy to use, if the application is not tailored for a particular site deployment, it also seems most likely to get you a visit from an angry network admin.
I'm wondering, therefore, since this is a relatively low bandwidth application — probably max one update every 4–5 minutes or so from each client, with likely no more than 25–50 clients — whether it might be "cheaper" in many ways to use another strategy:
Multicast: find a way to pick a well-known multicast address from 239.255/16 and have interested applications join the group when they start up.
Broadcast: send out a single UDP Broadcast message every time someone's status changes (and one "refresh" broadcast when the app launches, after which every client replies directly to the requesting user with their current status).
Unicast: send a UDP Broadcast at application start to announce interest, and when a client's status changes, it sends a UDP packet directly to every client who has announced. This results in the highest traffic, but might be less likely to annoy other systems with needless broadcast packets. It also introduces potential complications when apps crash (in terms of generating unnecessary traffic).
Multicast is most certainly the best technology for the job, but I'm wondering if the associated hassles are worth avoiding since this is just a "toy application," not a business-critical service intended for professional network admin deployment and configuration.

Monitoring network usage excluding local traffic

I am working on an app that monitors network usage. However I noticed many ways to do this does not allow exclusion of local traffic (say, Time Machine).
I am looking for a way to exclude local traffic, and only monitors usage that goes directly to/from the internet.
Update: Thank you for your replies, now I know how to find if the traffic is local, but I still don't know how I can calculate total in/out bytes (sorry if I didn't elaborate earlier). I have no way of knowing how many bytes are sent/received locally (or to the internet) in a certain period of time, or since the OS starts. This problem is further complicated by the fact processes are launched or killed when the OS is running.
The answer to the question How to get network adapter stats in linux/Mac OSX? gives an interesting way of summing up total usage but it doesn't help because the usage it sums up are interface statistics.
Update 2: I've posted my final solution to this. Please scroll down a bit to see.
you need to read the source for ifconfig(8), which describes how to get the status of every attached network interface.
pay particular attention to in_status(), which gets the inet address and netmask of an interface.
when the source or destination address in the traffic has the same host as a local interface
int is_local =
(src && netmask) == (ifaddr && netmask)
|| (dst && netmask) == (ifaddr && netmask)
then you can be sure that it is local
http://www.opensource.apple.com/source/network_cmds/network_cmds-307/ifconfig.tproj/ifconfig.c
Answering you comment about which interfaces carry local traffic is actually complicated, because it depends on what you mean by local traffic.
What “Local” Means
The easiest meaning of "local traffic" is traffic that does not leave the machine its generated on (two programs on the same machine talking to each other, for example). This traffic all goes over lo. This is one thing that people mean when they say local (and what I was thinking of when I answered).
The next easiest meaning would be "IP traffic destined to machines on the same subnet". That'd be traffic that has a destination address inside the local subnet. The easiest way to count this is going to be either the routing table (if Mac OS X counts traffic stats per route, the routes on the various gateways will give you non-local traffic) or with a firewall rule. This probably isn't want anyone means when they say "local traffic".
Another meaning would be "IP traffic destined to machines in this (physical) location". E.g., at my office we have several subnets in use, with routers between them, but traffic from one subnet to the other is still clearly local. You need network knowledge to distinguish local from non-local traffic with this definition.
Another meaning would be "IP traffic destined to machines in my organization". This is a reasonable meaning depending on how your network is set up (e.g., maybe you have fast fiber between your locations, but your Internet connections are much slower, or charged per-GB). Requires in-depth knowledge of the network to figure if a destination is going to be local or not—and, with things like VPNs, that may vary over time.
Finally, "Internet traffic" isn't the opposite of any of those. Sometimes, for example, what appears to be a local machine on your Ethernet segment is actually over a VPN, over the Internet (this isn't crazy, it's very useful for when remote users need to use various Windows services). Traffic inside your organization can easily travel over an Internet VPN.
Cheating in Simple Networks
If the network is very simple, with there being only one internal subnet, only one router, and all traffic not to that internal subnet being Internet traffic, you can cheat and solve this. This probably applies to the vast majority of home networks, and many small business ones as well.
Using firewall rules
In a simple network setup, you can probably make some assumptions, and get a close enough answer by counting traffic as non-local if:
the destination MAC address is the default gateway's MAC address; and
the destination IP address is not the default gateway's IP address
alternatively:
the destination IP address is not within the subnet of the network interface the default route goes out
You can probably create a firewall rule to count either of those. At least with Linux iptables you can, and I'm pretty sure BSD pf, and probably Mac OS X.
Alternate Approach: SNMP
Finally, if you can't use a firewall rule (as that'd require root), you could hope that the default gateway responds to SNMP community public, explore all its interfaces, and find the one with a off-subnet IP address, and then assume that is the Internet link. Then you can ask the router for traffic counts on that interface.
Of course, you'll find that many SOHO routers don't support SNMP, and those that do probably don't have it turned on.
The best way is to find the 'external' ip address through the eth0, eth1, or whatever adapter with a system call to ifconfig. Then pull logs for whatever system (messages, syslog, whatever) and write a filter for that external ip address. To make it nicer and more portable, write a regex that will filter for publicly routable IPs only and just filter messages log for that 'external' ip address.
I think, an approximate solution: getifaddrs can be used to get statistics on network usage.
It can get separate statistics for Wi-Fi and WWAN interfaces.
You might find more information from :
http://www.gsp.com/cgi-bin/man.cgi?section=3&topic=getifaddrs
It depends on how you define "local", but a common definition would be to look at the network mask.
For example, if your IP (ie the IP of the interface you monitor is
10.33.52.123
netmask 255.255.255.0
that would mean every IP-packet with both source-IP and destination-IP 10.33.52.xx is local.
I don't know cocoa or objective-c, but you can probably use some of these functions helping you extract the network from an IP-address: http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man3/inet_network.3.html
Don't know how to implement it in objective-c but the idea is that you get the address of the network you are in (you can figure this out from network class(A,B,C) based from your local ip or from bits in netmask if it's not standard), then just check the outgoing connection's address. If the destination is not in your local network, calculate traffic; if it's in, just do nothing.
There are three ranges of non-routable IP addresses, and they are commonly used as the address ranges for NAT services. Any address that is not in one of the non-routable address ranges is an external address.
Of course if you are not behind a NAT router, the task is harder (and technically all the addresses short of 127.0.0.1 are external at this point).
The non-routable IP ranges are:
10.0.0.0 - 10.255.255.255
172.16.0.0 - 172.31.255.255
192.168.0.0 - 192.168.255.255
The final working solution I have is to use libpcap to achieve this. Of course there are some downsides, which includes it requires elevated privileges and must capture all filtered packets to calculate statistics, but at least it works perfectly well.
Many documentations and tutorials on libpcap is fairly thorough and clear, I suggest every one interested in this solution to look at those with relatively little google-fu effort.
Also it may interest a few that my filter for internet traffic is simply the following -
- (NSString *)_internetFilterStringForInterface:(AKNetworkInterface *)interface
inOrOut:(BOOL)inYesOutNo
{
if (![interface net] || ![interface mask] || IsEmpty([interface addresses]))
{
return nil;
}
NSString *hostType = inYesOutNo ? #"dst" : #"src";
NSString *host = nil;
for (NSString *hostComponent in [interface addresses])
{
if (IsEmpty(hostComponent)) continue;
if (!host)
host = [NSString stringWithFormat:#"(%# host %#", hostType, hostComponent];
else
host = [host stringByAppendingFormat:#" or %# host %#", hostType, hostComponent];
}
host = [host stringByAppendingString:#")"];
NSString *net = [interface netString];
net = [net stringByReplacingOccurrencesOfString:#".0" withString:#""];
NSString *filter = [NSString stringWithFormat:
#"ip and (not %# net %#) and %#",
inYesOutNo ? #"src" : #"dst",
net, host];
return filter;
}
The filter is designed with some of the answers about what counts as 'local traffic', I know it does not encompass some edge cases such as double NAT configurations, etc., but I would like to see suggestions about this.
I know net = [net stringByReplacingOccurrencesOfString:#".0" withString:#""]; is just a quick hack which could easily fail under some peculiar circumstances but hey no one is complaining, at least not yet.

How do I get the external ip address with NSHost?

I'm trying to find my external IP address, but I get local ones only, behind the NAT.
NSArray *addresses = [[NSHost currentHost] addresses];
Is there a way to print out the public address? Using NSHost is it a good idea?
There may not always be a reliable way to get at your public IP, but the DNSService API in OS X will use UPnP and/or the NAT port mapping protocol to get the public IP (amongst other things). The code illustrating how to use the C API would be a bit large (50-60 lines) to show here, but there's some Apple sample code which implements a nice ObjC wrapper around the functionality, and even offers a pair of functions to directly return UInt32 and NSString representations of the public IP address.
The relevant code itself is located here, but you're probably best off just downloading the zip file and including PortMapper.h and PortMapper.m in your project and using them directly. Then all you'd need to do is:
NSString * publicAddressString = [PortMapper findPublicAddress];
If you ever get to look at a network topology chart for a major organization, do so. It's enlightening. The whole point of NAT, firewalls and all that other "black magic" is to allow the network to manage addresses (including protecting you) without your knowledge.
There are only three ways in which I've been able to reliably (and reliable is a relative concept here) get the external-facing IP address of a server.
The first is to as the network gods themselves (and make sure you refer to them as gods when asking, this will assist you in getting the information). Sometimes (not always), it's a simple mapping of the top bytes of your IP address whilst retaining the low-order bytes. Sometimes it's more complex, but still follows rules that you can use. Just keep in mind these rules can change at any time.
The second is to have a box outside of your network which you can query and it, in turn, can let you know your IP address.
The third is to specifically attach to an outside DNS server (not your corporate one) to retrieve the information.
Of course, you should question the need to know your external IP address. The whole point of DNS is to avoid having to worry too much about IP addresses and just refer to machines by the domain names.
There is no reliable way to get your public IP. Depending on your network topology, you may not be able to depend on the outgoing address for a number of reasons: IP address pools, dynamic routes, multiple layers of NAT, proxy servers, etc.