I would like to report the number of players in each of my game server nodes to a central authority that will be used to load balance connections to those nodes. This is a real time multiplayer web game. This is a game design question for a game that could go in any load balance direction and at this point any solution is possible. At a high level if you were designing this would it be better to open an outbound tcp connection to the authority and report game stats in the game loop OR would it be better to wait for the authority to ask the game server for stats over an incoming tcp connection? Which solution would be better if you had 1000 nodes?
If this were outbound from game server:
If the game server reports that this is my ip address, my port and my number of players the authority could catalog that data and update the number of players if the same ip/port combo exists. In order to prevent security issues with rogue agents reporting themselves as game servers you would then need to try and secure the connection with both encryption and some type of authorization mechanism.
If this were inbound to the game server
If the authority already knows the address of each node it needs to connect too then you may only need to worry about man in the middle servers pretending to be the authority. Given the data is only game stats though and nothing sensitive, encrypting it shouldn't matter in the grand scheme for a connection inbound to a game server. I could have for example a config file or database table with all the known game servers to connect too. A mitm would get game stats or maybe just block the connection. To me this seems like the better option to just connect out to your game server unencrypted and get the stats. The downside is anyone else could connect to your game server as well to ask for game stats. Then in your game server you'd have to figure out either which connection is the real authority or just broadcast the game stats to anyone asking for game stats. The game server from a security standpoint though could get overloaded with to many connections wanting to get game stats and you don't have a good way to know which authority is the one that matters for your load balancing vs adhoc agents. I FEEL LIKE THIS IS THE WAY TO GO.
I agree that inbound to the game servers are the way to go, for the reasons you stated.
Here are a few other considerations.
I think practically that it is a lot harder for the servers to connect to users' hosts because of ip changes and firewalls at the user-local level.
You should always encrypt your customer's over the wire. And speaking of MITM, you should have some way of preventing one user from spoofing another user. No-one wants their game play taken over by a hacker.
You could have, as part of your client-server protocol, a way for the server to dictate a minimum wait time before the client sends their next update.
You could have the client updates put into a queue (rabbit, kafka, etc.) and have the servers update at their own capabilities. Queues are a lot more forgiving of elastic traffic, if you can handle them.
Related
I am new to webrtc development, I would like to create a conference room with multiples connections. I have read about webrtc and peer communication. I would like to know if google meet used one peer connection for every participants one another. If so, how can they handle 250 participants without draining the browser resources?
For group calls, Google Meets does not use peer-to-peer, if that was the case, it would drain one's bandwidth very quickly and browser resources quickly.
Although I am not exactly sure about Meets' infrastructure, they most likely use a Selective Forwarding Unit (SFU) to be able to handle lots of connections, since it's common practice for this sort of application.
The SFU works as a webrtc client server-side, and it basically just forwards the stream to clients that are requesting it browser-side.
Signaling is not addressed by WebRTC (even if we do have JSEP as a starting point), but from what I understand, it works that way :
client tells the server it's available at X
server holds that information and maps it to an identifier
other client comes and sends an identifier to get connection information from the first client
other client uses it to create it's one connection information and sends it to the server
server sends this to first client
both client can now talk
This is all nice and well, but what happends if a 3rd client arrives ?
You have to redo the whole things. Which suppose the first two clients are STILL connected to the server, waiting for a 3rd client to signal itself, and start the exchanging process again so they can get the 3rd client connection information.
So does it mean you are required to have to sort of permanent link to the server for each client (long polling, websocket, etc) ? If yes, is there a way to do that efficiently ?
Cause I don't see the point of having webRTC if I have to setup nodejs or tornado and make it scales to the number of my users. It doesn't sound very p2pish to me.
Please tell me I missed something.
What about a chat system? Do you really need to keep a permanent link to the server for each client? Of course, because otherwise you have no way of keeping track of a user's status. This "permanent" link can be done different ways: you mentioned WebSocket and long polling, but simple periodic XHR polling works too (although this will affect the UX, depending on the interval).
So view it like a chat system, except that the media stream is P2P for reduced latency. Once a P2P WebRTC connection is established, the server may die and, of course, the P2P connection will be kept between the two clients. What I mean is: both users may always block your server once the P2P connection is established and still be connected together in the wild Internets.
Understand me well: once the P2P connection is established, your server will not be doing any more WebRTC signalling. The connection is only needed to keep track of the statuses.
So it depends on your application. If you want to keep the statuses of users and make them visible to others, then you're in the same situation as a chat system: you need to keep a certain link, somehow, to make sure their statuses are synced. Otherwise, your server exists to connect them together and is not needed afterwards. An example of the latter situation is: a user goes to a webpage, the webpage provides him with a new room URL, the user shares this URL to another peer by another mean, the other peer joins the room, server connects them together (manages WebRTC signalling) and then forgets them. They are now connected until one of them breaks the link. Just like this reference app.
Instead of a central server keeping one connection per client, a mesh network could also be considered, albeit difficult to implement.
I've read a lot about WebRTC, but there's one question that still remains. I hope you can help me with that:
Does WebRTC allow me to create a one-to-many connection? I don't mean "being able to have multiple connections to different computers", I really talk about having one connection that multicasts its data to multiple endpoints without the need to "upload" the data once for each endpoint. Will it be possible to send one single package to the web, that, when it reaches the web, magically splits itself into multiple packages with different targets?
I hope you get what I'm looking for :)
Until now, I've only seen one-to-one connections, or solutions that have one connection to a central server that does the multicast for them (which usually results in twice the ping).
But to me, one-to-one connections don't seem to be really useful (due to low upload-bandwith of clients), and solutions with a central server are also possible without WebRTC (using WebSockets), so the only real use case for WebRTC would be one-to-many connections.
So.. is this something that will be possible in the future? Or is it already possible today?
Three things:
IP multicast in the Internet is not possible at the moment (multicast addresses are not routed by ISPs)
WebRTC fits many use cases beyond one-to-many communication, just have a look at this document: https://datatracker.ietf.org/doc/html/draft-ietf-rtcweb-use-cases-and-requirements-06
WebRTC connections between browsers are always encrypted (using SRTP for A/V data and DTLS for generic data) and the encryption parameters (session keys etc.) are negotiated for every connection separately. How would you do that in a multicast environment (think of it as a distribution tree)?
So no, WebRTC cannot be used with IP multicast.
I would answer "It doesn't for now", because as a programmer, I can tell you, that there are number of ways browser devs to make it work if we (users) insist on it's importance. But how ? Since there's encryption, they could allow sharing of the session's encryption keys to the group of 'registered' (multicast) users. But how ? Well, Web was created for sharing. The most obvious way is through web server mediation and JS WebRTC API function (to load the user keys). Since multicast is most often used for efficient video distribution, you have a RTP/SRTP video server. The web server can coexist at the same machine. If they decide to extend it to web browsers - then just the "server" role can be done by the Web browser who created the multicast stream (the sender). The clients need to know who is it.
Again: In December 2013, this is still not possible. And multicasts are allowed on the Internet only in:
some experimental WAN nets
some internet+video ISP nets
LANs (when enabled at switch level, cheap switches transmit it to all ports). But you can be an ISP, researcher or LAN user, so it's necessary.
I'm looking to write a toy application for my own personal use (and possibly to share with friends) for peer-to-peer shared status on a local network. For instance, let's say I wanted to implement it for the name of the current building you're in (let's pretend the network topology is weird, and multiple buildings occupy the same LAN). The idea is if you run the application, you can set what building you're in, and you can see the buildings of every other user running the application on the local network.
The question is, what's the best transport/network layer technology to use to implement this?
My initial inclination was to use UDP Multicast, but the more research I do about it, the more I'm scared off by it: while the technology is great and seems easy to use, if the application is not tailored for a particular site deployment, it also seems most likely to get you a visit from an angry network admin.
I'm wondering, therefore, since this is a relatively low bandwidth application — probably max one update every 4–5 minutes or so from each client, with likely no more than 25–50 clients — whether it might be "cheaper" in many ways to use another strategy:
Multicast: find a way to pick a well-known multicast address from 239.255/16 and have interested applications join the group when they start up.
Broadcast: send out a single UDP Broadcast message every time someone's status changes (and one "refresh" broadcast when the app launches, after which every client replies directly to the requesting user with their current status).
Unicast: send a UDP Broadcast at application start to announce interest, and when a client's status changes, it sends a UDP packet directly to every client who has announced. This results in the highest traffic, but might be less likely to annoy other systems with needless broadcast packets. It also introduces potential complications when apps crash (in terms of generating unnecessary traffic).
Multicast is most certainly the best technology for the job, but I'm wondering if the associated hassles are worth avoiding since this is just a "toy application," not a business-critical service intended for professional network admin deployment and configuration.
We're adding a second web server for redundancy and load sharing purposes. All connections are mandated to be SSL, and adding a dedicated appliance is not possible at this moment.
I'd like to use round robin DNS, where both servers answer to the same domain using different IPs (we have a wildcard SSL certificate, so that's OK). I can get the DNS to return in random/round robin order no problem.
Is this a bad setup when using SSL?
Our user pattern is consistent -- users will consistently be utilizing the web app for 8-10 hours. We want each page view to be as fast as possible, and my concern is users could constantly flip between the servers, potentially negating any SSL handshake caching/keep alive.
Thanks!
Firstly, SSL has the ability to resume an earlier session, so flipping between servers will cost you a few hundred ms per request (longer if several clients are accessing the site simultaneously, since this is CPU time we're talking about).
Whether the clients will actually flip depends, though - DNS "load balancing" is fiddly business:
if many of your users are using the same recursive nameservers, they'll get the same "first IP" hence no load balancing
if the DNS record has a high TTL (several hours), caching nameservers will store a particular permutation of IP addresses until they expire (good so long as your users aren't all using the same recursive nameservers)
if your users have multiple recursive nameservers configured, they may flip if each nameserver has a different "first IP" (bad)
if you have no mechanism for removing "bad" records, and a low TTL, then if one server goes down 50% of your clients will get the "bad" server and have to wait for a timeout before they can see your site
As you can see there are various tradeoffs depending on whether you're more concerned about redundancy/failover or load balancing; DNS isn't really the best tool here - you really need the servers to share an IP using either a reverse proxy, or something like Heartbeat (assuming you're Linux-based).
An aside: if both servers are answering to the same domain then you don't need a wildcard cert, although CAs often charge more if you intend to use a cert on more than one server.
TLDR: You will be fine. The SSL renegotiations shouldn't happen frequently enough to be noticeable by your end user.
Rant starts here:
Load distribution using DNS is a commonly misunderstood topic that leads into a lot of anecdotal evidence and straw-man arguments. I've been in too many of these meetings.
Here's how I usually settle these arguments:
"Wow yeah that sounds really exoteric [long dramatic pause] but it really can't be that bad since google uses it"
$host encrypted.google.com
encrypted.google.com is an alias for www3.l.google.com.
www3.l.google.com has address 74.125.224.195
www3.l.google.com has address 74.125.224.202
www3.l.google.com has address 74.125.224.193
www3.l.google.com has address 74.125.224.197
www3.l.google.com has address 74.125.224.207
www3.l.google.com has address 74.125.224.206
www3.l.google.com has address 74.125.224.203
www3.l.google.com has address 74.125.224.204
www3.l.google.com has address 74.125.224.196
www3.l.google.com has address 74.125.224.199
www3.l.google.com has address 74.125.224.201
www3.l.google.com has address 74.125.224.194
www3.l.google.com has address 74.125.224.192
www3.l.google.com has address 74.125.224.200
www3.l.google.com has address 74.125.224.205
www3.l.google.com has address 74.125.224.198
Updates:
Is this setup redundant?
It is not inherently redundant in the engineering sense since if one of those ips were to fail it would continue to be served to the customer until a DNS zone change is performed and all downstream caches expire. With that said, most browsers are smart enough to try another ip under these circumstance - reference.
Moreover, a system could easily be devise that instead of requiring a DNS zone change to remove the failed node would, instead, route the ip of the failed instance to a servicing device by simple ip takeover.
Is this setup resilient?
Yes, resilience is achieved by minimizing your failure domain. Back to our example failure of a single ip (and remember these ips may represent load balancers backed by hundred of servers or even an entire data center) the likelihood of a customer hitting that ip is 1/16, or ~6%, (using the google example above). This is inherently more resilient than a system with a single A address, wich would impact 100% of the users, or a system with 2 A records in which the user has an even 50/50 change of hitting a failed resource.
Don't worry about it. There are multiple levels of DNS caches so user is not going to flip between 2 IPs on every request. The IP will stay the same for hours for each client.
We have an opposite problem. When server goes down, the user still has the bad IP. We set the TTL to 1 minute but very few browsers honor it. Due to this issue, VIP is a much better option than DNS for load-balancing on the same network.
DNS round robin does not provide redundancy.
Without substantial additional help it only provides dumb load sharing (nb: not load "balancing", which implies dynamic load distribution based on server load).
Having the same cert on two IPs should be no problem, though.