On the law lvl of SSL protocols there are 4 types of messages:
Handshake Protocol
ChangeCipherSpec Protocol
Alert Protocol
Application Data Protocol
After the handshaking is completed and the symmetric private key been exchanged, the client will send Application Data messages to the server.
How ever same server can handle multiple clients, and each of those clients got it's own symmetric key.
Does the server keeps the connection open with all of the clients? If not how does the server know what symmetric key to use for an incoming connection? Does Application Data Protocol provide some sort of session id that the server can use to map to the right key?
Does the server keeps the connection open with all of the clients?
It can, depending on how the server is implemented.
how does the server know what symmetric key to use for an incoming connection?
Imagine you have a multiplayer game. Your server code typically looks something like this:
sock = socket.listen(port)
clients = []
if sock.new_client_waiting:
new_client = sock.accept()
clients.append(new_client)
for client in clients:
if client.new_data_waiting:
data = client.read()
# handle incoming actions...
So if there are two clients, the server will just know about both and have a socket object for both. Therein lies your answer: the OS (its TCP stack) handles the concept of connections and by providing you a socket, you can read/write from and to that socket, and you know from which clients it originates (to some degree of certainty anyway).
Many servers work differently, e.g. web server code is much more like this:
sock = socket.listen(port)
while True: # infinitely loop...
client = sock.accept() # This call will block until a client is available
spawn_new_http_handler(client)
Whenever a new person connects, a worker thread will pick it up and manage things from there. But still, it will have its socket to read from and write to.
Does Application Data Protocol provide some sort of session id that the server can use to map to the right key?
I do not know these specs by heart, but I am pretty sure the answer is no. Session resumption is done earlier, at the handshake stage, and is meant for clients returning. E.g. if I connected to https://example.com 30 minutes ago and return now, it might have my session and we don't need to do the whole handshake again. It doesn't have anything to do with telling clients apart.
Related
I have a chat implementation working with CometD.
On front end I have a Client that has a clientId=123 and is talking to VirtualMachine-1
The longpolling connection between the VirtualMachine-1 and the Client is done through the clientId. When the connection is established during the handshake, VirtualMachine-1 registers the 123 clientId as it's own and accepts its data.
For some reason, if VM-1 is restarted or FAILS. The longpolling connection between Client and VM-1 is disconnected (since the VirtualMachine-1 is dead, the heartbeats would fail, thus it would become disconnected).
In which case, CometD loadBalancer will re-route the Client communication to a new VirtualMachine-2. However, since VirtualMachine-2 has different clientId it is not able to understand the "123" coming from the Client.
My question is - what is the cometD behavior in this case? How does it re-route the traffic from VM-1 to a new VM-2 to successfully go through handshaking process?
When a CometD client is redirected to the second server by the load balancer, the second server does not know about this client.
The client will send a /meta/connect message with clientId=123, and the second server will reply with a 402::unknown_session and advice: {reconnect: "handshake"}.
When receiving the advice to re-handshake, the client will send a /meta/handshake message and will get a new clientId=456 from the second server.
Upon handshake, a well written CometD application will subscribe (even for dynamic subscriptions) to all needed channels, and eventually be restored to function as before, almost transparently.
Messages published to the client during the switch from one server to the other are completely lost: CometD does not implement any persistent feature.
However, persisting messages until the client acknowledged them is possible: CometD offers a number of listeners that are invoked by the CometD implementation, and through these listeners an application can persist messages (or other information) into their own choice of persistent (and possibly distributed) store: Redis, RDBMS, etc.
CometD handles reconnection transparently for you - it just takes a few messages between client and the new server.
You also want to read about CometD's in-memory clustering features.
I’m writing a simple client-server app which for the time being will be for my own personal use. I’m using Winsock for the net communication. I have not done any networking for the last 10 years, so I am quite rusty. I’d like to use as little external code as possible, so I have written a home-made server discovery mechanism, as follows.
The client broadcasts a message containing the ‘name’ of a client UDP socket bound to an arbitrary port, which I will call the client’s discovery socket. The server recv() the broadcast and then sendto() the client discovery socket the ‘name’ of its listening socket. The client then uses this info to connect to the server (on a different socket). This mechanism should allow the server to bind its listening socket to the first port it can within the dynamic port range (49152-65535) and to the clients to discover where the server is and on which port it is listening.
The server part works fine: the server receives the broadcast messages and successfully sends its response.
On the client side the firewall log shows that the server’s response arrives to the machine and that it is addressed to the correct port (to the client’s discovery socket).
But the message never makes it to the client app. I’ve tried doing a recv() in blocking and non-blocking mode, and there is never any data available. ioctlsocket() always shows no data is available, even though I know the packet got it to the machine.
The server succeeds on doing a recv() on broadcasted data. But the client fails on doing a recv() of the server’s response which is addressed to its discovery socket.
The question is very vague: what gotchas should I watch for in this scenario? Why would recv() fail to get a packet which has actually arrived to the machine? The sockets are UDP, so the fact that they are not connected is irrelevant. Or is it?
Many thanks in advance.
The client broadcasts a message containing the ‘name’ of a client UDP socket bound to an arbitrary port, which I will call the client’s discovery socket.
The message doesn't need to contain anything. Just broadcast an empty message from the 'discovery socket'. recvfrom() will tell the server where it came from, and it can just reply directly.
The server recv() the broadcast and then sendto() the client discovery socket the ‘name’ of its listening socket.
Fair enough, although actually the server could just broadcast its own TCP listening port every 5 seconds or whatever.
On the client side the firewall log shows that the server’s response arrives to the machine and that it is addressed to the correct port (to the client’s discovery socket). But the message never makes it to the client app
If it got to the host it must get to the application. You must have got the ports mixed up somehow. Simplify it as above and retry.
Well, it was one of those stupid situations: Windows Firewall was active, besides the other firewall, and silently dropping packets. Deactivating it solved the problem.
But I still don’t understand how it works, as it was allowing the server to receive packets sent through broadcasting. And when I got at my wits end and set the server to answer back through a broadcast, THOSE packets got dropped.
Two days of frustration. I hope someone profits from my experience.
I'm trying to get some feedback on the recommendations for a service 'roster' in my specific application. I have a server app that maintains persistant socket connections with clients. I want to further develop the server to support distributed instances. Server "A" would need to be able to broadcast data to the other online server instances. Same goes for all other active instances.
Options I am trying to research:
Redis / Zookeeper / Doozer - Each server instance would register itself to the configuration server, and all connected servers would receive configuration updates as it changes. What then?
Maintain end-to-end connections with each server instance and iterate over the list with each outgoing data?
Some custom UDP multicast, but I would need to roll my own added reliability on top of it.
Custom message broker - A service that runs and maintains a registry as each server connects and informs it. Maintains a connection with each server to accept data and re-broadcast it to the other servers.
Some reliable UDP multicast transport where each server instance just broadcasts directly and no roster is maintained.
Here are my concerns:
I would love to avoid relying on external apps, like zookeeper or doozer but I would use them obviously if its the best solution
With a custom message broker, I wouldnt want it to become a bottleneck is throughput. Which would mean I might have to also be able to run multiple message brokers and use a load balancer when scaling?
multicast doesnt require any external processes if I manage to roll my own, but otherwise I would need to maybe use ZMQ, which again puts me in the situation of depends.
I realize that I am also talking about message delivery, but it goes hand in hand with the solution I go with.
By the way, my server is written in Go. Any ideas on a best recommended way to maintain scalability?
* EDIT of goal *
What I am really asking is what is the best way to implement broadcasting data between instances of a distributed server given the following:
Each server instance maintains persistent TCP socket connections with its remote clients and passes messages between them.
Messages need to be able to be broadcasted to the other running instances so they can be delivered to relavant client connections.
Low latency is important because the messaging can be high speed.
Sequence and reliability is important.
* Updated Question Summary *
If you have multiple servers / multiple end points that need to pub/sub between each other, what is a recommended mode of communication between them? One or more message brokers to re-pub messages to a roster of the discovered servers? Reliable multicast directly from each server?
How do you connect multiple end points in a distributed system while keeping latency low, speed high, and delivery reliable?
Assuming all of your client-facing endpoints are on the same LAN (which they can be for the first reasonable step in scaling), reliable UDP multicast would allow you to send published messages directly from the publishing endpoint to any of the endpoints who have clients subscribed to the channel. This also satisfies the low-latency requirement much better than proxying data through a persistent storage layer.
Multicast groups
A central database (say, Redis) could track a map of multicast groups (IP:PORT) <--> channels.
When an endpoint receives a new client with a new channel to subscribe, it can ask the database for the channel's multicast address and join the multicast group.
Reliable UDP multicast
When an endpoint receives a published message for a channel, it sends the message to that channel's multicast socket.
Message packets will contain ordered identifiers per server per multicast group. If an endpoint receives a message without receiving the previous message from a server, it will send a "not acknowledged" message for any messages it missed back to the publishing server.
The publishing server tracks a list of recent messages, and resends NAK'd messages.
To handle the edge case of a server sending only one message and having it fail to reach a server, server can send a packet count to the multicast group over the lifetime of their NAK queue: "I've sent 24 messages", giving other servers a chance to NAK previous messages.
You might want to just implement PGM.
Persistent storage
If you do end up storing data long-term, storage services can join the multicast groups just like endpoints... but store the messages in a database instead of sending them to clients.
I'm playing around with a webserver, using a unix socket and sendmsg / recvmsg to pass the socket file descriptor to a new server process without losing any requests. While testing it with ab I found that client connections would linger, and apachebench (ab) would show the error: "apr_poll: The timeout specified has expired (70007)".
I suspected that there was a change to the address of the file descriptor that would render open connections useless, however making sure the connections were closed at the end of every request didn't make a difference, a couple of the requests would fail.
Is there some extra oddity at the socket level or is ab just being weird? Is there anything else I should take into account?
Edit: Using PHP as a client to make requests also stalls during the cycle.
It does make sense if you have a master server which is listening on a socket (accepting incoming connections) and you have multiple worker processes.
You can select a suitable/free worker (for example, based on the number of TCP connections every worker is using) and pass the descriptor of the incoming connection from the master to the worker. This helps to avoid the "thundering herd" when multiple workers listen on the common endpoint.
That's equivalent to trying to send a telephone over a telephone line. It doesn't make any sense. A socket fd identifies the endpoint of a connection. If another host wants a connection it will have to mke its own. You can't give it one of yours.
I have a lot of client programs and one service.
This Client programs communicate with the server with http channel with WCF.
The clients have dynamic IP.
They are online 24h/day.
I need the following:
The server should notify all the clients in 3 min interval. If the client is new (started in the moment), is should notify it immediately.
But because the clients have dynamic IP and they are working 24h/day and sometimes the connection is unstable, is it good idea to use wcf duplex?
What happens when the connection goes down? Will it automatically recover?
Is is good idea to use remote MSMQ for this type of notification ?
Regards,
WCF duplex is very resource hungry and per rule of thumb you should not use more than 10. There is a lot of overhead involved with duplex channels. Also there is not auto-recover.
If you know the interval of 3 minutes and you want the client to get information when it starts why not let the client poll the information from the server?
When the connection goes down the callback will throw an exception and the channel will close.
I am not sure MSMQ will work for you unless each client will create an MSMQ queue for you and you push messages to each one of them. Again with an unreliable connection it will not help. I don't think you can "push" the data if you loose the connection to a client, client goes off-line or changes an IP without notifying your system.