Can someone point me to some documentation on how to implement a Bittorrent UDP application?
I am working on a Bittorrent application and I can successfully download using TCP but I want to implement UDP and I can't find any information on it. I am retrieving peers with UDP trackers but that approach appears different than downloading files.
If I sniff a UDP Handshake I see the following in Wireshark:
172.16.49.213 5.31.44.30 UDP 62 35507 → 18318 Len=20
5.31.44.30 172.16.49.213 UDP 62 18318 → 35507 Len=20
This is done before the handshake. Also, it looks like there are 20 bytes prepended to the Handshake (different than the 20 bytes above).
So I need some information to help me with what all this means.
The BitTorrent Peer Wire Protocol over UDP, called uTP is specified in:
BEP29 - uTorrent transport protocol
BitTorrentInc has also published uTP as a IETF RFC were they call it LEDBAT:
RFC 6817 - Low Extra Delay Background Transport (LEDBAT)
However, I wouldn't recommend anyone to do their own implementation, (except as a learning experience), as it involves a lot of time critical, low level network IO and is very tricky to get right.
Instead I recommend to use the library: https://github.com/bittorrent/libutp
Almost all clients implementing uTP uses this library. AFAIK, the only exception is libtorrent/rasterbar (used by qBittorrent and Deluge) and it don't work as good as libutp does.
Related
I was researching on QUIC protocol and my professor asked me this question. QUIC is said to be built on UDP and uses TLS packets. TLS itself requires reliable connection that is TCP. So, why QUIC is not said to be built on TCP?
PS: Please correct me if my concepts are wrong and if possible, please explain in deep how QUIC packets work.
QUIC is said to be built on UDP and uses TLS packets.
QUIC (RFC 9000) does not use TLS "packets".
Technically, TLS uses the term "record" to indicate a block of bytes that defines how the protocol is framed. A record specifies the TLS protocol version, the length of the record, etc.
Inside TLS frames there are one or more TLS messages that specify cryptographic information or commands.
The TLS records are transported by TCP.
What QUIC does instead is to reuse some of the TLS messages, but nothing of the TLS record layer.
For example, in TCP the first bytes sent by a client are typically a TLS record that wraps the ClientHello message.
In QUIC, instead, the first bytes are a QUIC Initial packet, that wraps a CRYPTO frame, that wraps the ClientHello message, and all of these bytes must fit into a UDP datagram (they typically do, and the Initial packet even carries a PADDING frame to make the initial bytes at least 1200).
Refer to RFC 9001 for more details about how TLS is used in QUIC.
More broadly about QUIC, it is based on UDP but borrows many TCP features (retransmissions, flow control, etc.) that basically make it an alternative to TCP in the sense that it is a reliable network protocol, with additional security (encryption) features built-in, borrowed by TLS.
TCP is clear-text and relies on TLS layered on top to provide encryption.
QUIC is a mix of TCP features and TLS features (there is no clear-text QUIC), without layering.
When you say "Why QUIC is not said to be built on TCP?", I say "QUIC is not built on TCP, it is built on UDP. However, QUIC borrows TCP features that make QUIC provide similar functionalities as TCP, in particular reliability and flow control, that are not provided by UDP".
For example, in Jetty we have implemented HTTP/1.1 and HTTP/2 (that are TCP-based protocols) on top of QUIC, using a single, persistent, QUIC stream.
This shows that QUIC can be indeed a replacement for TCP, as it can carry protocols that were designed for TCP.
QUIC includes TLS in it to allow it to be used over UDP in the same way as TCP works.
Why bother reinventing TCP and not just use TCP? Well TCP is kind of “stuck” as it’s slow to roll out new changes that fundamentally change how TCP works. Enhancing it to allow new features like multiplex streams will take a loooong time to roll out everywhere TCP is supported.
QUIC is built over simple UDP packets and everything else is handled at either end by QUIC and all the routers and networks in the middle don’t need to know about these new QUIC features.
The RFCs are all written by committee, and the structure and language is often confusing, so it is easy to mix things up.
A full TLS implementation requires a reliable transport, which as of today is provided by TCP and SCTP (RFC3436).
QUIC (RFC9001) doesn't actually require a full TLS implementation though, and instead simply re-uses parts of the TLSv1.3 standard:
4.1. Interface to TLS
As shown in Figure 4, the interface from QUIC to TLS consists of four
primary functions:
* Sending and receiving handshake messages
* Processing stored transport and application state from a resumed
session and determining if it is valid to generate or accept 0-RTT
data
* Rekeying (both transmit and receive)
* Updating handshake state
So, given the above, the actual process that QUIC takes to encrypting the packet data isn't TLS at all, but is instead provided by QUIC.
I am trying to do some tests in Chromium with HTTP/2, HTTP1.1, QUIC, and TCP. I want to try different combinations of the protocols. I am experiencing some very strange behaviour in my browser though. When I want to try HTTP1.1+QUIC I start my browser with:
chromium-browser --disable-http2 --enable-quic
And I can see at chrome://net-internals/ that HTTP2 is disabled and QUIC is enabled. However, when I do a web request to a server supporting HTTP2 and QUIC I get this:
Why would it say that the HTTP/2 is used when it so clearly says that http2 enabled: false at chrome://net-internals/ ?
I have previously been successful in running HTTP1.1 with QUIC. Has QUIC been updated to only work with HTTP/2? Or does the 'Protocol'-field display the wrong protocol?
I would love to if someone else have been successful is using QUIC with HTTP1.1
Many thanks!
QUIC only works with HTTP/2 and doesn’t make sense with HTTP/1.1.
HTTP/1.1 sends one request at a time on its one TCP connection. Browsers open 6-8 connections to allow some level of parallelisation but, other than that hack, HTTP/1.1 is basically synchronous - you send a request and can’t use that connection again until that request is responded to in its entirety. So HTTP/1.1 has a Head of Line (HOL) blocking problem as a slow, or delayed, response will block the connection so it can’t be used for a resource that could be responded to.
HTTP/2 is a binary, multiplexed protocol, which basically means requests and responses are split into packets which can be sent on a single TCP connection and intermingled. This is great as TCP connections are relatively expensive over the Internet due to the time to set up the TCP connection, to set up HTTPS potentially on top of that and then to build up the TCP slow start rate limiting window to an optimal size. So HTTP/2 being able to send and receive multiple HTTP requests and responses over one TCP connection is a massive improvement and solves the HOL Blocking problem at a HTTP level.
Unfortunately the HOL blocking problem is moved from HTTP layer to TCP layer. TCP is a guaranteed protocol - if a single TCP packet is lost then it is re-requested and the whole connection waits for that missed packet to come back. This is great as higher level protocols (like HTTP) can build upon this without worry about checking if pieces have made it or not or in the right order - TCP takes care of that. The downside with this is that protocols like HTTP/2 may not need this strict level of guarantees. If a server is sending 6 HTTP/2 responses on one connection at the same time, and one TCP packet gets dropped it will only be for one of those responses - the other 5 could in theory continue to be sent and then be processed by the browser. But TCP forbids this.
So QUIC aims to solve this by replacing the TCP part with a UDP based protocol that guarantees delivery at a stream level rather than a connection level. This could have been done by enhancing TCP but that’s so embedded into lots of servers, clients, operating systems, network infrastructure... etc. that it was easier to build upon UDP instead. Eventually learnings from this might be incorporated into TCP. Until then QUIC allows quick experimentation and innovation as UDP is very, very light and any additions to it (e.g. delivery guarantees) are basically implemented in the user-space land rather than lower level internals where it can’t be upgraded. QUIC will likely eventually be used for protocols other than HTTP but for the moment that’s it’s primary use case.
So QUIC effectively replaces the TCP part, but it also needs to change the lower level parts of HTTP/2 which means it also needs to implement the in between TLS part. The best diagram for explain it’s place is this diagram (taken from this presentation):
I have previously been successful in running HTTP1.1 with QUIC.
I seriously doubt this. As per above explanation QUIC only makes sense over a multiplexed protocol like HTTP/2. Perhaps Chrome used to call it QUIC rather than HTTP/2 + QUIC but it would always have been using HTTP/2 (or its similar predecessor SPDY).
Also HTTP/2 really only changes how the messages are sent on the wire. At a higher level, as used by web developers and users, it often acts the same way as HTTP/1.1 with the same verbs (GET, POST... etc.) and HTTP Headers (mostly). So since QUIC just improves how those HTTP/2 messages are sent on the wire, rather than the higher level “HTTP” protocol it really doesn’t make sense to discuss or measure how HTTP/1.1 over QUIC (if it did exist) would differ from HTTP/2 over QUIC - since HTTP/1.1 over QUIC would basically become HTTP/2 over QUIC.
Finally when QUIC has completed standardisation (as opposed to the non-standard Google QUIC version often called gQUIC), it will use a modified version of HTTP/2 which will be called HTTP/3. For now browsers will only use gQUIC and HTTP/2.
If you want a quick test, head to https://http3.is.
It’s a test website created by fastly.
I'm going to implement Java VoiP server to work with WebRtc. Implementation of browser p2p connection is really straightforward. Server to client connection is slightly more tricky.
After a quick look at RFC I wrote down what should be done to make Java server as browser. Kindly help me to complete list below.
Implement STUN server. Server should be abke to respond binding
request and keep-alive pings.
Implement DTLS protocol along with DTLS handshake. After the DTLS
handshake shared secret will be used as keying material within SRTP
and SRTCP.
Support multiplexing of SRTP and SRTCP stream. SRTP and SRTCP use
same port to adress NAT issue.
Not sure whether should I implement SRTCP. I believe connection will
not be broken, if server does not send SRTCP reports to client.
Decode SRTP stream to RTP.
Questions:
Is there anything else which should be done on server-side ?
How webRtc handles SRTCP reports ? Does it adjust sample rate/bit
rate depends on SRTCP report?
WebRtc claims that following issues will be addressed:
packet loss concealment
echo cancellation
bandwidth adaptivity
dynamic jitter buffering
automatic gain control
noise reduction and suppression
Is is webRtc internals or codec(Opus) internals? Do I need to do anything on server side to handle this issues, for example variable bitrate etc ?
The first step would be to implement Interactive Connectivity Establishement (RFC 5245). Whether you make use of a STUN/TURN server or not is irrelevant, your code needs to issue connectivity checks (which use STUN messages) to the browser and respond to the brower's connectivity checks. ICE is a fairly complex state machine, but it's doable.
You don't have to reinvent the wheel. STUN / TURN servers are external components. Use as they are. WebRTC source code is available which you can use in your application code and call the related methods.
Pls. refer to similar post - Server as WebRTC data channel peer
I have a client, whose code I can't change -- but I'd like to (re)write using ZeroMQ sockets.
The clients use both raw TCP and raw UDP sockets.
I know I can use ZMQ_ROUTER_RAW for raw TCP sockets, but what about for raw UDP datastreams?
Support for UDP in ZeroMQ is brand new. The documentation for the new socket types (Radio/Dish) was just pushed to the GitHub repository a few days ago. As of right now, though, it still looks like there's no raw UDP support, but perhaps it will stem out of this new functionality.
I'd recommend commenting on the existing thread about adding UDP support: https://github.com/zeromq/libzmq/issues/807 .
UDP Support for ZMQ is now documented here:
http://api.zeromq.org/master:zmq-udp
Pyzmq version 18 has it as well, stating: "Protocols supported include tcp, udp, pgm, epgm, inproc and ipc." That said, my experimentation with Python hasn't found a solution that works yet. I get "protocol is not compatible with socket type" errors.
This morning, there were big problems at work because an SNMP trap didn't "go through" because SNMP is run over UDP. I remember from the networking class in college that UDP isn't guaranteed delivery like TCP/IP. And Wikipedia says that SNMP can be run over TCP/IP, but UDP is more common.
I get that some of the advantages of UDP over TCP/IP are speed, broadcasting, and multicasting. But it seems to me that guaranteed delivery is more important for network monitoring than broadcasting ability. Particularly when there are serious high-security needs. One of my coworkers told me that UDP packets are the first to be dropped when traffic gets heavy. That is yet another reason to prefer TCP/IP over UDP for network monitoring (IMO).
So why does SNMP use UDP? I can't figure it out and can't find a good reason on Google either.
UDP is actually expected to work better than TCP in lossy networks (or congested networks). TCP is far better at transferring large quantities of data, but when the network fails it's more likely that UDP will get through. (in fact, I recently did a study testing this and it found that SNMP over UDP succeeded far better than SNMP over TCP in lossy networks when the UDP timeout was set properly). Generally, TCP starts behaving poorly at about 5% packet loss and becomes completely useless at 33% (ish) and UDP will still succeed (eventually).
So the right thing to do, as always, is pick the right tool for the right job. If you're doing routine monitoring of lots of data, you might consider TCP. But be prepared to fall back to UDP for fixing problems. Most stacks these days can actually use both TCP and UDP.
As for sending TRAPs, yes TRAPs are unreliable because they're not acknowledged. However, SNMP INFORMs are an acknowledged version of a SNMP TRAP. Thus if you want to know that the notification receiver got the message, please use INFORMs. Note that TCP does not solve this problem as it only provides layer 3 level notification that the message was received. There is no assurance that the notification receiver actually got it. SNMP INFORMs do application level acknowledgement and are much more trustworthy than assuming a TCP ack indicates they got it.
If systems sent SNMP traps via TCP they could block waiting for the packets to be ACKed if there was a problem getting the traffic to the receiver. If a lot of traps were generated, it could use up the available sockets on the system and the system would lock up. With UDP that is not an issue because it is stateless. A similar problem took out BitBucket in January although it was syslog protocol rather than SNMP--basically, they were inadvertently using syslog over TCP due to a configuration error, the syslog server went down, and all of the servers locked up waiting for the syslog server to ACK their packets. If SNMP traps were sent over TCP, a similar problem could occur.
http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downtime-last-week/
Check out O'Reilly's writings on SNMP: https://library.oreilly.com/book/9780596008406/essential-snmp/18.xhtml
One advantage of using UDP for SNMP traps is that you can direct UDP to a broadcast address, and then field them with multiple management stations on that subnet.
The use of traps with SNMP is considered unreliable. You really should not be relying on traps.
SNMP was designed to be used as a request/response protocol. The protocol details are simple (hence the name, "simple network management protocol"). And UDP is a very simple transport. Try implementing TCP on your basic agent - it's considerably more complex than a simple agent coded using UDP.
SNMP get/getnext operations have a retry mechanism - if a response is not received within timeout then the same request is sent up to a maximum number of tries.
Usually, when you're doing SNMP, you're on a company network, you're not doing this over the long haul. UDP can be more efficient. Let's look at (a gross oversimplification of) the conversation via TCP, then via UDP...
TCP version:
client sends SYN to server
server sends SYN/ACK to client
client sends ACK to server - socket is now established
client sends DATA to server
server sends ACK to client
server sends RESPONSE to client
client sends ACK to server
client sends FIN to server
server sends FIN/ACK to client
client sends ACK to server - socket is torn down
UDP version:
client sends request to server
server sends response to client
generally, the UDP version succeeds since it's on the same subnet, or not far away (i.e. on the company network).
However, if there is a problem with either the initial request or the response, it's up to the app to decide. A. can we get by with a missed packet? if so, who cares, just move on. B. do we need to make sure the message is sent? simple, just redo the whole thing... client sends request to server, server sends response to client. The application can provide a number just in case the recipient of the message receives both messages, he knows it's really the same message being sent again.
This same technique is why DNS is done over UDP. It's much lighter weight and generally it works the first time because you are supposed to be near your DNS resolver.