twisted server, nc client - twisted

Question

Ill demonstrate the problem I am facing with a small example.
class TestProtocol(basic.LineReceiver):
def lineReceived(self, line):
print line
Everything works fine as long as I use the telnet client to connect to the server. However, the line is not received connect and send the data using netcat. I have a feeling that this has something to do with the default delimiter being "\r\n" in twisted.
How could I make a server such that both the clients(telnet and nc) would behave in a similar manner when connecting to the client?

Answer 1

LineReceiver only supports one delimiter. You can specify it, but there can only be one at a time. In general, if you want to support multiple delimiters, you'll need to implement a new protocol that supports that. You could take a look at the implementation of LineReceiver for some ideas about how a line-based protocol is implemented.
netcat sends whatever you type, so the delimiter is often \n (but it may vary from platform to platform and terminal emulator to terminal emulator). For the special case of \n, which is a substring of the default LineReceiver delimiter \r\n, there's another trick you can use. Set the TestProtocol.delimiter to "\n" and then strip the "\r" off the end of the line passed to lineReceived if there is one.
class TestProtocol(basic.LineReceiver):
delimiter = "\n"
def lineReceived(self, line):
print line.rstrip("\r")

Answer 2

Another workaround is to use nc with the -C switch.
From the manual:
-C Send CRLF as line-ending
or as #CraigMcQueen suggested:
socket with -c switch (Ubuntu package).

Answer 3

Twisted's LineReceiver and LineOnlyReceiver only support one line ending delimiter.
Here is code for UniversalLineReceiver and UniversalLineOnlyReceiver, which override the dataReceived() method with support for universal line endings (any combination of CR+LF, CR or LF). The line breaks are detected with the regular expression object delimiter_re.
Note, they override functions with more code in them than I'd like, so there's a chance that they may break if the underlying Twisted implementation changes. I've tested they work with Twisted 13.2.0. The essential change is the use of delimiter_re.split() from the re module.
# Standard Python packages
import re
# Twisted framework
from twisted.protocols.basic import LineReceiver, LineOnlyReceiver
class UniversalLineReceiver(LineReceiver):
delimiter_re = re.compile(br"\r\n|\r|\n")
def dataReceived(self, data):
"""
Protocol.dataReceived.
Translates bytes into lines, and calls lineReceived (or
rawDataReceived, depending on mode.)
"""
if self._busyReceiving:
self._buffer += data
return
try:
self._busyReceiving = True
self._buffer += data
while self._buffer and not self.paused:
if self.line_mode:
try:
line, remainder = self.delimiter_re.split(self._buffer, 1)
except ValueError:
if len(self._buffer) > self.MAX_LENGTH:
line, self._buffer = self._buffer, b''
return self.lineLengthExceeded(line)
return
else:
lineLength = len(line)
if lineLength > self.MAX_LENGTH:
exceeded = self._buffer
self._buffer = b''
return self.lineLengthExceeded(exceeded)
self._buffer = remainder
why = self.lineReceived(line)
if (why or self.transport and
self.transport.disconnecting):
return why
else:
data = self._buffer
self._buffer = b''
why = self.rawDataReceived(data)
if why:
return why
finally:
self._busyReceiving = False
class UniversalLineOnlyReceiver(LineOnlyReceiver):
delimiter_re = re.compile(br"\r\n|\r|\n")
def dataReceived(self, data):
"""
Translates bytes into lines, and calls lineReceived.
"""
lines = self.delimiter_re.split(self._buffer+data)
self._buffer = lines.pop(-1)
for line in lines:
if self.transport.disconnecting:
# this is necessary because the transport may be told to lose
# the connection by a line within a larger packet, and it is
# important to disregard all the lines in that packet following
# the one that told it to close.
return
if len(line) > self.MAX_LENGTH:
return self.lineLengthExceeded(line)
else:
self.lineReceived(line)
if len(self._buffer) > self.MAX_LENGTH:
return self.lineLengthExceeded(self._buffer)