At the interface level DCCP is like TCP: you connect and then send/receive bytes.
I was wondering it's possible to make dccp connections in twisted by just adapting the wrappers for tcp...
According to the sample code (below) what needs to be changed is:
at socket instantiation: use different parameters
before using the socket: set some options
Then everything else would be the same...
Hints: I've spotted addressFamily and socketType in the sources of twisted but I have no idea on how to cleanly set them in the protocol factory. Also the protocol number, the 3rd parameter, here IPPROTO_DCCP, is always keeped to default. I have no clue either on how to access the socket to call setsockopt
import socket
socket.DCCP_SOCKOPT_PACKET_SIZE = 1
socket.DCCP_SOCKOPT_SERVICE = 2
socket.SOCK_DCCP = 6
socket.IPPROTO_DCCP = 33
socket.SOL_DCCP = 269
packet_size = 256
address = (socket.gethostname(),12345)
# Create sockets
server,client = [socket.socket(socket.AF_INET, socket.SOCK_DCCP,
socket.IPPROTO_DCCP) for i in range(2)]
for s in (server,client):
s.setsockopt(socket.SOL_DCCP, socket.DCCP_SOCKOPT_PACKET_SIZE, packet_size)
s.setsockopt(socket.SOL_DCCP, socket.DCCP_SOCKOPT_SERVICE, True)
# Connect sockets
server.bind(address)
server.listen(1)
client.connect(address)
s,a = server.accept()
# Echo
while True:
client.send(raw_input("IN: "))
print "OUT:", s.recv(1024)
More about DCCP:
https://www.sjero.net/research/dccp/
https://wiki.linuxfoundation.org/networking/dccp
TL;DR: dccp is a protocol that provides congestion control (like tcp) without guaranteeing reliability or in-order delivery of data (like udp). The standard linux kernel implements dccp.
Related
I’m using SSL for reading data from various remote services over secure websockets as follows: I create the socket, embed it in the SSL context and add the socket to the reading list for Unix.select. When the socket fires, I use Ssl.read to get the data.
4 services are working well. And with one I get Ssl.Read_error.Error_syscall: error:00000000:lib(0):func(0):reason(0) after receiving each websocket frame (size ~5-6Kb). By the way, frames here are much bigger than on other services, but I’m not sure it’s the reason.
I ignore syscall errors (and most probably loose some data) because frames continue to arrive. Then, always after one minute I get Ssl.Read_error.Error_zero_return: error:00000000:lib(0):func(0):reason(0), which means the peer closed SSL socket for writing and I have to restart the process because no new data will be received from this socket.
Problem is perfectly reproducible. At the same time examples for this service and my own test implementation with Node.JS receive the data for hours without any problems.
I assume I do something wrong or setup socket/SSL too straightforward (see below).
Any help or ideas would be strongly appreciated.
let sock = Unix.socket PF_INET SOCK_STREAM 0 in
let laddr = Unix.inet_addr_of_string p.interface in
Unix.bind sock (ADDR_INET (laddr,0));
Unix.connect sock addr;
let (sock, res) =
let req = Bytes.of_string http_request in
if ssl then begin
Ssl.init ();
let ctx = create_context TLSv1_2 Client_context in
let sock = Ssl.embed_socket sock ctx in
Ssl.connect sock;
(SslSock sock, (write sock req 0 http_request_len))
end else
(UnixSock sock, (Unix.write sock req 0 http_request_len))
WireShark did the trick: this “bad” service sends two websocket frames in one tcp packet where second frame has zero payload length. Naturally, my Websocket implementation improperly handled frames with zero payload which lead to missing of Ping frames and closing of TCP connection by remote server.
I am trying to establish a bluetooth serial communication link between a Raspberry Pi Zero W, running Raspbian Jessie [03-07-2017], and an Arduino (UNO).
I am currently able to write data to the Arduino using bluetoothctl.
The application requires that we are able to write data to a particular BLE Slave. There are multiple [HM-10] Slaves to switch between, the Slave needs to be chosen during the program execution.
There is no BAUD rate preference. Currently, we are using 9600 universally.
Functions have been created that automatically connect and then write data to an "attribute", this shows up as data on the Serial Monitor of the Arduino.
Python Code - using BlueZ 5.44 (manually installed):
import subprocess
from subprocess import Popen, PIPE
# Replaces the ':' with '_' to allow the MacAddress to be in the form
# of a "Path" when "selecting an attribute"
def changeMacAddr(word):
return ''.join(c if c != ':' else '_' for c in word)
# Connects to a given MacAddress and then selects the attribute to write to
def connBT(BTsubProcess, stringMacAddr):
BTsubProcess.stdin.write(bytes("".join("connect "+stringMacAddr +"\n"), "utf-8"))
BTsubProcess.stdin.flush()
time.sleep(2)
stringFormat = changeMacAddr(stringMacAddr)
BTsubProcess.stdin.write(bytes("".join("select-attribute /org/bluez/hci0/dev_"
+ stringFormat +
"/service0010/char0011" + "\n"), "utf-8"))
BTsubProcess.stdin.flush()
# Can only be run once connBT has run - writes the data in a list [must have numbers 0 - 255 ]
def writeBT(BTsubProcess, listOfData):
stringList = [str('{0} ').format(elem) for elem in listOfData]
BTsubProcess.stdin.write(bytes("".join("write " + "".join(stringList) + "\n"), "utf-8"))
BTsubProcess.stdin.flush()
# Disconnects
def clostBT(BTsubProcess):
BTsubProcess.communicate(bytes("disconnect\n", "utf-8"))
# To use the functions a subprocess "instance" of bluetoothctl must be made
blt = subprocess.Popen(["bluetoothctl"], stdin=subprocess.PIPE, shell=True)
# blt with then be passed into the function for BTsubProcess
# Note: the MacAddresses of the Bluetooth modules were pre-connected and trusted manually via bluetoothctl
This method works fine for small sets of data, but my requirements require me to stream data to the Arduino very quickly.
The current set up is:
Sensor data (accelerometer, EEG) via USB serial is received by the Pi
The Pi processes the data
Commands are then sent to the Arduino via the in built bluetooth of the Pi Zero W
However, while using this method the bluetooth data transmission would delay (temporarily freeze) when the sensor data changed.
The data transmission was flawless when using two pre-paired HM-10 modules, the Pi's GPIO serial port was configured using PySerial.
The following methods have also been tried:
Using WiringPi to set-up a bluetooth serial port on the /dev/ttyAMA0
using Python sockets and rfcomm
When attempting to use both of these methods. The Python code compiles, however, once the Serial Port is opened the data is seemingly not written and does not show up on the Arduino's Serial Monitor.
This then cripples the previous functions. Even when using bluetoothctl manually, the module cannot be unpaired/disconnected. Writing to the appropriate attribute does not work either.
A restart is required to regain normal function.
Is this approach correct?
Is there a better way to send data over BLE?
UPDATE: 05/07/2017
I am no longer working on this project. But troubleshooting has led me to believe that a "race condition" in the code may have led to the program not functioning as intended.
This was verified during the testing phase where a more barebones code was created that functioned very well.
netty 4.0.24
I am passing XML over UDP. When receiving the UPD packet, the packet is always of length 2048, truncating the message. Even though, I have attempted to set the receive buffer size to something larger (4096, 8192, 65536) but it is not being honored.
I have verified the UDP sender using another UDP ingest mechanism. A standalone Java app using java.net.DatagramSocket. The XML is around 45k.
I was able to trace the stack to DatagramSocketImpl.createChannel (line 281). Stepping into DatagramChannelConfig, it has a receiveBufferSize of whatever I set (great), but a rcvBufAllocator of 2048.
Does the rcvBufAllocator override the receiveBufferSize (SO_RCVBUF)? Is the message coming in multiple buffers?
Any feedback or alternative solutions would be greatly appreciated.
I also should mention, I am using an ESB called vert.x which uses netty heavily. Since I was able to trace down to netty, I was hopeful that I could find help here.
The maximum size of incoming datagrams copied out of the socket is actually not a socket option, but rather a parameter of the socket read() function that your client passes in each time it wants to read a datagram. One advantage of this interface is that programs accepting datagrams of unknown/varying lengths can adaptively change the size of the memory allocated for incoming datagram copies such that they do not over-allocate memory while still getting the whole datagram. (In netty this allocation/prediction is done by implementors of io.netty.channel.RecvByteBufAllocator.)
In contrast, SO_RCVBUF is the size of a buffer that holds all of the datagrams your client hasn't read yet.
Here's an example of how to configure a UDP service with a fixed max incoming datagram size with netty 4.x using a Bootstrap:
import io.netty.bootstrap.Bootstrap;
import io.netty.channel.ChannelOption;
import io.netty.channel.FixedRecvByteBufAllocator;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioDatagramChannel;
int maxDatagramSize = 4092;
String bindAddr = "0.0.0.0";
int port = 1234;
SimpleChannelInboundHandler<DatagramPacket> handler = . . .;
InetSocketAddress address = new InetSocketAddress(bindAddr, port);
NioEventLoopGroup group = new NioEventLoopGroup();
Bootstrap b = new Bootstrap()
.group(group)
.channel(NioDatagramChannel.class)
.handler(handler);
b.option(ChannelOption.RCVBUF_ALLOCATOR, new FixedRecvByteBufAllocator(maxDatagramSize));
b.bind(address).sync().channel().closeFuture().await();
You could also configure the allocator with ChannelConfig.setRecvByteBufAllocator
I have a rails application running on a server where I added some iptables rules to improve security. Now Omniauth callbacks stopped working. Every time I try to log in with any provider I get this error into my application log
Errno::ENETUNREACH (Network is unreachable - connect(2))
And this dropped package gets logged into syslog
IN=eth0 OUT= MAC=40:40:ea:31:ac:8d:64:00:f1:cd:1f:7f:08:00 SRC=66.220.147.99 DST=my_ip LEN=56 TOS=0x00 PREC=0x00 TTL=88 ID=0 DF PROTO=TCP SPT=443 DPT=37035 WINDOW=14480 RES=0x00 ACK SYN URGP=0
Can someone tell me what that entry in my syslog is about and what kind of iptables rule is needed to allow it.
If needed I could add also the rules I have applied this far.
EDIT:
The syslog line was incorrect, so I replaced it.
The answer to my original question found from http://lists.debian.org/debian-user/2002/07/msg01187.html
IN = interface the packet came in
OUT = interface used for sending the packet
MAC = MAC address for source and destination
SRC = IP of the sender
DST = IP of the receiver
LEN = Length of the packet
TOS = ?
PREC = Precedence
TTL = Time to live (hop count of the package)
ID = Packet ID number
DF = Don't fragment bit
PROTO = The protocol
SPT = Sender port
DPT = Receiving port
WINDOW = ?
RES = Received bits
And then some TCP flags in the end of the row. Didn't yet dig the meaning of those.
ACK = ?
SYN = ?
URGP = ?
I'm implementing a Twisted-based Heartbeat Client/Server combo, based on this example. It is my first Twisted project.
Basically it consists of a UDP Listener (Receiver), who calls a listener method (DetectorService.update) on receiving packages. The DetectorService always holds a list of currently active/inactive clients (I extended the example a lot, but the core is still the same), making it possible to react on clients which seem disconnected for a specified timeout.
This is the source taken from the site:
UDP_PORT = 43278; CHECK_PERIOD = 20; CHECK_TIMEOUT = 15
import time
from twisted.application import internet, service
from twisted.internet import protocol
from twisted.python import log
class Receiver(protocol.DatagramProtocol):
"""Receive UDP packets and log them in the clients dictionary"""
def datagramReceived(self, data, (ip, port)):
if data == 'PyHB':
self.callback(ip)
class DetectorService(internet.TimerService):
"""Detect clients not sending heartbeats for too long"""
def __init__(self):
internet.TimerService.__init__(self, CHECK_PERIOD, self.detect)
self.beats = {}
def update(self, ip):
self.beats[ip] = time.time()
def detect(self):
"""Log a list of clients with heartbeat older than CHECK_TIMEOUT"""
limit = time.time() - CHECK_TIMEOUT
silent = [ip for (ip, ipTime) in self.beats.items() if ipTime < limit]
log.msg('Silent clients: %s' % silent)
application = service.Application('Heartbeat')
# define and link the silent clients' detector service
detectorSvc = DetectorService()
detectorSvc.setServiceParent(application)
# create an instance of the Receiver protocol, and give it the callback
receiver = Receiver()
receiver.callback = detectorSvc.update
# define and link the UDP server service, passing the receiver in
udpServer = internet.UDPServer(UDP_PORT, receiver)
udpServer.setServiceParent(application)
# each service is started automatically by Twisted at launch time
log.msg('Asynchronous heartbeat server listening on port %d\n'
'press Ctrl-C to stop\n' % UDP_PORT)
This heartbeat server runs as a daemon in background.
Now my Problem:
I need to be able to run a script "externally" to print the number of offline/online clients on the console, which the Receiver gathers during his lifetime (self.beats). Like this:
$ pyhb showactiveclients
3 clients online
$ pyhb showofflineclients
1 client offline
So I need to add some kind of additional server (Socket, Tcp, RPC - it doesn't matter. the main point is that i'm able to build a client-script with the above behavior) to my DetectorService, which allows to connect to it from outside. It should just give a response to a request.
This server needs to have access to the internal variables of the running detectorservice instance, so my guess is that I have to extend the DetectorService with some kind of additionalservice.
After some hours of trying to combine the detectorservice with several other services, I still don't have an idea what's the best way to realize that behavior. So I hope that somebody can give me at least the essential hint how to start to solve this problem.
Thanks in advance!!!
I think you already have the general idea of the solution here, since you already applied it to an interaction between Receiver and DetectorService. The idea is for your objects to have references to other objects which let them do what they need to do.
So, consider a web service that responds to requests with a result based on the beats data:
from twisted.web.resource import Resource
class BeatsResource(Resource):
# It has no children, let it respond to the / URL for brevity.
isLeaf = True
def __init__(self, detector):
Resource.__init__(self)
# This is the idea - BeatsResource has a reference to the detector,
# which has the data needed to compute responses.
self._detector = detector
def render_GET(self, request):
limit = time.time() - CHECK_TIMEOUT
# Here, use that data.
beats = self._detector.beats
silent = [ip for (ip, ipTime) in beats.items() if ipTime < limit]
request.setHeader('content-type', 'text/plain')
return "%d silent clients" % (len(silent),)
# Integrate this into the existing application
application = service.Application('Heartbeat')
detectorSvc = DetectorService()
detectorSvc.setServiceParent(application)
.
.
.
from twisted.web.server import Site
from twisted.application.internet import TCPServer
# The other half of the idea - make sure to give the resource that reference
# it needs.
root = BeatsResource(detectorSvc)
TCPServer(8080, Site(root)).setServiceParent(application)