Parse XML stream from TCP socket - ruby-on-rails-3

I wrote a Ruby script to connect to an XML stream over a TCP socket. I want to parse this XML stream using LibXML). My problem is that I can't figure out how to pass this stream to LibXML. From the LibXML documentation XML::Document.io(io) this seems to be setup for HTTP, not a TCP socket. Any help on how to wrap my TCP socket so it can be used by LibXML? Or is there a better way to do parse this XML stream from my TCP socket? I'm learning Ruby now, so the code below isn't pretty in the way I authenticated either. Based on other questions on stackoverflow, it seems I will have to ensure that I have complete XML (proper closing tags) before sending that to the LibXML parser. Thanks in advance.
Here's what I have so far:
require 'socket'
require 'timeout'
username = "username\r\n"
password = "password\r\n"
port = 4500
server = 'xmlfeed.website.com'
tcp_client = TCPSocket.new(server, port)
all_data = []
value = 0
while true
partial_data = tcp_client.recv(1012)
if partial_data.length == 0
break
end
all_data << partial_data
puts(all_data)
if value == 0
tcp_client.puts(username)
value = 1
end
if value == 1
tcp_client.puts(password)
value = 2
end
if all_data.to_s.chomp.casecmp( "Starting Feed")
value = 3
puts "Begin the feed"
end
if value == 3
# Parse the data using LibXML
# this is the part I'm not sure how to approach
puts(all_data)
end
end
tcp_client.close
puts all_data.join()

Related

How to use DCCP with twisted ? (Datagram Congestion Control Protocol)

At the interface level DCCP is like TCP: you connect and then send/receive bytes.
I was wondering it's possible to make dccp connections in twisted by just adapting the wrappers for tcp...
According to the sample code (below) what needs to be changed is:
at socket instantiation: use different parameters
before using the socket: set some options
Then everything else would be the same...
Hints: I've spotted addressFamily and socketType in the sources of twisted but I have no idea on how to cleanly set them in the protocol factory. Also the protocol number, the 3rd parameter, here IPPROTO_DCCP, is always keeped to default. I have no clue either on how to access the socket to call setsockopt
import socket
socket.DCCP_SOCKOPT_PACKET_SIZE = 1
socket.DCCP_SOCKOPT_SERVICE = 2
socket.SOCK_DCCP = 6
socket.IPPROTO_DCCP = 33
socket.SOL_DCCP = 269
packet_size = 256
address = (socket.gethostname(),12345)
# Create sockets
server,client = [socket.socket(socket.AF_INET, socket.SOCK_DCCP,
socket.IPPROTO_DCCP) for i in range(2)]
for s in (server,client):
s.setsockopt(socket.SOL_DCCP, socket.DCCP_SOCKOPT_PACKET_SIZE, packet_size)
s.setsockopt(socket.SOL_DCCP, socket.DCCP_SOCKOPT_SERVICE, True)
# Connect sockets
server.bind(address)
server.listen(1)
client.connect(address)
s,a = server.accept()
# Echo
while True:
client.send(raw_input("IN: "))
print "OUT:", s.recv(1024)
More about DCCP:
https://www.sjero.net/research/dccp/
https://wiki.linuxfoundation.org/networking/dccp
TL;DR: dccp is a protocol that provides congestion control (like tcp) without guaranteeing reliability or in-order delivery of data (like udp). The standard linux kernel implements dccp.

BPF/Wincap filter not working in Pcap.net

I'm using a pcap.net process to broadcast out a packet, and listen in for a response. I want the response packets that it accepts to be a certain length. However, whenever I add a filter it only returns packets of length 100. Here's the code:
' Open the output device
Dim Communicator As PacketCommunicator = selectedOutputDevice.Open(100, PacketDeviceOpenAttributes.Promiscuous, 300)
Communicator.SetFilter("len >= 300")
' send broadcast packet
Communicator.SendPacket(BuildEthernetPacket(ReqInfoPkt, "ff:ff:ff:ff:ff:ff"))
' receive packets
While result = PacketCommunicatorReceiveResult.Ok And packetlist.Count < 500 And _
Not result = PacketCommunicatorReceiveResult.Timeout
result = Communicator.ReceivePacket(packet)
packetlist.Add(packet)
End While
Here's a link to the screen shot: link I tried asking on the pcap forums but didn't get a response.
So what am I doing wrong? Is there a synatx issue?
You open the device with the snapshot length set to 100 which means you cut all packets to be at most 100 bytes.
You should probably do
selectedOutputDevice.Open(65536, PacketDeviceOpenAttributes.Promiscuous, 300)
instead.
See the tutorial.

How to communicate with HiveServer2 running NONE authentication

I'm trying to communicate with HiveServer2 via ruby TCPSocket. As per Thrift SASL spec, I send START message and then plain auth information.
Server returns COMPLETE status with an empty payload. It should return challenge as a payload but an empty string.
START = 0x01
OK = 0x02
COMPLETE = 0x05
auth = 'PLAIN'
header = [START, auth.length].pack('cl>')
auth_string = ['anonymous'].pack('u')
auth_message = "[LOGIN] \u0000 #{auth_string} \u0000 #{auth_string}"
auth_header = [OK, auth_message.length].pack('cl>')
socket = TCPSocket.new localhost, 10000
socket.write header + auth
socket.write auth_header + auth_message
socket.read(5).unpack('cl>')
=> [5,0]
HiveServer2 returns 5 status that is COMPLETE.
No further communication is possible via this socket as the server returns nothing anymore.
I suspect auth_message constructed in a wrong way or something else is wrong.
Can anyone suggest the way HiveServer2 will understand my requests?
Any help will be appreciated.
UPD: Thrift SASL spec
UPD2: Solved! STARTTLS block should look like following below:
START = 0x01
OK = 0x02
COMPLETE = 0x05
auth = 'PLAIN'
header = [START, auth.length].pack('cl>')
auth_message = "[ANONYMOUS]\u0000anonymous\u0000anonymous"
auth_header = [OK, auth_message.length].pack('cl>')
socket = TCPSocket.new localhost, 10000
socket.write header + auth
socket.write auth_header + auth_message
socket.read(5).unpack('cl>')
=> [5,0]
After COMPLETE status received from the server, I can use TCLIService::Client to communicate with the HiveServer2. Only one thing to notice:
All writes to the underlying transport must be prefixed by the 4-byte length of the payload data, followed by the payload. All reads from this transport
should read the 4-byte length word, then read the full quantity of bytes
specified by this length word.
Try to use thrift gem and consider https://github.com/dallasmarlow/hiveserver2 instead of Ruby sockets.

How do TLS connections in EventMachine work?

I have a custom Protobuf-based protocol that I've implemented as an EventMachine protocol and I'd like to use it over a secure connection between the server and clients. Each time I send a message from a client to the server, I prepend the message with a 4-byte integer representing the size of the Protobuf serialized string to be sent such that the server knows how many bytes to read off the wire before parsing the data back into a Protobuf message.
I'm calling start_tls in the post_init callback method in both the client and server protocol handlers, with the one in the server handler being passed the server's private key and certificate. There seems to be no errors happening at this stage, based on log messages I'm printing out.
Where I get into trouble is when I begin parsing data in the receive_data callback in the server's handler code... I read 4 bytes of data off the wire and unpack it to an integer, but the integer that gets unpacked is not the same integer I send from the client (i.e. I'm sending 17, but receiving 134222349).
Note that this does not happen when I don't use TLS... everything works fine if I remove the start_tls calls in both the client and server code.
Is it the case that SSL/TLS data gets passed to the receive_data callback when TLS is used? If so, how do I know when data from the client begins? I can't seem to find any example code that discusses this use case...
OK, so via a cross-post to the EventMachine Google Group I figured out what my problem was here. Essentially, I was trying to send data from the client to the server before the TLS handshake was done because I wasn't waiting until the ssl_handshake_completed callback was called.
Here's the code I got to work, just in case anyone comes across this post in the future. :)
Handler code for the server-side:
require 'eventmachine'
class ServerHandler < EM::Connection
def post_init
start_tls :private_key_file => 'server.key', :cert_chain_file => 'server.crt', :verify_peer => false
end
def receive_data(data)
puts "Received data in server: #{data}"
send_data(data)
end
end
Handler code for the client-side:
require 'eventmachine'
class ClientHandler < EM::Connection
def connection_completed
start_tls
end
def receive_data(data)
puts "Received data in client: #{data}"
end
def ssl_handshake_completed
send_data('Hello World! - 12345')
end
end
Code to start server:
EventMachine.run do
puts 'Starting server...'
EventMachine.start_server('127.0.0.1', 45123, ServerHandler)
end
Code to start client:
EventMachine.run do
puts 'Starting client...'
EventMachine.connect('127.0.0.1', 45123, ClientHandler)
end

What this iptables log entry is about?

I have a rails application running on a server where I added some iptables rules to improve security. Now Omniauth callbacks stopped working. Every time I try to log in with any provider I get this error into my application log
Errno::ENETUNREACH (Network is unreachable - connect(2))
And this dropped package gets logged into syslog
IN=eth0 OUT= MAC=40:40:ea:31:ac:8d:64:00:f1:cd:1f:7f:08:00 SRC=66.220.147.99 DST=my_ip LEN=56 TOS=0x00 PREC=0x00 TTL=88 ID=0 DF PROTO=TCP SPT=443 DPT=37035 WINDOW=14480 RES=0x00 ACK SYN URGP=0
Can someone tell me what that entry in my syslog is about and what kind of iptables rule is needed to allow it.
If needed I could add also the rules I have applied this far.
EDIT:
The syslog line was incorrect, so I replaced it.
The answer to my original question found from http://lists.debian.org/debian-user/2002/07/msg01187.html
IN = interface the packet came in
OUT = interface used for sending the packet
MAC = MAC address for source and destination
SRC = IP of the sender
DST = IP of the receiver
LEN = Length of the packet
TOS = ?
PREC = Precedence
TTL = Time to live (hop count of the package)
ID = Packet ID number
DF = Don't fragment bit
PROTO = The protocol
SPT = Sender port
DPT = Receiving port
WINDOW = ?
RES = Received bits
And then some TCP flags in the end of the row. Didn't yet dig the meaning of those.
ACK = ?
SYN = ?
URGP = ?