Good Morning,
I have a Zeek machine generating logs on a Modbus traffic.
Currently, my script generates logs looking like this :
ts tid id.orig_h id.orig_p id.resp_h id.resp_p unit_id func network_direction
1342774501.072580 32 10.2.2.2 51411 10.2.2.3 502 255 READ_HOLDING_REGISTERS request
1342774501.087014 32 10.2.2.2 51411 10.2.2.3 502 255 READ_HOLDING_REGISTERS response
'tid' is the transaction id to identify a request/response couple. I want to know if a robot hasn't responded to the Controller by logging only requests that did not have a response within 1 second.
My code is :
module Modbus_Extended;
export {
redef enum Log::ID += { LOG_DETAILED,
LOG_MASK_WRITE_REGISTER,
LOG_READ_WRITE_MULTIPLE_REGISTERS};
type Modbus_Detailed: record {
ts : time &log; # Timestamp of event
tid : count &log; # Zeek unique ID for connection
id : conn_id &log; # Zeek connection struct (addresses and ports)
unit_id : count &log; # Modbus unit-id
func : string &log &optional; # Modbus Function
network_direction : string &log &optional; # Message direction (request or response)
address : count &log &optional; # Starting address for value(s) field
quantity : count &log &optional; # Number of addresses/values read or written to
values : string &log &optional; # Coils, discrete_inputs, or registers read/written to
};
global log_modbus_detailed: event(rec: Modbus_Detailed);
global transaction_ids: set[string, string] = {};
event modbus_message(c: connection,
headers: ModbusHeaders,
is_orig: bool) &priority=-5 {
local modbus_detailed_rec: Modbus_Detailed;
if(headers$tid !in transaction_ids[count]){
add transaction_ids[headers$tid, c$modbus$ts]
}else{
delete transaction_ids[headers$tid, c$modbus$ts]
}
for(i in transaction_ids[timestamp]){
if(c$modbus$ts > transactions_ids[headers$tid, i] +1)
{
Log::write(LOG_DETAILED, modbus_detailed_rec);
}
}
}
}
My guess is that I have to store transaction ids and check if I get only one occurence within this timelapse and then log it into a file, but I can figure out how to do it. Currently I can only generate logs with all the modbus traffic.
Thank you for your help
Related
I'm using RabbitMQ. I've defined a queue with priority, and I can send messages to this queue with some priority value using RMQ GUI, and consumers also get the messages in sorted order, but when I try to send the message from my java code using Stream bridge, I don't know how to specify the priority with the message.
Here's what I have tried :
I have added x-max-priority: 10 to the queue while creating the queue.
Consumer example =
#Bean
public Consumer<Message<String>> testListener() {
return (m) -> {
System.out.println("inside consumer with message : " + m);
System.out.println("headers : " + m.getHeaders());
System.out.println("payload : " + m.getPayload());
};
}
Producer example =
#GET
#Path("test/")
public void test(#Context HttpServletRequest request) {
System.out.println("inside test");
try {
String payload = "hello world";
logger.info("going to send a message : {}", payload);
int priority = 5;
Message<String> message = MessageBuilder.withPayload(payload)
.setHeader("priority", priority)
.build();
boolean res = STREAM_BRIDGE.send("testWriter-out-0", message);
System.out.println(message);
System.out.println(res);
} catch (Exception e) {
logger.error(e);
}
}
The output of the Producer =
-> inside test
-> GenericMessage [payload=hello world, headers={priority=5, id=some_id, timestamp=epoch}]
-> true
The output of the Consumer =
-> inside consumer with message : GenericMessage [payload=hello world, headers={amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=test_exchange, amqp_deliveryTag=1, deliveryAttempt=1, amqp_consumerQueue=test_exchange.ats, amqp_redelivered=false, amqp_receivedRoutingKey=test_exchange, amqp_timestamp=date_time, amqp_messageId=some_id, id=some_id, amqp_consumerTag=some_tag, sourceData=(Body:'hello world' MessageProperties [headers={}, timestamp=date_time, messageId=some_id, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=test_exchange, receivedRoutingKey=test_exchange, deliveryTag=1, consumerTag=some_tag, consumerQueue=test_exchange.ats]), contentType=application/json, timestamp=epoch}]
-> headers : {amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=test_exchange, amqp_deliveryTag=1, deliveryAttempt=1, amqp_consumerQueue=test_exchange.ats, amqp_redelivered=false, amqp_receivedRoutingKey=test_exchange, amqp_timestamp=date_time, amqp_messageId=some_id, id=some_id, amqp_consumerTag=tag, sourceData=(Body:'hello world' MessageProperties [headers={}, timestamp=date_time, messageId=some_id, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=test_exchange, receivedRoutingKey=test_exchange, deliveryTag=1, consumerTag=tag, consumerQueue=test_exchange.ats]), contentType=application/json, timestamp=epoch}
-> payload : hello world
So the message goes to RMQ and the consumer also gets the message, but on RMQ GUI when I perform Get-message operation on the Queue, I get this result =>
Message 1
The server reported 0 messages remaining.
Exchange test_exchange
Routing Key test_exchange
Redelivered ○
Properties
timestamp: timestamp
message_id: some_id
priority: 0
delivery_mode: 2
headers:
content_type: application/json
Payload hello world
11 bytes
Encoding: string
As we can see in the above result, priority is set to 0 by RMQ (and hence in the Consumer, I get the messages in the FIFO manner, not in a priority-based manner) and inside headers : only one header is present "content_type: application/json", so I think the priority is not a part of the header but is a part of properties, then how to set message properties using StreamBridge?
To conclude, I am trying to figure out how to set the priority of a message dynamically while sending it using StreamBridge, any help would be appreciated, thanks in advance !
Please, consider to use the latest Spring Cloud Stream: https://spring.io/projects/spring-cloud-stream#learn.
Apparently your spring-cloud-starter-stream-rabbit = 3.0.3.RELEASE is old enough to suffer from the issue https://github.com/spring-cloud/spring-cloud-stream/issues/1931.
Have just tested with the latest one and I got the proper priority property on the message posted into RabbitMQ queue by the mentioned StreamBridge.
I'm trying to write a Zeek script to divide the dns traffic into two log files (query and reply)
The error is "Field missing value" for the code $TTL=c$dns$TTLs in dns_query_reply event.
I don't understand the reason for this error since the dns.log file correctly contains the value.
The code is as follows:
module DnsFeatureExtractor;
export{
redef enum Log::ID += {QueryDNS};
redef enum Log::ID += {ReplyDNS};
type InfoQuery: record{
uid: string &log;
id: conn_id &log;
domain: string &log &optional;
query_type: string &log &optional;
timestamp: time &log;
};
type InfoReply: record{
uid: string &log;
id: conn_id &log;
response_code: count &log &optional;
TTL: vector of interval &log &optional;
resolved_IP: vector of string &log &optional;
timestamp: time &log;
};
}
event zeek_init(){
Log::create_stream(QueryDNS, [$columns=InfoQuery, $path="QueryDNS"]);
Log::create_stream(ReplyDNS, [$columns=InfoReply, $path="ReplyDNS"]);
}
event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count){
local name_qtype = DNS::query_types[qtype];
local rec: DnsFeatureExtractor::InfoQuery = [$uid=c$uid, $id=c$id, $domain=query, $query_type=name_qtype, $timestamp=c$start_time];
Log::write(DnsFeatureExtractor::QueryDNS, rec);
}
event dns_query_reply(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count){
local rec_r: DnsFeatureExtractor::InfoReply = [$uid=c$uid, $id=c$id, $response_code=msg$rcode, $TTL=c$dns$TTLs, $resolved_IP=c$dns$answers, $timestamp=c$start_time];
Log::write(DnsFeatureExtractor::ReplyDNS, rec_r);
}
This one's subtle: you're seeing an ordering problem. The dns_query_reply event handler is invoked before that of the event that actually populates the TTLs (such as dns_A_reply).
The c$dns fields in question get populated in the DNS::do_reply hook, so your best bet is to ensure you add to the hook. If you use the default priority, it'll run after the one that populates the fields. Try this instead of the dns_query_reply handler:
hook DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string) {
local rec_r: DnsFeatureExtractor::InfoReply = [$uid=c$uid, $id=c$id, $response_code=msg$rcode, $TTL=c$dns$TTLs, $resolved_IP=c$dns$answers, $timestamp=c$start_time];
Log::write(DnsFeatureExtractor::ReplyDNS, rec_r);
}
Note that you're still dealing with optional record values, so it's possible that despite the above you still don't see all field values. To guard against that, you could check whether the optional values are actually defined, and copy them over only in that case:
hook DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string)
{
local rec_r: DnsFeatureExtractor::InfoReply = [$uid=c$uid, $id=c$id, $response_code=msg$rcode, $timestamp=c$start_time];
if ( c$dns?$TTLs )
rec_r$TTL = c$dns$TTLs;
if ( c$dns?$answers )
rec_r$resolved_IP = c$dns$answers;
Log::write(DnsFeatureExtractor::ReplyDNS, rec_r);
}
I am trying to read a file with a list of IP addresses and another one with domains, as a proof of concept of the Input Framework defined in https://docs.zeek.org/en/stable/frameworks/input.html
I´ve prepared the following bro scripts:
reading.bro:
type Idx: record {
ip: addr;
};
type Idx: record {
domain: string;
};
global ips: table[addr] of Idx = table();
global domains: table[string] of Idx = table();
event bro_init() {
Input::add_table([$source="read_ip_bro", $name="ips",
$idx=Idx, $destination=ips, $mode=Input::REREAD]);
Input::add_table([$source="read_domain_bro", $name="domains",
$idx=Idx, $destination=domains, $mode=Input::REREAD]);
Input::remove("ips");
Input::remove("domains");
}
And the bad_ip.bro script, which check if an IP is in the blacklist, which loads the previous one:
bad_ip.bro
#load reading.bro
module HTTP;
event http_reply(c: connection, version: string, code: count, reason: string)
{
if ( c$id$orig_h in ips )
print fmt("A malicious IP is connecting: %s", c$id$orig_h);
}
However, when I run bro, I get the error:
error: Input stream ips: Table type does not match index type. Need type 'string':string, got 'addr':addr
Segmentation fault (core dumped)
You cannot assign a string type to an addr type. In order to do so, you must use the utility function to_addr(). Of course, it would be wise to verify that that string contains a valid addr first. For example:
if(is_valid_ip(inputString){
inputAddr = to_addr(inputString)
} else { print "addr expected, got a string"; }
The client program below receives messages from a WebSocket server.
It doesn't send any messages.
CLIENT
use v6;
use Cro::WebSocket::Client;
constant WS-URL = 'ws://localhost:20000/status';
constant TIMEOUT-TO-CONNECT = 5; # seconds
my $timeout;
my $connection-attempt;
await Promise.anyof(
$connection-attempt = Cro::WebSocket::Client.connect(WS-URL),
$timeout = Promise.in(TIMEOUT-TO-CONNECT));
if $timeout.status == Kept
{
say "* could not connect to server in ', TIMEOUT-TO-CONNECT, ' seconds";
exit 1;
}
if $connection-attempt.status != Kept
{
my $cause = $connection-attempt.cause;
say '"* error when trying to connect to server';
say '"* --------------------------------------';
# $cause is a long string, how do we get a simple numeric code ?
say $cause;
say '"* ======================================';
exit 1;
}
my $connection = $connection-attempt.result;
my $peer = 'localhost:20000';
say '* connected with ', $peer;
react
{
whenever $connection.messages -> $message
{
my $body = await $message.body;
say '* received message=[' ~ $body ~ '] from server';
LAST { say '* LAST'; done; }
QUIT { default { say '* QUIT'; done; }}
}
CLOSE { say '* CLOSE: leaving react block';}
} # react
SERVER
use Cro::HTTP::Router;
use Cro::HTTP::Server;
use Cro::HTTP::Router::WebSocket;
my $application =
route
{
get -> 'status'
{
web-socket -> $incoming
{
my $counter = 0;
my $timer = Supply.interval(1);
supply
{
whenever $incoming -> $thing
{
LAST { note '* LAST: client connection was closed'; done; }
QUIT { default { note '* QUIT: error in client connection'; done; } }
}
whenever $timer
{
$counter++;
say '* sending message ', $counter;
emit $counter.Str;
}
CLOSE { say '* CLOSE: leaving supply block'; }
} # supply
} #incoming
} # get -> status
}
my $server = Cro::HTTP::Server.new: :port(20000), :$application;
$server.start;
say '* serving on port 20000';
react whenever signal(SIGINT)
{
$server.stop;
exit;
}
Now, when the server goes out (say, by Ctrl+C) the client sees nothing.
Setting CRO_TRACE=1 in the client gives this:
TRACE(anon 2)] Cro::WebSocket::MessageParser EMIT WebSocket Message - Text
* received message=[4] from server
[TRACE(anon 1)] Cro::TCP::Connector DONE
[TRACE(anon 2)] Cro::WebSocket::FrameParser DONE
[TRACE(anon 2)] Cro::WebSocket::MessageParser DONE
[TRACE(anon 1)] Cro::HTTP::ResponseParser DONE
^C
The client showed nothing more (and then I cancelled it).
So, the question is: how should the client deal with this scenario ?
UPDATE
Edited the question, now showing the server code
Also, I'm in Fedora 28.
When I first cancel the server, netstat shows
$ netstat -ant | grep 20000
tcp6 0 0 ::1:20000 ::1:56652 TIME_WAIT
$
Tcpdump shows
IP6 ::1.20000 > ::1.56652: Flags [F.], seq 145, ack 194, win 350, options [nop,nop,TS val 1476681452 ecr 1476680552], length 0
IP6 ::1.56652 > ::1.20000: Flags [F.], seq 194, ack 146, win 350, options [nop,nop,TS val 1476681453 ecr 1476681452], length 0
IP6 ::1.20000 > ::1.56652: Flags [.], ack 195, win 350, options [nop,nop,TS val 1476681453 ecr 1476681453], length 0
It seems the last ACK from the client to the server is missing, I guess the client didn't close the connection.
Also, I'm curious as to why Cro chooses to work with IPv6 by default.
This is a bug that has been fixed since this question was posted, but I'm leaving an answer because of this part of the question that may confuse people when dealing with networking in Raku:
Also, I'm curious as to why Cro chooses to work with IPv6 by default.
localhost will resolve to an IPv6 address first if that's what the first address for localhost in your hosts file is. As of writing, IO::Socket::Async (which Cro uses internally) only allows PF_UNSPEC to be specified as a family, and the only address that will ever used from the results of hostname resolution is the first one in the list of addresses received. This will be changed at some point in the future as part of the work for my IP6NS grant and a problem solving issue to improve how DNS is handled, but for now, if you want to use IPv4/IPv6 only, you should specify 127.0.0.1/::1 instead of using localhost (or whichever addresses your machine resolves it to if they're different).
We have a typical web-service which serves JSON data read from a remote database. I was trying out returning Result and AsyncResult, each with the following configuration:
play {
akka {
event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
loglevel = WARNING
actor {
default-dispatcher = {
fork-join-executor {
parallelism-factor = 1.0
parallelism-max = 1
}
}
}
}
}
and one with
parallelism-factor = 1.0
parallelism-max = 5
Following are observations where the time taken to complete 500 requests is given (average of 5 readings):
1. parallelism-max=1 and parallelism-factor=1.0
Result :
Completion time = 291662 ms.
AsyncResult:
Completion time = 55601 ms
2. parallelism-max=5 and parallelism-factor=1.0
Result :
Completion time = 46419 ms.
AsyncResult:
Completion time = 46977 ms
We can see that with parallelism-max=1, AsyncResult clearly takes very less time compare to Result. However, with parallelism-max=5, Result and AsyncResult give very similar timings.
Shouldn't the time required become less as the number of threads increases, for AsyncResult also ?
Requesting for help to understand the reasons behind this observation.