Syslog-NG: 2 logs from the same source written differently - splunk

I have 2 sets of logs. Each is going to their own syslog server. But the source of the logs is the same - a palo alto prisma vpn.
For whatever reason, Syslog-Server A (the oldest source) writes the logs like this (in bold):
Nov 22 15:08:03 34 456
But my newest Syslog Server, B, writes the logs like this (in bold):
Nov 22 15:08:03 34.0.0.1 456
This is a problem. Because, on each syslog-ng server, we have a Splunk Universal Forwarder that forwards the logs to an index. We use the Palo Tech Add-On to parse the data.
It appears the TA is expected to parse the incoming data with this field:
Nov 22 15:08:03 34 456
Any other way breaks parsing. I have my new syslog-ng file (for the new server - B) written as below:
#version:3.31
#include "scl.conf"
options {
flush_lines (0);
time_reopen (10);
log_fifo_size (1000);
chain_hostnames (off);
use_dns (no); #was yes
use_fqdn (no); #was yes
create_dirs (no);
keep_hostname (no); #was yes
};
source vpn_encrypted_log_traffic {
network(
ip(0.0.0.0)
port(6514)
transport("tls")
tls(
cert-file("/etc/syslog-ng/certs/prv.cer")
key-file("/etc/syslog-ng/certs/prv.key")
peer_verify(optional-untrusted)
)
);
};
destination prisma{ file("/directory/log.log") create_dir(yes) ); }
log { source(vpn_encrypted_log_traffic); destination(prisma); };
And the old syslog server (A) just has this:
#version:3.5
#include "scl.conf"
options {
time-reap (30);
keep_hostname (no); #was yes
};
source vpn_encrypted_log_traffic {
network(
ip(0.0.0.0)
port(6514)
transport("tls")
tls(
cert-file("/etc/syslog-ng/certs/prv.cer")
key-file("/etc/syslog-ng/certs/prv.key")
peer_verify(optional-untrusted)
)
);
};
destination prisma{ file("/directory/log.log") create_dir(yes) ); }
log { source(vpn_encrypted_log_traffic); destination(prisma); };
I can only think the problem exists in Prisma. But the configs look 1-to-1 to me.

34.0.0.1 is the hostname/IP address part of a BSD syslog message.
use_dns(no); keep_hostname(no); means this part of the message will be replaced with server B's IP address.
keep_hostname(yes) can be used to leave the hostname intact.

Related

Syslog-ng row message required to send- no timestamp - no header require

I am using below configuration of syslog-ng OS. Our purpose is to get the syslog message from device and relay the same message to analytic tool. We want to have row log message as shown below , to be sent to analytic tool without removing any character (i.e. ',") from original message. providing configuration file , original log and processed log (by syslog-ng). We also want to get rid of additional header or timestamp added by syslog-ng.
Configuration file
Used version:- Version: 3.2.5
options {flush_lines (0);time_reopen (10);log_fifo_size (1000);long_hostnames (off);use_dns (no); use_fqdn (no);create_dirs (no);keep_hostname (yes);keep-timestamp(no);};
source slocal{syslog(port(514) transport("udp")flags(no-parse) );};
template t_syslog {template("${MESSAGE}\n");template-escape(yes);};
destination dfgtall { file("/var/netwitness/fgtall.log" template(t_syslog)); };
log { source(slocal);destination(dfgtall); };
Original log
date=2020-03-07 time=20:46:02 devname="ABCD" devid="FGT" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="VDOM-Int" eventtime=1583594162 srcip=1.1.1.1 srcport=55498 srcintf="LAN" srcintfrole="lan" dstip=10.10.10.1 dstport=21 dstintf="EXTERNAL" dstintfrole="wan" sessionid=583411984 proto=6 action="deny" policyid=0 policytype="policy" service="FTP" dstcountry="United States" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high"
Received log message
<5>Jul 20 14:41:42 root: date=2020-03-07 time=20:46:02 devname=ABCD devid=FGT logid=0000000013 type=traffic subtype=forward level=notice vd=VDOM-Int eventtime=1583594162 srcip=1.1.1.1 srcport=55498 srcintf=LAN srcintfrole=lan dstip=10.10.10.1 dstport=21 dstintf=EXTERNAL dstintfrole=wan sessionid=583411984 proto=6 action=deny policyid=0 policytype=policy service=FTP dstcountry=United States srccountry=Reserved trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat=unscanned crscore=30 craction=131072 crlevel=high
syslog-ng v3.2.5 is really old. Please upgrade to a newer version.
Using flags(no-parse) in the source, and the proper template in the destination config ($MESSAGE\n) are the key here.
The following snippet works as expected with syslog-ng v3.28:
source s_udp {
syslog(
port(514)
transport("udp")
flags(no-parse)
);
};
destination dfgtall { file("/tmp/fgtall.log" template("${MESSAGE}\n")); };
log {
source(s_udp);
destination(dfgtall);
};

My output filter is not working on (apache)Oracle http server 12.1.3

I have written a customized Apache module with outputfilter. This is for OHS 12.1.3 SPARC 64bit. Upon LoadModule module get loaded and logs in FilterRegisterHooks get generated. But After accessing the web pages output filter never gets called.
This works fine on Apache on other machines.
//This log get generated on LoadModule
static void myFilterRegisterHooks(apr_pool_t *p)
{
apr_file_t *filetest = NULL;
apr_status_t rv;
ap_register_output_filter(Out_myFilterName, myFilterOutFilter, NULL, AP_FTYPE_RESOURCE);
{
rv = apr_file_open(&filetest, FILEDIRE, APR_FOPEN_WRITE | APR_FOPEN_CREATE | APR_FOPEN_APPEND, 0, p);
rv = apr_file_printf(filetest, "\nMessage --> %s", "Inside myFilterRegisterHooks");
apr_file_close(filetest);
}
}
But on accessing website below log never get generated
static apr_status_t myFilterOutFilter(ap_filter_t *f, apr_bucket_brigade *bbIn)
{
apr_file_t *filetest123 = NULL;
apr_status_t rv123;
{
rv123 = apr_file_open(&filetest123, FILEDIRE, APR_FOPEN_WRITE | APR_FOPEN_CREATE | APR_FOPEN_APPEND, 0, f->r->pool);
rv123 = apr_file_printf(filetest123, "\nMessage --> %s", "##### Inside myFilterOutFilter #####");
apr_file_close(filetest123);
}
{
apr_bucket *bcktIn;
//remaining code...........
I tried giving directive "AllowOverride All" in perticular directory.
Checked all error log files no any error log generated for this.
Please let me know if something is not configured properly.

Cro WebSocket client doesn't see when the server goes out

The client program below receives messages from a WebSocket server.
It doesn't send any messages.
CLIENT
use v6;
use Cro::WebSocket::Client;
constant WS-URL = 'ws://localhost:20000/status';
constant TIMEOUT-TO-CONNECT = 5; # seconds
my $timeout;
my $connection-attempt;
await Promise.anyof(
$connection-attempt = Cro::WebSocket::Client.connect(WS-URL),
$timeout = Promise.in(TIMEOUT-TO-CONNECT));
if $timeout.status == Kept
{
say "* could not connect to server in ', TIMEOUT-TO-CONNECT, ' seconds";
exit 1;
}
if $connection-attempt.status != Kept
{
my $cause = $connection-attempt.cause;
say '"* error when trying to connect to server';
say '"* --------------------------------------';
# $cause is a long string, how do we get a simple numeric code ?
say $cause;
say '"* ======================================';
exit 1;
}
my $connection = $connection-attempt.result;
my $peer = 'localhost:20000';
say '* connected with ', $peer;
react
{
whenever $connection.messages -> $message
{
my $body = await $message.body;
say '* received message=[' ~ $body ~ '] from server';
LAST { say '* LAST'; done; }
QUIT { default { say '* QUIT'; done; }}
}
CLOSE { say '* CLOSE: leaving react block';}
} # react
SERVER
use Cro::HTTP::Router;
use Cro::HTTP::Server;
use Cro::HTTP::Router::WebSocket;
my $application =
route
{
get -> 'status'
{
web-socket -> $incoming
{
my $counter = 0;
my $timer = Supply.interval(1);
supply
{
whenever $incoming -> $thing
{
LAST { note '* LAST: client connection was closed'; done; }
QUIT { default { note '* QUIT: error in client connection'; done; } }
}
whenever $timer
{
$counter++;
say '* sending message ', $counter;
emit $counter.Str;
}
CLOSE { say '* CLOSE: leaving supply block'; }
} # supply
} #incoming
} # get -> status
}
my $server = Cro::HTTP::Server.new: :port(20000), :$application;
$server.start;
say '* serving on port 20000';
react whenever signal(SIGINT)
{
$server.stop;
exit;
}
Now, when the server goes out (say, by Ctrl+C) the client sees nothing.
Setting CRO_TRACE=1 in the client gives this:
TRACE(anon 2)] Cro::WebSocket::MessageParser EMIT WebSocket Message - Text
* received message=[4] from server
[TRACE(anon 1)] Cro::TCP::Connector DONE
[TRACE(anon 2)] Cro::WebSocket::FrameParser DONE
[TRACE(anon 2)] Cro::WebSocket::MessageParser DONE
[TRACE(anon 1)] Cro::HTTP::ResponseParser DONE
^C
The client showed nothing more (and then I cancelled it).
So, the question is: how should the client deal with this scenario ?
UPDATE
Edited the question, now showing the server code
Also, I'm in Fedora 28.
When I first cancel the server, netstat shows
$ netstat -ant | grep 20000
tcp6 0 0 ::1:20000 ::1:56652 TIME_WAIT
$
Tcpdump shows
IP6 ::1.20000 > ::1.56652: Flags [F.], seq 145, ack 194, win 350, options [nop,nop,TS val 1476681452 ecr 1476680552], length 0
IP6 ::1.56652 > ::1.20000: Flags [F.], seq 194, ack 146, win 350, options [nop,nop,TS val 1476681453 ecr 1476681452], length 0
IP6 ::1.20000 > ::1.56652: Flags [.], ack 195, win 350, options [nop,nop,TS val 1476681453 ecr 1476681453], length 0
It seems the last ACK from the client to the server is missing, I guess the client didn't close the connection.
Also, I'm curious as to why Cro chooses to work with IPv6 by default.
This is a bug that has been fixed since this question was posted, but I'm leaving an answer because of this part of the question that may confuse people when dealing with networking in Raku:
Also, I'm curious as to why Cro chooses to work with IPv6 by default.
localhost will resolve to an IPv6 address first if that's what the first address for localhost in your hosts file is. As of writing, IO::Socket::Async (which Cro uses internally) only allows PF_UNSPEC to be specified as a family, and the only address that will ever used from the results of hostname resolution is the first one in the list of addresses received. This will be changed at some point in the future as part of the work for my IP6NS grant and a problem solving issue to improve how DNS is handled, but for now, if you want to use IPv4/IPv6 only, you should specify 127.0.0.1/::1 instead of using localhost (or whichever addresses your machine resolves it to if they're different).

"TLS support is not available" when creating GTlsClientConnection with libnice

I have working code where two peers are connecting over a relay server (coturn) and everything seems to be fine over pseudo-tcp. I've tested message exchange successfully with nice_agent_attach_recv() and nice_agent_get_io_stream().
But when I try to create a GTlsClientConnection I get back: 0:TLS support is not available
Here is some partial code:
if(!nice_agent_set_relay_info(agent, stream_id,
NICE_COMPONENT_TYPE_RTP,
"my.coturn.server",
5349, //tls-listener-port (I also tried the non tls port: 3478)
username.c_str(),
password.c_str(),
NICE_RELAY_TYPE_TURN_TCP))
{
printf("error setting up relay info\n");
}
...
//after state has changed to NICE_COMPONENT_STATE_READY
...
io_stream = nice_agent_get_io_stream (agent, stream_id, component_id);
input = g_io_stream_get_input_stream (G_IO_STREAM (io_stream));
output = g_io_stream_get_output_stream (G_IO_STREAM (io_stream));
GIOStream* tlsConnection = g_tls_client_connection_new
(G_IO_STREAM (io_stream), NULL, &error);
/////////////////////////
/// error == 0 (TLS support is not available)
I am new to libnice and glib. So, I may be missing something basic.
Probably need the glib-networking package installed.

Processing-java sketch( server ) not responding in the way I want it to

I have created a processing-java sketch. This sketch is the server. All I want this program to do is that the client and server can connect and write messages(sentences) between each other. Case 1 was successful, but case 2 was not. I have explained the process for each case and what went wrong/successful.
Case 1) On the same computer(Mac), I started the server program and on Terminal("Command Prompt" on Mac), I typed telnet local host 5204 and the client(Mac) connected with the server(Mac). I was able to type sentences (or Strings) between the server and client and it was successful. So whatever sentence I type in the server, it was visible to the client and vice versa. Note: The server and client were both in the same computer.
Case 2) On the Mac, I started the server program. On another computer(Windows 7)
I connected to the server via Command Prompt. The connection was successful. In this case, the Strings could be sent from the server to the client and the Strings were visible to the client. But when I tried to send Strings to the server from the client, the server could only receive the information character by character, not as an entire sentence/String. I tried changing the port number, the client device, the frameRate, but I still had no success.
This is my problem. Please comment if my question could be clearer or if I need to give more details. Thank you for answering.
Below is my Server code:
import processing.net.*;
Server myServer;
//Strings from server and client
String typing = "";
String c = "";
void setup() {
size(400, 400);
//creating server on port 5204
myServer = new Server(this, 5204);
}
void draw() {
background(255);
//displaying server's text and client's text
fill(0);
text(typing, 100, 100);
text("Client: " + c, 100, 150);
Client client = myServer.available();
if(client != null) {
//reading input from client
c = client.readString();
c.trim();
}
}
void keyPressed() {
//Server can type sentences to client
if(key == '\n') {
myServer.write(typing + '\n');
typing = "";
}else{
typing = typing + key;
}
}
Did you try ncat for Windows?
With it you can try: echo Text to send & echo. | ncat localhost 5204
Source