I am trying to leverage the parsing of structured data feature in syslog-ng. From my firewall, I am forwarding the following message:
<14>1 2012-10-06T11:03:56.493 SRX100 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos#2636.1.1.1.2.36 reason="TCP FIN" source-address="192.168.199.207" source-port="59292" destination-address="184.73.190.157" destination-port="80" service-name="junos-http" nat-source-address="50.193.12.149" nat-source-port="19230" nat-destination-address="184.73.190.157" nat-destination-port="80" src-nat-rule-name="source-nat-rule" dst-nat-rule-name="None" protocol-id="6" policy-name="trust-to-untrust" source-zone-name="trust" destination-zone-name="untrust" session-id-32="9375" packets-from-client="9" bytes-from-client="4342" packets-from-server="7" bytes-from-server="1507" elapsed-time="1" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="vlan.0"]
Based on the format of the IETF logs, it appears to be correct, but for some reason the structured data is actually being parsed as the message portion of the log and not being parsed as structured data.
On the syslog-ng side, you need to use either a syslog() source, or a tcp() source with flags(syslog-proto) set, and then the stuff will end up in variables like ${.SDATA.junos#2636.1.1.1.2.36.reason} and so on and so forth, which then you can use as you see fit.
Related
Using flink 1.7.0, but also seen on flink 1.8.0. We are getting frequent but somewhat random errors when reading gzipped objects from S3 through the flink .readFile source:
org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException: Data read has a different length than the expected: dataLength=9713156; expectedLength=9770429; includeSkipped=true; in.getClass()=class org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client$2; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0
at org.apache.flink.fs.s3base.shaded.com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:151)
at org.apache.flink.fs.s3base.shaded.com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:93)
at org.apache.flink.fs.s3base.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:76)
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AInputStream.closeStream(S3AInputStream.java:529)
at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AInputStream.close(S3AInputStream.java:490)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.flink.fs.s3.common.hadoop.HadoopDataInputStream.close(HadoopDataInputStream.java:89)
at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227)
at java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
at org.apache.flink.api.common.io.InputStreamFSInputWrapper.close(InputStreamFSInputWrapper.java:46)
at org.apache.flink.api.common.io.FileInputFormat.close(FileInputFormat.java:861)
at org.apache.flink.api.common.io.DelimitedInputFormat.close(DelimitedInputFormat.java:536)
at org.apache.flink.streaming.api.functions.source.ContinuousFileReaderOperator$SplitReader.run(ContinuousFileReaderOperator.java:336)
ys
Within a given job, we generally see many / most of the jobs read successfully, but there's pretty much always at least one failure (say out of 50 files).
It seems this error is actually originating from the AWS client, so perhaps flink has nothing to do with it, but I'm hopeful someone might have an insight as to how to make this work reliably.
When the error occurs, it ends up killing the source and canceling all the connected operators. I'm still new to flink, but I would think that this is something that could be recoverable from a previous snapshot? Should I expect that flink will retry reading the file when this kind of exception occurs?
Maybe you can try to add more connection for s3a like
flink:
...
config: |
fs.s3a.connection.maximum: 320
We parse millions of messages a day using syslog-ng, and are in the process of implementing patterndb.
Due to inconsistency in how the messages are composed, in a small percentage of cases, my patterns are insufficient to capture the fields of the message (spacing is off, or sometimes a field is missing altogether).
How can I deal with these cases? Ideally, the parser entry in my log destination would evaluate to false (like a filter) and it would be captured by my fallback log destination.
Try setting drop-unmatched(yes) (needs syslog-ng OSE 3.11 or later):
parser pattern_db {
db-parser(
file("/opt/syslog-ng/var/db/patterndb.xml")
drop-unmatched(yes)
);
};
Also, recent syslog-ng versions have several different parsers that might be better for certain log messages than patterndb, for example, JSON and key=value parsers.
Trying to use the mv.NET by bluefinity tools. Made some integration packages with it for importing data from a d3 multi-value database into MS SQL 2012 but seem to be having some trouble with the mapping.
For the VOYAGES table have some commentX fields in the D3 application that are acting quite unwieldy and the INSERT fails after a certain number of rows with the following message
>Error: 0xC0047062 at INSERT, mvNET Source[354]: System.Exception: Error #8: dataReader[0] = LTPAC002 ci.BufferColumnIndex = 52, ci.ColumnName = COMMGROUP(Error #8: dataReader[0] = LTPAC002 ci.BufferColumnIndex = 52, ci.ColumnName = COMMGROUP(The value is too large to fit in the column data area of the buffer.))
at mvNETDataSource.mvNETSource.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers)
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPrimeOutput(IDTSManagedComponentWrapper100 wrapper, Int32 outputs, Int32[] outputIDs, IDTSBuffer100[] buffers, IntPtr ppBufferWirePacket)
Error: 0xC0047038 at INSERT, SSIS.Pipeline: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED.The PrimeOutput method on mvNET Source returned error code 0x80131500.The component returned a failure code when the pipeline engine called PrimeOutput().The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.There may be error messages posted before this with more information about the failure.
The value is too large to fit in the column data area of the buffer. -> tried changing the input / outputs types but can't seem to get it right.
In the SQL table the columns are of type ntext.
In the .dtsx job the data type for the columns are of type Unicode String [DT_WSTR] with length 4000 , I guess these are auto-detected.
The import worked for other D3 files like this not sure why it fails for these comment fields.
Running the query on the mv.NET Data Manager ( on the d3 server) times out after 240 seconds so maybe this is the underlying issue?
Any ideas how to proceed? Thank you ~
Most like reason is column COMMGROUP does not have correct data type or some record in source do not fit in output type
To find error record (causing) you have to use on redirect row (property of component failing component ) and get the result set in some txt.csv /or tsv file .
then check data
The exception is being thrown from mv.NET so I suggest you call (or ask your reseller) to call Bluefinity support and ask them about this. You're paying for support, might as well use it. Those programs shouldn't be allowed to throw exceptions like that.
D3 doesn't export Unicode, that might be one issue. But if the Data Manager times-out then I suspect something is wrong in the connectivity into D3. Open a Connection Monitor from the Session Monitor and watch the connection when you make the request. I'm guessing it's either hanging or more probably it's falling into BASIC Debug.
Make sure all D3-side programs related to this are either all Flash-compiled, or all Not Flashed. Your app code will fall into Debug if it's not Flashed but MVNET.BP is.
If it's your program that's in Debug, fix it. If you're not sure which program it is, LIST-RUNTIME-ERRORS in DM.
If it's a MVNET.BP program, again work with Bluefinity. If you are using MVSP for connectivity then the Connection Monitor may be useless, you'll need to change that to an IP (Telnet) connection to see the raw data exchange.
I'm trying to handle UDH data for multipart messages and I want to use an appropriate error code when there's a problem with the multipart fields. There are several error codes for bad TLV's, but I don't see anything except generic failure messages that I could use for UDH. Is that the best I can do here, or is there a more direct message I could use?
Your observation is correct - there is not a single error defined for UDH.
If you want, you can use error codes from reserved section and assign specific meaning to one/some of them. I think the "Reserved for SMSC vendor specific errors" (0x400-0x4FF) can be a good candidate.
Actually it looks to me like ESME_RX_R_APPN = 0x66 "ESME receiver reject message error." is the best error code to throw here.
It should correspond to a permanent error for this message only and let further messages be processed.
I am reading some lines from a CSV file, converting them to business objects, aggregating these to batches and passing the resulting aggregates to a bean, which may throw an PersistenceException.
Somehow like this:
from(file:inputdir).split().tokenize("\n").bean(a).aggregate(constant(true), new AbstractListAggregationStrategy(){...}).completionSize(3).bean(b)
I have a onException(Exception.class).handled(true).to("file:failuredir").log(). If an exception occurs on bean(a), everything is handled as expected: wrong lines in inputdir/input.csv are written to failuredir/input.csv.
Now if bean(b) fails, Camel seems to fail reconstructing the original message:
message.org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file: target/failure/ID-myhostname-34516-1372093690069-0-7
Having tried various attempts to get this working, like using HawtDBAggregationRepository, toggling useOriginalMessage at onException and propagating back the exception in my AggregationStrategy, I am out of ideas.
How can I achieve the same behaviour for bean(b) which can be seen with bean(a)?
The aggregator is a stateful EIP pattern, so when it sends out a message, then its a new Exchange. So the bean(b) cannot get access to the original message that came from the file route.