Is there any way to determine the current bro version from the conn.log file?
I have an application that parses conn.logs and I don't currently see anything in the header that details a version number
If your logs are in TSV format, the header information provides a schema:
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn
#open 2018-07-30-17-02-19
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto serviceduration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents
#types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string]
In JSON, no such header or schema currently exists. I would suggest basing your processing on known fields and their types, e.g. based on the TSV output.
Related
There are transactional input CSV files coming on a daily basis on an FTP location. I need to read these input files and process them on daily batch execution. The name of the files remains the same every day, but the date gets appended at the end of the filenames every day,
Ex:
Day1
General_Ledger1_2020-07-01,
General_Ledger2_2020-07-01,
General_Ledger3_2020-07-01,
General_Ledger4_2020-07-01,
General_Ledger5_2020-07-01
Day2
General_Ledger1_2020-07-02,
General_Ledger2_2020-07-02,
General_Ledger3_2020-07-02,
General_Ledger4_2020-07-02,
General_Ledger5_2020-07-02
How can I append this Date information to the input file name every time the job runs?
I have faced similar problem earlier and this can be solved using calculated parameter in the file path. Here, you can create expressions that will retrieve the file dynamically.
Example,
CONCAT( UPPER(lit('$(Prefix)')), ADD_DAYS( TODATE(lit('$(currentTime)'), 'yyyy-mm-dd'), 'yyyy-mm-dd' ,-1),'.csv')
Breaking of the expression :
$(currentTime) : this system parameter will get the current date (this will also include timestamp).
(TODATE(lit('$(currentTime)'), 'yyyy-mm-dd') : TODATE will get only date from the whole timestamp with format as ‘yyyy-mm-dd’.
ADD_DAYS(TODATE(lit('$(currentTime)'), 'yyyy-mm-dd'), 'yyyy-mm-dd' ,-1) : ADD_DAYS here will add -1 to the date retrieved from. TODATE(). Hence (2020-04-24) + (-1) would give us 2020-04-23
$(Prefix) : $(Prefix) will be an user defined input parameter of type String which user will be providing at runtime – Since the
prefix will be always dynamic.
CONCAT() : Finally to combine all the results and form the exact file path CONCAT() can be used. Also in between some static
string is added as it will always be fixed for every file to be read.
is there a way to extract only IPv4 from a file in JSON language using VB.net
For example I would like that when I open a JSON file from VB I can filter only IPv4 from this text for example: https://pastebin.com/raw/S7Vnnxqa
& i expect the results like this https://pastebin.com/raw/8L8Ckrwi i founded this website that he offer a tool to do that https://www.toolsvoid.com/extract-ip-addresses/ i put the link here to understand more what i mean but i don't want to use an external tool i want it to be converted from VB directly thanks for your help in advance.
Your "text" is JSON. Load it using the JSON parser of your choice (google VB.NET parse JSON), loop over the matches array and read the IP address from the http.host property of each element.
Here is an example how to do it using the Newtonsoft.Json package (see it working here on DotNetFiddle):
' Assume that the variable myJsonString contains the original string
Dim myJObject = JObject.Parse(myJsonString)
For Each match In myJObject("matches")
Console.WriteLine(match("http")("host"))
Next
Output:
62.176.84.198
197.214.169.59
46.234.76.75
122.136.141.67
219.73.94.83
2402:800:621b:33f1:d1e3:5544:4fcf:526e
178.136.75.125
188.167.212.252
...
If you want to extract only IPv4 and not IPv6, you can use a regular expression to check whether it matches:
Dim IPV4Regex = New Regex("^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$")
Dim ip = match("http")("host")
If IPV4Regex.Match(ip).Success Then
Console.WriteLine(ip)
End If
62.176.84.198
197.214.169.59
46.234.76.75
122.136.141.67
219.73.94.83
178.136.75.125
188.167.212.252
...
Of course it's always recommended to parse the input data in a structured way, to avoid surprises such as false positives. But if you just want to match anything that looks like an IP address, regardless of the input format (even if you just put hello1.2.3.4world in the textbox), then you could use just the regular expression and skip the structured approach (see it working here on DotNetFiddle):
Dim IPV4RegexWithWordBoundary = New Regex("\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b")
Dim match = IPV4RegexWithWordBoundary.Match(myJsonString)
Do While match.Success
Console.WriteLine(match.Value)
match = match.NextMatch()
Loop
Here I modified the regular expression to use \b...\b instead of ^...$ so that it matches word boundaries instead of start/end of string. Note however that now we get IP addresses twice with the input that you provided, because the addresses exist more than once:
62.176.84.198
62.176.84.198
197.214.169.59
197.214.169.59
46.234.76.75
46.234.76.75
...
I have a input folder in ADLS in the format year/month/date eg: 2017/07/11. I want to pass this input folder as a parameter to my usql script. I am not using ADF. I dont want to generate current date from within Usql script as i am not sure if the input folder is of the current date. How to do it effectively?
One way I thought of was uploading a "done" file after all my input folder is uploaded to ADLS account and that "done" file will contain the date. But i am not able to use that date to form my input data path. Please help.
Let's assume you have several csv files in your folder structure (structured as yyyy/MM/dd) and you want to extract all the files in a folder of a specific date. You can do it in two ways (depending in whether you need to have exact datetime semantics or if you are fine with path concat).
First the path concat example:
DECLARE EXTERNAL #folder = "2017/07/11"; // Script parameter with default value.
// You can specify the value also with constant-foldable expression on Datetime.Now.
DECLARE #path = "/constantpath/"+#folder+"/{*.csv}";
#data = EXTRACT I int, s string // or whatever your schema is...
FROM #path
USING Extractors.Csv();
...
And here is the example with a file set virtual column:
DECLARE EXTERNAL #date = "2017/07/11"; // Script parameter with default value.
// You can specify the value also with constant-foldable expression on Datetime.Now and string serialization (I am not sure if the ADF parameter model supports DateTime values).
DECLARE #path = "/constantpath/{date:yyyy}/{date:MM}/{date:dd}/{*.csv}";
#data = EXTRACT I int, s string // or whatever your schema is...
, date DateTime // virtual column for the date pattern
FROM #path
USING Extractors.Csv();
// Now apply the requested filter to reduce the files to the requested set
#data = SELECT * FROM #data WHERE date == DateTime.Parse(#date);
...
In both cases, you pass the parameter via the ADF parameterization model and you can decide to wrap the code into a U-SQL stored procedure or TVF as suggested by Bob.
I'm currently creating a flat file export for one of our clients, i've managed to get the file in the format they want, i'm trying to get the easiest way of creating a dynamic file name. I've got the date in as a variable and the path ect but they want a count in the file name. For example
File name 1 : TDY_11-02-2013_{1}_T1.txt. The {} being the count. So next weeks file would be TDY_17-02-2013_{2}_T1.txt
I cant see an easy way of doing this!! any idea's??
EDIT:
on my first answer, I thought you meant count of values returned on a query. My bad!
two ways to achieve this, you could loop into the destination folder, select the last file by date, get its value and increase 1, which sound like a lot of trouble. Why not a simple log table on the DB with last execution date and ID and then you compose your file name base on the last row of this table?
where exactly is your problem?
you can make a dynamic file name using expressions:
the count, you can use a "row count" component inside your data flow to assign the result to a variable and use the variable on your expression:
Use Script task and get the number inside the curly braces of the file name and store it in a variable.
Create a variable(FileNo of type int) which stores the number for the file
Pseudo code
string name = string.Empty;
string loction = #"D:\";
/* Get the path from the connection manager like the code below
instead of hard coding like D: above
string flatFileConn =
(string(Dts.Connections["Yourfile"].AcquireConnection(null) as String);
*/
string pattern = string.Empty;
int number = 0;
string pattern = #"{([0-9])}"; // Not sure about the correct regular expression to retrieve the number inside braces
foreach (string s in Directory.GetFiles(loction,"*.txt"))
{
name = Path.GetFileNameWithoutExtension(s);
Match match = Regex.Match(name, pattern );
if (match.Success)
{
dts.Variables["User::FileNo"].Value = int.Parse(match.Value)+1;
}
}
Now once you get the value use it in your file expression in the connection manager
#[User::FilePath] +#[User::FileName]
+"_{"+ (DT_STR,10,1252) #[User::FileNo] + "}T1.txt"
I'm working with an Oracle database, and I would like to read the contents of a BLOB. How do I do this?
When I do a simple select statement, it merely returns "(BLOB)" (without the quotes). How do I read the actual contents?
You can dump the value in hex using UTL_RAW.CAST_TO_RAW(UTL_RAW.CAST_TO_VARCHAR2()).
SELECT b FROM foo;
-- (BLOB)
SELECT UTL_RAW.CAST_TO_RAW(UTL_RAW.CAST_TO_VARCHAR2(b))
FROM foo;
-- 1F8B080087CDC1520003F348CDC9C9D75128CF2FCA49D1E30200D7BBCDFC0E000000
This is handy because you this is the same format used for inserting into BLOB columns:
CREATE GLOBAL TEMPORARY TABLE foo (
b BLOB);
INSERT INTO foo VALUES ('1f8b080087cdc1520003f348cdc9c9d75128cf2fca49d1e30200d7bbcdfc0e000000');
DESC foo;
-- Name Null Type
-- ---- ---- ----
-- B BLOB
However, at a certain point (2000 bytes?) the corresponding hex string exceeds Oracle’s maximum string length. If you need to handle that case, you’ll have to combine How do I get textual contents from BLOB in Oracle SQL with the documentation for DMBS_LOB.SUBSTR for a more complicated approach that will allow you to see substrings of the BLOB.
SQL Developer can show the blob as an image (at least it works for jpegs). In the Data view, double click on the BLOB field to get the "pencil" icon. Click on the pencil to get a dialog that will allow you to select a "View As Image" checkbox.
If the content is not too large, you can also use
SELECT CAST ( <blobfield> AS RAW( <maxFieldLength> ) ) FROM <table>;
or
SELECT DUMP ( CAST ( <blobfield> AS RAW( <maxFieldLength> ) ) ) FROM <table>;
This will show you the HEX values.
If you use the Oracle native data provider rather than the Microsoft driver then you can get at all field types
Dim cn As New Oracle.DataAccess.Client.OracleConnection
Dim cm As New Oracle.DataAccess.Client.OracleCommand
Dim dr As Oracle.DataAccess.Client.OracleDataReader
The connection string does not require a Provider value so you would use something like:
"Data Source=myOracle;UserID=Me;Password=secret"
Open the connection:
cn.ConnectionString = "Data Source=myOracle;UserID=Me;Password=secret"
cn.Open()
Attach the command and set the Sql statement
cm.Connection = cn
cm.CommandText = strCommand
Set the Fetch size. I use 4000 because it's as big as a varchar can be
cm.InitialLONGFetchSize = 4000
Start the reader and loop through the records/columns
dr = cm.ExecuteReader
Do while dr.read()
strMyLongString = dr(i)
Loop
You can be more specific with the read, eg dr.GetOracleString(i) dr.GetOracleClob(i) etc. if you first identify the data type in the column. If you're reading a LONG datatype then the simple dr(i) or dr.GetOracleString(i) works fine. The key is to ensure that the InitialLONGFetchSize is big enough for the datatype. Note also that the native driver does not support CommandBehavior.SequentialAccess for the data reader but you don't need it and also, the LONG field does not even have to be the last field in the select statement.
What client do you use? .Net, Java, Ruby, SQLPLUS, SQL DEVELOPER? Where did you write that simple select statement?
And why do you want to read the content of the blob, a blob contains binary data so that data is unreadable. You should use a clob instead of a blob if you want to store text instead of binary content.
I suggest that you download SQL DEVELOPER: http://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html . With SQL DEVELOPER you can see the content.
If you're interested to get the plaintext (body part) from a BLOB, you could use the CTX_DOC package.
For example, the CTX_DOC.FILTER procedure can "generate either a plain text or a HTML version of a document". Be aware that CTX_DOC.FILTER requires an index on the BLOB column. If you don't want that, you could use the CTX_DOC.POLICY_FILTER procedure instead, which doesn't require an index.