EVAL inside grok logstash - variables

I am trying to add new filed in grok filter which supposed to an arithmetic expression of the fields that are extracted by grok match command.
Unfortunately was not able to figure out the correct syntax for that... Anybody?
I found somewhere that {(8*6)} supposed to return 48, but what about variables instead of constants?
====
`if [type] == "f5" {
grok {
match => [ message, "...%{WORD:was_status}...%{NUMBER:hour}hr:%{NUMBER:min}min:%{NUMBER:sec}sec" ]
add_field => [ "duration_%{was_status}", "\{((%{hour} * 3600) + (%{min} * 60) + %{sec})}" ]
}
}`
====
got the result, but EVAL obviously not working correctly:
message: .... [ was down for 0hr:0min:4sec ]
duration_down \`{((0 * 3600) + (0 * 60) + 4)}`
Thanks a lot,
Yuri

There is an outstanding feature request for a math filter, but I'm not aware of any such feature at this time.
In the meantime, you can use the ruby filter to run arbitrary Ruby code on your event.
Here's a simple example:
input {
generator {
count => 1
message => "1 2 3"
}
}
filter {
grok {
match => ["message", "%{NUMBER:a:int} %{NUMBER:b:int} %{NUMBER:c:int}"]
}
ruby {
code => "event['sum'] = event['a'] + event['b'] + event['c']"
}
}
output {
stdout {
codec => rubydebug{}
}
}
Note that grok will usually parse values into strings. If I hadn't converted them to integers, Ruby would have handled the + operator as a string concatenation (and sum would end up equaling 123).
Your ruby filter might look more like this:
ruby {
code => "event['duration_down'] = event['hour']*3600 + event['min']*60 + event['sec']"
}

Related

Laravel Excel: how to have all zeroes printed in a exported csv?

I'm exporting to csv using Laravel Excel.
I'm creating the out value as
'amount' => number_format(-$ai->final_invoice_amount, 2, ".", "")
But, for example for 204.00, in the exported csv I got only 204, without dot and leading zeroes.
I know that it's a valid number for a computer; but our client has a strict parser and it wants a 204.00 value.
I tried, but not works, to add an explicit cast to string, but it's useless because number_format outputs a string in any case
'amount' => (string)number_format(-$ai->final_invoice_amount, 2, ".", "")
Laravel Excel's Events give you access to PHPSpreadSheet under the hood.
https://docs.laravel-excel.com/3.1/exports/extending.html#events
With Events, you can apply the desired formatting to a cell or group of cells. It's easier to demonstrate than explain, so I have created a simple example.
<?php
namespace App\Export;
use Maatwebsite\Excel\Concerns\FromCollection;
use Maatwebsite\Excel\Concerns\WithEvents;
use Maatwebsite\Excel\Events\BeforeSheet;
class SampleExport implements FromCollection, WithEvents
{
public function collection()
{
return collect([
[
'id' => 1,
'amount_1' => 100,
'amount_2' => 75.20,
'amount_3' => -23.10,
],
[
'id' => 2,
'amount_1' => -60,
'amount_2' => 50.40,
'amount_3' => 110,
],
]);
}
public function registerEvents(): array
{
return [
BeforeSheet::class => function (BeforeSheet $event) {
// format columns B-D to two decimal places
$event->sheet
->getDelegate()
->getStyle('B:D')
->getNumberFormat()
->setFormatCode('0.00');
},
];
}
}
This is the resulting CSV as output to a file.
"1","100.00","75.20","-23.10"
"2","-60.00","50.40","110.00"
NOTE: The values being formatted MUST BE A NUMBER TYPE! Attempting to format strings will not work.

Sequelize Querying with Op.or and Op.ne with same array of numbers

I'm having trouble getting the correct query with sequelize.
I have an array representing ids of entries lets say its like this -
userVacationsIds = [1,2,3]
i made the first query like this
Vacation.findAll({
where: {
id: {
[Op.or]: userVacationsIds
}
}
})
.then(vacationSpec => {
Vacation.findAll({
where:{
//Here i need to get all entries that DONT have the ids from the array
}
}
})
I can't get the correct query as specified in my code "comment"
I've tried referring to sequelize documentation but i can't understand how to chain these queries specifically
Also tried an online converter but that failed too.
Specified the code i have above
So i just need some help getting this query correct please.
I eventually expect to get 2 arrays - one containing all entries with the ids from the array, the other containing everything else (as in id is NOT in the array)
I figured it out.
I feel silly.
This is the query that worked
Vacation.findAll({
where: {
id: {
[Op.or]: userVacationsIds
}
}
}).then(vacationSpec => {
Vacation.findAll({
where: {
id: {
[Op.notIn]: userVacationsIds
}
}
})

MongoDB like statement with multiple fields

With SQL we can do the following :
select * from x where concat(x.y ," ",x.z) like "%find m%"
when x.y = "find" and x.z = "me".
How do I do the same thing with MongoDB, When I use a JSON structure similar to this:
{
data:
[
{
id:1,
value : "find"
},
{
id:2,
value : "me"
}
]
}
The comparison to SQL here is not valid since no relational database has the same concept of embedded arrays that MongoDB has, and is provided in your example. You can only "concat" between "fields in a row" of a table. Basically not the same thing.
You can do this with the JavaScript evaluation of $where, which is not optimal, but it's a start. And you can add some extra "smarts" to the match as well with caution:
db.collection.find({
"$or": [
{ "data.value": /^f/ },
{ "data.value": /^m/ }
],
"$where": function() {
var items = [];
this.data.forEach(function(item) {
items.push(item.value);
});
var myString = items.join(" ");
if ( myString.match(/find m/) != null )
return 1;
}
})
So there you go. We optimized this a bit by taking the first characters from your "test string" in each word and compared the tokens to each element of the array in the document.
The next part "concatenates" the array elements into a string and then does a "regex" comparison ( same as "like" ) on the concatenated result to see if it matches. Where it does then the document is considered a match and returned.
Not optimal, but these are the options available to MongoDB on a structure like this. Perhaps the structure should be different. But you don't specify why you want this so we can't advise a better solution to what you want to achieve.

ElasticSearch analyzed fields

I'm building my search but need to analyze 1 field with different analyzers. My problem is for a field I need to have an analyzer on it for stemming (snowball) and then also one to keep the full word as one token (keyword). I can get this to work by the following index settings:
curl -X PUT "http://localhost:9200/$IndexName/" -d '{
"settings":{
"analysis":{
"analyzer":{
"analyzer1":{
"type":"custom",
"tokenizer":"keyword",
"filter":[ "standard", "lowercase", "stop", "snowball", "my_synonyms" ]
}
}
},
"filter": {
"my_synonyms": {
"type": "synonym",
"synonyms_path ": "synonyms.txt"
}
}
}
},
"mappings": {
"product": {
"properties": {
"title": {
"type": "string",
"search_analyzer" : "analyzer1",
"index_analyzer" : "analyzer1"
}
}
}
}
}';
The problem comes when searching on a single word in the title field. If it's populated with The Cat in the Hat it will store it as "The Cat in the Hat" but if I search for cats I get nothing returned.
Is this even possible to accomplish or do I need to have 2 separate fields and analyze one with keyword and the other with snowball?
I'm using nest in vb code to index the data if that matters.
Thanks
Robert
You can apply two different analyzers to the same using the fields property (previously known as multi fields).
My VB.NET is a bit rusty, so I hope you don't mind the C# examples. If you're using the latest code from the dev branch, Fields was just added to each core mapping descriptor so you can now do this:
client.Map<Foo>(m => m
.Properties(props => props
.String(s => s
.Name(o => o.Bar)
.Analyzer("keyword")
.Fields(fs => fs
.String(f => f
.Name(o => o.Bar.Suffix("stemmed"))
.Analyzer("snowball")
)
)
)
)
);
Otherwise, if you're using NEST 1.0.2 or earlier (which you likely are), you have to accomplish this via the older multi field type way:
client.Map<Foo>(m => m
.Properties(props => props
.MultiField(mf => mf
.Name(o => o.Bar)
.Fields(fs => fs
.String(s => s
.Name(o => o.Bar)
.Analyzer("keyword"))
.String(s => s
.Name(o => o.Bar.Suffix("stemmed"))
.Analyzer("snowball"))
)
)
)
);
Both ways are supported by Elasticsearch and will do the exact same thing. Applying the keyword analyzer to the primary bar field, and the snowball analyzer to the bar.stemmed field. stemmed of course was just the suffix I chose in these examples, you can use whatever suffix name you desire. In fact, you don't need to add a suffix, you can name the multi field something completely different than the primary field.

Getting SQL MIN() and MAX() from DBIx::Class::ResultSetColumn in one query

I want to select the MIN() and MAX() of a column from a table. But instead of querying the database twice I'd like to solve this in just one query.
I know I could do this
my $col = $schema->result_source("Birthday")->get_column("birthdate");
my $min = $col->min();
my $max = $col->max();
But it would query the database twice.
The only other solution I found is quite ugly, by messing around with the select and as attributes to search(). For example
my $res = $rs->search({}, {
select => [ {min => "birthdate"}, {max => "birthdate"},
as => [qw/minBirthdate maxBirthdate/]
});
say $res->get_column("minBirthdate")->first() . " - " . $res->get_column("maxBirthdate")->first();
Which produces this - my wanted SQL
SELECT MIN(birthdate), MAX(birthdate) FROM birthdays;
Is there any more elegant way to get this done with DBIx::Class?
And to make it even cooler, is there a way to respect the inflation/deflation of the column?
You can use columns as a shortcut to combine select and as attributes as such:
my $res = $rs->search(undef, {
columns => [
{ minBirthdate => { min => "birthdate" } },
{ maxBirthdate => { max => "birthdate" } },
]
});
Or, if you prefer more control over the SQL, use string refs, which can help with more complex calculations:
my $res = $rs->search(undef, {
columns => [
{ minBirthdate => \"MIN(birthdate)" },
{ maxBirthdate => \"MAX(birthdate)" },
]
});
Now, if you really want to clean it up a bit, I highly recommend DBIx::Class::Helpers, which allows you to write it as such:
my $minmax = $rs->columns([
{minBirthdate=>\"MIN(birthdate)"},
{maxBirthdate=>\"MAX(birthdate)"},
])->hri->single;
say "$minmax->{minBirthdate} - $minmax->{maxBirthdate}";