I have 3 measures, and I want to add measure1 and measure2 and then subtract measure3 from the result. I'm unable to get the composite metric below to work. Any help would be appreciated.
subtract([
sum([
sum(s("measure1", "*")),
sum(s("measure2", "*")),
]),
sum(s("measure3", "*"))
])
Figured out what the issue was. Missing the aggregate function within the series.
subtract([
sum([
s("measure1", "*", {function:"sum"}),
s("measure2", "*", {function:"sum"})
]),
s("measure3", "*", {function:"sum"})
])
Related
Here is a Sequalize query below which retrieves a transformed value based on the table column value.
courses.findAll({
attributes: [ [sequelize.fn('to_char', sequelize.col('session_date'), 'Day'), 'days']]
});
The above sequlaize query will return result equal to as followed SQL query.
select to_char(bs.session_date, 'Day') as days from courses bs;
Expected output:
I want the transformed value which is in attributes along with all records like below. I know we can mention all the column names in attributes array but it is a tedious job. Any shortcut similar to asterisk in SQL query.
select to_char(bs.session_date, 'Day') as days,* from courses bs;
I tried the below sequalize query but no luck.
courses.findAll({
attributes: [ [sequelize.fn('to_char', sequelize.col('session_date'), 'Day'), 'days'],'*']
});
The attributes option can be passed an object as well as an array of fields for finer tuning in situations like this. It's briefly addressed in the documentation.
courses.findAll({
attributes: {
include: [
[ sequelize.fn('to_char', sequelize.col('session_date'), 'Day'), 'days' ]
]
}
});
By using include we're adding fields to the courses.* selection. Likewise we can also include an exclude parameter in the attributes object which will remove fields from the courses.* selection.
There is one shortcut to achieve the asterisk kind of selection in Sequalize. Which can be done as follows...
// To get all the column names in an array
let attributes = Object.keys(yourModel.rawAttributes);
courses.findAll({
attributes: [...attributes ,
[sequelize.fn('to_char', sequelize.col('session_date'), 'Day'), 'days']]
});
This is a work around there may be a different option.
I followed these instructions to get my AWS WAF data into an Athena table.
I would like to query the data to find the latest requests with an action of BLOCK. This query works:
SELECT
from_unixtime(timestamp / 1000e0) AS date,
action,
httprequest.clientip AS ip,
httprequest.uri AS request,
httprequest.country as country,
terminatingruleid,
rulegrouplist
FROM waf_logs
WHERE action='BLOCK'
ORDER BY date DESC
LIMIT 100;
My issue is cleanly identifying the "terminatingrule" - the reason the request was blocked. As an example, a result has
terminatingrule = AWS-AWSManagedRulesCommonRuleSet
And
rulegrouplist = [
{
"nonterminatingmatchingrules": [],
"rulegroupid": "AWS#AWSManagedRulesAmazonIpReputationList",
"terminatingrule": "null",
"excludedrules": "null"
},
{
"nonterminatingmatchingrules": [],
"rulegroupid": "AWS#AWSManagedRulesKnownBadInputsRuleSet",
"terminatingrule": "null",
"excludedrules": "null"
},
{
"nonterminatingmatchingrules": [],
"rulegroupid": "AWS#AWSManagedRulesLinuxRuleSet",
"terminatingrule": "null",
"excludedrules": "null"
},
{
"nonterminatingmatchingrules": [],
"rulegroupid": "AWS#AWSManagedRulesCommonRuleSet",
"terminatingrule": {
"rulematchdetails": "null",
"action": "BLOCK",
"ruleid": "NoUserAgent_HEADER"
},
"excludedrules":"null"
}
]
The piece of data I would like separated into a column is rulegrouplist[terminatingrule].ruleid which has a value of NoUserAgent_HEADER
AWS provide useful information on querying nested Athena arrays, but I have been unable to get the result I want.
I have framed this as an AWS question but since Athena uses SQL queries, it's likely that anyone with good SQL skills could work this out.
It's not entirely clear to me exactly what you want, but I'm going to assume you are after the array element where terminatingrule is not "null" (I will also assume that if there are multiple you want the first).
The documentation you link to say that the type of the rulegrouplist column is array<string>. The reason why it is string and not a complex type is because there seems to be multiple different schemas for this column, one example being that the terminatingrule property is either the string "null", or a struct/object – something that can't be described using Athena's type system.
This is not a problem, however. When dealing with JSON there's a whole set of JSON functions that can be used. Here's one way to use json_extract combined with filter and element_at to remove array elements where the terminatingrule property is the string "null" and then pick the first of the remaining elements:
SELECT
element_at(
filter(
rulegrouplist,
rulegroup -> json_extract(rulegroup, '$.terminatingrule') <> CAST('null' AS JSON)
),
1
) AS first_non_null_terminatingrule
FROM waf_logs
WHERE action = 'BLOCK'
ORDER BY date DESC
You say you want the "latest", which to me is ambiguous and could mean both first non-null and last non-null element. The query above will return the first non-null element, and if you want the last you can change the second argument to element_at to -1 (Athena's array indexing starts from 1, and -1 is counting from the end).
To return the individual ruleid element of the json:
SELECT from_unixtime(timestamp / 1000e0) AS date, action, httprequest.clientip AS ip, httprequest.uri AS request, httprequest.country as country, terminatingruleid, json_extract(element_at(filter(rulegrouplist,rulegroup -> json_extract(rulegroup, '$.terminatingrule') <> CAST('null' AS JSON) ),1), '$.terminatingrule.ruleid') AS ruleid
FROM waf_logs
WHERE action='BLOCK'
ORDER BY date DESC
I had the same issue but the solution posted by Theo didn't work for me, even though the table was created according to the instructions linked to in the original post.
Here is what worked for me, which is basically the same as Theo's solution, but without the json conversion:
SELECT
from_unixtime(timestamp / 1000e0) AS date,
action,
httprequest.clientip AS ip,
httprequest.uri AS request,
httprequest.country as country,
terminatingruleid,
rulegrouplist,
element_at(filter(ruleGroupList, ruleGroup -> ruleGroup.terminatingRule IS NOT NULL),1).terminatingRule.ruleId AS ruleId
FROM waf_logs
WHERE action='BLOCK'
ORDER BY date DESC
LIMIT 100;
I have a list of countries which will be separated by starting letter so for example when you click on 'A' it will make an API call to return all the countries beginning with 'A'.
However there are some letters that don't have any countries in our system, and these may change as we update out data.
I want to have a query that will let me know which letters do not have any countries that begin with them, so that I can disable them.
I can do this be running a findOne query for every single letter in the alphabet... but that is not neat or performant. Is there a way to get the data from a single query?
I am able to get the desired result by using a substring function within a distinct function.
const result = await Countries.findAll({
attributes: [
[
sequelize.fn(
'DISTINCT',
sequelize.fn('substring', sequelize.col('countryName'), 1, 1),
),
'letter',
],
],
group: [sequelize.fn('substring', sequelize.col('countryName'), 1, 1)],
raw: true,
})
I have a table with a jsonb column with data from one row like
[
{
"a": [],
"c_id": 624,
"ps": [{"": 0, "pr": "73", "f": "M", "s": "M"}],
"g_n": "K L Mish",
"g_num": 1
},
{
"a": [],
"c_id": 719,
"ps": [{"": 0, "pr": "65433", "f": "R", "s": "W"}],
"g_n": "S H Star",
"g_num": 2
},
]
I want to update c_id in the table wherever it is 719 to 720.
How can I do it?
I am using Postgres 12.1
If it is only one single occurrence, you could do it using a Regular Expression:
Click: demo:db<>fiddle
UPDATE mytable
SET mydata = s.result::jsonb
FROM (
SELECT
regexp_replace(mydata::text, '(.*)("c_id"\s*:\s*)(719)(.*)','\1\2720\4') AS result
FROM
mytable
) s;
RegExp Groups:
(.*) All characters before the relevant key
("c_id"\s*:\s*) The relevant key incl. possible spaces
(719) The relevant value to be replaced
(.*) Everything after the relevant point
With \1\2720\4 you put the first two groups together, followed by the new value (instead of group 3) and the fourth group.
Disclaimer:
I fully agree with #a_horse_with_no_name: You should think about storing all values in separate and normalized tables/columns. You would gain a lot of benefits (much better search and update handling, indexing, performance, ...). If you need this JSON output, just handle it as output: Generate it when you need it, do not store it. Maybe a view could help a lot
My data has 2 arrays (may have different size) like so:
{
"depotCode":"0001",
"in":[
{
"partCode":"AX-01",
"qty":15
},
{
"partCode":"AX-02",
"qty":77
},
{
"partCode":"AX-03",
"qty":100
} ],
"out":[
{
"partCode":"AX-01",
"qty":7
},
{
"partCode":"TB-77",
"qty":5
}
]
}
and expected result looks like this:
depotCode,partCode,in,out
0001,AX-01,15,7
0001,AX-02,77,0
0001,AX-03,100,0
0001,TB-77,0,5
what's the efficient hive query for this?
if you notice, since AX-01 is present in both "in" and "out", then the value is non-zero for both field.
as for the other, either one is zero
PS: if possible, I need this done in a single query and efficiently since we have more than a billion rows already (so subquery that may be slow should be avoided, unless subquery is somehow not slow in hive or spark, etc)
#srilanka_ratnapura; use concat_ws function
here is a documentation about it: http://www.folkstalk.com/2011/11/string-functions-in-hive.html
Example:
select concat_ws(',', col1, col2, col3, col4) from tbl
Returns:
col1,col2,col3,col4
Hope this helps. Thanks.