Does BigQuery have a safe navagation operator, i.e. a null-safe variant of its field navigation operator?
Ideally I'm looking for an operator akin to ?. in Swift/TypeScript, &. in Ruby, etc., but a function I could call would suffice as well.
Right now my query looks like:
SELECT a.b.c.d.e
FROM myTable AS a
WHERE
a.b IS NOT NULL
&& a.b.c IS NOT NULL
&& a.b.c.d IS NOT NULL
&& a.b.c.d.e = "my desired value"
Edit: This doesn't actually work.
Name b not found inside a at [12:34]
I'd wish it could be something like:
SELECT a.b.c.d.e
FROM myTable AS a
WHERE a?.b?.c?.d?.e = "my desired value"
afaik, there is no safe navagation operator for STRUCT type in bigquery.
what I can come up with is to conver nested STRUCT type to JSON type and utilize json path with which you can navigate safely.
WITH myTable AS (
SELECT STRUCT(STRUCT(STRUCT('my_desired_value' AS e) AS d) AS c) AS b
)
SELECT TO_JSON(b).c.d.e, --
TO_JSON(b).f.d.e, -- non-existing path
-- b.f.d.e --> error - Field name f does not exist ...
FROM myTable AS a;
To check field path of struct type, you can use INFORMATION_SCHEMA.COLUMN_FIELD_PATHS.
SELECT *
FROM `your-project.your_dataset.INFORMATION_SCHEMA.COLUMN_FIELD_PATHS`
WHERE table_name = 'myTable';
Related
I have function module which imports my_values
my_values is an custom internal table type of string.
This "my_values" variable contains for example: ["foo", "bar"]
I want to select all values from table Z_MYTAB where the column my_col is in my_values.
I tried this:
SELECT * FROM Z_MYTAB WHERE
my_col in #my_values INTO TABLE #DATA(my_rows).
But this fails with an error message:
table my_values has wrong row structure
(The message was translated to English. The original could be slightly different)
I could loop over my_values but I would like to avoid this.
How to do SQL IN with host variables which are internal tables?
Selection with IN is possible only with a range table.
Conversion of an internal table into a range table can be done like this:
DATA ltr_value TYPE RANGE OF string.
ltr_value = VALUE #( FOR <my_value> IN my_values
( sign = 'I'
option = 'EQ'
low = <my_value> )
).
IN openands could be of 2 types:
SELECT ... WHERE my_col IN ( value1, value2 , value3)
in this case no host expression can be used as right operand
SELECT ... WHERE my_col IN sel_tab[]
in this case sel_tab is a range like
So you could use the following:
DATA sel_tab type range of string.
sel_tab = value #( for ls in my_values ( sign = 'I' option = 'EQ' low = ls ) ).
SELECT * FROM Z_MYTAB WHERE
my_col in #sel_tab[] INTO TABLE #DATA(my_rows).
Best regards
User JozsefSzikszai pointed me to "SELECT FOR ALL ENTRIES".
I found this in the docs:
For an elementary row type, the pseudo component table_line must be specified for comp.
See: https://help.sap.com/doc/abapdocu_752_index_htm/7.52/en-US/abenwhere_logexp_itab.htm
IF my_values is initial.
exit.
endif.
SELECT * FROM Z_MYTAB
FOR ALL ENTRIES IN #my_values WHERE
column_name = #my_values-table_line
INTO TABLE #DATA(result_rows).
I am using Azure CosmosDB to store documents (JSON).
I am trying to query all documents that contain the field "abc", and not return the documents that do not have the field "abc". For example, return the first object below and not the second
{
"abc": "123"
}
{
"jkl": "098"
}
I am trying to use the following code:
client.queryDocuments(
collectionUrl,
`SELECT r.id, r.authToken.instagram,r.userName FROM root r WHERE r.abc`
)
I assumed the above would check if abc exists similar to if (r.abc) {}
I have tried using WHERE r.abc IS NOT NULL
Thanks in advance
If you want to know if a field exists you should use the IS_DEFINED("FieldName")
If you want to know if the field's value has a value the
FieldName != null or
FieldName <> null (apparently)
I use variations of this in production:
SELECT c.FieldName
FROM c
WHERE IS_DEFINED(c.FieldName)
All you need to do is change your query to
SELECT r.id, r.authToken.instagram,r.userName FROM root r WHERE r.abc != null
or
SELECT r.id, r.authToken.instagram,r.userName FROM root r WHERE r.abc <> null
Both operators work (tested on the Data Explorer)
Add the NOT operator in the SQL query to negate.
SELECT r.id, r.authToken.instagram,r.userName
FROM root r
WHERE NOT IS_DEFINED(r.abc)
to include all entries where the FieldName abc doesn't exist.
I have got a big CSVs that contain big strings. I wanna parse them in U-SQL.
#t1 =
SELECT
Regex.Match("ID=881cf2f5f474579a:T=1489536183:S=ALNI_MZsMMpA4voGE4kQMYxooceW2AOr0Q", "ID=(?<ID>\\w+):T=(?<T>\\w+):S=(?<S>[\\w\\d_]*)") AS p
FROM
(VALUES(1)) AS fe(n);
#t2 =
SELECT
p.Groups["ID"].Value AS gads_id,
p.Groups["T"].Value AS gads_t,
p.Groups["S"].Value AS gads_s
FROM
#t1;
OUTPUT #t
TO "/inhabit/test.csv"
USING Outputters.Csv();
Severity Code Description Project File Line Suppression State
Error E_CSC_USER_INVALIDCOLUMNTYPE:
'System.Text.RegularExpressions.Match' cannot be used as column type.
I know how to do it in a SQL way with EXPLODE/CROSS APPLY/GROUP BY. But may be it is possible to do without these dances?
One more update
#t1 =
SELECT
Regex.Match("ID=881cf2f5f474579a:T=1489536183:S=ALNI_MZsMMpA4voGE4kQMYxooceW2AOr0Q", "ID=(?<ID>\\w+):T=(?<T>\\w+):S=(?<S>[\\w\\d_]*)").Groups["ID"].Value AS id,
Regex.Match("ID=881cf2f5f474579a:T=1489536183:S=ALNI_MZsMMpA4voGE4kQMYxooceW2AOr0Q", "ID=(?<ID>\\w+):T=(?<T>\\w+):S=(?<S>[\\w\\d_]*)").Groups["T"].Value AS t,
Regex.Match("ID=881cf2f5f474579a:T=1489536183:S=ALNI_MZsMMpA4voGE4kQMYxooceW2AOr0Q", "ID=(?<ID>\\w+):T=(?<T>\\w+):S=(?<S>[\\w\\d_]*)").Groups["S"].Value AS s
FROM
(VALUES(1)) AS fe(n);
OUTPUT #t1
TO "/inhabit/test.csv"
USING Outputters.Csv();
This wariant works fine. But there is a question. Will the regex evauated 3 times per row? Does exists any chance to hint U-SQL engine - the function Regex.Match is deterministic.
You should probably be using something more efficient than Regex.Match. But to answer your original question:
System.Text.RegularExpressions.Match is not part of the built-in U-SQL types.
Thus you would need to convert it into a built-in type, such as string or SqlArray<string> or wrap it into a udt that provides an IFormatter to make it a user-defined type.
Looks like it is better to use something like this to parse the simple strings. Regexes are slow for the task and if i will use simple string expressions (instead of CLR calls) they probably will be translated into c++ code at codegen phase... and .net interop will be eliminated (i'm not sure).
#t1 =
SELECT
pv.cust_gads != null ? new SQL.ARRAY<string>(pv.cust_gads.Split(':')) : null AS p
FROM
dwh.raw_page_view_data AS pv
WHERE
pv.year == "2017" AND
pv.month == "04";
#t3 =
SELECT
p != null && p.Count == 3 ? p[0].Split('=')[1] : null AS id,
p != null && p.Count == 3 ? p[1].Split('=')[1] : null AS t,
p != null && p.Count == 3 ? p[2].Split('=')[1] : null AS s
FROM
#t1 AS t1;
OUTPUT #t3
TO "/tmp/test.csv"
USING Outputters.Csv();
I have a Field Field<T>. I want to create a named value for that field, to be able to use it in a query. The name of the value should be the name of the field.
select value as field from ...
Is the the correct way to do it?
public <T> Field<T> namedValue(Field<T> field, T value) {
return DSL.val(value, field).as(field);
}
Although it works, I was wondering if there is a shorter way to do this. I might be pedantic here :).
update
I am creating the following construction:
UPADTE table SET x = alias.x, y = alias.y
FROM (SELECT constant value for x, table2.y FROM table2 WHERE ...) AS alias.
Let's simplify this to (for the sake of this example, to focus on the constant selection):
SELECT
FROM (SELECT constant value for x) AS alias.
First, I started with:
Select s1 = context.select(DSL.val("TEST"));
Select s2 = context.select(s1.fields()).from(s1);
This resulted in an incorrect query:
select "alias_66794930"."TEST" from (select 'TEST') as "alias_66794930"
(I am not really sure if this is correct behavior from jOOQ.)
So, I added an alias:
Select s1 = context.select(DSL.val("TEST").as(X));
Select s2 = context.select(s1.fields()).from(s1);
This resulted in:
select "alias_76324565"."x" from (select 'TEST' as "x") as "alias_76324565"
This works fine. Then, I ran into problems when the constant vale was null:
Select s1 = context.select(DSL.val(null).as(X));
Select s2 = context.select(s1.fields()).from(s1);
This resulted in:
select "alias_85795854"."x" from (select cast(? as varchar) as "x") as "alias_85795854"
1400 [localhost-startStop-1] TRACE org.jooq.impl.DefaultBinding - Binding variable 1 : null (class java.lang.Object)
This makes sense, the field type is not known. So I added the field (with its type) as following:
Select s1 = context.select(DSL.val(null, X).as(X));
Select s2 = context.select(s1.fields()).from(s1);
Binding is now correct:
1678 [localhost-startStop-1] TRACE org.jooq.impl.DefaultBinding - Binding variable 1 : null (class java.lang.String)
All done!
I don't think you can get much shorter than what you already have. I mean, your SQL reads:
value as field
And your Java/jOOQ code reads:
DSL.val(value, field).as(field)
You could of course static import DSL.val or DSL.*:
import static org.jooq.impl.DSL.*;
And then shorten things to:
val(value, field).as(field)
And if you're very sure about value's type, you don't need to coerce it to that of field
val(value).as(field)
Now, you definitely can't go any shorter, and there's no more need for your namedValue() function...
Let's say I want a JasperReport that lets the user filter on a date if they so wish. The SQL is as follows:
select * from foo where bar = $P{bar} and some_date > $P{some.date}
Now, I don't want to filter by some date if they didn't pass the date in. I found the following kludge that people use:
select * from foo where bar = $P{bar} $P!{some.date.fragment}
And the some.date.fragment parameter is defined with the following default:
($P{some.date} == null || $P{some.date}.equals("")) ? "" : "AND some_date >'" + new java.sql.Date($P{some.date}.getTime()).toString() + "'"
This is not working as the toString doesn't output the date in a format that my SQL server understands. I would like to have the conditional still use a prepared statement with the jdbc driver and toss the parameter in, I just want the prepared statement to be dependent on if the parameter is null or not. Can this be done?
Before you have used the $P!{} expression the JDBC-Driver does all formatting for you.
But if you use the $P!{} expression you have to format yourself.
Something like this should work:
(
$P{some.date} == null
?
""
:
"AND some_date >'" + (new SimpleDateFormat("dd.MM.yyyy HH:mm:ss.SSS")).format($P{some.date}) + "'"
)
Depending on your data type you have to customize dd.MM.yyyy HH:mm:ss.SSS.
If you don't want to use the $P!{} expression you can avoid it with the solution below.
I personally don't like this way. It also may cause a bad execution plan.
If don't want to use $P!{} because you worry about sql injection. It's needless as long your parameter $P{some.date} contains a safe data type like java.lang.Date.
Create a parameter. Let's call it ${is_null_pram} and add a default expression with param class Integer:
($P{some.date} == null ? 1 : 0)
Now you can query:
SELECT
*
FROM foo
WHERE
bar = $P{bar}
AND
(
some_date > $P{some.date}
OR 1 = $P{is_null_pram}
)
I think you can use the function:
$X{EQUAL, <column_name>, <parameter_name>}
It optimizes the query as you can see in this help page.
You can use this conditional statement
select *
from foo
where bar = $P{bar} and some_date > $P{some.date} or $P{some.date} is null