How can I remove [" and "] in KQL from the below column values?
["xyz#test.com"]
["abc#test.com"]
["123#test.com"]
Thanks
I guess you would also like to remove the double quotes.
datatable(col:string)
[
'["xyz#test.com"]'
,'["abc#test.com"]'
,'["123#test.com"]'
]
| extend option_1 = trim(#'^\["|"]$', col) // Trim opening [" and closing "]
| extend option_2 = trim(#'[[\]"]+', col) // Trim sequences of the characters [, ] and "
| extend option_3 = extract(#'^\["(.*)"]$', 1, col) // Extract the expression between the opening [" and closing "]
| extend option_4 = tostring(todynamic(col)[0]) // Convert to json (resulting in json array) and retrieve the 1st element
col
option_1
option_2
option_3
option_4
["xyz#test.com"]
xyz#test.com
xyz#test.com
xyz#test.com
xyz#test.com
["abc#test.com"]
abc#test.com
abc#test.com
abc#test.com
abc#test.com
["123#test.com"]
123#test.com
123#test.com
123#test.com
123#test.com
Fiddle
Related
I want to remove an object from a json column in Sqlite and I can't make it work. The json column contains a nested object, has the following type:
{
a: number;
pair: {
field1: string;
field2: string;
}[]
}
I want to update the column "ArrayColumn" with the same values but remove the object that has field1 equal to "0" and field2 equal to "1" . Every row contains the "pair" array, but not all the "pair" arrays in ArrayColumn contain this value ({"field1":"0", "field2":"1"})
I have the following structure:
Id| ArrayColumn
--------------------------------------------------------------------------------------------
1 | { "a":1, "pair":[{"field1":"0", "field2":"1"},{"field1":"C", "field2":"D"},{"field1":"E", "field2":"F"}] }
2 | { "a":5, "pair":[{"field1":"C", "field2":"D"},{"field1":"E", "field2":"F"}] }
3 | { "a":8, "pair":[{"field1":"G", "field2":"G"},{"field1":"0", "field2":"1"},{"field1":"A", "field2":"A"}] }
4 | { "a":1, "pair":[{"field1":"F", "field2":"T"},{"field1":"C", "field2":"D"},{"field1":"0", "field2":"1"}] }
5 | { "a":1, "pair":[{"field1":"A", "field2":"B"}] }
After updating the rows, the values would be:
Id| ArrayColumn
--------------------------------------------------------------------------------------------
1 | { "a":1, "pair":[{"field1":"C", "field2":"D"},{"field1":"E", "field2":"F"}] }
2 | { "a":5, "pair":[{"field1":"C", "field2":"D"},{"field1":"E", "field2":"F"}] }
3 | { "a":8, "pair":[{"field1":"G", "field2":"G"},{"field1":"A", "field2":"A"}] }
4 | { "a":1, "pair":[{"field1":"F", "field2":"T"},{"field1":"C", "field2":"D"}] }
5 | { "a":1, "pair":[{"field1":"A", "field2":"B"}] }
I tried with JSON_TREE but can't make it work.
I was thinking that the first step would be to select all the rows that contain that value, I retreived them using these 2 ways:
With LIKE operator searching for the stringified form:
select Id, json_extract(json(par), '$.pair') as pair from Table pair like '%{"field1":"0","field2":"1"}%'
Using json_tree
select Id, value from Table, json_tree(Table.ArrayColumn, '$.pair' ) where json_extract(value, '$.field1' ) = '0' AND json_extract(value, '$.field2' ) = '1'
I tried using json_remove with this small example but no luck:
SELECT json_remove('[{"field1":"1","field2":"0"},{"field1":"A","field2":"B"}]', '${"field1":"1","field2":"0"}' )
I tried using json_remove but had no luck.
Thank you
For this sample data the simplest way to do this is to treat the json column as a string and use string functions to remove the value that you want:
UPDATE tablename
SET ArrayColumn = REPLACE(REPLACE(REPLACE(ArrayColumn, ']', ',]'), '{"field1":"0", "field2":"1"},', ''), ',]', ']')
WHERE ArrayColumn LIKE '%{"field1":"0", "field2":"1"}%';
See the demo.
I'm trying to parse a simple sql via this grammar:
grammar = ```
program : stmnt*
stmnt : select_stmnt | drop_stmnt
select_stmnt : select_clause from_clause? group_by_clause? having_clause? order_by_clause? limit_clause? SEMICOLON
select_clause : "select"i selectables
selectables : column_name ("," column_name)*
from_clause : "from"i source where_clause?
where_clause : "where"i condition
group_by_clause : "group"i "by"i column_name ("," column_name)*
having_clause : "having"i condition
order_by_clause : "order"i "by" (column_name ("asc"i|"desc"i)?)*
limit_clause : "limit"i INTEGER_NUMBER ("offset"i INTEGER_NUMBER)?
// NOTE: there should be no on-clause on cross join and this will have to enforced post parse
source : joining? table_name table_alias?
joining : source join_modifier? JOIN source ON condition
//source : table_name table_alias? joined_source?
//joined_source : join_modifier? JOIN table_name table_alias? ON condition
join_modifier : "inner" | ("left" "outer"?) | ("right" "outer"?) | ("full" "outer"?) | "cross"
condition : or_clause+
or_clause : and_clause ("or" and_clause)*
and_clause : predicate ("and" predicate)*
// NOTE: order of operator should be longest tokens first
predicate : comparison ( ( EQUAL | NOT_EQUAL ) comparison )*
comparison : term ( ( LESS_EQUAL | GREATER_EQUAL | LESS | GREATER ) term )*
term : factor ( ( "-" | "+" ) factor )*
factor : unary ( ( "/" | "*" ) unary )*
unary : ( "!" | "-" ) unary
| primary
primary : INTEGER_NUMBER | FLOAT_NUMBER | STRING | "true" | "false" | "null"
| IDENTIFIER
drop_stmnt : "drop" "table" table_name
FLOAT_NUMBER : INTEGER_NUMBER "." ("0".."9")*
column_name : IDENTIFIER
table_name : IDENTIFIER
table_alias : IDENTIFIER
// keywords
// define keywords as they have higher priority
SELECT.5 : "select"i
FROM.5 : "from"i
WHERE.5 : "where"i
JOIN.5 : "join"i
ON.5 : "on"i
// operators
STAR : "*"
LEFT_PAREN : "("
RIGHT_PAREN : ")"
LEFT_BRACKET : "["
RIGHT_BRACKET : "]"
DOT : "."
EQUAL : "="
LESS : "<"
GREATER : ">"
COMMA : ","
// 2-char ops
LESS_EQUAL : "<="
GREATER_EQUAL : ">="
NOT_EQUAL : ("<>" | "!=")
SEMICOLON : ";"
IDENTIFIER.9 : ("_" | ("a".."z") | ("A".."Z"))* ("_" | ("a".."z") | ("A".."Z") | ("0".."9"))+
%import common.ESCAPED_STRING -> STRING
%import common.SIGNED_NUMBER -> INTEGER_NUMBER
%import common.WS
%ignore WS
However, when I call the parser with text,
"""select cola, colb from foo left outer join bar b on x = 1 join jar j on jb > xw where cola <> colb and colx > coly""",
it parses the second join as a term, i.e. as part of the first join's condition. Any thoughts on how to do this correctly?
This is as example taken from another thread, but essentially I would like to achieve this:
Sample data
ID Name Value
1 TV1 {"URL": "www.url.com", "Icon": "some_icon"}
2 TV2 {"URL": "www.url.com", "Icon": "some_icon", "Facebook": "Facebook_URL"}
3 TV3 {"URL": "www.url.com", "Icon": "some_icon", "Twitter": "Twitter_URL"}
..........
Expected output
ID Name URL Icon Facebook Twitter
1 TV1 www.url.com some_icon NULL NULL
2 TV2 www.url.com some_icon Facebook_URL NULL
3 TV3 www.url.com some_icon NULL Twitter_URL
I'm totally new to Snowflake so I'm shaking my head on how to do this easily (and hopefully automatically, in the case where some rows might have more elements in the json than other rows, which would be tedious to assign manually). Some lines might have sub-categories too.
I found the parse_json function for Snowflake, but it's only giving me the same json column in a new column, still in json format.
TIA!
You can create a view over your table with the following SELECT:
SELECT ID,
Name,
Value:URL::varchar as URL,
Value:Icon::varchar as Icon,
Value:Facebook::varchar as Facebook,
Value:Twitter::varchar as Twitter
FROM tablename;
Additional attributes will be ignored unless you add them to the view. There is no way to "automatically" include them into the view, but you could create a stored procedure that dynamically generates the view based on all the attributes that are in the full variant content of a table.
You can create a SP to automatically build the CREATE VIEW for you based on the JSON data in the VARIANT.
I have some simple example below:
-- prepare the table and data
create or replace table test (
col1 int, col2 string,
data1 variant, data2 variant
);
insert into test select 1,2, parse_json(
'{"URL": "test", "Icon": "test1", "Facebook": "http://www.facebook.com"}'
), parse_json(
'{"k1": "test", "k2": "test1", "k3": "http://www.facebook.com"}'
);
insert into test select 3,4,parse_json(
'{"URL": "test", "Icon": "test1", "Twitter": "http://www.twitter.com"}'
), parse_json(
'{"k4": "v4", "k3": "http://www.ericlin.me"}'
);
-- create the SP, we need to know which table and
-- column has the variant data
create or replace procedure create_view(
table_name varchar
)
returns string
language javascript
as
$$
var final_columns = [];
// first, find out the columns
var query = `SHOW COLUMNS IN TABLE ${TABLE_NAME}`;
var stmt = snowflake.createStatement({sqlText: query});
var result = stmt.execute();
var variant_columns = [];
while (result.next()) {
var col_name = result.getColumnValue(3);
var data_type = JSON.parse(result.getColumnValue(4));
// just use it if it is not a VARIANT type
// if it is variant type, we need to remember this column
// and then run query against it later
if (data_type["type"] != "VARIANT") {
final_columns.push(col_name);
} else {
variant_columns.push(col_name);
}
}
var columns = {};
query = `SELECT ` + variant_columns.join(', ') + ` FROM ${TABLE_NAME}`;
stmt = snowflake.createStatement({sqlText: query});
result = stmt.execute();
while (result.next()) {
for(i=1; i<=variant_columns.length; i++) {
var sub_result = result.getColumnValue(i);
if(!sub_result) {
continue;
}
var keys = Object.keys(sub_result);
for(j=0; j<keys.length; j++) {
columns[variant_columns[i-1] + ":" + keys[j]] = keys[j];
}
}
}
for(path in columns) {
final_columns.push(path + "::STRING AS " + columns[path]);
}
var create_view_sql = "CREATE OR REPLACE VIEW " +
TABLE_NAME + "_VIEW\n" +
"AS SELECT " + "\n" +
" " + final_columns.join(",\n ") + "\n" +
"FROM " + TABLE_NAME + ";";
snowflake.execute({sqlText: create_view_sql});
return create_view_sql + "\n\nVIEW created successfully.";
$$;
Execute the SP will return below string:
call create_view('TEST');
+---------------------------------------+
| CREATE_VIEW |
|---------------------------------------|
| CREATE OR REPLACE VIEW TEST_VIEW |
| AS SELECT |
| COL1, |
| COL2, |
| DATA1:Facebook::STRING AS Facebook, |
| DATA1:Icon::STRING AS Icon, |
| DATA1:URL::STRING AS URL, |
| DATA2:k1::STRING AS k1, |
| DATA2:k2::STRING AS k2, |
| DATA2:k3::STRING AS k3, |
| DATA1:Twitter::STRING AS Twitter, |
| DATA2:k4::STRING AS k4 |
| FROM TEST; |
| |
| VIEW created successfully. |
+---------------------------------------+
Then query the VIEW:
SELECT * FROM TEST_VIEW;
+------+------+-------------------------+-------+------+------+-------+-------------------------+------------------------+------+
| COL1 | COL2 | FACEBOOK | ICON | URL | K1 | K2 | K3 | TWITTER | K4 |
|------+------+-------------------------+-------+------+------+-------+-------------------------+------------------------+------|
| 1 | 2 | http://www.facebook.com | test1 | test | test | test1 | http://www.facebook.com | NULL | NULL |
| 3 | 4 | NULL | test1 | test | NULL | NULL | http://www.ericlin.me | http://www.twitter.com | v4 |
+------+------+-------------------------+-------+------+------+-------+-------------------------+------------------------+------+
Query the source table:
SELECT * FROM TEST;
+------+------+------------------------------------------+-----------------------------------+
| COL1 | COL2 | DATA1 | DATA2 |
|------+------+------------------------------------------+-----------------------------------|
| 1 | 2 | { | { |
| | | "Facebook": "http://www.facebook.com", | "k1": "test", |
| | | "Icon": "test1", | "k2": "test1", |
| | | "URL": "test" | "k3": "http://www.facebook.com" |
| | | } | } |
| 3 | 4 | { | { |
| | | "Icon": "test1", | "k3": "http://www.ericlin.me", |
| | | "Twitter": "http://www.twitter.com", | "k4": "v4" |
| | | "URL": "test" | } |
| | | } | |
+------+------+------------------------------------------+-----------------------------------+
You can refine this SP to detect nested data and have them added to the columns list as well.
{
"address" : <address>
}
Examples:
| address |
| 00247 |
And match $.address == <address>
I have above case where I need to pass 'address' attribute as integer and not a string.
But script passes it as 247 and not 00247.
actual: '247', expected: '00247', reason: not equal
Your help would be appreciated. Thank you!
Covered in the docs: https://github.com/intuit/karate#scenario-outline-enhancements
Make this change:
Scenario Outline:
* def body = { address: '#(address)' }
* And match $.address == address
Examples:
| address! |
| 00247 |
I have one column table in my snowflake database that contain a JSON mapping structure as following
ColumnMappings : {"Field Mapping": "blank=Blank,E=East,N=North,"}
How to write a query that if I feed the Field Mapping a value of E I will get East or if the value if N I will get North so on and so forth without hard coding the value in the query like what CASE statement provides.
You really want your mapping in this JSON form:
{
"blank" : "Blank",
"E" : "East",
"N" : "North"
}
You can achieve that in Snowflake e.g. with a simple JS UDF:
create or replace table x(cm variant) as
select parse_json(*) from values('{"fm": "blank=Blank,E=East,N=North,"}');
create or replace function mysplit(s string)
returns variant
language javascript
as $$
res = S
.split(",")
.reduce(
(acc,val) => {
var vals = val.split("=");
acc[vals[0]] = vals[1];
return acc;
},
{});
return res;
$$;
select cm:fm, mysplit(cm:fm) from x;
-------------------------------+--------------------+
CM:FM | MYSPLIT(CM:FM) |
-------------------------------+--------------------+
"blank=Blank,E=East,N=North," | { |
| "E": "East", |
| "N": "North", |
| "blank": "Blank" |
| } |
-------------------------------+--------------------+
And then you can simply extract values by key with GET, e.g.
select cm:fm, get(mysplit(cm:fm), 'E') from x;
-------------------------------+--------------------------+
CM:FM | GET(MYSPLIT(CM:FM), 'E') |
-------------------------------+--------------------------+
"blank=Blank,E=East,N=North," | "East" |
-------------------------------+--------------------------+
For performance, you might want to make sure you call mysplit only once per value in your mapping table, or even pre-materialize it.