I have input fixed width txt file as source.
test file sample below
column_1
12ABC3455
13XYZ5678
How to build dynamic column pattern to produce derived columns.
column Name : empId -> substring(column_1,1,2)
derive Column setting
I can hardcode the empid in & substring(column_1,1,2) in expression.
but i need to make it dynamic with the JSON input to derive dynamic derived columns with column pattern.
Below sample JSON input parameter.
My input JSON formatted parameter
[
{
"colname": "empid",
"startpos": 1,
"length": 2
},
{
"colname": "empname",
"startpos": 3,
"length": 3
},
{
"colname": "empSal",
"startpos": 6,
"length": 4
}
]
help me to build the column pattern with the json input
I tested many times and can't achieve that.
Just per my experience, I'm afraid to tell you that it's impossible to to that in Data Factory actives or Data Flow with json parameter.
Related
I have a table in postgres called day, which contains a jsonb column called plan_activities.
I have a day record with day.id = 18 and plan_activities contains the following JSON:
[
{
"activity": "Gym",
"cardio": false,
"strength": true,
"quantity": 20,
"units": "mins",
"timeOfDay": "Evening",
"summary": "Gym - 20 mins - Evening",
"timeOfDayOrder": 4
},
{
"activity": "Walk",
"cardio": true,
"strength": false,
"quantity": 15,
"units": "minutes",
"timeOfDay": "morning",
"summary": "Walk - 15 minutes - Lunchtime",
"timeOfDayOrder": 1
}
]
When I execute the following query:
select jsonb_array_elements(day.plan_activities) as activities
from day
where day.id = 18;
I get the following error:
Failed to run sql query: cannot extract elements from a scalar
The JSON contains a valid JSON array as far as I can tell. What am I doing wrong?
My eventual goal if I can extract this list is to create separate records elsewhere, each of which contains all the fields plus a reference back to the day record.
This error happens when you try to treat a JSON scalar, like a single string or number, as an array.
-- ERROR: cannot extract elements from a scalar
select jsonb_array_elements('23'::jsonb);
One of the rows of your query does not contain a JSON array.
Check with select plan_activities from day where id = 18. Although id is normally a unique primary key and it should be impossible to have more than one row returned.
Another way this could happen is if the JSON structure was accidentally added as a single JSON string.
-- 1, 2, 3
select jsonb_array_elements('[1, 2, 3]'::jsonb);
-- Note the extra quotes.
-- ERROR: cannot extract elements from a scalar
select jsonb_array_elements('"[1, 2, 3]"'::jsonb);
I have a table name ‘my doc’. And there is a column which is a nested json called ‘element’. The structure is as below.
I want to extract league_id. How to make it with SQL? Thanks
{
“api”: {
“Results
Different SQL implementations have different standards..
E.g. mysql can handle a JSON field in their table
-- returns {"a": 1, "b": 2}:
SELECT JSON_OBJECT('a', 1, 'b', 2);
https://www.sitepoint.com/use-json-data-fields-mysql-databases/
I have the below sample JSON:
{
"Id1": {
"name": "Item1.jpg",
"Status": "Approved"
},
"Id2": {
"name": "Item2.jpg",
"Status": "Approved"
}
}
and I am trying to get the following output:
_key name Status
Id1 Item1.jpg Approved
Id2 Item2.jpg Approved
Is there any way I can achieve this in Snowflake using SQL?
You should use Snowflake's VARIANT data type in any column holding JSON data. Let's break this down step by step:
create temporary table FOO(v variant); -- Temp table to hold the JSON. Often you'll see a variant column simply called "V"
-- Insert into the variant column. Parse the JSON because variants don't hold string types. They hold semi-structured types.
insert into FOO select parse_json('{"Id1": {"name": "Item1.jpg", "Status": "Approved"}, "Id2": {"name": "Item2.jpg", "Status": "Approved"}}');
-- See how it looks in its raw state
select * from FOO;
-- Flatten the top-level JSON. The flatten function breaks down the JSON into several usable columns
select * from foo, lateral flatten(input => (foo.v)) ;
-- Now traverse the JSON using the column name and : to get to the property you want. Cast to string using ::string.
-- If you must have exact case on your column names, you need to double quote them.
select KEY as "_key",
VALUE:name::string as "name",
VALUE:Status::string as "Status"
from FOO, lateral flatten(input => (FOO.V)) ;
I try to export the big query table to Google Cloud storage with format specified as JSON
Here, I noticed like columns with null values are not included in resulting JSON files
So Is there a way to get all the fields of row to be converted into JSON ?
My intention is to export data from table , do some transformations and reload the data back into new table . So , i basically need all fields to be included in generated JSON files .
For example , I tried exporting Bigquery-public-data.samples.wikipedia table
After exporting, JSON rows include only columns with non-null value
{
"title": "Strait of Messina Bridge",
"id": "1462053",
"language": "",
"wp_namespace": "0",
"revision_id": "115349459",
"contributor_ip": "80.129.30.196",
"timestamp": "1173977859",
"comment": "/* Controversy and concerns */",
"num_characters": "20009"
}
Few columns like contrinutor_id , contributor_username , others with null values are not included in generated JSON
I have a table say types, which had a JSON column, say location that looks like this:
{ "attribute":[
{
"type": "state",
"value": "CA"
},
{
"type": "distance",
"value": "200.00"
} ...
]
}
Each row in the table has the data, and all have the "type": "state" in it. I want to just extract the value of "type": "state" from every row in the table, and put it in a new column. I checked out several questions on SO, like:
Query for element of array in JSON column
Index for finding an element in a JSON array
Query for array elements inside JSON type
but could not get it working. I do not need to query on this. I need the value of this column. I apologize in advance if I missed something.
create table t(data json);
insert into t values('{"attribute":[{"type": "state","value": "CA"},{"type": "distance","value": "200.00"}]}'::json);
select elem->>'value' as state
from t, json_array_elements(t.data->'attribute') elem
where elem->>'type' = 'state';
| state |
| :---- |
| CA |
dbfiddle here
I mainly use Redshift where there is a built-in function to do this. So on the off-chance you're there, check it out.
redshift docs
It looks like Postgres has a similar function set:
https://www.postgresql.org/docs/current/static/functions-json.html
I think you'll need to chain three functions together to make this work.
SELECT
your_field::json->'attribute'->0->'value'
FROM
your_table
What I'm trying is a json extract by key name, followed by a json array extract by index (always the 1st, if your example is consistent with the full data), followed finally by another extract by key name.
Edit: got it working for your example
SELECT
'{ "attribute":[
{
"type": "state",
"value": "CA"
},
{
"type": "distance",
"value": "200.00"
}
]
}'::json->'attribute'->0->'value'
Returns "CA"
2nd edit: nested querying
#McNets is the right, better answer. But in this dive, I discovered you can nest queries in Postgres! How frickin' cool!
I stored the json as a text field in a dummy table and successfully ran this:
SELECT
(SELECT value FROM json_to_recordset(
my_column::json->'attribute') as x(type text, value text)
WHERE
type = 'state'
)
FROM dummy_table