I'm using the OpenJSON function in SQL to import various JSON format files into SQL and can usually handle the variations in formats from various sources, however I've got an example where I can't reach a certain value.
Example JSON format file:
{
"bob": {
"user_type": "standard",
"user_enabled": "true",
"last_login": "2021-07-25"
},
"claire": {
"user_type": "administrator",
"user_enabled": "true",
"last_login": "2021-09-17"
}
}
One of the values I want to return as one of my columns is the user's name;
I believe it's called the key but not entirely sure, because when I execute the following having loaded the json string into the #json variable:
select *
from openjson(#json)
I get two columns, one labelled key containing the username, the other containing my nested json string within {} braces.
Usually, to run my select statement, I would do something like
select username,user_type,user_enabled,last_login
from openjson(#thisjson)
with (
username nvarchar(100),
user_type nvarchar(100),
user_enabled nvarchar(100),
last_login nvarchar(100)
)
I get that sometimes I have to put the base in the brackets after openjson, and sometimes I have to follow the input column definitions with something like '$.last_login' to help traverse the structure, but can't work out how to identify or select the placeholder for the username.
Related
I have a very simple table on BigQuery that I've created from another table. One of the fields is the user_email (string), which I've extracted from a JSON field with JSON_QUERY(json_field, "$.email").
I would like to get some specific emails from the table with a query that looks like this:
SELECT user_email
FROM my_table
WHERE user_email IN ("blabla#example.com", "blabla2#example.com"...)
The emails in the list from above DO exist, but I'm not getting any results from the query.
Any idea?
You should use JSON_VALUE instead of JSON_QUERY to extract a value as a BigQuery String. JSON QUERY returns a value as JSON-compatible String, not BigQuery String.
For example,
SELECT JSON_QUERY('{ "email": "blabla#example.com" }', '$.email') email;
Note that returned value is wrapped with double quotes.
But if you use JSON_VALUE,
SELECT JSON_VALUE('{ "email": "blabla#example.com" }', '$.email') email;
You can get a value without double quotes and be able to compare it with other BigQuery Strings IN ("blabla#example.com", "blabla2#example.com"...)
I have an SQL table called FileGroups. I will query for certain columns such as Id, Name and Version. In my Azure Data Factory pipeline, I want to map the column names and row values from the SQL query results to key-value pairs for a JSON string. I also need to include a couple pipeline parameters in the JSON string. I will then pass this JSON string as input for a stored procedure at the end of my pipeline.
The resulting JSON string will look like this:
{
"id": "guid1",
"name": "fileGroup1",
"version": 1.0,
"pipeline_param_1": "value1",
"pipeline_param_2": "value2"
},
{
"id": "guid2",
"name": "fileGroup2",
"version": 2.0,
"pipeline_param_1": "value1",
"pipeline_param_2": "value2"
}
How do I query the SQL table and construct this JSON string all within my ADF pipeline? What activities or data flow transformations do I need to achieve this?
the easiest way to implement it is by using a "copy activity"
Here is a quick demo that i created, i want to transform SQL data into Json, i copied SalesLT.Customer data from sql sample data
created SQL database with sample data in azure portal.
In azure data factory, i added the database as a dataset.
created a pipeline and i named it "mapSQLDataToJSON"
in the pipeline , i added a "Copy activity"
in copy activity, i added the sql db as a source dataset and added a query option , Query : "#concat('select CustomerID,Title, pipeId= ''', pipeline().RunId,''' from SalesLT.Customer')"
here you can select the columns that you need and add to the data a new column like i did , added a new column "pipId" and used pipeline params.
in copy activity i added the blob storage as a sink and data type to be "Json"
tested connections and triggered the pipeline
i opened the blob storage , and i clicked on the copied Json data , and it worked.
Copy activity in ADF:
Data in blob storage:
you can read here about copy activity and pipeline params , links:
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-system-variables
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview
If your source database is a Microsoft SQL database, like Azure SQL DB, Sql Server, Managed Instance, Azure Synapse Analytics etc, then it is quite capable manipulating JSON. The FOR JSON clause constructs valid JSON and you can use options like WITHOUT_ARRAY_WRAPPER to produce clean output.
A simple example:
DROP TABLE IF EXISTS #tmp;
CREATE TABLE #tmp (
id VARCHAR(10) NOT NULL,
[name] VARCHAR(20) NOT NULL,
[version] VARCHAR(5) NOT NULL,
pipeline_param_1 VARCHAR(20) NOT NULL,
pipeline_param_2 VARCHAR(20) NOT NULL
);
INSERT INTO #tmp VALUES
( 'guid1', 'fileGroup1', '1.0', 'value1.1', 'value1.2' ),
( 'guid2', 'fileGroup2', '2.0', 'value2.1', 'value2.2' )
SELECT *
FROM #tmp
FOR JSON PATH, WITHOUT_ARRAY_WRAPPER;
Sample output:
I currently have a table sessions with a column actions(JSONB).
I am able to properly store an array from my frontend to my database using array_to_jsonb(array value).
The structure of my array (array value) looks like this:
var arrayStructure = [
{name: 'email client', check: false},
{name: 'send contract', check: false}
]
Unlike a lot of the questions I've seen on SO about modifying JSONB or accessing certain keys, I am only interested in converting the entire JSONB data back to an array for the frontend.
My temporary fix is to map through the array and use JSON.parse() in Javascript to restructure each object in the array. I cannot use JSON.parse() on the entire array, as it throws an error.
I'm looking for a query to bring back the array format to the JSONB data type stored. I've seen JSONB_SET, LATERAL, AND JSONB_ARRAY_ELEMENTS_TEXT. But not in a way that worked to bring back a proper array.
Starts As: JSONB in actions column in table named sessions
Should Result In: A query that brings back all rows, but with the actions column (JSONB) converted back to an array for the frontend:
select session_id, session_name, someFunction or lateral here(actions) from sessions
Screenshot of Sessions Table
I've tried queries like this:
SELECT
session_id,
actions::jsonb -> 'name' as name
FROM sessions;
And receive back null for name. I've tried ->> to access a deeper level, but that didn't work either.
This is half of the correct query result:
select session_id, jsonb_array_elements_text(actions)
from sessions
group by session_id;
Which results in this (only pay attention to results for session_id of 264):
query result
Now I have objects in their own rows as:
{"name": "some task", "check": "false}
When what I want for the actions column is:
[ {name: "some task", check: false}, {name: "other task", check: true} ]
So I need to further parse the JSON and group by session_id. I'm just struggling to build a sub-query that does that.
Steps to Create Set Up:
create table fakeSessions (
session_id serial primary key,
name varchar(20),
list jsonb
)
insert into fakeSessions(name, list)
VALUES(
'running',
'["{\"name\":\"inquired\",\"check\":false}", "{\"name\":\"sent online guide\",\"check\":false}", "{\"name\":\"booked!\",\"check\":false}"]'
)
insert into fakeSessions(name, list)
VALUES(
'snowboarding',
'["{\"name\":\"rental\",\"check\":false}", "{\"name\":\"booked ski passes\",\"check\":false}", "{\"name\":\"survey\",\"check\":false}"]'
)
The closest query I've created:
with exports as (
select jsonb_array_elements_text(actions)::jsonb as doc from sessions
)
select array_agg(doc) from
exports, sessions
group by session_id;
Get the text values, and then apply an aggregate function to those returned rows. Just can't get the select array_agg(doc) to work as expected. Most likely because I need a different function in that place.
Does this help?
demo:db<>fiddle
SELECT
jsonb_agg(elem)
FROM
sessions, jsonb_array_elements(actions) as elem
jsonb_array_elements() expands the jsonb array into one row each jsonb element
jsonb_agg() aggregates these jsonb elements into one big array.
I was able to reach the answer by building off of your query! Thank you so much. The query that got the expected outcome was this:
with exports as (
select session_id, jsonb_array_elements_text(actions)::jsonb as doc from sessions
)
select session_id, jsonb_agg(doc) from exports
group by session_id;
I wasn't using the jsonb_agg() function once I got the elements. The difference for getting the exact format was just using jsonb_array_elements_text::jsonb
I am attempting to insert a new row into a table with a column defined as an array or UUIDs:
alter table medias add column "order" uuid[];
I am using Objection.js ORM and attempting to execute the following query:
const order = [
'BFAD6B0D-D3E6-4EB3-B3AB-108244A5DD7F'
]
Medias
.query()
.insert({
order: lit(order.map(id => lit(id).castType('uuid'))).castArray()
})
But the query is malformed and therefore does not execute:
INSERT INTO xxx ("order")
VALUES (ARRAY [
{"_value":"BFAD6B0D-D3E6-4EB3-B3AB-108244A5DD7F","_cast":"uuid","_toJson":false,"_toArray":false}
])
As can be seen, the query contains the JSON-stringified representation of the LiteralBuilder object and not something that the SQL syntax understands as a typecast.
If I skip casting the individual UUID strings and just cast the whole column into an array, then Postgres rejects the query because the column is of type uuid[] but I am attempting to insert the column as text[].
How can I format this query using Objection.js ORM?
My goal is to keep the column definition untouched and be able to insert a Postgres' array of UUIDs using Objection.js, either through its API or via raw query. If this is not currently possible with Objection, I am willing, as a last resort, to re-define the column as text[], but I would like to make sure I really have no other option.
I have an app that uses a local sqlite database which has a particular table called 'Theme' for each project. Sometimes that table has a column 'startDate' and sometimes it does not. I need to have that 'startDate' returned in a particular format if it does exist. My problem is, when I query this table, specifying the neccessary format, if the column does not exist, the query returns an error "NO SUCH COLUMN".
HOW DO I CHECK FOR COLUMN EXISTENCE, IF IT DOES EXIST, RETURN THE 'startDate' PROPERLY FORMATTED ALONG WITH THE REST OF THE DATA, IF IT DOES NOT EXIST, RETURN THE REST OF THE DATA WITHOUT THE 'startDate'???
This must be done in 1 query!
Something like this...
SELECT * (if exists STRFTIME('%Y/%m/%d %H:%M:%S', startDate) AS sDate FROM Theme
Only one query:
Cursor cursor = database.query(TABLE_NAME, null, null, null, null, null, null);
if(cursor.moveToFirst()) {
do {
for(String columnName : cursor.getColumnNames()) {
// do something
}
} while(cursor.moveToNext());
} else {
// your actions for empty table
}
cursor.close();