There are a lot of posts on unnesting repeated fields in BigQuery -- but, being new to this environment, I have tried almost every code variation I found to flatten a data file. But, I cannot seem to produce one without creating blanks in the id field. It seem like I need to unflatten a nested variable?
I'm using a COVID Dimensions data set that is part of the public collection. Here is some minimal code that produces my problem:
SELECT
id,
authors
FROM
`covid-19-dimensions-ai.data.publications`
CROSS JOIN
UNNEST(authors)
LIMIT 1000
And, here is the JSON structure after running this query. Everything is flattened with the structure I want, but I don't know how to fill in / avoid the blank id variables.
{
"id": "pub.1130234899",
"authors": {
"first_name": "Eric M",
"last_name": "Yoshida",
"initials": null,
"researcher_id": "ur.01071531321.03",
"grid_ids": [
"grid.17091.3e"
],
"corresponding": false,
"raw_affiliations": [
"Division of Gastroenterology, University of British Columbia, Vancouver, British Columbia, Canada"
],
"affiliations_address": [
{
"grid_id": "grid.17091.3e",
"city_id": "6173331",
"state_code": "CA-BC",
"country_code": "CA",
"raw_affiliation": "Division of Gastroenterology, University of British Columbia, Vancouver, British Columbia, Canada"
}
]
}
}
See small correction to your original query
SELECT
id,
author
FROM
`covid-19-dimensions-ai.data.publications`
CROSS JOIN
UNNEST(authors) author
LIMIT 1000
Related
I have a posts table with few columns including a liked_by column which's type is an int array.
As I can't post the table here I'll post a single post's JSON structure which comes as below
"post": {
"ID": 1,
"CreatedAt": "2022-08-15T11:06:44.386954+05:30",
"UpdatedAt": "2022-08-15T11:06:44.386954+05:30",
"DeletedAt": null,
"title": "Pofst1131",
"postText": "yyhfgwegfewgewwegwegwegweg",
"img": "fegjegwegwg.com",
"userName": "AthfanFasee",
"likedBy": [
3,
1,
4
],
"createdBy": 1,
}
I'm trying to send posts in the order they are liked (Most Liked Posts). Which should order the posts according to the number of values inside the liked_by array. How can I achieve this in Postgres?
For a side note, I'm using Go lang with GORM ORM but I'm using raw SQL builder instead of ORM tools. I'll be fine with solving this problem using go lang as well. The way I achieved this in MongoDB and NodeJS is to group by the size of liked by array and add a total like count field and sort using that field as below
if(sort === 'likesCount') {
data = Post.aggregate([
{
$addFields: {
totalLikesCount: { $size: "$likedBy" }
}
}
])
data = data.sort('-totalLikesCount');
} else {
data = data.sort('-createdAt') ;
}
Use a native query.
Provided that the table column that contains the sample data is called post, then
select <list of expressions> from the_table
order by json_array_length(post->'likedBy') desc;
Unrelated but why don't you try a normalized data design?
Edit
Now that I know your table structure here is the updated query. Use array_length.
select <list of expressions> from public.posts
order by array_length(liked_by, 1) desc nulls last;
You may also wish to add a where clause too.
Last thing in our SQL Beginners course was to tip our toes on few other DB:s and I chose MongoDB. The last and the "Hardest" thing I can do as bonus round is to turn this sqlite command to MongoDB collection line.
sqlite> SELECT ore, COUNT(*), MAX(price) FROM Database GROUP BY ore;
I created the DB with these values:
db.Database.insertMany( [
{ biome: "Desert", ore: "Silver", price: 8000 },
{ biome: "Forest", ore: "Gold", price: 5000 },
{ biome: "Meadow" , ore: "Silver", price: 7000 },
{ biome: "Swamp", ore: "Bronze", price: 6000 },
{ biome: "Mountains", ore: "Gold", price: 9000 },
{ biome: "Arctic" , ore: "Gold", price: 6500 }
] )
So yeah... I have been reading about pipelines and aggregation operations, but it is flying over my head xP. This is not vital for this course but I would love to learn how this goes. My school has very bad habit of teaching everything in our native language, even if no one in their right minds would ever use that terminology in real life. This makes it sometimes extra hard for me to learn these things while trying to study on my own. If anyone want to give any examples I would be grateful!
End result should look something like this:
sqlite> SELECT ore, COUNT(*), MAX(price) FROM Database GROUP BY ore;
ore COUNT(*) MAX(price)
---------- ---------- -----------
Silver 2 8000
Bronze 1 6000
Gold 3 9000
What you are looking for is the $group stage.
The $group stage is used to group multiple documents in a collection based on one or many keys. You can learn more about this pipeline stage here.
You will mention the keys you want to group by in the _id key of the Group stage.
all the rest of the keys are user-defined with can be accumulated with MongoDB's built-in operators.
In your case, you can make use of the $sum operator and pass in the value 1 to add one value to the user-defined count key for each document grouped.
And to find the max price, make use of the $max key and pass in $price (note $ prefix since you are self-referencing a key in the source document) to get the max value of a single group.
db.collection.aggregate([
{
"$group": {
"_id": "$ore",
"count": {
"$sum": 1
},
"max": {
"$max": "$price"
}
},
},
])
Mongo Playground Sample Execution
I am new to couchbase and to non-relational DB.
I have a bucket with players and teams(2 types of documents).
each player has type, playedFor(an array with all the teams he played) and a name for example:
{
"type":"player"
"name":"player1"
"playedFor": [
"England/Manchester/United"
"England/Manchester/City"
]
}
each team has type, name and category for example:
{
"type": "team"
"name": "England/Manchester/City"
"category": "FC"
}
I want to know how many players played for each team of category FC.
I made this query to calc for specific team:
SELECT COUNT(1) AS total
FROM bucket AS a
WHERE a.type='player'
AND (any r in a.playedFor satisfies r in ["England/Manchester/United"] end)
but how can i make this query for all teams?
The wrinkle in the way you've modeled this data is that player can play for 1 or more teams (hence the array).
One way to approach this is to use Couchbase's UNNEST clause to "flatten" these arrays (it's basically joining the document to each of the items in the array).
At that point, it becomes as easy as a standard GROUP BY. Here's an example:
SELECT team, count(1) AS totalPlayers
FROM `bucket` AS a
UNNEST a.playedFor team
WHERE a.type='player'
GROUP BY team
This query would generate output like:
[
{
"team": "Pittsburgh/Pirates",
"totalPlayers": 8
},
{
"team": "England/Manchester/United",
"totalPlayers": 10
},
{
"team": "England/Manchester/City",
"totalPlayers": 15
},
{
"team": "Cincinnati/Reds",
"totalPlayers": 21
}
]
(Sorry, I used MLB teams to augment your sample, since I don't know much about soccer teams).
Notice that the separate team documents don't figure into this query, but you could also JOIN to them if you need information from them for your quer(ies).
I have a S3 bucket with 500,000+ json records, eg.
{
"userId": "00000000001",
"profile": {
"created": 1539469486,
"userId": "00000000001",
"primaryApplicant": {
"totalSavings": 65000,
"incomes": [
{ "amount": 5000, "incomeType": "SALARY", "frequency": "FORTNIGHTLY" },
{ "amount": 2000, "incomeType": "OTHER", "frequency": "MONTHLY" }
]
}
}
}
I created a new table in Athena
CREATE EXTERNAL TABLE profiles (
userId string,
profile struct<
created:int,
userId:string,
primaryApplicant:struct<
totalSavings:int,
incomes:array<struct<amount:int,incomeType:string,frequency:string>>,
>
>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ( 'ignore.malformed.json' = 'true')
LOCATION 's3://profile-data'
I am interested in the incomeTypes, eg. "SALARY", "PENSIONS", "OTHER", etc.. and ran this query changing jsonData.incometype each time:
SELECT jsonData
FROM "sampledb"."profiles"
CROSS JOIN UNNEST(sampledb.profiles.profile.primaryApplicant.incomes) AS la(jsonData)
WHERE jsonData.incometype='SALARY'
This worked fine with CROSS JOIN UNNEST which flattened the incomes array so that the data example above would span across 2 rows. The only idiosyncratic thing was that CROSS JOIN UNNEST made all the field names lowercase, eg. a row looked like this:
{amount=1520, incometype=SALARY, frequency=FORTNIGHTLY}
Now I have been asked how many users have two or more "SALARY" entries, eg.
"incomes": [
{ "amount": 3000, "incomeType": "SALARY", "frequency": "FORTNIGHTLY" },
{ "amount": 4000, "incomeType": "SALARY", "frequency": "MONTHLY" }
],
I'm not sure how to go about this.
How do I query the array of structures to look for duplicate incomeTypes of "SALARY"?
Do I have to iterate over the array?
What should the result look like?
UNNEST is a very powerful feature, and it's possible to solve this problem using it. However, I think using Presto's Lambda functions is more straight forward:
SELECT COUNT(*)
FROM sampledb.profiles
WHERE CARDINALITY(FILTER(profile.primaryApplicant.incomes, income -> income.incomeType = 'SALARY')) > 1
This solution uses FILTER on the profile.primaryApplicant.incomes array to get only those with an incomeType of SALARY, and then CARDINALITY to extract the length of that result.
Case sensitivity is never easy with SQL engines. In general I think you should not expect them to respect case, and many don't. Athena in particular explicitly converts column names to lower case.
You can combine filter with cardinality to filter array elements having incomeType = 'SALARY' more than once.
This can be further improve so that intermediate array is not materialized by using reduce (see examples in the docs; I'm not quoting them here, since they do not directly answer your question).
Query:
"SELECT * FROM [table] ORDER BY id DESC LIMIT 10"
AllowLargeResults = true
FlattenResults = false
table schema:
[
{
"name": "id",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "repeated_field_1",
"type": "STRING",
"mode": "REPEATED"
},
{
"name": "repeated_field_2",
"type": "STRING",
"mode": "REPEATED"
}
]
The query "SELECT * FROM [table] LIMIT 10" works just fine. I get this error when I add an order by clause, even though the order by does not mention either repeated field.
Is there any way to make this work?
The ORDER BY clause causes BigQuery to automatically flatten the output of a query, causing your query to attempt to generate a cross product of repeated_field_1 and repeated_field_2.
If you don't care about preserving the repeatedness of the fields, you could explicitly FLATTEN both of them, which will cause your query to generate the cross product that the original query is complaining about.
SELECT *
FROM FLATTEN(FLATTEN([table], repeated_field_1), repeated_field_2)
ORDER BY id DESC
LIMIT 10
Other than that, I don't have a good workaround for your query to both ORDER BY and also output repeated fields.
See also: BigQuery flattens result when selecting into table with GROUP BY even with “noflatten_results” flag on