Extract JSON content in Metabase SQL query - sql

Using: Django==2.2.24, Python=3.6, PostgreSQL is underlying DB
Working with Django ORM, I can easily make all sort of queries, but I started using Metabase, and my SQL might be a bit rusty.
The problem:
I am trying to get a count of the items in a list, under a key in a dictionary, stored as a JSONField:
from django.db import models
from jsonfield import JSONField
class MyTable(models.Model):
data_field = JSONField(blank=True, default=dict)
Example of the dictionary stored in data_field:
{..., "my_list": [{}, {}, ...], ...}
Under "my_list" key, the value stored is a list, which contains a number of other dictionaries.
In Metabase, I am trying to get a count for the number of dictionaries in the list, but even more basic things, none of which work.
Some stuff I tried:
Attempt:
SELECT COUNT(elem->'my_list') as my_list_count
FROM my_table, json_object_keys(data_field:json) AS elem
Error:
ERROR: syntax error at or near ":" Position: 226
Attempt:
SELECT ARRAY_LENGTH(elem->'my_list') as my_list_count
FROM my_table, JSON_OBJECT_KEYS(data_field:json) AS elem
Error:
ERROR: syntax error at or near ":" Position: 233
Attempt:
SELECT JSON_ARRAY_LENGTH(data_field->'my_list'::json)
FROM my_table
Error:
ERROR: invalid input syntax for type json Detail: Token "my_list" is invalid. Position: 162 Where: JSON data, line 1: my_list
Attempt:
SELECT ARRAY_LENGTH(JSON_QUERY_ARRAY(data_field, '$.my_list'))
FROM my_table
Error:
ERROR: function json_query_array(text, unknown) does not exist Hint: No function matches the given name and argument types. You might need to add explicit type casts. Position: 140
Basically, I think the issue is that I am using the wrong signatures (most of the time) in the methods I am trying to use.
I used this query to make sure I can at least get the keys from the dictionary:
SELECT JSON_OBJECT_KEYS(data_field::json)
FROM my_table
I was not able to use JSON_OBJECT_KEYS() without adding the ::json cast, I was getting this error:
ERROR: function json_object_keys(text) does not exist Hint: No function matches the given name and argument types. You might need to add explicit type casts. Position: 127
But with the json cast, I am getting all the keys as intended.
Thank you for taking a look!
EDIT:
I also found this interesting article with different solution but none of the solutions worked.
Also seen this SO post which did not help.

Ok, after some more digging around, I found this article, which had the correct format/syntax.
This code is what I used to fetch the list from the JSON object successfully:
select data_field::json->'my_list' as the_list
from my_table
Then, I used json_array_length() to get the number of elements:
select json_array_length(data_field::json->'my_list') as number_of_elements
from my_table
All done! :)
EDIT:
I just found the reason to this whole shenanigan.
In the code (which goes years back) we used this package:
jsonfield==1.0.3
And used this way:
from jsonfield import JSONField
The issue is that in the background, Postgres saves the data as a string, so it needs to be cast into a JSON.
Later Django introduced its own JSONField, which stores data as you would expect, without a need to cast:
from django.contrib.postgres.fields import JSONField

Related

Error : Field "S_MARA-MATNR" is unkown during FOR statement

DATA: t_mara type STANDARD TABLE OF mara WITH EMPTY KEY.
DATA(t_data1) = VALUE ty_data( FOR s_mara IN t_mara ( s_mara–matnr ) ).
I am trying to implement a similar code using FOR statement but I am getting an error that the field is unknown in the work area even though it would be declared inline.
Can you please let me know what went wrong? This is my first time I am facing this error on FOR loop.
Not sure because you are not providing too much detail but try this:
DATA t_mara type STANDARD TABLE OF mara WITH EMPTY KEY.
DATA(t_data1) = VALUE ty_data( FOR s_mara IN t_mara ( matnr = s_mara-matnr ) ).

PostgresSQL - Converting String in JSONB to Actual JSONB, Error: Token is invalid

I am having trouble working with the JSONB structure in PostgreSQL. currently my data is saved as follows:
"{\"Hello\":\"World\",\"idx\":0}"
Which obviously is not correct 😀 so I am trying to "repair" this and get the actual JSON representation for querying with:
SELECT regexp_replace(trim('"' FROM json_data::text), '\\"', '"', 'g')::jsonb FROM My_table
However when trying this, I get the following error:
ERROR: invalid input syntax for type json
DETAIL: Token "Рыба" is invalid.
CONTEXT: JSON data, line 1: ...х, как : Люди X, Пароль \\"Рыба...
SQL state: 22P02
So I am thinking that this is due to the character encoding that is not being accepted by the JSONB standard.
My main question then is though, how can I repair this kind of table so that I am still able to query it? I tried utilizing conver_from and convert_to but am unable to figure out how to fix this error... did anyone encounter this already?
Update: Found it! (thanks to Convert JSON string to JSONB), utilizing
SELECT (json_data#>>'{}')::jsonb FROM my_table
fixed it

Google Cloud Datalow:Getting a below error at runtime

I am writing data into nested array BQ table(array name inside the table is -merchant_array)using my dataflow template.
Sometime its running fine and loading the data but sometime its giving me that error at run time.
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: com.fasterxml.jackson.databind.JsonMappingException: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?) (through reference chain: com.google.api.services.bigquery.model.TableRow["null"])
"message" : "Error while reading data, error message: JSON parsing error in row starting at position 223615: Only optional fields can be set to NULL. Field: merchant_array; Value: NULL",
Anyone has any idea why I am getting this error.
Thanks in advance.
here I got the issue that was causing error so I am posting my own question's answer,it might be helpful for anyone.
So the error was like-
Only optional fields can be set to NULL. Field: merchant_array; Value: NULL",
And here merchant_array is defined as an array that contains record (repetitive) data.
As per google doc the the array can not be-
ARRAYs cannot be NULL.
NULL ARRAY elements cannot persist to a table.
At the same time I was using arraylist in my code, that allows null values. So before making a record type data in code or setting the data in arraylist, just remove the NULL tablerows if exist.
hope this will helpful.

react-native-realm pattern matching causes invalid predicate

I have this line of code:
realm.objects('Users').filtered("profile LIKE '%athletic%'")
and I tried this line of code:
realm.objects('Users').filtered("profile LIKE '*athletic*'")
It gave the error profile LIKE '%athletic%':1:0: Invalid predicate when running the program. I am unable to find documentation for pattern matching in react-native-realm. How do I find records that return users who have profiles with the string athletic in it?
For that kind of query, you can use CONTAINS
realm.objects('Users').filtered("profile CONTAINS 'athletic'")
See more on the query language used in Realm-JS here

ActiveRecord where clause error

I'm a newbie to rails.I have a model called OfflineExport. In the table I have data like this
#<OfflineExport id: 2,
parameters:
{"project_id"=>"3",
"type"=>"submissions",
"filters"=>{"task_type"=>"",
"corrections"=>"", "grade"=>"",
"min_duration"=>"", "after"=>"",
"max_duration"=>"", "reviews"=>"",
"before"=>""},
"send_email"=>"true",
"options"=>{"offline_record_id"=>2}}>
Am trying to fetch parameters["project_id"] in where clause like
OfflineExport.where("parameters[project_id] = '3'")
But am getting error like:
ActiveRecord::StatementInvalid: PGError: ERROR: cannot subscript type text because it is not an array
can anyone help me in solving this?
It seems that you have a serialized column in your model. Something like this: serialize :parameters. This means the data is stored in the database not in an easily readable format, which leads that you can not query on it. Same as here.
Solution: extract the field you want to query on, and make a column for it.