I am having trouble working with the JSONB structure in PostgreSQL. currently my data is saved as follows:
"{\"Hello\":\"World\",\"idx\":0}"
Which obviously is not correct 😀 so I am trying to "repair" this and get the actual JSON representation for querying with:
SELECT regexp_replace(trim('"' FROM json_data::text), '\\"', '"', 'g')::jsonb FROM My_table
However when trying this, I get the following error:
ERROR: invalid input syntax for type json
DETAIL: Token "Рыба" is invalid.
CONTEXT: JSON data, line 1: ...х, как : Люди X, Пароль \\"Рыба...
SQL state: 22P02
So I am thinking that this is due to the character encoding that is not being accepted by the JSONB standard.
My main question then is though, how can I repair this kind of table so that I am still able to query it? I tried utilizing conver_from and convert_to but am unable to figure out how to fix this error... did anyone encounter this already?
Update: Found it! (thanks to Convert JSON string to JSONB), utilizing
SELECT (json_data#>>'{}')::jsonb FROM my_table
fixed it
Related
I have data in the comment column of the payments table. The data is stored as plain text in the following format:
{"foo"=>"bar"}
I need to query the value of the specific "foo" key and tried the following:
select comment::json -> 'foo' from payments
but because the data stored is not in JSON format I get the following error:
invalid input syntax for type json DETAIL: Token "=" is invalid. CONTEXT: JSON data, line 1: {"foo"=>"bar"}
which refers to the => that Ruby uses for Hashes.
Is there a way to convert the text data to JSON data on-the-fly so I can then access the specific keys I need?
You can replace the => with a : to make that example a valid JSON value:
replace(comment, '=>', ':')::jsonb ->> 'foo'
It sounds like the data is technically valid ruby which means we can do something a bit clever.
require 'json'
def parse_data(data_string)
eval(data_string).to_json
end
Should do the trick so long as the data is trusted.
Using: Django==2.2.24, Python=3.6, PostgreSQL is underlying DB
Working with Django ORM, I can easily make all sort of queries, but I started using Metabase, and my SQL might be a bit rusty.
The problem:
I am trying to get a count of the items in a list, under a key in a dictionary, stored as a JSONField:
from django.db import models
from jsonfield import JSONField
class MyTable(models.Model):
data_field = JSONField(blank=True, default=dict)
Example of the dictionary stored in data_field:
{..., "my_list": [{}, {}, ...], ...}
Under "my_list" key, the value stored is a list, which contains a number of other dictionaries.
In Metabase, I am trying to get a count for the number of dictionaries in the list, but even more basic things, none of which work.
Some stuff I tried:
Attempt:
SELECT COUNT(elem->'my_list') as my_list_count
FROM my_table, json_object_keys(data_field:json) AS elem
Error:
ERROR: syntax error at or near ":" Position: 226
Attempt:
SELECT ARRAY_LENGTH(elem->'my_list') as my_list_count
FROM my_table, JSON_OBJECT_KEYS(data_field:json) AS elem
Error:
ERROR: syntax error at or near ":" Position: 233
Attempt:
SELECT JSON_ARRAY_LENGTH(data_field->'my_list'::json)
FROM my_table
Error:
ERROR: invalid input syntax for type json Detail: Token "my_list" is invalid. Position: 162 Where: JSON data, line 1: my_list
Attempt:
SELECT ARRAY_LENGTH(JSON_QUERY_ARRAY(data_field, '$.my_list'))
FROM my_table
Error:
ERROR: function json_query_array(text, unknown) does not exist Hint: No function matches the given name and argument types. You might need to add explicit type casts. Position: 140
Basically, I think the issue is that I am using the wrong signatures (most of the time) in the methods I am trying to use.
I used this query to make sure I can at least get the keys from the dictionary:
SELECT JSON_OBJECT_KEYS(data_field::json)
FROM my_table
I was not able to use JSON_OBJECT_KEYS() without adding the ::json cast, I was getting this error:
ERROR: function json_object_keys(text) does not exist Hint: No function matches the given name and argument types. You might need to add explicit type casts. Position: 127
But with the json cast, I am getting all the keys as intended.
Thank you for taking a look!
EDIT:
I also found this interesting article with different solution but none of the solutions worked.
Also seen this SO post which did not help.
Ok, after some more digging around, I found this article, which had the correct format/syntax.
This code is what I used to fetch the list from the JSON object successfully:
select data_field::json->'my_list' as the_list
from my_table
Then, I used json_array_length() to get the number of elements:
select json_array_length(data_field::json->'my_list') as number_of_elements
from my_table
All done! :)
EDIT:
I just found the reason to this whole shenanigan.
In the code (which goes years back) we used this package:
jsonfield==1.0.3
And used this way:
from jsonfield import JSONField
The issue is that in the background, Postgres saves the data as a string, so it needs to be cast into a JSON.
Later Django introduced its own JSONField, which stores data as you would expect, without a need to cast:
from django.contrib.postgres.fields import JSONField
I have a requirement where integer value should be converted to date type in Snowflake.Initially I used following query to get the desired result:
SELECT TO_DATE(TO_varchar(19000000+1200034),'YYYYMMDD')
2019-07-09
Now when I used same query for the input - "20200034", I am getting following error:
select TO_DATE(TO_varchar(19000000+1200034),'YYYYMMDD')
Can't parse '20200034' as date with format 'YYYYMMDD'
"20200034" is actually coming from one of the columns in snowflake table. To resolve this issue I tried using "TRY_TO_DATE" function, but output of "TRY_TO_DATE" function is giving incorrect result. Please find details below:
select TRY_TO_DATE(TO_varchar(19000000+1200034))
1970-08-22
As per Snowflake documentation, error handling function does not support optional format argument supported by TO_DATE , DATE.
https://docs.snowflake.com/en/sql-reference/functions/try_to_date.html
You can set the DATE_INPUT_FORMAT for the session before calling the TRY_TO_DATE function.
I suggest you contact Snowflake support and ask them to enable try_to_date with format string - it's available but needs to be enabled manually.
However you have to be aware that TRY_TO_DATE on '20200034' will be resolved to NULL.
I'm attempting to query against a materialized path stored with postgres ltree type from a Grails application. Unfortunately, my query uses the "?" operator which is being captured by GORM as a parameter
sqlRestriction("materialized_path ? (SELECT ARRAY(SELECT CAST(CAST(subpath(?,0,generate_series) AS text) ||'.*{1}' AS lquery) FROM generate_series(1,nlevel(CAST(? AS lquery)))))"
,[vertex.materializedPath,vertex.materializedPath])
Where that first question mark should be escaped and the error being thrown is
org.postgresql.util.PSQLException: No value specified for parameter 4.
at org.postgresql.core.v3.SimpleParameterList.checkAllParametersSet(SimpleParameterList.java:246)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:272)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:116)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)
Found it myself with a little experimentation. It just takes a double-question mark. So,
"materialized_path ? (SELECT ARRAY(...
becomes
"materialized_path ?? (SELECT ARRAY(
I'm a newbie to rails.I have a model called OfflineExport. In the table I have data like this
#<OfflineExport id: 2,
parameters:
{"project_id"=>"3",
"type"=>"submissions",
"filters"=>{"task_type"=>"",
"corrections"=>"", "grade"=>"",
"min_duration"=>"", "after"=>"",
"max_duration"=>"", "reviews"=>"",
"before"=>""},
"send_email"=>"true",
"options"=>{"offline_record_id"=>2}}>
Am trying to fetch parameters["project_id"] in where clause like
OfflineExport.where("parameters[project_id] = '3'")
But am getting error like:
ActiveRecord::StatementInvalid: PGError: ERROR: cannot subscript type text because it is not an array
can anyone help me in solving this?
It seems that you have a serialized column in your model. Something like this: serialize :parameters. This means the data is stored in the database not in an easily readable format, which leads that you can not query on it. Same as here.
Solution: extract the field you want to query on, and make a column for it.