Scan unstructured JSON BYTEA into map[string]string - sql

This seems like a common problem and may be posted somewhere already, but I can't find any threads talking about it, so here is the problem:
I have a Postgres table storeing a column of type BYTEA.
CREATE TABLE foo (
id VARCHAR PRIMARY KEY,
json_data BYTEA
)
The column json_data is really just JSON stored as BYTEA (It's not ideal I know). It is unstructured, but guaranteed to be of string -> string JSON.
When I query this table, I need to scan the query SELECT * FROM foo WHERE id = $1 into the following struct:
type JSONData map[string]string
type Foo struct {
ID string `db:"id"`
Data JSONData `db:"json_data"`
}
I'm using sqlx's Get method. When I execute a query I'm getting the error message sql: Scan error on column index 1, name "json_data": unsupported Scan, storing driver.Value type []uint8 into type *foo.JSONData.
Obviously, the scanner is having trouble scanning the JSON BYTEA into a map. I can implement my own scanner and call my custom scanner on the json_data column, but I'm wondering if there are better ways to do this. Could my JSONData type implement an existing interface to do this automatically?

As suggested by #iLoveReflection, implementing the Scanner interface on *JSONData worked. Here is the actual implementation:
func (j *JSONData) Scan(src interface{}) error {
b, ok := src.([]byte)
if !ok {
return errors.New("invalid data type")
}
return json.Unmarshal(b, j)
}

Related

Postgres | V9.4 | Extract value from json

I have a table that one of the columns is in type TEXT and holds a json object inside.
I what to reach a key inside that json and ask about it's value.
The column name is json_representation and the json looks like that:
{
"additionalInfo": {
"dbSources": [{
"user": "Mike"
}]
}
}
I want to get the value of the "user" and ask if it is equal to "Mike".
I tried the following:
select
json_representation->'additionalInfo'->'dbSources'->>'user' as singleUser
from users
where singleUser = 'Mike';
I keep getting an error:
Query execution failed
Reason:
SQL Error [42883]: ERROR: operator does not exist: text -> unknown
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Position: 31
please advice
Thanks
The error message tells you what to do: you might need to add an explicit type cast:
And as you can not reference a column alias in the WHERE clause, you need to wrap it into a derived table:
select *
from (
select json_representation::json ->'additionalInfo'->'dbSources' -> 0 ->>'user' as single_user
from users
) as t
where t.single_user = 'Mike';
:: is Postgres' cast operator
But the better solution would be to change the column's data type to json permanently. And once you upgrade to a supported version of Postgres, you should use jsonb.

Handling Null DataType

I'm using the Over function from Piggybank to get the Lag of a row
res= foreach (group table by fieldA) {
Aord = order table by fieldB;
generate flatten(Stitch(Aord, Over(Aord.fieldB, 'lag'))) as (fieldA,fieldB,lag_fieldB) ;}
This works correctly and when I do a dump I get the expected result, the problem is when I want to use lag_fieldB for any comparison or transformation I get datatype issues.
If I do a describe it returns fieldA: long,fieldB: chararray,lag_fieldB: NULL
I'm new with PIG but I already tried casting to chararray and using ToString() and I keep getting errors like these:
ERROR 1052: Cannot cast bytearray to chararray
ERROR 1051: Cannot cast to bytearray
Thanks for your help
Ok after some looking around into the code of the Over function I found that you can instantiate the Over class to set the return type. What worked for me was:
DEFINE ChOver org.apache.pig.piggybank.evaluation.Over('chararray');
res= foreach (group table by fieldA) {
Aord = order table by fieldB;
generate flatten(Stitch(Aord, ChOver(Aord.fieldB, 'lag'))) as (fieldA,fieldB,lag_fieldB) ;}
Now the describe is telling me
fieldA: long,fieldB: chararray,lag_fieldB: chararray
And I'm able to use the columns as expected, hope this can save some time for someone else.

How to scan SQL array to []int64 in Go?

I am scanning an array of int from Postgres DB and it is returning as []uint8. I need them in []int64, how can I convert them into []int64 or how can I return them from the DB as []int64? In my query I am selecting using the Array function in Postgres: Array(col1) where col1 is serial.
The error I am getting is:
unsupported Scan, storing driver.Value type []uint8 into type []int64
If you're using github.com/lib/pq, just use Int64Array.
col1arr := []int64{}
arr := pq.Int64Array{}
err := rows.Scan(&arr)
// ...
col1arr = []int64(arr)

Setting a Clob value in a native query

Oracle DB.
Spring JPA using Hibernate.
I am having difficulty inserting a Clob value into a native sql query.
The code calling the query is as follows:
#SuppressWarnings("unchecked")
public List<Object[]> findQueryColumnsByNativeQuery(String queryString, Map<String, Object> namedParameters)
{
List<Object[]> result = null;
final Query query = em.createNativeQuery(queryString);
if (namedParameters != null)
{
Set<String> keys = namedParameters.keySet();
for (String key : keys)
{
final Object value = namedParameters.get(key);
query.setParameter(key, value);
}
}
query.setHint(QueryHints.HINT_READONLY, Boolean.TRUE);
result = query.getResultList();
return result;
}
The query string is of the format
SELECT COUNT ( DISTINCT ( <column> ) ) FROM <Table> c where (exact ( <column> , (:clobValue), null ) = 1 )
where "(exact ( , (:clobValue), null ) = 1 )" is a function and "clobValue" is a Clob.
I can adjust the query to work as follows:
SELECT COUNT ( DISTINCT ( <column> ) ) FROM <Table> c where (exact ( <column> , to_clob((:stringValue)), null ) = 1 )
where "stringValue" is a String but obviously this only works up to the max sql string size (4000) and I need to pass in much more than that.
I have tried to pass the Clob value as a java.sql.Clob using the method
final Clob clobValue = org.hibernate.engine.jdbc.ClobProxy.generateProxy(stringValue);
This results in a java.io.NotSerializableException: org.hibernate.engine.jdbc.ClobProxy
I have tried to Serialize the Clob using
final Clob clob = org.hibernate.engine.jdbc.ClobProxy.generateProxy(stringValue);
final Clob clobValue = SerializableClobProxy.generateProxy(clob);
But this appears to provide the wrong type of argument to the "exact" function resulting in (org.hibernate.engine.jdbc.spi.SqlExceptionHelper:144) - SQL Error: 29900, SQLState: 99999
(org.hibernate.engine.jdbc.spi.SqlExceptionHelper:146) - ORA-29900: operator binding does not exist
ORA-06553: PLS-306: wrong number or types of arguments in call to 'EXACT'
After reading some post about using Clobs with entities I have tried passing in a byte[] but this also provides the wrong argument type (org.hibernate.engine.jdbc.spi.SqlExceptionHelper:144) - SQL Error: 29900, SQLState: 99999
(org.hibernate.engine.jdbc.spi.SqlExceptionHelper:146) - ORA-29900: operator binding does not exist
ORA-06553: PLS-306: wrong number or types of arguments in call to 'EXACT'
I can also just pass in the value as a String as long as it doesn't break the max string value
I have seen a post (Using function in where clause with clob parameter) which seems to suggest that the only way is to use "plain old JDBC". This is not an option.
I am up against a hard deadline so any help is very welcome.
I'm afraid your assumptions about CLOBs in Oracle are wrong. In Oracle CLOB locator is something like a file handle. And such handle can be created by the database only. So you can not simply pass CLOB as bind variable. CLOB must be somehow related to database storage, because this it can occupy up to 176TB and something like that can not be held in Java Heap.
So the usual approach is to call either DB functions empty_clob() or dbms_lob.create_temporary (in some form). Then you get a clob from database even if you think it is "IN" parameter. Then you can write as many data as you want into that locator (handle, CLOB) and then you can use this CLOB as a parameter for a query.
If you do not follow this pattern, your code will not work. It does not matter whether you use JPA, SpringBatch or plan JDBC. This constrain is given by the database.
It seems that it's required to set type of parameter explicitly for Hibernate in such cases. The following code worked for me:
Clob clob = entityManager
.unwrap(Session.class)
.getLobHelper()
.createClob(reader, length);
int inserted = entityManager
.unwrap(org.hibernate.Session.class)
.createSQLQuery("INSERT INTO EXAMPLE ( UUID, TYPE, DATA) VALUES (:uuid, :type, :data)")
.setParameter("uuid", java.util.Uuid.randomUUID(), org.hibernate.type.UUIDBinaryType.INSTANCE)
.setParameter("type", java.util.Uuid.randomUUID(), org.hibernate.type.StringType.INSTANCE)
.setParameter("data", clob, org.hibernate.type.ClobType.INSTANCE)
.executeUpdate();
Similar workaround is available for Blob.
THE ANSWER: Thank you both for your answers. I should have updated this when i solved the issue some time ago. In the end I used JDBC and the problem disappeared in a puff of smoke!

NHibernate Like with integer

I have a NHibernate search function where I receive integers and want to return results where at least the beginning coincides with the integers, e.g.
received integer: 729
returns: 729445, 7291 etc.
The database column is of type int, as is the property "Id" of Foo.
But
int id = 729;
var criteria = session.CreateCriteria(typeof(Foo))
criteria.Add(NHibernate.Criterion.Expression.InsensitiveLike("Id", id.ToString() + "%"));
return criteria.List<Foo>();
does result in an error (Could not convert parameter string to int32). Is there something wrong in the code, a work around, or other solution?
How about this:
int id = 729;
var criteria = session.CreateCriteria(typeof(Foo))
criteria.Add(Expression.Like(Projections.Cast(NHibernateUtil.String, Projections.Property("Id")), id.ToString(), MatchMode.Anywhere));
return criteria.List<Foo>();
Have you tried something like this:
int id = 729;
var criteria = session.CreateCriteria(typeof(Foo))
criteria.Add(NHibernate.Criterion.Expression.Like(Projections.SqlFunction("to_char", NHibernate.NHibernateUtil.String, Projections.Property("Id")), id.ToString() + "%"));
return criteria.List<Foo>();
The idea is convert the column before using a to_char function. Some databases do this automatically.
AFAIK, you'll need to store your integer as a string in the database if you want to use the built in NHibernate functionality for this (I would recommend this approach even without NHibernate - the minute you start doing 'like' searches you are dealing with a string, not a number - think US Zip Codes, etc...).
You could also do it mathematically in a database-specific function (or convert to a string as described in Thiago Azevedo's answer), but I imagine these options would be significantly slower, and also have potential to tie you to a specific database.