I have a JSON which I am trying to parse in HIVE using get_json_object built-in function. In my JSON there is a key like "timestamp", where my parsing is failing because "timestamp" is identified as a keyword.
I am trying to use select get_json_object(col1,'$.timestamp') as ts from table1.
How can the keyword be handled while using get_json_object in HIVE?
Run this to have the keywords ignored.
SET hive.support.sql11.reserved.keywords=false;
Related
I want to write a PySpark dataframe into an Impala table, but I am getting an error message. Basically, my code looks like:
properties = {'user': os.getenv('USERNAME), 'password': os.getenv('SECRET), 'driver': 'com.cloudera.impala.jdbc41.Driver'}
df.write.jdbc(url=os.getenv('URL'), table=os.getenv('URL'), mode='append', properties=properties)
The problem seems that the "Create table" statement that is generated has some bad syntax:
When I run the same query in DBeaver, I get the same error message. Only when I delete the quotation marks, the table gets created. I have no idea how to solve this. I created the dataframe by flattening a json file, using the withColumn and explode functions. Can I somehow avoid these quotation marks from being generated?
As an alternative, would it be possible to write the dataframe in an already existing table, using an insert into query instead?
Edit: Another issue I just realized: when it comes to string columns, the "create table"-statements contains the word "TEXT", instead of "STRING" or "varchar", which is also not recognized as a proper data type by Impala...
Thanks a lot!
I've got a table which labeled a col...sigh... date.
Does presto have a setting similar to hive that allows for keywords to be escaped?
e.g. a presto variant of:
set hive.support.sql11.reserved.keywords=false;
so my table looks like this
{"#timestamp":"2018-08-08T09:21:57.947+00:00","#version":"1","message":"bla bla"}
i can extract for example the message part using the json functions like below
JSON_EXTRACT_SCALAR(log,'$.message') AS message
but when i try to extract the timestamp the same way
JSON_EXTRACT_SCALAR(log,'$.#timestamp') AS timestamp
i get the error "Error: Unsupported operator in JSONPath: #"
any ideas on the correct syntax?
In cases where a JSON key uses invalid JSONPath characters, you can escape those characters using single quotes and brackets, [' ']. For example:
$['#timestamp']
SELECT
json_extract_scalar('{"#timestamp":"2018-08-08T09:21:57.947+00:00"}',"$['#timestamp']")
See more examples on: JSON Functions in Standard SQL
My table consists of 253 columns so loaded data to pig bag without schema method, when I apply filter condition to any data getting this error how to avoid it? Any jars needs to be added ?
Error: IMPLICIT_CAST_TO_CHARARRAY
Since schema is not defined in the load statement, so default data type for all the fields are bytearray.
Pig tries to implicitly cast from one type to another while using in the script.
All castings are not possible, so in that case, warnings are thrown.
You can refer the Pig Cast operators for details
When the schema is not specified the fields are loaded as bytearray.When you apply filter you will have to cast the field you are using for the filter.If you are filtering using a string use (chararray) in front of the field.
I am exploring oracle 12c for storing json data in a json aware clob column named METADATA. My existing data in that column looks something like this:
{
"com.xyz.abc.key": {
"key_a": "value_a",
"key_b": ["value_b_1", "value_b_2", "value_c_2"]
}
}
The problem is that if I were to construct a query using JSON_VALUE
JSON_VALUE(METADATA ,'$.com.xyz.abc.key.key_a')
then I rightly get null as result as oracle will interpret a different json path. I tried escaping characters but that also doesn't work.
I went through oracle's whitepaper on this topic but it doesn't cover this case.
I need help in correctly constructing my SQL/JSON query in this case.
Edit
Solution : '$."com.xyz.abc.key".key_a' is the correct way for doing this.