I have a table in athena aws where the column 'metadata_stopinfo' has the structure that you can see in the image.
I am trying to extract values that are inside that array, however when I try
SELECT
"json_extract_scalar"(metadata_stopinfo, '$.city')
FROM "table"
I have the following problem
SYNTAX_ERROR: line 2:5: Unexpected parameters (array(row("address" row("addressline" varchar,"city" varchar,"countrycode" varchar,"countrycodeoriginal" varchar,"state" varchar,"zipcode" varchar),"carrierreference" varchar,"contacts" array(row("contacttype" varchar,"email" varchar,"fax" varchar,"mobilephone" varchar,"name" varchar,"officephone" varchar,"userid" varchar)),"containerinfo" array(row("containerid" varchar,"containeridtype" varchar,"equipmentcode" varchar,"equipmenttype" varchar)),"conveyancelinenumber" varchar,"conveyancetype" varchar,"conveyancetypeoriginal" varchar,"dateinfo" row("arrivalestimateddate" varchar,"arrivalestimateddateend" varchar,"arrivalestimatedendoffset" varchar,"arrivalestimatedoffset" varchar,"arrivalrequesteddate" varchar,"deliveryestimateddate" varchar,"deliveryestimateddateend" varchar,"deliveryestimatedendoffset" varchar,"deliveryestimatedoffset" varchar,"deliveryrequesteddate" varchar,"deliveryrequesteddateend" varchar,"deliveryrequestedendoffset" varchar,"deliveryrequestedoffset" varchar,"departureestimateddate" varchar,"departureestimateddateend" varchar,"departureestimatedendoffset" varchar,"departureestimatedoffset" varchar,"departurerequesteddate" varchar,"pickuprequesteddate" varchar,"pickuprequesteddateend" varchar,"pickuprequestedendoffset" varchar,"pickuprequestedoffset" varchar,"pickupestimateddate" varchar,"pickupestimateddateend" varchar,"pickupestimatedendoffset" varchar,"pickupestimatedoffset" varchar),"deliverynotenumber" varchar,"instructions" array(row("customerspecificsubtype" varchar,"header" boolean,"instructionsubtype" varchar,"instructiontype" varchar,"text" varchar)),"locationid" varchar,"partnercarrieraddress" row("addressline" varchar,"city" varchar,"countrycode" varchar,"countrycodeoriginal" varchar,"state" varchar,"zipcode" varchar),"partnercarriercontacts" array(row("contacttype" varchar,"email" varchar,"fax" varchar,"name" varchar,"officephone" varchar)),"partnercarrierid" varchar,"partnercarriername" varchar,"partnerid" varchar,"partnername" varchar,"partnertimezone" varchar,"partnertype" varchar,"productquantity" row("number" double,"originalunitofmeasure" varchar,"quantitytype" varchar,"unitofmeasure" varchar),"sequencenumber" bigint,"shipmentidentifier" varchar,"stoptype" varchar,"transportinfo" row("description" varchar,"transportcode" varchar,"transportoriginalcode" varchar),"vesselinfo" row("lloydsnumber" varchar,"shipsradiocallnumber" varchar,"vesselname" varchar,"vesselnumber" varchar,"voyagetripnumber" varchar))), varchar(6)) for function json_extract_scalar. Expected: json_extract_scalar(varchar(x), JsonPath) , json_extract_scalar(json, JsonPath)
My question is, how can i extract values inside de column ?
json_extract_scalar unsurprisingly works with json (note that even if yur data was in json format, json_extract_scalar(metadata_stopinfo, '$.city') still would not have worked cause your data is an array), while your column contains array's of row's, so you need to work with it correspondingly. For example you can use indexes to access elements in array (in presto array indexes start from 1):
SELECT
metadata_stopinfo[1] r
FROM "table"
And then access the fields:
The fields may be of any SQL type, and are accessed with field reference operator .
SELECT
metadata_stopinfo[1].city city
FROM "table"
Also you can flatten the array with unnest:
SELECT r.city
FROM "table",
unnest(metadata_stopinfo) as t(r)
I have a table with many columns stored as text. These columns have information which i am planning to cast and copy the data into a fresh table with the same column names but with correct data types ( float8 )
I tested the SELECT with CAST operator "::" and it works fine. All the columns are being converted as you can see in the picture
However , when I uncomment the INSERT statement ( above it ) to start writing in the target table, it throws an error. The target table has identical column names and only float8 column types.
The expression from the source table is indeed of type text but i am using the cast operator so why does it not work like before when only running the SELECT statement?
My query below:
INSERT INTO "PM"."new_VM_gcell_evolution_hourly_BSC"
(select
"CR3120:Channel_Assignment_Failures_All_Channels_Busy_or_Channel"::float8,
"DL_Mean_Quality"::float8,
"A312Ca:Failed_Assignments_during_MTC_on_the_A_Interface_Includi"::float8,
"nsp_Urban_TA_4number"::float8,
"A3129C:Failed_Assignments_First_Assignment,_Assignment_Timed_Ou"::float8,
"R3120C:Channel_Assignment_Failures_All_Channels_Busy_or_Channel"::float8,
"TSs_Interf_B5"::float8,
"R3120D:Channel_Assignment_Failures_All_Channels_Busy_or_Channel"::float8,
"HO_Out_Internal_Succ_Rate"::float8,
"CH_Req_Protocol_Undefinednbnumber"::float8,
"UL_Drop_Congnumber"::float8,
"Timing_Adv"::float8,
"DL_Qual"::float8,
"A3100C:Assignment_Requests_TCHH_Only"::float8,
"MS_to_BTS_max_distance"::float8,
"TSs_Interf_B2"::float8,
"UL_Qual"::float8,
"UL_Drop_EDGE_Rate"::float8,
"Avg_BTS_Power_Level_AMR"::float8,
"DL_Drop_N3105number"::float8,
"nsp_Urban_TA_2_R"::float8,
"CH_Req_CallReestabnbnumber"::float8,
"TCH_Drops_cause_Timing_Advance"::float8,
"TCH_Drops_per_Erlang"::float8,
"UL_Drop_Flushnumber"::float8,
"A312F:Number_of_Assignment_Failures_No_Abis_Resource_Available"::float8,
"UL_Drop_GPRS_Rate"::float8,
"nsp_Urban_TA_3number"::float8,
"Call_Drop_rate"::float8,
"DL_Fail_Assingnumber"::float8,
"SDCCH_Radio_Failures"::float8,
"HO_Cmd_UL_Qual"::float8,
"UL/DL_RxQualnumber"::float8,
"UL/DL_RxLevnumber"::float8,
"UL_Fail_EDGE_Rate"::float8,
"Call_Drop_Total"::float8,
"HO_Out_Succ_Rate"::float8,
"Date"::text,
"A3127E:Failed_Assignments_during_Call_Reestablishment_on_the_Um"::float8,
"UL_Drop_N3103number"::float8,
"Configured_TCH"::float8,
"SDCCH_Drop_Call_Rate_only_LU"::float8,
"A312M:Failed_Assignments_Reconnection_to_Old_Channels,_No_Chann"::float8,
"SDCCH_Blocking_Rate"::float8,
"TCH_Drops_cause_DL_FER"::float8,
"TCH_Fail_rate"::float8,
"DL_Drop_Preemnumber"::float8,
"FR_Traffic_totalErl"::float8,
"DL_Congnumber"::float8,
"TCH_Assign_Unsuccnumber"::float8,
"nsp_Urban_TA_0_R"::float8,
"DL_Drop_GPRS_Rate"::float8,
"HR_TraficErl"::float8,
"SDCCH_Cong_Rate"::float8,
"Abis_and_Ater_Interface_Anlysistimes"::float8,
"A3100B:Assignment_Requests_TCHF_Only"::float8,
"AS4300D:Mean_Uplink_Level_during_Radio_Link_Failure_SDCCHdB"::float8,
"nsp_Urban_TA_89number"::float8,
"TCH_Drops_cause_DownQual"::float8,
"SDCCH_Drop_RxLev"::float8,
"Max_UL_Pwr_Duration"::float8,
"nsp_Channel_Req_LUnumber"::float8,
"TCH_HR_Initialy_Config"::float8,
"HO_Inc_Cong_Rate"::float8,
"DL_RxLev_avgdBm"::float8,
"TCH_Assign_Requestsnumber"::float8,
"A312K:Failed_Assignments_First_Assignment,_No_Channel_Available"::float8,
"Call_Drop_TCH_Quality_Rate"::float8,
"AMR_HR_TraficErl"::float8,
"nsp_Urban_TA_67_R"::float8,
"UL_Fail_MSnumber"::float8,
"Time"::text,
"TCH_Drops_cause_UpLevel"::float8,
"AMR_FR_TraficErl"::float8,
"TCH_FR_Initialy_Config"::float8,
"UL_Drop_N3101number"::float8,
"DL_FERnumber"::float8,
"SDCCH_Drop_Call_Rate_wo_LU"::float8,
"DL_Mean_StrengthdB"::float8,
"Call_Drop_Abis"::float8,
"SDCCH_Initialy_Confignumber"::float8,
"Call_Drop_TCH_RxLev_Rate"::float8,
"TA_Meannumber"::float8,
"nsp_Urban_TA_67number"::float8,
"AS3240NA:Average_MS_Power_Level_of_NonAMR_Call"::float8,
"Load"::float8,
"A3129Q:Failed_Assignments_Reconnection_to_Old_Channels,_Timer_E"::float8,
"CH_Request_11_bit"::float8,
"AS4340D:Mean_TA_during_Radio_Link_Failure_SDCCH"::float8,
"S4210A:Uplink_Interference_Indication_Messages_SDCCH"::float8,
"UL_FERnumber"::float8,
"A3129P:Failed_Assignments_Reconnection_to_Old_Channels,_Timer_E"::float8,
"K3003A:Successful_SDCCH_Seizures_Call_Type"::float8,
"CH_Req_LAUnbnumber"::float8,
"SDCCH_Drop_Rate"::float8,
"GBSC"::text,
"CS_CSSR_with_SDCCH_blocks_wo_LU"::float8,
"A312Aa:Failed_Assignments_during_MOC_on_the_A_Interface_Includi"::float8,
"Avg_BTS_Power_Level"::float8,
"R3120A:Channel_Assignment_Failures_All_Channels_Busy_or_Channel"::float8,
"A312S:Failed_Assignments_Signaling_Channel"::float8,
"SD_Fail_MoC"::float8,
"UL_RxQual_avg"::float8,
"Call_Drop_HO"::float8,
"AS4330D:Mean_Downlink_Quality_during_Radio_Link_Failure_SDCCH"::float8,
"CH_Req_Emerg_Callsnbnumber"::float8,
"UL_RxQualnumber"::float8,
"UL_Fail_OtherCausenumber"::float8,
"A312Da:Failed_Assignments_during_Emergency_Call_on_the_A_Interf"::float8,
"UL_Drop_Preemnumber"::float8,
"TCH_Drops_cause_UpDown_FER"::float8,
"AS4320D:Mean_Downlink_Level_during_Radio_Link_Failure_SDCCHdB"::float8,
"A3129O:Failed_Assignments_First_Assignment,_Directed_Retry_Time"::float8,
"Max_DL_Pwr_Duration"::float8,
"UL_Drop_Suspendnumber"::float8,
"TCH_Assign_Congestionnumber"::float8,
"Transmission_Resource_Analysistimes"::float8,
"HO_Out_RxQual_rate"::float8,
"nsp_Urban_TA_1013number"::float8,
"A3129I:Failed_Assignments_Invalid_State"::float8,
"DL_Fail_OtherCausenumber"::float8,
"CR3129:Channel_Assignment_Failures_All_Channels_Busy_or_Channel"::float8,
"HO_Inc_Successnumber"::float8,
"A3129B:Failed_Assignments_First_Assignment,_Terrestrial_Resourc"::float8,
"Trafic_totalErl"::float8,
"nsp_urban_TA_1463all_R"::float8,
"HO_Out_RxQualnumber"::float8,
"DL_RxQualnumber"::float8,
"HO_Inc_Succ_Rate"::float8,
"SDCCH_Fail_Rate"::float8,
"SDCCH_Non_Radio_Drops"::float8,
"nsp_Urban_TA_0number"::float8,
"A3129E:Failed_Assignments_CIC_Unavailable"::float8,
"nsp_CH_Request_PSnumber"::float8,
"TCH_Assign_Unsucc_rate"::float8,
"Call_Drop_Radio"::float8,
"EFR_TraficErl"::float8,
"A312L:Failed_Assignments_Reconnection_to_Old_Channels,_No_Chann"::float8,
"TCH_Drops_cause_DownLevel"::float8,
"nsp_Urban_TA_5number"::float8,
"Abnormal_Terminals_Analysistimes"::float8,
"HO_Inc_Congnumber"::float8,
"CSSR_Rate"::float8,
"M3020C:Call_Drops_on_SDCCHQuality"::float8,
"RH333:Handover_Drop_Rate_of_TCH"::float8,
"UL_Drop_EDGE_Abisnumber"::float8,
"A3129J:Failed_Assignments_Invalid_Message"::float8,
"TCH_Availability"::float8,
"UL_Level"::float8,
"nsp_Urban_TA_4_R"::float8,
"UL_Drop_OtherCausenumber"::float8,
"HR_Traffic_Rate"::float8,
"DL_Level"::float8,
"CH_Request_8_bit"::float8,
"UL_Fail_MS_Assingnumber"::float8,
"DL_Fail_BSC_Commandnumber"::float8,
"A3129R:Failed_Assignments_Reconnection_to_Old_Channels,_Reconne"::float8,
"UL_Fail_BSC_Commandnumber"::float8,
"DL_Drop_Suspendnumber"::float8,
"nsp_urban_TA_1463allnumber"::float8,
"Direct_Retry"::float8,
"nsp_Urban_TA_89_R"::float8,
"CH_Req_PSnbnumber"::float8,
"A3129N:Failed_Assignments_Reconnection_to_Old_Channels,_Terrest"::float8,
"A312A:Failed_Assignments_First_Assignment,_No_Channel_Available"::float8,
"nsp_Urban_TA_2number"::float8,
"DL_RxQual_avg"::float8,
"UL_Mean_StrengthdB"::float8,
"DL_Drop_EDGE_Rate"::float8,
"A3129T:Failed_Assignments_No_Ater_Resource_Available"::float8,
"TSs_Interf_B1"::float8,
"A3129D:Failed_Assignments_Reconnection_to_Old_Channels,_Reconne"::float8,
"nsp_Channel_Req_MOCnumber"::float8,
"UL_Fail_GPRS_Rate"::float8,
"S4210B:Downlink_Interference_Indication_Messages_SDCCH"::float8,
"R3120E:Channel_Assignment_Failures_All_Channels_Busy_or_Channel"::float8,
"HO_Inc_Failnumber"::float8,
"DL_Drop_Flushnumber"::float8,
"S3655:Number_of_configured_TRXs_in_a_cell"::float8,
"TCH_Drops_cause_UpDown_Qual"::float8,
"CR3005:Number_of_Initially_Configured_Channels_Static_PDTCH_Sup"::float8,
"UL/DL_FERnumber"::float8,
"SDCCH_Blocking_"::float8,
"HO_Out_InterRAT_Succ_Rate"::text,
"CM30E:Call_Drops_on_SDCCH_Location_Updating"::float8,
"AS4310D:Mean_Uplink_Quality_during_Radio_Link_Failure_SDCCH"::float8,
"UL_Congnumber"::float8,
"TAnumber"::float8,
"Configured_SDCCH"::float8,
"Othernumber"::float8,
"TCH_Assign_Fail_Radionumber"::float8,
"HO_Out_Internal_Req_Nb"::float8,
"SDCCH_Cong_Nbnumber"::float8,
"UL_Mean_Quality"::float8,
"HO_Out_External_Req_Nb"::float8,
"nsp_Urban_TA_1013_R"::float8,
"SDCCH_Drops_wo_LU"::float8,
"Call_Drop_TCH_Qualitynumber"::float8,
"A3100A:Assignment_Requests_Signaling_Channel_TCH"::float8,
"Better_Cell"::float8,
"Call_Drop_TCH_RxLevnumber"::float8,
"nsp_Channel_Req_MTCnumber"::float8,
"DL_Fail_EDGE_Rate"::float8,
"ZTR104B:Call_Drop_Rate_on_SDCCH_Call_Type"::float8,
"Call_Drop_no_MR"::float8,
"DL_Drop_Congnumber"::float8,
"HO_Inc_InterRAT_reqnumber"::float8,
"TCH_Assign_Cong_rate"::float8,
"HO_Inc_Fail_Rate"::float8,
"TCH_Assign_Fail_sp_ver_unav"::float8,
"HO_Inc_InterRAT_unsucc"::float8,
"A3170A:Number_of_Completed_TCH_Assignments_CSFB_MOC"::float8,
"HO_Inc_Unsucc_Rate"::float8,
"TSs_Interf_B4"::float8,
"A3100K:Assignment_Requests_Signaling_Channel_SDCCH"::float8,
"TCH_Drops_cause_UpDown_Level"::float8,
"SDCCH_Fail_Nbnumber"::float8,
"TCH_Non_Radio_Drops"::float8,
"DL_Fail_MSnumber"::float8,
"HO_Cmd_DL_Qual"::float8,
"DL_Drop_OtherCausenumber"::float8,
"SMS_on_SDCCH"::float8,
"FERnumber"::float8,
"HO_Inc_Reqnumber"::float8,
"nsp_Urban_TA_3_R"::float8,
"M3020D:Call_Drops_on_SDCCHOther"::float8,
"HO_Out_External_Succ_Rate"::float8,
"M3020A:Call_Drops_on_SDCCHTA"::float8,
"TCH_Assign_Fail_Radio_rate"::float8,
"CR3001:Number_of_Initially_Configured_Channels_Static_PDCH"::float8,
"HR_Traffic_totalErl"::float8,
"CH_Req_MTCnbnumber"::float8,
"SDCCH_Drop_RxQual"::float8,
"UL_RxLev_avgdBm"::float8,
"CH_Req_MOCnbnumber"::float8,
"SDCCH_Dropnumber"::float8,
"A3169A:Failed_Assignments_Um_Cause"::float8,
"A3129H:Failed_Assignments_Clear_Commands_Sent_By_MSC"::float8,
"TCH_Radio_Drops"::float8,
"HO_Out_InterRAT_Req_Nb"::float8,
"HO_Inc_InterRAT_succnumber"::float8,
"DL_Drop_Abisnumber"::float8,
"TCH_Drops_cause_UL_FER"::float8,
"UL_RxLevelnumber"::float8,
"nsp_CH_Request_CSnumber"::float8,
"A3170B:Number_of_Completed_TCH_Assignments_CSFB_MTC"::float8,
"A3129F:Failed_Assignments_CIC_Allocated"::float8,
"DL_Fail_GPRS_Rate"::float8,
"M3020B:Call_Drops_on_SDCCHReceived_Level"::float8,
"nsp_Urban_TA_1number"::float8,
"SDCCH_Availability"::float8,
"Call_Drop_forced_HO"::float8,
"SDCCH_Drop_Others"::float8,
"Call_Drop_Equip."::float8,
"CH_Req_PS"::float8,
"SDCCH_Drop_Call_Rate"::float8,
"TCH_Drops_cause_UpQual"::float8,
"CH_Req_LMU_Reservednbnumber"::float8,
"DL_RxLevnumber"::float8,
"nsp_Urban_TA_1_R"::float8,
"A312Ea:Failed_Assignments_during_Call_Reestablishment_on_the_A_"::float8,
"TSs_Interf_B3"::float8,
"SDCCH_Drop_TimingAdvance"::float8,
"Avg_BTS_Power_Level_NAMR"::float8,
"TCH_Drops_cause_Other"::float8,
"FR_TraficErl"::float8,
"Avg_MS_Power_Level"::float8,
"HO_Outgoing_Requestsnumber"::float8,
"A3129G:Failed_Assignments_A_Interface_Failure"::float8,
"CH_Req_LAU"::float8,
"nsp_Urban_TA_5_R"::float8,
"HO_Inc_Unsuccessnumber"::float8,
"AS3240A:Average_MS_Power_Level_of_AMR_Call"::float8
from "PM"."VM_gcell_evolution_hourly_BSC_recovered")
It says "column is of type double but expression is of type text" so what's really happening is that it's trying to insert one of the expressions where you cast to ::text into a column that is of type double.
If the problem was converting from text to double, you'd get a different message. Besides, the SELECT worked fine, which means it didn't encounter any text data that it couldn't convert to double.
Since you don't specify the target table columns in your INSERT, and you have so many of them, most likely you missed a column or got the order wrong.
Honestly if the table has 270 columns and they have the same name in the source and destination tables, you should really generate the query using something like python from a list of columns, that will be faster than proofreading the 270 lines...
There is PII data that exists inside a key-value pair of a json object stored inside my Hive table in a column.Is there any way that I can mask only the value of the required key-value pair while keeping others unmasked?
{ "name":"John", "age":30, "car":null, "mob_no": "99999999"}
I require field "mob_no" which is PII data and its associated value to be masked. Other key-value pairs of the json should be visible
You can use regexp_replace to replace some values in json with some string (see json_with_number_replaced column in the example below).
Better apply sha256 instead of simple replacement, because this function is cryptographically strong (not reversible one way function), deterministic and collision tolerant. These properties of sha256 hash function allow you to join on obfuscated values(if using the same obfuscation method) and do aggregations. You can still count different mobile_numbers after obfuscation. See mobile_number_obfuscated and json_with_number_obfuscated columns calculation in the example below:
select json original_json,
s.original_mobile_number,
s.mobile_number_obfuscated,
regexp_replace(s.json, '(mob_no":[\\s]?")([+]?[\\d]+)','$1xx') as json_with_number_replaced,
regexp_replace(s.json, '(mob_no":[\\s]?")([+]?[\\d]+)',concat('$1',mobile_number_obfuscated)) as json_with_number_obfuscated
from
(
select regexp_extract(s.json, 'mob_no":[\\s]?"([+]?[\\d]+)',1) as original_mobile_number,
java_method('org.apache.commons.codec.digest.DigestUtils', 'sha256Hex', regexp_extract(s.json, 'mob_no":[\\s]?"([+]?[\\d]+)',1)) mobile_number_obfuscated,
s.json
from
(--original data
select '{ "name":"John", "age":30, "car":null, "mob_no": "+99999999"}' as json
)s
)s;
OK
original_json original_mobile_number mobile_number_obfuscated json_with_number_replaced json_with_number_obfuscated
{ "name":"John", "age":30, "car":null, "mob_no": "+99999999"} +99999999 98ae38dddc1994179e21d104feb7b09e5627953d9fe9b9851239ac445b6de3cd { "name":"John", "age":30, "car":null, "mob_no": "xx"} { "name":"John", "age":30, "car":null, "mob_no": "98ae38dddc1994179e21d104feb7b09e5627953d9fe9b9851239ac445b6de3cd"}
Time taken: 1.988 seconds, Fetched: 1 row(s)
In this example I demonstrated how you can use regexp_replace to replace value for some key in the JSON, also how to use sha256 to obfuscate your PII data.
I'm attempting to write a program that will grab the content from fields from a table both specified by the user on the selection screen.
For example, the user could specify the fields equnr, b_werk, b_lager from the table eqbs.
I've been able to accomplish this like so:
" Determine list of fields provided by user
DATA(lv_fields) = COND string(
WHEN p_key3 IS NOT INITIAL AND p_string IS NOT INITIAL THEN
|{ p_key1 }, { p_key2 }, { p_key3 }, { p_string }|
WHEN p_key2 IS NOT INITIAL AND p_string IS NOT INITIAL THEN
|{ p_key1 }, { p_key2 }, { p_string }|
WHEN p_key2 IS NOT INITIAL AND p_string IS NOT INITIAL THEN
|{ p_key1 }, { p_string }| ).
DATA: lv_field_tab TYPE TABLE OF line.
APPEND lv_fields TO lv_field_tab.
" Determine table specified by user and prepare for Open SQL query
DATA t_ref TYPE REF TO data.
FIELD-SYMBOLS: <t> TYPE any,
<comp> TYPE any.
CREATE DATA t_ref TYPE (p_table).
ASSIGN t_ref->* TO <t>.
ASSIGN COMPONENT lv_fields OF STRUCTURE <t> TO <comp>.
" Prepare result container
DATA: lt_zca_str_to_char TYPE TABLE OF zca_str_to_char,
ls_zca_str_to_char TYPE zca_str_to_char.
SELECT (lv_field_tab) FROM (p_table) INTO (#ls_zca_str_to_char-key1, #ls_zca_str_to_char-key2, #ls_zca_str_to_char-key3, #ls_zca_str_to_char-string).
APPEND ls_zca_str_to_char TO lt_zca_str_to_char.
ENDSELECT.
This will correctly populate lt_zca_str_to_char with data from the table specified by the user.
However, this implies that the user is always providing p_key1, p_key2, and p_key3. I could perform a different selection statement based on how many key fields the user provides, but what's the fun in that?
I set out to solve this like this:
DATA(lv_results) = COND string(
WHEN p_key3 IS NOT INITIAL AND p_string IS NOT INITIAL THEN
|(#ls_zca_str_to_char-key1, #ls_zca_str_to_char-key2, #ls_zca_str_to_char-key3, #ls_zca_str_to_char-string)|
WHEN p_key2 IS NOT INITIAL AND p_string IS NOT INITIAL THEN
|(#ls_zca_str_to_char-key1, #ls_zca_str_to_char-key2, #ls_zca_str_to_char-string)|
WHEN p_key2 IS NOT INITIAL AND p_string IS NOT INITIAL THEN
|(#ls_zca_str_to_char-key1, #ls_zca_str_to_char-string)| ).
SELECT (lv_field_tab) FROM (p_table) INTO (#lv_results).
APPEND ls_zca_str_to_char TO lt_zca_str_to_char.
ENDSELECT.
This will activate, and when I get to my Open SQL query (from a Z table, only filling out the first two of three possible key fields), the values are the following:
lv_field_tab = GUID, TEXT_ID, TEXT_DATA (Good)
p_table = ZCR_TRANS_TEXT (Good)
lv_results = (#ls_zca_str_to_char-key1, #ls_zca_str_to_char-key2, #ls_zca_str_to_char-string) (Good, 3 = 3!)
But, since I'm assuming the compiler is seeing (#lv_results) as one single variable, the program dumps with the following error:
The current ABAP program attempted to execute an Open SQL statement
containing a dynamic entry. The parser returned the following error:
"The field list and the INTO list must have the same number of
elements."
Is it possible for me to use the new Open SQL syntax to accomplish my dynamic INTO clause in harmony with my dynamic field list?
The brackets on the INTO do not do what you expect, from the ABAP help:
... INTO (#dobj1, #dobj2, ... )
Effect
If the results set consists of multiple columns or aggregate expressions specified explicitly in the SELECT list, a list of elementary data objects dobj1, dobj2, ... (in parentheses and separated by commas) can be specified after INTO.
In your case you only have one value in there so you can only select one column and the data will be passed in the variable LV_RESULT. Not what you are looking for. Since you want to fill the fields of an existing structure the INTO CORRESPONDING FIELDS OF construct will work here. And you can use TABLE to make your command more efficient as well. This leads to:
SELECT (lv_field_tab) FROM (p_table)
INTO CORRESPONDING FIELDS OF TABLE #lt_zca_str_to_char.
As said previously, you may use INTO CORRESPONDING FIELDS OF ..., but it's not mandatory, it's only for simplifying the code.
So, instead of using CORRESPONDING FIELDS, you may create a structure dynamically (RTTC) with its components corresponding to the columns in LV_FIELD_TAB, and you may then use:
SELECT (lv_field_tab) FROM (p_table) INTO #<structure> ... ENDSELECT.
But of course, as explained by Gert Beukema, you should better do only one SELECT, by creating an internal table dynamically with the same logic as for the structure above, and you may then use:
SELECT (lv_field_tab) FROM (p_table) INTO TABLE #<internal table> ...
Refer to the many examples in the web how to create data objects dynamically with RTTC.
Do not use a fields list for your INTO clause.
Try with
INTO CORRESPONDING FIELDS OF TABLE
must be a FIELD-SYMBOL type any table, and the rest of the logic is up to you (to put the proper information from your generic and almost-empty to your specific destination one).
I have searched extensively (in Postgres docs and on Google and SO) to find examples of JSON functions being used on actual JSON columns in a table.
Here's my problem: I am trying to extract key values from an array of JSON objects in a column, using jsonb_to_recordset(), but get syntax errors. When I pass the object literally to the function, it works fine:
Passing JSON literally:
select *
from jsonb_to_recordset('[
{ "id": 0, "name": "400MB-PDF.pdf", "extension": ".pdf",
"transferId": "ap31fcoqcajjuqml6rng"},
{ "id": 0, "name": "1000MB-PDF.pdf", "extension": ".pdf",
"transferId": "ap31fcoqcajjuqml6rng"}
]') as f(name text);`
results in:
400MB-PDF.pdf
1000MB-PDF.pdf
It extracts the value of the key "name".
Here's the JSON in the column, being extracted using:
select journal.data::jsonb#>>'{context,data,files}'
from journal
where id = 'ap32bbofopvo7pjgo07g';
resulting in:
[ { "id": 0, "name": "400MB-PDF.pdf", "extension": ".pdf",
"transferId": "ap31fcoqcajjuqml6rng"},
{ "id": 0, "name": "1000MB-PDF.pdf", "extension": ".pdf",
"transferId": "ap31fcoqcajjuqml6rng"}
]
But when I try to pass jsonb#>>'{context,data,files}' to jsonb_to_recordset() like this:
select id,
journal.data::jsonb#>>::jsonb_to_recordset('{context,data,files}') as f(name text)
from journal
where id = 'ap32bbofopvo7pjgo07g';
I get a syntax error. I have tried different ways but each time it complains about a syntax error:
Version:
PostgreSQL 9.4.10 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2, 64-bit
The expressions after select must evaluate to a single value. Since jsonb_to_recordset returns a set of rows and columns, you can't use it there.
The solution is a cross join lateral, which allows you to expand one row into multiple rows using a function. That gives you single rows that select can act on. For example:
select *
from journal j
cross join lateral
jsonb_to_recordset(j.data#>'{context, data, files}') as d(id int, name text)
where j.id = 'ap32bbofopvo7pjgo07g'
Note that the #>> operator returns type text, and the #> operator returns type jsonb. As jsonb_to_recordset expects jsonb as its first parameter I'm using #>.
See it working at rextester.com
jsonb_to_recordset is a set-valued function and can only be invoked in specific places. The FROM clause is one such place, which is why your first example works, but the SELECT clause is not.
In order to turn your JSON array into a "table" that you can query, you need to use a lateral join. The effect is rather like a foreach loop on the source recordset, and that's where you apply the jsonb_to_recordset function. Here's a sample dataset:
create table jstuff (id int, val jsonb);
insert into jstuff
values
(1, '[{"outer": {"inner": "a"}}, {"outer": {"inner": "b"}}]'),
(2, '[{"outer": {"inner": "c"}}]');
A simple lateral join query:
select id, r.*
from jstuff
join lateral jsonb_to_recordset(val) as r("outer" jsonb) on true;
id | outer
----+----------------
1 | {"inner": "a"}
1 | {"inner": "b"}
2 | {"inner": "c"}
(3 rows)
That's the hard part. Note that you have to define what your new recordset looks like in the AS clause -- since each element in our val array is a JSON object with a single field named "outer", that's what we give it. If your array elements contain multiple fields you're interested in, you declare those in a similar manner. Be aware also that your JSON schema needs to be consistent: if an array element doesn't contain a key named "outer", the resulting value will be null.
From here, you just need to pull the specific value you need out of each JSON object using the traversal operator as you were. If I wanted only the "inner" value from the sample dataset, I would specify select id, r.outer->>'inner'. Since it's already JSONB, it doesn't require casting.