My input is long "20190503143744" and wanted to convert to format "2019-09-06 11:46:22"
Trying below code:
A = LOAD 'stp_master_subscriber_profile' using org.apache.hive.hcatalog.pig.HCatLoader() as (mdn:chararray, imei:chararray, imsi:chararray, subscriber_id:long, manufacturer:chararray, model:chararray, update_time:long, scenario:chararray, vz_customer:chararray, commit_time:long);
B = FOREACH A GENERATE ToString(ToDate((chararray)commit_time,'yyyyMMdd'),'yyyy-MM-dd HH:mm:ss') as event_date_gmt:chararray;
Getting error:
ERROR 1066: Unable to open iterator for alias B. Backend error : org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POUserFunc (Name: POUserFunc(org.apache.pig.builtin.ToDate2ARGS)[datetime] - scope-15 Operator Key: scope-15) children: null at []]: java.lang.IllegalArgumentException: Invalid format: "20190503143744" is malformed at "143744"
The issue is that you're specifying the format as yyyyMMdd but your original input is in yyyyMMddHHmmss format, so you get an error when Pig reaches 143744 instead of the end of your string. Try this:
B = FOREACH A GENERATE ToString(ToDate((chararray)commit_time,'yyyyMMddHHmmss'),
'yyyy-MM-dd HH:mm:ss') as event_date_gmt;
Related
I am getting an error while trying to convert string array to json array in postgresql.
SQL:
select * from
jsonb_array_elements_text(to_jsonb('[{\"Apr2021\":\"1.2\",\"Aug2000\":\"1.3\",\"Dec2023\":\"22.5\",\"Feb2023\":\"66.7\",\"Jan2023\":\"99.1\",\"Jul2023\":\"11.0\",\"Jun2021\":\"44.2\",\"Mar2023\":\"55\",\"May2023\":\"10\",\"Nov2023\":\"44\",\"Oct2023\":\"99\",\"Sep2023\":\"33\"}]'::json))
Error:
> Invalid operation: invalid input syntax for type json Details: Token
> "Apr2021" is invalid.
I have a requirement where I need to invoke UDT type defined in stored Procedure on Oracle, from Mule flow. To attain this I am creating a JDBC array type like:
%dw 2.0
output application/java
---
{
inParam: Db::createArray("Database_Config","ADDRESS_TAB",[ Db::createStruct("Database_Config","ADDRESS_TAB_TYPE",
["TESt-Mule",**("2015-07-04T21:01:01") as DateTime**,"WB"])
])
}
the oracle ADDRESS_TAB_TYPE is
ADDRESS_TAB TABLE OF ADDRESS_TAB_TYPE
Name Null? Type
------------------- ----- -------------
IADDRESS_NAME VARCHAR2(240)
IINACTIVE_DATE DATE
ISTATE VARCHAR2(150)
type of date defined on Oracle side is DATE
after invoking the flow I am getting below error:
ERROR 2021-03-18 23:14:06,971 [[MuleRuntime].uber.04: [playground].oracle-db-loc-testFlow.CPU_INTENSIVE #4d9589a8] [processor: ; event: 876ad840-8811-11eb-a282-a483e7749b4e] org.mule.runtime.core.internal.exception.OnErrorPropagateHandler:
********************************************************************************
Message : "org.mule.weave.v2.el.ExpressionFunctionCallException: Exception while executing createStruct("Database_Config","ADDRESS_TAB_TYPE",
["TESt-Mule",**("2015-07-04T21:01:01") as DateTime**,"WB"])
]) cause: An error occurred when trying to create JDBC Structure. Fail to convert to internal representation: 2015-07-04T21:01:01Z
Trace:
at callFunction (Unknown)
at createStruct (line: -1, column: -1)
at createArray (line: 5, column: 73)
at main (line: 5, column: 12)
I am trying to pass this datevalue to oracle, Any help is appreciated. Thanks!
I used this transformation:
fun dateTimeFormat(inputDateTime) = inputDateTime as DateTime as String{format: "yyyy-MM-dd HH:mm:ss.SSS"}
and it worked.
Thanks!
Would the suggestion here help you out maybe ?
https://help.mulesoft.com/s/question/0D52T00004mXXV4/how-to-insert-the-date-value-into-oracle-database-using-mule-4
Below is my kafka-connect-sink.properties file
I am using confluent-6.0.1.
name=enba-sink-postgres
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
connection.url=jdbc:postgresql://IP:PORT/DB
connection.user=USERNAME
connection.password=PASSWORD
tasks.max=1
topics=postgresInsert
insert.mode=INSERT
table.name.format=schema."tableName"
auto.create=false
key.converter.schema.registry.url=http://localhost:8081
key.converter.schemas.enable=false
value.converter.schemas.enable=false
config.action.reload=restart
value.converter.schema.registry.url=http://localhost:8081
errors.tolerance=all
errors.log.enable=true
errors.log.include.messages=true
print.key=true
# Transforms
transforms=TimestampConverter
transforms.TimestampConverter.type=org.apache.kafka.connect.transforms.TimestampConverter$Value
transforms.TimestampConverter.format=yyyy-MM-dd HH:mm:ss
transforms.TimestampConverter.target.type=Timestamp
transforms.TimestampConverter.target.field=DATE_TIME
I am using avro data and schema is :
{\"type\":\"record\",\"name\":\"log\",\"namespace\":\"transform.name.space\",\"fields\":[{\"name\":\"TRANSACTION_ID\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"MSISDN\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"TRIGGER_NAME\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"W_ID\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"STEP\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"REWARD_ID\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"CAM_ID\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"STATUS\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"COMMENTS\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"CCR_JSON\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"},{\"name\":\"DATE_TIME\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}
Basically DATE_TIME column in Postgres is of type Timestamp and from avro I tried sending String date and also of type long .
DATE_TIME = 2022-12-15 14:38:02
Issue is If I dont use transform then I am getting error :
ERROR: column "DATE_TIME" is of type timestamp with time zone but expression is of type character varying
And If I use transforms as mentioned above then error is :
[2021-02-06 21:47:41,897] ERROR Error encountered in task enba-sink-postgres-0. Executing stage 'TRANSFORMATION' with class 'org.apache.kafka.connect.transforms.TimestampConverter$Value', where consumed record is {topic='enba', partition=0, offset=69, timestamp=1612628261605, timestampType=CreateTime}. (org.apache.kafka.connect.runtime.errors.LogReporter:66)
org.apache.kafka.connect.errors.ConnectException: Schema Schema{com.package.kafkaconnect.Enbalog:STRUCT} does not correspond to a known timestamp type format
I got it working using :
# Transforms
transforms= timestamp
transforms.timestamp.type= org.apache.kafka.connect.transforms.TimestampConverter$Value
transforms.timestamp.target.type= Timestamp
transforms.timestamp.field= DATE_TIME
transforms.timestamp.format= yyyy-MM-dd HH:mm:ss
For some reason transforms=TimestampConverter was not working.
When I run this query on BigQuery UI:
DELETE FROM `A.Books.items` where Date='2018-08-31'
The query works great.
However when I'm trying to do it on Airflow:
delete_sql = '''DELETE FROM `A.Books.items` where Date = {0}'''.format('2018-08-31') # // later this will be variable
delete_old= BigQueryOperator(
task_id='bigquery_delete',
bql=delete_sql,
destination_dataset_table=False,
bigquery_conn_id=CONNECTION_ID,
delegate_to=False,
use_legacy_sql = 'False',
udf_config=False,
dag=dag,
)
it returns:
Exception: BigQuery job failed. Final error was: {u'reason': u'invalidQuery', u'message': u'No matching signature for operator = for argument types: DATE, INT64. Supported signatures: ANY = ANY at [1:52]', u'location': u'query'
Date is a column with type DATE in items table.
How do I fix this error?
The following should work:
delete_sql = '''DELETE FROM A.Books.items where Date = '{0}' '''.format('2018-08-31')
You need quotes when you want to replace the strings.
In BigQuery, I ran this query
SELECT SEC_TO_TIMESTAMP(start_time), resource
FROM [logs.requestlogs_20140305]
and received the error
Error: Invalid function name: SEC_TO_TIMESTAMP
SEC_TO_TIMESTAMP is listed in the date functions reference, so why does this error come up?
Turns out my start_time column was of type float, not an integer which SEC_TO_TIMESTAMP expects. Changing the query to
SELECT SEC_TO_TIMESTAMP(INTEGER(start_time)), resource
FROM [logs.requestlogs_20140305]
fixed the problem.