I run the following code snippet for Pig:
publisher_hour_listings_cdf = foreach rscf_pub_hours_cumsum_proj generate
rscf_publisher_id,
rscf_is_uw,
rscf_hour,
(int)rscf_hour as rscf_hour_int,
rscf_cum_listings,
total_daily_listings,
(rscf_cum_listings*1.0)/(total_daily_listings*1.0) as cdf,
'$handledDate$';
and I get an error:
2020-05-02 04:31:53,130 INFO - [Step] - thread 74038 -
/opt/pig/pig-0.17.0/bin/pig: [main] ERROR
org.apache.pig.tools.pigstats.PigStats - ERROR 0:
org.apache.pig.backend.executionengine.ExecException: ERROR 0:
Exception while executing (Name: publisher_hour_listings_cdf: New For
Each(false,false,false,false,false,false,false,false)[bag] - scope-203
Operator Key: scope-203):
org.apache.pig.backend.executionengine.ExecException: ERROR 0:
Exception while executing [POCast (Name: Cast[long] - scope-171
Operator Key: scope-171) children: [[POProject (Name: Project[int][3]
- scope-170 Operator Key: scope-170) children: null at []]] at [rscf_cum_listings[-1,-1]]]: java.lang.ClassCastException:
java.lang.Long cannot be cast to java.lang.Integer
We thought that Pig fails to cast Long to Integer.
We tried to cast with "biginteger" instead of "int":
(biginteger)rscf_hour as rscf_hour_int
and the script also failed with the exact same error message.
We also tried to remove the int casting at all:
rscf_hour as rscf_hour_int
and we still got the same error.
Do you know why?
Related
I have been trying to log an error from a web activity (POST method) into a field in a synapse table. The problem is, there are some special characters in the message key string like:
{
"value": [
{
"id": "",
"runId": "",
"debugRunId": ,
"runGroupId": "",
"pipelineName": "my_dynamic_pipeline_name",
"parameters": {
"region_code": "",
"data_start_date": "",
"data_end_date": "",
"etl_insert_batch_id": "",
"pipeline_subject_area": "",
"type_of_request": "",
"pipeline_name": "",
"pipeline_requested_by": "",
"debug": "",
"cdmloadtype": ""
},
"invokedBy": {
"id": "",
"name": "",
"invokedByType": ""
},
"runStart": "",
"runEnd": "",
"durationInMs": ,
"status": "",
"message": "Operation on target my_dynamic_pipeline_name failed: Operation on target my_dynamic_dataflow_name failed: {\"StatusCode\":\"DFExecutorUserError\",\"Message\":\"Job failed due to reason: at Sink 'SinkutilFailedDummy': java.sql.BatchUpdateException: Execution Status - FAILED ;Error number - 15165 ;Pipeline name - my_dynamic_pipeline_name; Stored procedure name - my_stored_proc_name ; Error step - Step 3: Key Hash and Util Type Hash Generation ; Insert batch ID - 1816 ; Error Message - Could not find object 'my object' or you do not have permission.\",\"Details\":\"java.sql.BatchUpdateException: Execution Status - FAILED ;Error number - 15165 ;Pipeline name - my_dynamic_pipeline_name; Stored procedure name - my_stored_proc_name ; Error step - Step 3: Key Hash and Util Type Hash Generation ; Insert batch ID - 1816 ; Error Message - Could not find object 'my_object' or you do not have permission.\\n\\tat shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerStatement.executeBatch(SQLServerStatement.java:1845)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter.executeBatchSQLs(JDBCStore.scala:462)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter.executeSQL(JDBCStore.scala:440)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostSQLAndDDL$2.apply$mcV$sp(JDBCStore.scala:494)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostSQLAndDDL$2.apply(JDBCStore.scala:494)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostS\"}",
...
}
So I can filter down the output with:
#activity('pingPL').output.value[0].message
but there are {} and $ special characters that the Data Flow expression is trying to evaluate.
I already try to use replace or string functions in the pipeline expression or in the dataflow expression without success.
Is there a way to parse this as a string or get to the Message key?, something like:
#activity('pingPL').output.value[0].message*.failed*.failed.Message
Update:
This seems to be working:
#json(split(activity('pingPL').output.value[0].message, 'failed: ')[2]).Message
I can split by failed: and the index 2 will give me the error logs within the {...}. I can parse that as a json and use the Message key. It is working but it is not the ideal dynamic solution since the error message wouldn't have always the same structure.
Got a solution using substring and indexof to extract the {...} info:
substring(activity('pingPL').output.value[0].message,indexof(activity('pingPL').output.value[0].message,'{'),sub(indexof(activity('pingPL').output.value[0].message,'}'),sub(indexof(activity('pingPL').output.value[0].message,'{'),1)))
Getting this string as the output:
{\"StatusCode\":\"DFExecutorUserError\",\"Message\":\"Job failed due to reason: at Sink 'SinkutilFailedDummy': java.sql.BatchUpdateException: Execution Status - FAILED ;Error number - 15165 ;Pipeline name - my_dynamic_pipeline_name; Stored procedure name - my_stored_proc_name ; Error step - Step 3: Key Hash and Util Type Hash Generation ; Insert batch ID - 1816 ; Error Message - Could not find object 'my object' or you do not have permission.\",\"Details\":\"java.sql.BatchUpdateException: Execution Status - FAILED ;Error number - 15165 ;Pipeline name - my_dynamic_pipeline_name; Stored procedure name - my_stored_proc_name ; Error step - Step 3: Key Hash and Util Type Hash Generation ; Insert batch ID - 1816 ; Error Message - Could not find object 'my_object' or you do not have permission.\\n\\tat shaded.msdataflow.com.microsoft.sqlserver.jdbc.SQLServerStatement.executeBatch(SQLServerStatement.java:1845)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter.executeBatchSQLs(JDBCStore.scala:462)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter.executeSQL(JDBCStore.scala:440)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostSQLAndDDL$2.apply$mcV$sp(JDBCStore.scala:494)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostSQLAndDDL$2.apply(JDBCStore.scala:494)\\n\\tat com.microsoft.dataflow.transformers.store.JDBCWriter$$anonfun$executeTableOpAndPostS\"}
Then I used json expression to extract the key message:
json('extracted string').message
Then use replace to remove the single quotations ' to avoid a sql error.
This is the final expression I got to extract the error message:
#replace(json(substring(activity('pingPL').output.value[0].message,indexof(activity('pingPL').output.value[0].message,'{'),sub(indexof(activity('pingPL').output.value[0].message,'}'),sub(indexof(activity('pingPL').output.value[0].message,'{'),1)))).message,'''','-')
I have a simple array as such :
qux > [1,2,3]
upon deleting the array via Client.json.arrPop('qux', '.' , 1) I receive
F:\vite-multiple-chat\backendofmultiplchat\node_modules\.pnpm\#node-redis+json#1.0.1_#node-redis+client#1.0.1\node_modules\#node-redis\json\dist\commands\index.js:87
return jsons.map(transformRedisJsonNullReply);
^
TypeError: jsons.map is not a function
at Object.transformRedisJsonNullArrayNullReply (F:\vite-multiple-chat\backendofmultiplchat\node_modules\.pnpm\#node-redis+json#1.0.1_#node-redis+client#1.0.1\node_modules\#node-redis\json\dist\commands\index.js:87:18)
at transformCommandReply (F:\vite-multiple-chat\backendofmultiplchat\node_modules\.pnpm\#node-redis+client#1.0.1\node_modules\#node-redis\client\dist\lib\commander.js:100:20)
at Commander.commandsExecutor (F:\vite-multiple-chat\backendofmultiplchat\node_modules\.pnpm\#node-redis+client#1.0.1\node_modules\#node-redis\client\dist\lib\client\index.js:160:54)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
The element is popped successfully however the error is thrown anyway. How can I resolve this issue?
I have a DataWeave message transformer, let's say:
%dw 2.0
import fail from dw::Runtime
output application/java
fun isValuePresent(value, message: String) = if ( value == null or isEmpty(value) ) fail(message) else value
---
{
brand: isValuePresent(payload.document[0].brand, p('import.error.missing.brand')),
...
I also have an error handler for this kind of errors.
Errors in Mule have their properties, like: description or detailedDescription.
Now normally, when I am catching other errors (like those from is true component) - everything is fine, error.description holds my error message, everything is fine.
But when an error produced by fail() is produced, I get a very big error description message:
""my error message here
Trace:
at fail (Unknown)
at isValuePresent (line: 13, column: 85)
at main (line: 23, column: 7)" evaluating expression: "%dw 2.0
import fail from dw::Runtime
output application/java
fun isValuePresent(value, message: String) = if ( value == null or isEmpty(value) ) fail(message) else value
---
{
brand: isValuePresent(payload.document[0].brand, p('import.error.missing.brand')),
...
...
etc, etc
It looks like the whole content of my dataweave script is added to the trace. And I just want to have:
my error message here
Trace:
at fail (Unknown)
at isValuePresent (line: 13, column: 85)
at main (line: 23, column: 7)" evaluating expression: "%dw 2.0
Is it possible to achieve this? Or I have I made some mistakes when designing this behaviour? Is there a way to fix this?
I don't think you can do anything about that. It is the way the error is reported with fail() and it is not customizable. Probably it depends on how DataWeave itself reports errors rather than fail() itself.
I'm new in Mulesoft, I'm following Quickstart guide. In Step 2 (https://developer.mulesoft.com/guides/quick-start/developing-your-first-mule-application), I need to receive variables from URI in this way:
[{'id' : attributes.uriParams.productId}]
But when I try my GET I have the following error in console:
**Message : "Cannot coerce Array ([{id: "2" as String {class: "java.lang.String"}}]) to Object 1| [{'id' : attributes.uriParams.productId}] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Trace: at main (line: 1, column: 1)" evaluating expression: "[{'id' : attributes.uriParams.productId}]". Error type : MULE:EXPRESSION Element : get:\products(productId):test_daniel-config/processors/1 # test6_db_connection:test_daniel.xml:133 (Select) Element XML : SELECT product.,CONCAT('["', (GROUP_CONCAT(variant.picture SEPARATOR '","')),'"]') AS pictures,CONCAT('[', GROUP_CONCAT('{"',variant.identifierType, '":"', variant.identifier, '"}'),']') AS identifiersFROM product INNER JOIN variant ON product.uuid = variant.productUUIDWHERE product.uuid = :id; #[[{'id' : attributes.uriParams.productId}]] *
Any Ideas? Thanks!
cannot coerce Array to object error pop's up when you are using an array where you were supposed to use an object.
in the exception above the uri-param should be treated as ab object i.e. enclosed in {} but its being treated as an array of objects [{}].
this is causing the error.
We have successfully executed the DatabaseTablesPreparer and inited the tables in the DB, but when we try to init the indexes on the table with SQLScriptPreparer, we get the following exception:
ES1 dbinit [] [] com.intershop.platform.cartridge.internal.CartridgeImpl [] [] [] [] "main" Neither Ivy descriptor nor cartridge properties found for cartridge 'app_core_a1'!
ES1 dbinit [] [app_core_a1:Class1 DatabaseIndexesPreparer [hr/a1/core/dbinit/scripts/dbindex.ddl] Version:null] com.intershop.beehive.core.dbinit.preparer.database.DatabaseIndexesPreparer [] [] [] [] "main" [core] Exception java.lang.NullPointerException: null
at com.intershop.beehive.core.dbinit.preparer.database.SQLScriptPreparer.getCommand(SQLScriptPreparer.java:158)
at com.intershop.beehive.core.dbinit.preparer.database.SQLScriptPreparer.process(SQLScriptPreparer.java:353)
We had the similar problem with DatabaseTablesPreparer (Cartridge was null), and we solved it by adding cartridge.properties file, but now we are getting the same error ("Neither Ivy descriptor nor cartridge properties found for cartridge 'app_core_a1'") even though the cartridge properties file is defined.
There are the lines in decompiled preparer code where the null pointer exception occurs:
getCartridge().getVersion() + (getCartridge().getBuild().isEmpty() ? "" : new StringBuilder().append(".").append(getCartridge().getBuild()).toString()) };
This is the preparer from dbinit.properties:
Class1 = com.intershop.beehive.core.dbinit.preparer.database.DatabaseIndexesPreparer \
hr/a1/core/dbinit/scripts/dbindex.ddl
And this is the dbinit command we are executing:
dbinit.bat --exec-id=app_core_a1:Class1
DatabaseTablesPreparer from the same cartridge, defined in the same dbinit executes successfully.
Problem was fixed by publishing cartridge. It seems that ivy descriptor was deleted and it had to be republished.