how to use dirSize() EL function in oozie decision node - el

I tried to use dirSize() in oozie decision node. but it does not work.
When I use ${fs:dirSize(InputDir) gt 10 * KB} in the decision node. The oozie workflow status moves to failed state and no error is displayed.
Please find below the code snippet from workflow.
<action name="L1_check" cred="hcat_creds">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<script>l1_check_code.sql</script>
<param>PI_DB=${PI_DB}</param>
<param>PRD_DB=${PRD_DB}</param>
<param>PROMO_START_DATE=${PROMO_START_DATE}</param>
<param>PROMO_END_DATE=${PROMO_END_DATE}</param>
</hive>
<ok to="decision-node" />
<error to="fail" />
</action>
<decision name="decision-node">
<switch>
<case to="L1_exists">
${fs:dirSize(InputDir) gt 10 * KB}
</case>
<default to="fail"/>
</switch>
</decision>
InputDir is a defined in the properties file and the path exists.
First action L1_check executes and transitioned to decision node. then workflow status changes to Failed state.
Is there any error in the way function is used? If yes, what should be right function to use?
Also is there a way to give a path with combination of parameter and string
eg: dirSize(InputDir/l1_check)
where InputDir is a parameter and l1_check is a static name

Related

Oozie - EL_ERROR: cannot convert String to type Double

Getting oozie EL_ERROR. Please find below the oozie workflow details. Please advise.
Error Code: EL_ERROR
Error Message: An exception occured trying to convert String "/tmp/dir" to type "java.lang.Double"
Expected Result: /tmp/dir/0007735-200221155317872-oozie-oozi-W/destination
workflow.xml
<decision name="On_Success">
<switch>
<case to="success">${fs:dirSize(hdfs_path_variable+wf:id()+hdfs_dir) gt 0 }</case>
<default to="fail"/>
</switch>
</decision>
job.properties
hdfs_path_variable=/tmp/dir
hdfs_dir=destination

Apache oozie Launcher ERROR

I am want to run sqoop action using apache oozie but it always giving me error:
Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], main() threw exception, null
my workflow.xml is:
<workflow-app name="Datashop-Core-Sqoop" xmlns="uri:oozie:workflow:0.1">
<!-- <start to="inputAvailableCheckDecision"/>
<decision name="inputAvailableCheckDecision">
<switch>
<case to="sqoopAction">
${sqoopInputRecordCount gt minRequiredRecordCount}
</case>
<default to="end"/>
</switch>
</decision> -->
<start to="sqoopAction"/>
<action name="sqoopAction">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>oozie.libpath</name>
<value>${oozieLibPath}</value>
</property>
</configuration>
<command>import --connect jdbc:mysql://${mysqlServer}/${mysqlServerDB} --username ${mysqlServerDBUID} --password ${mysqlServerDBPwd} --tabltable ${mysqlTable} -m 1 --target-dir ${targetDir}</command>
</sqoop>
<ok to="end"/>
<error to="killJob"/>
</action>
<kill name="killJob">
<message>"Killed job due to error: ${wf:errorMessage(wf:lastErrorNode())}"</message>
</kill>
<end name="end" />
</workflow-app>
this always throwing sqoopmain class exception which is finally treating as Error:
#sqoopAction] ERROR is considered as FAILED for SLA
In some stack question i found that external library is need to be included with <file>
so i changed my workflow.xml and add a line below <command> tag:
<file>${oozieLibPath}/sqoop/mysql-connector-java-5.1.37-bin.jar#mysql-connector-java-5.1.37-bin.jar</file>
But still same error is producing what may be the reason for this error. is there any thing else i need to configure?

Oozie EL function: An exception occured trying to convert String to type "java.lang.Double"

I am trying to run an Oozie workflow that calls EL function replaceAll(). The action that's using the replaceAll() is this
<action name="createSuccess">
<fs>
<configuration>
<property>
<name>runDate</name>
<value>${replaceAll(hdfsDir, nameNode + '/(.+)/' + region + '/([0-9\\-]+)/?', '$2')}</value>
</property>
</configuration>
<mkdir path="${nameNode}/path/run/${region}/${runDate}"/>
<touchz path="${nameNode}/path/run/${region}/${runDate}/success.txt"/>
</fs>
<ok to="end"/>
<error to="sendEmailKill"/>
</action>
hdfsDir is something like hdfs://nameNode:8020/some/path/region/2015-04-22 and I need to grab that date at the end as a property and use it.
But when I run the above action, I got this exception:
javax.servlet.jsp.el.ELException: An exception occured trying to convert String "hdfs://nameNode:8020" to type "java.lang.Double"
at org.apache.commons.el.Logger.logError(Logger.java:481)
at org.apache.commons.el.Logger.logError(Logger.java:498)
at org.apache.commons.el.Logger.logError(Logger.java:566)
at org.apache.commons.el.Coercions.coerceToPrimitiveNumber(Coercions.java:440)
at org.apache.commons.el.Coercions.applyArithmeticOperator(Coercions.java:852)
at org.apache.commons.el.ArithmeticOperator.apply(ArithmeticOperator.java:83)
at org.apache.commons.el.BinaryOperatorExpression.evaluate(BinaryOperatorExpression.java:170)
at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:163)
at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:175)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:60)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Any ideas why I'm getting this exception and how to fix it?
After trial an error, I figured out that I can not use "+" to concatenate two strings. I have to use this:
${replaceAll(hdfsDumpDir, concat(concat(concat(nameNode, '/(.+)/'), region), '/'), '')}

Failed to use log4crc configuration file

I am trying to use log4c for my C application logging. For implementing rolling policy, I used log4c rolling file APIs. I understood from Log4C developers guide that the rolling policy could be achieved by editing the configuration file (log4crc). I tried editing the appender name to a log file path in the configuration file. But the application run without logging. Can anyone please tell me how to implement rolling log mechanism using log4c?
Please see my helloworld application sample and the configuration file below:
#include<stdio.h>
#include"log4c.h"
int main(int argc, char** argv)
{
int rc = 0;
log4c_category_t* mycat = NULL;
if (log4c_init()){
printf("log4c_init() failed");
rc = 1;
}else{
mycat = log4c_category_get("log4c.examples.helloworld");
log4c_category_log(mycat, LOG4C_PRIORITY_ERROR, "Hello World!");
/* Explicitly call the log4c cleanup routine */
if ( log4c_fini()){
printf("log4c_fini() failed");
}
}
return 0;
}
The configuration file looks as follows:
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE log4c SYSTEM "">
<log4c version="1.2.3">
<config>
<bufsize>0</bufsize>
<debug level="2"/>
<nocleanup>0</nocleanup>
<reread>1</reread>
</config>
<category name="root" priority="notice"/>
<category name="six13log.log" priority="error" appender="stdout" />
<rollingpolicy name="myrollingpolicy" type="sizewin" maxsize="1024" maxnum="10" />
<appender name="myrollingfileappender" type="rollingfile" logdir="." prefix="myprefix" layout="dated" rollingpolicy="myrollingpolicy" />
<appender name="stdout" type="stream" layout="basic"/>
<appender name="stderr" type="stream" layout="dated"/>
<appender name="syslog" type="syslog" layout="basic"/>
<appender name="s13file" type="s13_file" layout="basic"/>
<appender name="plain_stderr" type="s13_stderr" layout="none"/>
<appender name="cat_stderr" type="s13_stderr" layout="catlayout"/>
<appender name="xml_stderr" type="s13_stderr" layout="xmllayout"/>
<appender name="user_stderr" type="s13_stderr" layout="userlayout"/>
<layout name="basic" type="basic"/>
<layout name="dated" type="dated"/>
<layout name="catlayout" type="s13_cat"/>
<layout name="xmllayout" type="s13_xml"/>
<layout name="none" type="s13_none"/>
<layout name="userlayout" type="s13_userloc"/>
<category name="six13log.log.app.application2" priority="debug" appender="cat_stderr" />
<category name="six13log.log.app.application3" priority="debug" appender="user_stderr" />
<category name="six13log.log.app" priority="debug" appender="myrollingfileappender" />
<category name="log4c.examples.helloworld" priority="debug" appender="stdout"/>
<category name="log4c.examples.helloworld" priority="debug" appender="/home .. /..filename.txt"/>
</log4c>
I edited the last two lines in the configuration file for logging.
Please let me know what is wrong with my log4crc configuration file and how I can use configuration file to implement logging without using log4c rolling policy apis?
I am also new to log4c. But what I understand from your log4c configuration file
you have category name "log4c.examples.helloworld" twice. So when you write log4c_category_get("log4c.examples.helloworld") it is going to check from CRC file and will get appender name.
Corresponding to log4c.examples.helloworld the appender names are "stdout" and "/home .. /..filename.txt". First one is proper as it will give output to screen. The last one is improper. Do not specify file path there. Put name/ string that will correspoond to appender name. And then in appender name you have logdir value where you can specify the directory where your file will get created.
Check whether your log4crc file is included in the helloworld directory
Removing the version number completely works for me:
Change
<log4c version="1.2.3">
to
<log4c>

cannot retrieve mapping for action error

im using struts 1.2 and trying to create a link in jsp page with code
<html:link href="AdminModule/UserCreation.jsp" >
Create User </html:link> `
so that the link will move to page UserCreation in which the code like this is present
<html:form action="/adminUserCreation">
AdminName : <html:text property="username"/><html:errors property="username"/><br/>
Password : <html:password property="password"/><html:errors property="password"/><br/>
<html:submit/>
</html:form>
the struts-config file contain code like this
<form-beans>
<form-bean name="adminUserCreationForm" type="Admin.Form.AdminUserCreationForm" />
</form-beans>
<action-mappings>
<action
attribute="adminUserCreationForm"
input="/AdminModule/AdminHomePage.jsp"
name="adminUserCreationForm"
path="/adminUserCreation"
scope="request"
type="Admin.Action.AdminUserCreationAction"
validate="false">
<set-property property="cancellable" value="true" />
<forward name="failure" path="/AdminModule/AdminErrorPage.jsp" />
<forward name="success" path="/AdminSuccessPage.jsp" />
</action>
</action-mappings>
im getng exception
An exception occurred processing JSP page /AdminModule/UserCreation.jsp at line 30
30: <html:form action="/adminUserCreation">
31: AdminName : <html:text property="username"/><html:errors property="username"/><br/>
32: Password : <html:password property="password"/><html:errors property="password"/><br/>
33: <html:submit/>
root cause is
javax.servlet.ServletException: javax.servlet.jsp.JspException: Cannot retrieve mapping for action /adminUserCreation
How to open one jsp from another through struts
This helped me to solve it..
Maybe it will be helpful for anyone else who has the same problem.