Hue: oozie parameters - hive

I want to pass 2 parameters to my Hiveql script in oozie,
my script:
ALTER TABLE default.otarie_appsession
ADD IF NOT EXISTS PARTITION ( insert_date=${dt},hr=${hr} );
My Oozie workflow :
When i send the job it ask for parameter values, so i put:
And this is the error:
2016-02-05 18:41:55,460 WARN org.apache.oozie.action.hadoop.HiveActionExecutor: SERVER[DVS1VM65] USER[root] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000290-160122145737153-oozie-oozi-W] ACTION[0000290-160122145737153-oozie-oozi-W#hive-a586] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]
This is the XML of the workflow:
<workflow-app name="My_Workflow" xmlns="uri:oozie:workflow:0.5">
<start to="hive-a586"/>
<kill name="Kill">
<message>L'action a échoué, message d'erreur[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="hive-a586">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/user/oozie/some_scripts/hive-site.xml</job-xml>
<script>/user/oozie/some_scripts/addpart.hql</script>
<param>hr=</param>
<param>dt=</param>
</hive>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
When i remove all parmeters and use hard coded value, the script works fine, so its clear that i have problem to pass parameters. And my final goal is to pass current date and hour.
Thank you.

What about setting Hive parameters with Oozie parameters...
<param>hr=${bilouteHR}</param>
<param>dt=${bilouteDT}</param>
...then setting values for these Oozie parameters at submission time?
bilouteHR
00
bilouteDT
20160105
Hope that solves your issue, biloute.

Related

What causes "Value cannot be negative" in cloudwatch agent logs?

I see the following log entries in /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log
2021-10-06T13:18:00Z W! Value cannot be negative: -1
2021-10-06T13:18:00Z W! Value cannot be negative: -1
2021-10-06T13:18:00Z W! Value cannot be negative: -1
Is there anyway to find out what is the source of this -1 value?
I don't think there is a good way to find the actual source in the general case. That's why I opened an issue in aws/amazon-cloudwatch-agent.
But there are some guesses that you can do:
First, check if you are using collectd GenericJMX to collect the NonHeapMemoryUsage attribute from java.lang:type=Memory MBean, that is part of the usual collectd's GenericJMX example configuration. The NonHeapMemoryUsage is a composite attribute with committed,init, max and used. And max is always -1.
You can verify this using jmx-term cli utility:
java -jar jmxterm-1.0.2-uber.jar
$>open localhost:8993
$>get -d java.lang -b java.lang:type=Memory NonHeapMemoryUsage
#mbean = java.lang:type=Memory:
NonHeapMemoryUsage = {
committed = 312860672;
init = 7667712;
max = -1;
used = 304634560;
};
If this is the case you can use collectd Chains functionality to "drop" the TypeInstance=nonheap-max, by adding the following to your collectd.conf:
<Chain "PostCache">
# only show system and user
<Rule "jmx_memory">
<Match "regex">
Plugin "GenericJMX"
Type "jmx_memory"
</Match>
<Match "regex">
TypeInstance "nonheap-max"
Invert true
</Match>
# If a jmx_memory metric and it's not the nonheap-max then send it to the amazon-cloudwatch-agent 25826
<Target "write">
Plugin network
</Target>
</Rule>
# Default target drop all metrics
Target "stop"
</Chain>
Second, if it not the NonHeapMemoryUsage is likely to be any other collectd metric, you can drop all metrics with negative values with collectd's <Match "Value"> although I don't recommend it:
<Chain "PostCache">
# only show system and user
<Rule "drop negative">
<Match "value">
Min 0
Satisfy "All"
</Match>
# send positive values to amazon-cloudwatch-agent 25826
<Target "write">
Plugin network
</Target>
</Rule>
# Default target drop all metrics
Target "stop"
</Chain>

failed to schedual xaction file for run PDI job

I want to run my Pentaho-data-integration job with xaction file.
my job(schedualeJob.kjb) has one transformation(repository2.ktr)( without any schedule).
the contetnt of my schedule.xaction file is:
<?xml version="1.0" encoding="UTF-8"?>
<action-sequence>
<title>My scheduled job</title>
<version>1</version>
<logging-level>ERROR</logging-level>
<documentation>
<author>mzy</author>
<description>Sequence for running daily job.</description>
<help/>
<result-type/>
<icon/>
</documentation>
<inputs>
</inputs>
<outputs>
<logResult type="string">
<destinations>
<response>content</response>
</destinations>
</logResult>
</outputs>
<resources>
<job-file>
<solution-file>
<location>schedualeJob.kjb</location>
<mime-type>text/xml</mime-type>
</solution-file>
</job-file>
</resources>
<actions>
<action-definition>
<component-name>KettleComponent</component-name>
<action-type>Pentaho Data Integration Job</action-type>
<action-inputs>
</action-inputs>
<action-resources>
<job-file type="resource"/>
</action-resources>
<action-outputs>
<kettle-execution-log type="string" mapping="logResult"/>
<kettle-execution-status type="string" mapping="statusResult"/>
</action-outputs>
<component-definition>
<kettle-logging-level><![CDATA[info]]></kettle-logging-level>
</component-definition>
</action-definition>
</actions>
</action-sequence>
I uploaded all of these 3 file in my Pentaho cde repository(beside of my report).
and set schedule for schedule.xaction file from pentaho cde. but the schedule does not run correctly. any body can help me?

"file or LOB operation FILEOPEN failed" when loading an XML file into a table

I am using 11gr2 and trying to load an XML file into a table.
Here is the code:
connected as sys:
GRANT ALL ON DIRECTORY XMLDIR TO user_1;
switch to user_1:
CREATE TABLE mytable1 (key_column VARCHAR2(10) PRIMARY KEY,
xml_column XMLType);
CREATE DIRECTORY xmldir AS 'F:\OracleTestFiles\XMLs\';
the XML file:
<PurchaseOrder
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation=
"http://localhost:8080/source/schemas/poSource/xsd/purchaseOrder.xsd">
<Reference>SBELL-2002100912333601PDT</Reference>
<Actions>
<Action>
<User>SVOLLMAN</User>
</Action>
</Actions>
<Reject/>
<Requestor>Sarah J. Bell</Requestor>
<User>SBELL</User>
<CostCenter>S30</CostCenter>
<ShippingInstructions>
<name>Sarah J. Bell</name>
<address>400 Oracle Parkway
Redwood Shores
CA
94065
USA</address>
<telephone>650 506 7400</telephone>
</ShippingInstructions>
<SpecialInstructions>Air Mail</SpecialInstructions>
<LineItems>
<LineItem ItemNumber="1">
<Description>A Night to Remember</Description>
<Part Id="715515009058" UnitPrice="39.95" Quantity="2"/>
</LineItem>
<LineItem ItemNumber="2">
<Description>The Unbearable Lightness Of Being</Description>
<Part Id="37429140222" UnitPrice="29.95" Quantity="2"/>
</LineItem>
<LineItem ItemNumber="3">
<Description>Sisters</Description>
<Part Id="715515011020" UnitPrice="29.95" Quantity="4"/>
</LineItem>
</LineItems>
</PurchaseOrder>
the insert statement:
INSERT INTO mytable1 VALUES ('1a', XMLType(bfilename('XMLDIR', 'PurchaseOrder.xml'), nls_charset_id('AL32UTF8')));
When I am executing the INSERT statement, I receive the following error:
Error report - ORA-22288: file or LOB operation FILEOPEN failed The
system cannot find the path specified. ORA-06512: at "SYS.XMLTYPE",
line 296 ORA-06512: at line 1
Can someone explain what is wrong, please?
Should I grant other privileges in the OS (Windows 7)?
I found the problem.
The phisically location of the directory from the hard drive was on a mapped driver (F:).
I changed it on C: and now it is working like a charm.

custom properties in liquibase.properties file

I am using liquibase 2.0.5 and my properties file looks like below
#liquibase.properties
url=jdbc:oracle:thin:#//localhost:1521/orcl
username=myschama
password=myschama
masterUsername=system
masterPassword=system
I want to use system user to create myschema user. Currently, I get the following error.
Unexpected error running Liquibase: Unknown parameter: 'masterUsername'
SEVERE 12/21/17 12:59 PM:liquibase: Unexpected error running Liquibase: Unknown parameter: 'masterUsername'
liquibase.exception.CommandLineParsingException: Unknown parameter: 'masterUsername'
at liquibase.integration.commandline.Main.parsePropertiesFile(Main.java:387)
at liquibase.integration.commandline.Main.main(Main.java:122)
Is it possible to have such custom values in the properties file
If you want to use it in your changeSets as property, e.g.
<changeSet ...>
<sql>
INSERT INTO myTab(some_column) VALUES ('${masterUsername}')
</sql>
</changeSet>
then try putting this into your liquibase.properties:
parameter.masterUsername=your_desired_value
As an alternative, you can pass these on the command line as Java properties, e.g.
liquibase update ... -DmasterUsername=your_desired_value
See http://www.liquibase.org/documentation/changelog_parameters.html and http://www.liquibase.org/documentation/command_line.html for details.

Not null sqlCheck in Liquibase precondition

I'm wondering if there is an easy way to create liquibase precondition checking if particular value is NULL or NOT NULL. My approach doesn't work.
I created the following test changeset:
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">
<changeSet id="1" author="test">
<createTable tableName="test_table">
<column name="id" type="int">
<constraints primaryKey="true" nullable="false" />
</column>
<column name="test_text" type="varchar(50)">
<constraints nullable="true" />
</column>
</createTable>
</changeSet>
<changeSet id="2" author="test">
<insert tableName="test_table">
<column name="id" value="1" />
<column name="test_text" value="test1" />
</insert>
<insert tableName="test_table">
<column name="id" value="2" />
<column name="test_text" value="null" />
</insert>
</changeSet>
<changeSet id="3" author="test">
<preConditions onFail="HALT">
<sqlCheck expectedResult="test1">select test_text from test_table where id=1</sqlCheck>
</preConditions>
<insert tableName="test_table">
<column name="id" value="3" />
<column name="test_text" value="foo" />
</insert>
</changeSet>
<changeSet id="4" author="test">
<preConditions onFail="HALT">
<sqlCheck expectedResult="NULL">select test_text from test_table where id=2</sqlCheck>
</preConditions>
<insert tableName="test_table">
<column name="id" value="4" />
<column name="test_text" value="bar" />
</insert>
</changeSet>
</databaseChangeLog>
Everything works fine except the changeset 4, which is failing with surprising error message "No rows returned from SQL Precondition". However the data is there:
$ psql -U test -c 'select * from test_table'
id | test_text
----+-----------
1 | test1
2 |
3 | foo
(3 rows)
It seems the result is interpreted as empty. In fact it returns one row with one column containing NULL value. Any ideas how to make this working?
I'm using Liquibase 3.3.5 with PostgreSQL 9.3 on Ubuntu 14.04.
Here is the full output (info level):
$ ./liquibase --driver=org.postgresql.Driver --url="jdbc:postgresql://localhost:5432/test" --username=test --password="" --changeLogFile=/tmp/test.changeset.xml --logLevel=info update
Liquibase Home is not set.
INFO 02.06.15 09:57: liquibase: Successfully acquired change log lock
INFO 02.06.15 09:57: liquibase: Creating database history table with name: public.databasechangelog
INFO 02.06.15 09:57: liquibase: Reading from public.databasechangelog
INFO 02.06.15 09:57: liquibase: /tmp/test.changeset.xml: /tmp/test.changeset.xml::1::test: Table test_table created
INFO 02.06.15 09:57: liquibase: /tmp/test.changeset.xml: /tmp/test.changeset.xml::1::test: ChangeSet /tmp/test.changeset.xml::1::test ran successfully in 15ms
INFO 02.06.15 09:57: liquibase: /tmp/test.changeset.xml: /tmp/test.changeset.xml::2::test: New row inserted into test_table
INFO 02.06.15 09:57: liquibase: /tmp/test.changeset.xml: /tmp/test.changeset.xml::2::test: New row inserted into test_table
INFO 02.06.15 09:57: liquibase: /tmp/test.changeset.xml: /tmp/test.changeset.xml::2::test: ChangeSet /tmp/test.changeset.xml::2::test ran successfully in 10ms
INFO 02.06.15 09:57: liquibase: /tmp/test.changeset.xml: /tmp/test.changeset.xml::3::test: New row inserted into test_table
INFO 02.06.15 09:57: liquibase: /tmp/test.changeset.xml: /tmp/test.changeset.xml::3::test: ChangeSet /tmp/test.changeset.xml::3::test ran successfully in 3ms
SEVERE 02.06.15 09:57: liquibase: /tmp/test.changeset.xml: /tmp/test.changeset.xml::4::test: Change Set /tmp/test.changeset.xml::4::test failed. Error: Migration failed for change set /tmp/test.changeset.xml::4::test:
Reason:
/tmp/test.changeset.xml : No rows returned from SQL Precondition
liquibase.exception.MigrationFailedException: Migration failed for change set /tmp/test.changeset.xml::4::test:
Reason:
/tmp/test.changeset.xml : No rows returned from SQL Precondition
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:485)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:43)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:73)
at liquibase.Liquibase.update(Liquibase.java:200)
at liquibase.integration.commandline.Main.doMigration(Main.java:1044)
at liquibase.integration.commandline.Main.run(Main.java:175)
at liquibase.integration.commandline.Main.main(Main.java:94)
Caused by: liquibase.exception.PreconditionFailedException: Preconditions Failed
at liquibase.precondition.core.AndPrecondition.check(AndPrecondition.java:51)
at liquibase.precondition.core.PreconditionContainer.check(PreconditionContainer.java:201)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:471)
... 6 more
INFO 02.06.15 09:57: liquibase: /tmp/test.changeset.xml::4::test: Successfully released change log lock
Unexpected error running Liquibase: Preconditions Failed
SEVERE 02.06.15 09:57: liquibase: /tmp/test.changeset.xml::4::test: Preconditions Failed
liquibase.exception.MigrationFailedException: Migration failed for change set /tmp/test.changeset.xml::4::test:
Reason:
/tmp/test.changeset.xml : No rows returned from SQL Precondition
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:485)
at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:43)
at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:73)
at liquibase.Liquibase.update(Liquibase.java:200)
at liquibase.integration.commandline.Main.doMigration(Main.java:1044)
at liquibase.integration.commandline.Main.run(Main.java:175)
at liquibase.integration.commandline.Main.main(Main.java:94)
Caused by: liquibase.exception.PreconditionFailedException: Preconditions Failed
at liquibase.precondition.core.AndPrecondition.check(AndPrecondition.java:51)
at liquibase.precondition.core.PreconditionContainer.check(PreconditionContainer.java:201)
at liquibase.changelog.ChangeSet.execute(ChangeSet.java:471)
... 6 more
Might not be the most simple or elegant way but you could use a <customPrecondition>.
You would write your own class that has to implement CustomPrecondition.
Checkout this example class: ExampleCustomPrecondition as a reference.
You get a Database object at hand and could do a query to the database yourself and throw a CustomPreconditionFailedException.
(Alternatively create a Jira-Ticket to file a request for changing this behavior or adding a way to achieve the null preCondition check.)
This workaround might work
<sqlCheck expectedResult="1">select COUNT(id) from test_table where id=2 AND test_text IS NULL</sqlCheck>