I am new to oozie and want to add a hive job to my workflow. Could you please tell me where could I find or how could I create the hive-default.xml. I have actually installed everthing via cloudera manager and I am not sure where to find this file. I have looked for it in /etc/hive/conf which seems to be its usual directory but it is not there in that folder. I also did a find command in the terminal and it didn't pull up any file. Please help.
In /etc/hive/conf you should have hive-site.xml. You can copy this file to your HDFS workflow directory and rename it to hive-default.xml, should work.
Related
Trying to get the sidekick image built and having some issues. Is there any documentation other than the README.md file?
My current problem is with getting the JRE requirement working but there are others. The page says "download Oracle JRE and place it inside the working directory. Optionally if you have a company wide distribution url, use that one at a later step." and the help says "Java (JRE) download url or path inside working directory". Have not been able to get this to work.
I went to the JRE link provided and was presented with options to download a rpm file or a tar.gz file. Which is expected (was unable to get either one working)?
It says to place the file in the "working directory" but not sure where exactly. Tried in sidekick folder and in sidekick/jre both without success no matter what I used after the -j command. Is this just the path or should the filename be included as well? Can I get an example?
I'm running this script using my login but noticed the output folder is being created with root user and group. I see no indication that this should be run with sudo. What is the correct way to run this script?
Using debug, I see the function "download if not cached". Can I save these files (JRE, Bamboo jar file, etc.) somewhere so I don't have to worry about downloading them? If so, where should they go? Looks like I might have a problem with the wget to d/l the jar file so would like to just be able to place all these in a folder and be done with it.
It looks like the major problem is the script didn't clean up after itself if it fails. The issue was the first time it failed then that caused subsequent issues as the output folder was already there. Removing this directory between each attempt help.
As for the correct syntax for the -j JRE option I manually downloaded the JRE and placed in a folder called per-build-container/sidekick/stuff/. For the command line it is not just the path but the file name as well (the tar.gz and not the RPM). For my case it was
-j stuff/jre-8u251-linux-x64.tar.gz
Note I also ran the script as sudo. Wasn't stated but seemed to work OK.
Another issue I ran into was the download of the agent jar file. There is a redirect in the wget file that was not working for us. I ended up editing the script and replacing the Altassian based url with the redirected one.
This addresses all the issues I ran into with the initial question.
We know that
add jar s3://elasticmapreduce/samples/hive-ads/libs/jsonserde.jar;
is only effective during the current session.
Is there a way to add jar to Hive permanently and globally so that the jar will be available during the lifecycle of the cluster?
UPDATE:
I figured out a way: download the jar by using aws cli, aws s3 cp s3://elasticmapreduce/samples/hive-ads/libs/jsonserde.jar ., then copy the jar to /usr/lib/hive/lib of all nodes of the EMR cluster
Is there a better way to do this?
insert your ADD JAR commands in your .hiverc file and start hive.
add jar yourjarName.jar
What is .hiverc file?
It is a file that is executed when you launch the hive shell - making it an ideal place for adding any hive configuration/customization you want set, on start of the hive shell. This could be:
Setting column headers to be visible in query results
Making the current database name part of the hive prompt
Adding any jars or files
Registering UDFs
2 .hiverc file location
The file is loaded from the hive conf directory.
I have the CDH4.2 distribution and the location is:
/etc/hive/conf.cloudera.hive1
If the file does not exist, you can create it. It needs to be
deployed to every node from where you might launch the Hive shell.
ref-http://hadooped.blogspot.in/2013/08/hive-hiverc-file.html
I am trying to install Yii framework via archive file in Windows 7...after downloading the file I tried extracting it into a folder in my C:\wamp\www\bid location, however, I encounter an error saying this:
! C:\wamp\www\bid\yii-basic-app-2.0.2.tgz: Cannot create symbolic link C:\wamp\www\bid\basic\vendor\bin\markdown
A required privilege is not held by the client.
! C:\wamp\www\bid\yii-basic-app-2.0.2.tgz: Cannot create symbolic link C:\wamp\www\bid\basic\vendor\bin\yii
A required privilege is not held by the client.
I thought that perhaps Winrar can't extract tgz files so I downloaded 7-zip...Using 7-zip it extracted into a tar file and the tar file was extracted with no errors...For some reason though I do not seem to have the framework folder that seem to be in other people's directory structure...Moreover after trying create my own framework folder and performing this command:
yiic webapp C:\wamp\www\bid
It states that yiic is not recognized as an internal or external command
Can someone tell me what I'm doing wrong...I've tried setting this up numerous times and failed
You need to run 7-Zip File Manager in administrator mode.
Right-click the icon of 7-Zip File Manager, and then click "Run as administrator".
you need to call the command with php since it is actually a external command, i do it like this
php ./yiic webapp NameOfApp
This way your terminal understand that yii is a program that runs with php.
Another solution is to add the yii path to your environmental variable, regards
I am working on Pentaho Kettle version 5.0.1. In one of my transformation I am using javascript component where I am calling a method located in the JAR which I have copied to the lib folder of data-integration and everything is working fine in my local. But in my dev environment(I run it using kitchen) I don't have permission to copy my Jar file to the lib folder due to the restrictions on the server. Is there any other way using which I can supply the path of my custom Jar during run time so that the Kettle Job/Transformation can use it while being executed. Is there a way Kettle can pick the Jar location other than data-integration/lib?. Any help will be appreciated.
Take a look into kitchen.sh (and pan.sh). At some point the script starts adding stuff to the classpath. You can add more folders to the classpath there.
You still need permissions to edit the kitchen.sh file, though. If you can't do that, I suggest creating a copy of kitchen.sh you can write, in a separate location, and change the $BASEDIR folder to the actual PDI installation, so that kitchen can be located elsewhere.
If you have permission you can put your jar in another directory and after you specify this directory in the launcher.properties which you find in data-integration\launcher.
For exemple: if you put your jar in this directory: /export/home.
In launcher.properties: you will add this path and precisely libraries=../test:../lib:../libswt:../export/home
I wanner to write a pig script which should load a jar file. The following is my code:
Register /aa/bb/cc/ex.jar
I run the pig by hui and the jar file exists in the hdfs. However, it always remind that the file doesn't exist.
I am not sure if I use correct method to register a jar file from hdfs. Could you please give me ideas?
Thanks in advance.
According to
http://pig.apache.org/docs/r0.12.0/basic.html#register, you have to specify a full location uri for the jar file. For example,
register hdfs://namenode:port/aa/bb/cc/ex.jar;