After setting Hadoop Home path and Prefix path in .bashrc and /etc/profile also im getting the same error - Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path
If i run the script from crontab im facing this error from hive> prompt its working fine
plz help with the regarding how to solve this
Set $HADOOP_HOME in $HIVE_HOME/conf/hive-env.sh
try loading user bash profile in the script, as below,
. ~/.bash_profile
bash is included in user bash_profile and it will have user specific configurations as well.
see the similar question Hbase commands not working in script executed via crontab
Related
I am trying to use the SSHOperator to SSH into a remote machine and run an external application through the command line. I have setup the SSH connection via the admin page.
This section of code is used to define the commands and the SSH connection to the external machine.
sshHook = SSHHook(ssh_conn_id='remote_comp')
command_1 ="""
cd /files/232-065/Rans
bash run.sh
"""
Where 'run.sh' runs the shell script:
#!/bin/sh
starccm+ -batch run_export.java Rans_Model.sim
Which simply runs the commercial software starccm+ with some options I have specified.
This section defines the task:
inlet_profile = SSHOperator(
task_id='inlet_profile',
ssh_hook=sshHook,
command=command_1
)
I have confirmed the SSH connection works by giving a simple 'ls' command and checking the output.
The error that I get is:
bash run.sh, error: run.sh: line 2: starccm+: command not found
The command in 'run.sh' works when I am logged into the machine (it does not require a GUI). This makes me think that there is a problem with the SSH session and it is not the same as the one that Apache Airflow logs into, but I am not sure how to solve this problem.
Does anyone have any experience with this?
There is no issue with SSH connection (at least from the error message). However, the issue is with starccm+ installation path.
Please check the installation path of starccm+ .
Check if the installation path is part of $PATH env variable
$ echo $PATH
If not, then install it in the standard locations like /bin or /usr/bin etc (provided they are included in $PATH variable), or export the installed director into PATH variable like this,
$ export PATH=$PATH:/<absolute_path>
It is not ideal but if you struggle with setting the path variable you can run starccm stating the full path like:
/directory/where/star/is/installed/starccm+ -batch run_export.java Rans_Model.sim
When i run command hive it is only able to start from bin folder beacause metastore is created in bin only if i run it from home its not able to get start and shows error.
I have added these lines in my .bashrc file for hive
HIVE env variables
export HIVE_HOME=/opt/hadoop/hive/apache-hive-2.3.4-bin
export PATH=$HIVE_HOME/bin:$PATH
Can you try to setup path as mentioned below and retry,
user#ubuntu:~$ sudo gedit ~/.bashrc
Copy and paste the following lines at end of the file
# Set HIVE_HOME
export HIVE_HOME="/opt/hadoop/hive/apache-hive-2.3.4-bin"
PATH=$PATH:$HIVE_HOME/bin
export PATH
But here my suggestion is, instead of using hive command prompt try to use recommended way that is beeline client. If you have hiveserver2 configured you can connect using beeline client and query to hive.
I am installing Apache nutch 1.11 binary distribution in an AIX machine. After unzipping the zip file "apache-nutch-1.11-bin.zip " under /usr directory. I just tried to run the bin/nutch command as per the instruction at https://wiki.apache.org/nutch/NutchTutorial
I am getting the following exception : bin/nutch: not found
I am using root user and have also tried running the "chmod +x bin/nutch" for permissions.
Can anyone help me in understanding what I am missing here ? Thanks in advance.
Just to be sure, after unzipping the zip file, you did changed the directory to "apache-nutch-1.11" in your command prompt and then entered the bin/nutch command. right?
if you did please look for a "bin" subdirectory in "apache-nutch-1.11" directory. If there isn't any, I guess the zip file you downloaded has a problem. If there is, in your command prompt first change the directory to where it is in, and then run the command. I hope it helps.
I am setting up Apche Hadoop 2.6 for the Psuedo Distributed Operation by following the instructions provided in the link:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
I am facing an issue after I execute the command: $ bin/hdfs dfs -put etc/hadoop input
The error message is: put:'input': No such file or directory
How to resolve this?
Also, I have edited the hadoop-env.sh with the statement: export HADOOP_PREFIX=/usr/local/hadoop, but cannot understand that why shell prints out the warning: /usr/local/hadoop/etc/hadoop/hadop-env.sh: line 32: export:='/usr/local/hadoop': not a valid identifier
Thanks for the help.
I have fixed this problem.
I created the directory: $ bin/hdfs dfs -mkdir /user/root and the problem got solved, as I was logged in as the root in ubuntu. Earlier, I was giving wrong username, hence, facing the issue.
I just installed a single node Hadoop 2.2.0 cluster running on ubuntu.
I tried a couple of basic example calculations and it works fine.
I then tried to setup hive 0.12.0, that includes hcatalog.
I actually follow this tutorial.
And when I try to start hcatalog, I always get the following error :
bash $HIVE_HOME/hcatalog/sbin/hcat_server.sh start
dirname: missing operand
Try `dirname --help' for more information.
Started metastore server init, testing if initialized correctly...
/usr/local/hive/hcatalog/sbin/hcat_server.sh: line 91: /usr/local/hive-0.12.0/hcatalog/sbin/../var/log/hcat.out: No such file or directory
Metastore startup failed, see /usr/local/hive-0.12.0/hcatalog/sbin/../var/log/hcat.err
But there's no hcat.err file at all, I'm kind of blocked right now.
Any help would be much appreciated !
Thanks in advance,
Guillaume
I worked out that hcat was not executable in the hive installation I have downloaded.
S just sudo chmod A+X hcat and it works