As Cloudera closes all public repositories for ambari (see https://community.cloudera.com/t5/Support-Questions/Hortonworks-Repositories-can-t-be-accessed/td-p/311058) I tried to build ambari from scratch by my self (see Ambari 2.7.5 Installation Guide
I struggled with the problem described in Install Ambari, can't download hortonworks HDP from amazon S3
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project ambari-metrics-timelineservice: An Ant BuildException has occured: Can't get https://s3.amazonaws.com/dev.hortonwo
[ERROR] around Ant part ...<get usetimestamp="true" src="https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.1-1/tars/hbase/hbase-2.0.2.3.1.4.1-1-bin.tar.gz" dest="/home/...
But the suggested solution does work for me. It turns out that the repositories https://s3.amazonaws.com/dev.hortonworks.com/HDP seem to be closed too.
Is there any chance to build ambari at the moment?
Are there different public repositories to download the binaries but the closed repositories of hortonworks/cloudera?
These configurations solved this problem to me:
vi ambari-metrics/ambari-metrics-timelineservice/pom.xml
Change:
<hadoop.version>3.1.1.3.1.4.0-315</hadoop.version>
<phoenix.version>5.0.0.3.1.4.0-315</phoenix.version>
<hbase.version>2.0.2.3.1.4.0-315</hbase.version>
To:
<hadoop.version>3.1.1</hadoop.version>
<phoenix.version>5.0.0-HBase-2.0</phoenix.version>
<hbase.version>2.0.2</hbase.version>
vi ambari-metrics/pom.xml
Change:
<hbase.tar>https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz</hbase.tar>
<hbase.folder>hbase-2.0.2.3.1.4.0-315</hbase.folder>
<hadoop.tar>https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz</hadoop.tar>
<hadoop.folder>hadoop-3.1.1.3.1.4.0-315</hadoop.folder>
To:
<hbase.tar>https://archive.apache.org/dist/hbase/2.0.2/hbase-2.0.2-bin.tar.gz</hbase.tar>
<hbase.folder>hbase-2.0.2</hbase.folder>
<hadoop.tar>https://archive.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz</hadoop.tar>
<hadoop.folder>hadoop-3.1.1</hadoop.folder>
On same file, change:
<phoenix.tar>https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/phoenix/phoenix-5.0.0.3.1.4.0-315.tar.gz</phoenix.tar>
<phoenix.folder>phoenix-5.0.0.3.1.4.0-315</phoenix.folder>
To:
<phoenix.tar>https://downloads.apache.org/phoenix/apache-phoenix-5.0.0-HBase-2.0/bin/apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz</phoenix.tar>
<phoenix.folder>apache-phoenix-5.0.0-HBase-2.0-bin</phoenix.folder>
Related
When I running command below,
s3-dist-cp --src s3://test/9.19 --dest hdfs:///user/hadoop/test
I got a error about auxService.
20/02/03 07:52:13 INFO mapreduce.Job: Task Id : attempt_1580716305878_0001_m_000000_2, Status : FAILED
Container launch failed for container_1580716305878_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
In many QnA, I found a solution like this
link.
But there is no process for nodemanager.
[hadoop#ip-172-31-37-115 ~]$ initctl list | grep yarn
hadoop-yarn-timelineserver start/running, process 8149
hadoop-yarn-resourcemanager start/running, process 17331
hadoop-yarn-proxyserver start/running, process 8147
My EMR was created by quick menu with emr-5.28.0.
Is there anyone knows about this problem?
Thanks!
I'm sure there's some way to update the configs, but what I did was create a cluster using the 'advanced' setup and chose these software packages:
Ganglia
Hive
Hue
Mahout
Pig
Tez
Spark
Hadoop
(8 in total)
Most of those, except spark, are installed with the default settings (the first radio button for software packages in quick setup). One of these software packages or something related to it is what causes s3-dist-cp to be installed, and I was able to use it with no problems with that setup.
I did all the setup for oozie 4.3.0 on Apache hadoop single node cluster, when tried running any standard example workflow.xml that comes with oozie, it is throwing below error.
WARN ActionStartXCommand:523 - SERVER[data01.teg.io] USER[hadoop] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-161215143751620-oozie-hado-W] ACTION[0000000-161215143751620-oozie-hado-W#mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
I looked at the parameter "mapreduce.framework.name" and it is set to yarn everywhere in all config files. I checked Sharelib is created properly and can see when queried with shareliblist command, i dont see where exactly the problem is. Tried every solution came up in google and could not solve it even after struggling for 2 days with it.
I can start and stop oozie daemon with out any problem.
Any insights are greatly helpful.
I figured out the solution. Unlike the prior versions of Oozie before 4.x.x, 4.3.0 does not generate hadoop-libs.jar file when we run the build command.
In the beginning, i copied jar files only from my hadoop's
/srv/hadoop-2.7.3/share/hadoop/common to oozie's libext folder. After i copied jar files from all the below paths to oozie's libext folder, i was able to successfully setup the Oozie.
/srv/hadoop-2.7.3/share/hadoop/common/*.jar
/srv/hadoop-2.7.3/share/hadoop/common/lib/*.jar
/srv/hadoop-2.7.3/share/hadoop/hdfs/*.jar
/srv/hadoop-2.7.3/share/hadoop/hdfs/lib/*.jar
/srv/hadoop-2.7.3/share/hadoop/mapreduce/*.jar
/srv/hadoop-2.7.3/share/hadoop/mapreduce/lib/*.jar
/srv/hadoop-2.7.3/share/hadoop/yarn/*.jar
/srv/hadoop-2.7.3/share/hadoop/yarn/lib/*.jar
Relatively new to Apache OOZIE and did an installation on Ubuntu 14.04, Hadoop 2.6.0, JDK 1.8. I was able to install oozie and the web console is visible at the 11000 port of my server.
Now while i copied the examples bundled with oozie and tried to run them i am running into an error which says no sharedlib exists.
Installed the sharedlib as below-
bin/oozie-setup.sh sharelib create -fs hdfs://localhost:54310
(my namenode is running on localhost 54310 and JT on localhost 54311)
hadoop fs -ls /user/hduser/share/lib is showing shared library created as per the oozie-site.xml file. However when i check the shared library using the command -
oozie admin -oozie http://localhost:11000/oozie -shareliblist the list is blank and also jobs are failing for the same reason.
Any clues on how should i approach this problem?
Thanks.
The sharelib create command looks fine.
If you havent done so already copy the core-site.xml from your hadoop installation folder into $OOZIE_HOME/conf/hadoop-conf/.
There might already be a "placeholder" core-site.xml in the hadoop-conf folder, delete or rename that one. Oozie doesnt get its hadoop configuration directly from your hadoop install (like hive for example) but from the core-site.xml you place in that hadoop-conf folder.
Okay i got a solution for this.
So when i was trying to create the sharedlib directory it was doing on HDFS but while running the job local path was being refereed. So i extracted the oozie-sharedlib tar.gz file in my local /user/hduser/share/lib directory and its working now.
But did not get the reason so its still an open question.
I have encountered the same issue and it turned out that
oozie was not able to communicate with hdfs, as it was not able to find the location for core-site.xml or any other hadoop configuration which has to be declared inside oozie-site.xml.
Corresponding property in oozie-site.xml is oozie.service.HadoopAccessorService.hadoop.configurations
this property was defined wrongly in my case.
changed it to point to where my Hadoop configuration xmls are present and then it started communicating with hdfs and hence was able to locate the sharelib on hdfs
When running the top level maven target
test
I get the following error:
FATAL: command execution failed
java.io.IOException: Cannot run program "mvn" (in directory "/var/lib/jenkins /jobs/selenium/workspace"): java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at hudson.Proc$LocalProc.<init>(Proc.java:244)
at hudson.Proc$LocalProc.<init>(Proc.java:216)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:709)
at hudson.Launcher$ProcStarter.start(Launcher.java:338)
at hudson.Launcher$ProcStarter.join(Launcher.java:345)
at hudson.tasks.Maven.perform(Maven.java:263)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1502)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
... 15 more
Build step 'Invoke top-level Maven targets' marked build as failure
This seems to be an issue concerning the maven pathway, but I've setup the maven pathway on my host machine. M2_HOME, M2, and PATH are all correct. I know they are correct because I can run the maven commands from the command line. When I try to invoke maven commands in jenkins though I get the error.
So I went into Jenkins->Manage Jenkins->Configure System and I clicked on Maven installations...
I checked off on
Install automatically
Version 2.2.1
I clicked save and tried to run my project again with the same error. When I do mvn -version I get 2.2.1 so that should be right.
From the Configure System page I have also tried
Name default
MAVEN_HOME /usr/local/apache-maven/apache-maven-2.2.1
Any ideas?
The solution to my question has two parts. First I needed to make sure that after creating the maven Installation setup on the Configure System page, that I specified that same configuration in the build itself. Second Jenkins does not seem to have sufficient privileges on the redhat box I'm running it on. Once I finally got it pointed to the right maven instance I got a lot of unable to create file/folder errors. These permission errors could be the real reason I had so much trouble with maven on this machine. I have not solved these permission errors and will create a new question for them.
I have a Maven project which generates a 413.06 KB jar file. I have to deploy it on Apache Archiva based managed repository. I have tried to deploy different versions, and it created required layout and structure, uploaded some files, even it uploaded that jar with 200~ KB. every time the jar file size changes but always it fails to upload 413.06 KB jar file.
Information:-
I am running standalone Archiva
I have given guest account to Global Repository Manager & "Repository Manager - MYREPO"
I have also tried a separate account in Archiva with "Repository Manager - MYREPO" rights and configured it in maven's settings.xml file to set custom timeout.
I am getting following error
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy
(default-deploy) on project SharedshelfRepository: Error deploying artifact: Transfer error:
The server did not respond within the configured timeout. -> [Help 1]
that might be maven-deploy-plugin issue, resources plugin itself needs several dependencies,try manually jar nad p
What version of Maven are you using? You might try 3.0.4 as it has a different HTTP library. I'm also not sure if there's more context for what was happening when it timed out (it seems more request oriented rather than deploy oriented, and deploy does request some metadata).
I can't see that you'd need to alter the timeout, as none of the defaults should apply to such a small file. How long does it take to fail?