I am using Hadoop 3.2.0 and trying to run a simple application in a docker container and I have made the required configuration changes both in yarn-site.xml and container-executor.cfg to choose LinuxContainerExecutor and docker runtime.
I use the example of distributed shell in one of the hortonworks blog. https://hortonworks.com/blog/trying-containerized-applications-apache-hadoop-yarn-3-1/
The problem I face here is when the application is submitted to YARN it fails with a reason related to directory creation issue with the below error
2019-02-14 20:51:16,450 INFO distributedshell.Client: Got application
report from ASM for, appId=2, clientToAMToken=null,
appDiagnostics=Application application_1550156488785_0002 failed 2
times due to AM Container for appattempt_1550156488785_0002_000002
exited with exitCode: -1000 Failing this attempt.Diagnostics:
[2019-02-14 20:51:16.282]Application application_1550156488785_0002
initialization failed (exitCode=20) with output: main : command
provided 0 main : user is myuser main : requested yarn user is
myuser Failed to create directory
/data/yarn/local/nmPrivate/container_1550156488785_0002_02_000001.tokens/usercache/myuser
- Not a directory
I have configured yarn.nodemanager.local-dirs in yarn-site.xml and I can see the same reflected in YARN web ui localhost:8088/conf
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/yarn/local</value>
<final>false</final>
<source>yarn-site.xml</source>
</property>
I do not understand why is it trying to create usercache dir inside the nmPrivate directory.
Note : I have verified the permissions for myuser to the directories and also have tried clearing the directories manually as suggested in a related post. But no fruit. I do not see any additional information about container launch failure in any other logs.
How do I debug why the usercache dir is not resolved properly??
Really appreciate any help on this.
Realized that this is all because of the users the services were started with and the permissions to the directories the services work on.
After making sure the required changes are done, I am able to seamlessly run the examples and other applications..
Thanks Hadoop user community for the direction. Adding the link here for more details.
http://mail-archives.apache.org/mod_mbox/hadoop-user/201902.mbox/browser
Unable to execute sonar run for ms build when i change the sonar host from localhost to my machine ip.
tried changing the host in sonar.properties file and also in sonarqube_analysis.xml.
c:\HID\project-test\PACS\sonar_opencover\Project>MSBuild.SonarQube.Runner.exe end
Default properties file was found at C:\HID\project-test\dotnet\MSBuild.SonarQube.Runner-1.0.1\SonarQube.Analysis.xml
Loading analysis properties from C:\HID\project-test\dotnet\MSBuild.SonarQube.Runner-1.0.1\SonarQube.Analysis.xml
Post-processing started.
Execution failed. The specified executable does not exist: c:\HID\project-test\PACS\sonar_opencover\Project\.sonarqube\bin\MSBuild.SonarQube.Internal.PostProcess.exe
Post-processing failed. Exit code: 1
Scripts i executed
#call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\VsDevCmd.bat"
MSBuild.SonarQube.Runner.exe begin /k:"jenkins_test5" /n:"jenkins_test5" /v:"1.0" /d:sonar.cs.opencover.reportsPaths="%CD%\opencover.xml"
msbuild.exe
"C:\HID\project-test\PACS\sonar_opencover\opencover.4.6.210\OpenCover.Console.exe" -output:"%CD%\opencover.xml" -register:user -target:"vstest.console.exe" -targetargs:"UnitTestProject1\bin\Debug\UnitTestProject1.dll"
MSBuild.SonarQube.Runner.exe end
I ran into this issue today. Turns out that I had miss configured SonarQube.Analysis.xml. Essentially, it expects
<Property Name="sonar.host.url">http://{server}:9000/sonar</Property>
including /sonar (which comes from sonar.web.context setting of sonar.properties on the server).
I had missed out /sonar in SonarQube.Analysis.xml and had exact same symptom as yours.
I have this error because the sonarcube url is wrong, it can't connect to the conarcube host:
<Property Name="sonar.host.url">http://{server}:{Port}</Property>
After asking my colleague I found that the url has changed, I correct the url and it works succesfully.
Trying to set up Jenkins on one of my servers for the first time and think I might be missing something.
Jenkins 1.545
Phing 2.6.1
Jenkins builds give me the following output.
Building in workspace /var/www/vhosts/domain.co.uk/httpdocs
looking for '/var/www/vhosts/domain.co.uk/httpdocs/build.xml' ...
looking for '/var/www/vhosts/domain.co.uk/httpdocs/build.xml' ...
looking for 'build.xml' ...
buildfile 'build.xml' not found.
Build step 'Invoke Phing targets' marked build as failure
Finished: FAILURE
If I run my build.xml on it's own it works fine.
I'm using a custom workspace at the moment, before I tried a symlink from the default workspace to my webroot, when I did that it found the build file but failed when trying to run phing. I know it's a problem with permissions but I'm not sure exactly what.
I'm running this on a plesk web server and have tried adding the jenkins user to the psacln and psaserv groups but that didn't work either.
I use hudson but I think is the same problem.
Provide to ant job the full path (advanced settings)
${WORKSPACE}/buil.xml
Assuming the correct set of jenkins user
RUN_AS_USER=jenkins
Go to the custom workspace and
chown -R jenkins:jenkins myworkspace
if it doesn't work
chmod -R 777 myworkspace
then you will fix later.
I hope it helps.
When running the top level maven target
test
I get the following error:
FATAL: command execution failed
java.io.IOException: Cannot run program "mvn" (in directory "/var/lib/jenkins /jobs/selenium/workspace"): java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at hudson.Proc$LocalProc.<init>(Proc.java:244)
at hudson.Proc$LocalProc.<init>(Proc.java:216)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:709)
at hudson.Launcher$ProcStarter.start(Launcher.java:338)
at hudson.Launcher$ProcStarter.join(Launcher.java:345)
at hudson.tasks.Maven.perform(Maven.java:263)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1502)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
... 15 more
Build step 'Invoke top-level Maven targets' marked build as failure
This seems to be an issue concerning the maven pathway, but I've setup the maven pathway on my host machine. M2_HOME, M2, and PATH are all correct. I know they are correct because I can run the maven commands from the command line. When I try to invoke maven commands in jenkins though I get the error.
So I went into Jenkins->Manage Jenkins->Configure System and I clicked on Maven installations...
I checked off on
Install automatically
Version 2.2.1
I clicked save and tried to run my project again with the same error. When I do mvn -version I get 2.2.1 so that should be right.
From the Configure System page I have also tried
Name default
MAVEN_HOME /usr/local/apache-maven/apache-maven-2.2.1
Any ideas?
The solution to my question has two parts. First I needed to make sure that after creating the maven Installation setup on the Configure System page, that I specified that same configuration in the build itself. Second Jenkins does not seem to have sufficient privileges on the redhat box I'm running it on. Once I finally got it pointed to the right maven instance I got a lot of unable to create file/folder errors. These permission errors could be the real reason I had so much trouble with maven on this machine. I have not solved these permission errors and will create a new question for them.
I want to prevent maven from displaying INFO messages, I want to see only WARNINGS and ERRORS (if any).
How can I achieve this, preferably by changing the command line that calls maven?
Answering your question
I made a small investigation because I am also interested in the solution.
Maven command line verbosity options
According to http://books.sonatype.com/mvnref-book/reference/running-sect-options.html#running-sect-verbose-option
-e for error
-X for debug
-q for only error
Maven logging config file
Currently maven 3.1.x uses SLF4J to log to the System.out .
You can modify the logging settings at the file:
${MAVEN_HOME}/conf/logging/simplelogger.properties
According to the page : http://maven.apache.org/maven-logging.html
Command line setup
I think you should be able to setup the default Log level of the simple logger via a command line parameter, like this:
$ mvn clean package -Dorg.slf4j.simpleLogger.defaultLogLevel=debug
But I could not get it to work. I guess the only problem with this is, maven picks up the default level from the config file on the classpath. I also tried a couple of other settings via System.properties, but all of them were unsuccessful.
Appendix
You can find the source of slf4j on github here : slf4j github
The source of the simplelogger here : slf4j/jcl-over-slf4j/src/main/java/org/apache/commons/logging/impl/SimpleLog.java
The plexus loader loads the simplelogger.properties.
Edit: this answer was from 2013, there are probably better ways to do this now, please consider the other answers.
Linux:
mvn validate clean install | egrep -v "(^\[INFO\])"
or
mvn validate clean install | egrep -v "(^\[INFO\]|^\[DEBUG\])"
Windows:
mvn validate clean install | findstr /V /R "^\[INFO\] ^\[DEBUG\]"
You can achieve this with MAVEN_OPTS, for example
MAVEN_OPTS=-Dorg.slf4j.simpleLogger.defaultLogLevel=warn mvn clean
Rather than putting the system property directly on the command line. (At least for maven 3.3.1.)
Consider using ~/.mavenrc for setting MAVEN_OPTS if you would like logging changed for your login across all maven invocations.
If you are using Logback, just put this logback-test.xml file into src/test/resources directory:
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="STDOUT" />
</root>
</configuration>
you can achieve this by using below in the commandline itself
-e for error
-X for debug
-q for only error
e.g :
mvn test -X -DsomeProperties='SomeValue' [For Debug level Logs]
mvn test -e -DsomeProperties='SomeValue' [For Error level Logs]
mvn test -q -DsomeProperties='SomeValue' [For Only Error Logs]
The simplest way is to upgrade to Maven 3.3.1 or higher to take advantage of the ${maven.projectBasedir}/.mvn/jvm.config support.
Then you can use any options from Maven's SL4FJ's SimpleLogger support to configure all loggers or particular loggers. For example, here is a how to make all warning at warn level, except for a the PMD which is configured to log at error:
cat .mvn/jvm.config
-Dorg.slf4j.simpleLogger.defaultLogLevel=warn -Dorg.slf4j.simpleLogger.log.net.sourceforge.pmd=error
See here for more details on logging with Maven.
Unfortunately, even with maven 3 the only way to do that is to patch source code.
Here is short instruction how to do that.
Clone or fork Maven 3 repo: "git clone https://github.com/apache/maven-3.git"
Edit org.apache.maven.cli.MavenCli#logging, and change
cliRequest.request.setLoggingLevel( MavenExecutionRequest.LOGGING_LEVEL_INFO );
to
cliRequest.request.setLoggingLevel( MavenExecutionRequest.LOGGING_LEVEL_WARN );
In current snapshot version it's at line 270
Then just run "mvn install", your new maven distro will be located in "apache-maven\target\" folder
See this diff for the reference: https://github.com/ushkinaz/maven-3/commit/cc079aa75ca8c82658c7ff53f18c6caaa32d2131
Go to simplelogger.properties in ${MAVEN_HOME}/conf/logging/ and set the following properties:
org.slf4j.simpleLogger.defaultLogLevel=warn
org.slf4j.simpleLogger.log.Sisu=warn
org.slf4j.simpleLogger.warnLevelString=warn
And beware: warn, not warning
In Debian this worked for me like a charm to change the log level for Maven 3.6 at runtime without having to change the simplelogger.properties file:
MAVEN_OPTS='-Dorg.slf4j.simpleLogger.defaultLogLevel=error' mvn test
I have noticed when using the 2.20.1 version of the maven sunfire plugin, all warnings are written down to a dumpstream file. e.g. /myproject/target/surefire-reports/2017-11-11T23-02-19_850.dumpstream
Changing the info to error in simplelogging.properties file will help in achieving your requirement.
Just change the value of the below line
org.slf4j.simpleLogger.defaultLogLevel=info
to
org.slf4j.simpleLogger.defaultLogLevel=error