Do we have anything to prevent the map-reduce being shown when we use DUMP command in PIG. Just wanted to see the output. Anything that could be run in silent mode?
Make changes in the /conf/pig.properties
$PIG_HOME/conf/pig.properties :
enable:
# log4jconf=./conf/log4j.properties
rename: log4j.properties.template -> log4j.properties
log4j.properties :
set info to error:
log4j.logger.org.apache.pig=info, A
May not work with pig version 0.12+.
Related
This is a follow up question of:
Clean error output in ansible-playbook
The above post proposed a way to change the output format of ansible-playbook once it encounter an error. But I found that it is largely ineffective for ansible-molecule (https://molecule.readthedocs.io/en/latest/index.html). When I run the test suite:
$ molecule converge
It still generates the error message with lots of escaped characters:
...
TASK [solo : copy wifi.sh] *****************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: #bash /tmp/setupwifi.sh > /log/setupwifi.log 2>&1
fatal: [instance]: FAILED! => {"changed": false, "msg": "AnsibleError: template error while templating string: unexpected char '#' at 1526. String: \ncat > /tmp/timeout.sh << 'SCRIPT'\n#!/bin/sh\n\n# Execute a command with a timeout\n\n# License: LGPLv2\n# Author:\n# http://www.pixelbeat.org/\n# Notes:\n# Note there is a timeout command packaged with coreutils since v7.0\n# ...
It may be possible that ansible-molecule fail to identify ansible.cfg file under the directory. How should I configure it properly in my testing?
Quick and dirty background info (dig into the doc to find more info if you want extra details): molecule does not read your ansible.cfg at role level but will reconstruct its own config file in molecule/<scenario_dir>/.molecule/ansible.cfg.
You cannot change everything in that file (and certainly not edit it yourself manually), but there are quite a few config options you can set in the provisionner.config_options section of your molecule.yml file. I did not check specifically but I believe that changing the stdout_callback and stderr_callback should be possible. This is described in the configuration page in documentation. Give it a try:
provisionner:
name: ansible
config_options:
defaults:
stdout_callback: debug
stderr_callback: debug
having a small problem where you can help me out. On our new cluster we use LMod as environmental module system.
Creating a Module TCL Script for OpenFOAM, a system-dependent bashrc file need to be loaded.
This is the TCL script which I am using on another module system, it works fine. I am not able to execute the "source" command line in Lmod, what I am missing here?
#%Module1.0#####################################################################
##
## modules software/openfoam_v1812
##
## /opt/software/openfoam/openfoamv1812/OpenFOAM-v1812
proc ModulesHelp { } {
global version modroot
puts stderr "software/OpenFOAM-v1812 - sets the Environment for OpenFOAM-v1812 (openfoam.com)"
}
module-whatis "Sets the environment for using OpenFOAM-v1812"
# for Tcl script use only
set VERSION v1812
set OpenFOAM_PATH /opt/software/openfoam/openfoam${VERSION}/OpenFOAM-${VERSION}
set FOAM_INST_DIR /opt/software/openfoam/openfoam${VERSION}
puts stdout "source /opt/software/openfoam/openfoam${VERSION}/OpenFOAM-${VERSION}/etc/bashrc;"
I am not an expert, but I have recently come across a similar problem, in my case for activating Anaconda Python in a model. In my case, the solution was to use the 'execute' command in LMod
https://lmod.readthedocs.io/en/latest/050_lua_modulefiles.html
which has the documentation:
execute {cmd=”<any command>”,modeA={“load”}}
Run any command with a certain mode. For example execute {cmd=”ulimit
-s unlimited”,modeA={“load”}} will run the command ulimit -s unlimited as the last thing that the loading the module will do.
Hope this helps
I have the problem that I want to enable logging of a screen session at the start of it which then saves the log to a specific file.
What I have until now was:
screen -AmdSL cod2war /home/cod2server/scripts/service_28969.sh
while service_28969.sh is a shell script that will call other scripts which produce output.
I started multiple of those screen-sessions with different names, for example
screen -AmdSL cod2sd /home/cod2server/scripts/service_28962.sh
-L enables logging as the screen's man say, and will safe the ouput in a file called 'screenlog.0', now since I have multiple of those screens only one of it produces output saved in that log file (I can't find other 'screenlog.*' files in that folder).
I thought to use the -Logfile "file" option from the same man page, but it doesn't work for me and I can't find out what I'm doing wrong..
screen -Logfile cod2sd.log -AmdS cod2sd /home/u268450/cod2server/scripts/service_28962.sh
will produce the following error:
Use: screen [-opts] [cmd [args]]
or: screen -r [host.tty]
Options:
[...]
Error: Unknown option Logfile
and
screen -AmdS cod2sd /home/u268450/cod2server/scripts/service_28962.sh -Logfile cod2sd.log
will run without any error and start the screen but without the logging at all..
You can specify a logfile from within the default startup ~/.screenrc file using a line like
logfile mylog.log
To do this from the command line you can create a file mystartup to hold the above line, then use option -c mystartup to tell screen to read this file for setup instead of the default. If you also need to have ~/.screenrc read, you can add the source command to your startup file. The final result would look something like:
echo 'logfile mylog.log
source ~/.screenrc' >mystartup
screen -AmdSL cod2war -c mystartup /home/cod2server/scripts/service_28969.sh
This works for me:
screen -L -Logfile /Logs/Screen/`date +%Y%m%d`_screen.log
The configs I checked:
screen version 4.08.00 (GNU) 05-Feb-20 on FreeBSD 12.2
and
version 4.06.02 (GNU) 23-Oct-17 on Debian GNU/Linux 10 (buster)
and
version 4.00.03 (FAU) 23-Oct-06 on Mac OS X 10.9.5.
I just ran into this error myself and found this solution that worked with my python file, wanted to share for anyone else who might run into this issue:
screen -L -Logfile LOGFILENAME.LOG -dmS SCREENNAME python3 ./FILENAME.PY
I have no idea if this is the 'correct' way but it works.
-L enables logging
-Logfile LOGFILENAME.LOG declares what to call the log file and file format
-dmS SCREENNAME, dm runs in detached mode and S allows you to name the session
python3 ./FILENAME.PY in this case is my script but I assume that any other script here functions
I have tried a different ordering of these commands and this was the only way I managed to have them all run without issues. Hopes this helps.
I have been trying to omit logs from console while querying in hive, but still it is showing up.
If you are opening the hive console by typing
> hive
in your terminal and then write queries, you can solve this by simply using
> hive -S
This basically means that you are starting hive in silent mode.
Hope that helps.
You could increase the polling interval to minutes or hours:
SET hive.exec.counters.pull.interval=[millis];
The default is 1000 milliseconds, but you can increase it to anything you like. That should decrease the number of logs written to stdout.
If you don't want any logs on the console while starting the shell you can set the hive.root.logger property
$HIVE_HOME/bin/hive --config hive.root.logger=INFO,DRFA
hive.root.logger specifies the logging level as well as the log
destination. Specifying console as the target sends the logs to the
standard error (instead of the log file).
If you want to see ERROR messages on console you can set this command
$HIVE_HOME/bin/hive --config hive.root.logger=ERROR,console
Start hive in silent mode using
$ hive -S
then Set logger level to Error, which will avoid Warnings/Info from printing.
hive> set logger.PerfLogger.level = ERROR;
If there is "SLF4J: Class path contains multiple SLF4J bindings." in your log, it means that there are multiple log4j jars (different versions, different behaviors) in the class path
I don't know the principle of log4j, but according to the Hadoop configuration file, perform the following steps:
cd $HIVE_HOME/conf
cat > log4j.properties <<EOL
log4j.rootLogger=WARN, CA
log4j.appender.CA=org.apache.log4j.ConsoleAppender
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
EOL
After starting hive (Hive 3.1.2 Apache), the log is set to WARN level, which may not necessarily work, but you can try it.
I am trying to run my .ktr file in cmd line. I have my data-integration setup in this path:
C:\Users\dhamodharan.a\Desktop\pdi-ce-4.4.0-stable\data-integration
and my .ktr file in this path:
C:\Users\dhamodharan.a\Desktop\test.krt
while am trying to run that in cmd line I am getting the following error
DEBUG: Using PENTAHO_JAVA_HOME
DEBUG: _PENTAHO_JAVA_HOME=C:\Program Files (x86)\Java\jre7
DEBUG: _PENTAHO_JAVA=C:\Program Files (x86)\Java\jre7\bin\java.exe
WARN 11-08 11:47:09,728 - Unable to load Hadoop Configuration from "file:///C:/
Users/dhamodharan.a/Desktop/pdi-ce-4.4.0-stable/data-integration/plugins/pentaho
-big-data-plugin/hadoop-configurations/mapr". For more information enable debug
logging.
INFO 11-08 11:47:09,759 - Pan - Start of run.
ERROR: No repository provided, can't load transformation.
C:\Users\dhamodharan.a\Desktop\pdi-ce-4.4.0-stable\data-integration>e:C:\Users\d
hamodharan.a\Desktop.test.ktr /level:Basic
I am trying to run an input excel file and make the output as excel. Do I also need to create repository for that?
If I try to create repository option I saw only for dbms not for excel.
Make sure the environment variable PENTAHO_JAVA_HOME is set correctly and then it'll work.
For some reason the java install is not on your path. But if spoon works you must have it somewhere.
i did the environment variable for JAVA_HOME now pan.bat and kitchen.bat works fine.
here the command:
pan.bat /file:C:\Users\dhamodharan.a\Desktop\dhamu\test.ktr /level:Basic > C:\Users\dhamodharan.a\Desktop\dhamu\test.log
thanks