What is the difference between these two redirections?
[localhost ~]$ echo "something" > a_file.txt
[localhost ~]$ echo "something" >| a_file.txt
I can't seem to find any documentation about >| in the help.
>| overrides the noclobber option in the shell (set with $ set -o noclobber, indicates that files can not be written over).
Basically, with noclobber, you get an error if you try to overwrite an existing file using >:
$ ./program > existing_file.txt
bash: existing_file.txt: cannot overwrite existing file
$
Using >| will override that error and force the file to be written over:
$ ./program >| existing_file.txt
$
It's analogous to using the -f or --force option on many shell commands.
From the Bash Reference Manual Section "3.6.2 Redirecting Output":
If the redirection operator is >, and the noclobber option to the set builtin has been enabled, the redirection will fail if the file whose name results from the expansion of word exists and is a regular file. If the redirection operator is >|, or the redirection operator is > and the noclobber option is not enabled, the redirection is attempted even if the file named by word exists.
Searching for "bash noclobber" generally brings up articles that mention this somewhere. See this question on SuperUser, this section in O'Reilly's "Unix Power Tools", and this Wikipedia article on Clobbering for examples.
Related
Httpd processes use a non-default configuration file if they are run with the -f flag.
For example
/home/myuser/apache/httpd-2.4.8/bin/httpd -f /confFiles/apache/2.4.8/apache.conf -k start
will use this configuration file: /confFiles/apache/2.4.8/apache.conf
I need to get this location and would rather not have to check for possible -f flags used to start httpd.
The answer here says to run /path/to/httpd -V and concatenate
-D SERVER_CONFIG_FILE="conf/httpd.conf"
with
-D HTTPD_ROOT="/etc/httpd"
to get the final path to the config file.
However, this path will not be the correct one if the -f flag is used to start the httpd process.
Is there a command that can get the config file that is actually being used by the process?
The answer you refer to mentions the paths httpd was compiled with, but as you say those can be manually changed with parameters.
The simple way to check is the command line, if process is called "httpd" (standard name), a simple ps will reveal the config file being used:
ps auxw | grep httpd
Or querying the server if server has mod_info loaded, in command line or with your favourite browser:
curl "http://yourserver.example.com/server-info?server" | grep -i "config file"
Note: mod_info should not be publicaly available for everyone to see.
I just installed Home-brew and now I'm trying to insert the home-brew directory at the top of my path environment variable by typing in two commands inside my terminal. My questions are these:
What is a path environment variable?
Are the two codes provided me correct?
echo "export Path=/usr/local/bin:$PATH" >> ~/.bash_profile && source ~/.bash_profile
After this I am to type in brew doctor. Nothing is happening as far as I can see.
Can anyone offer me some advice or direction?
I installed brew in my new Mac M1 and ask me to put /opt/homebrew/bin in the path, so the right command for this case is:
echo "export PATH=/opt/homebrew/bin:$PATH" >> ~/.bash_profile && source ~/.bash_profile
TL;DR
echo "export PATH=/usr/local/bin:$PATH" >> ~/.bash_profile && source ~/.bash_profile
is what you want.
To answer your first question; in order to run (execute) a program (executable) the shell must know exactly where it is in your filesystem in order to run it. The PATH environment variable is a list of directories that the shell uses to search for executables. When you use a command that is not built into the shell you are using the shell will search through these directories in order and will execute the first matching executable it finds.
For example when you type: mv foo bar the shell is almost certainly actually using an executable located in the /bin directory. Thus fully the command is
/bin/mv foo bar
The PATH environment variable therefore saves you some extra typing. You can see what is in your PATH currently (as you can with all environment variables) by entering:
echo $<NAME OF VARIABLE>
So in this instance:
echo $PATH
As I mentioned earlier, ordering is important. Adding /usr/local/bin to the beginning of PATH means that the shell will search there first and so if you have an executable foo in that folder it will be used in preference to any other foo executables you may have in the folders in your path. This means that any executables you install with brew will be used in preference to the system defaults.
On to your second question. What the command you have provided is trying to do is add a line to your .bash_profile and then source it. The .bash_profile is a text file stored in your home directory that is sourced (read) every time bash (your shell) starts. The mistake in the line you've provided is that only the first letter of PATH is capitalised. To your shell Path and PATH are very different things.
To fix it you want:
echo "export PATH=/usr/local/bin:$PATH" >> ~/.bash_profile && source ~/.bash_profile
To explain
echo "export PATH=/usr/local/bin:$PATH"
simply prints or echoes what follows to stdout, which in the above instance is the terminal. (stdout, stderr and stdin are very important concepts on UNIX systems but rather off topic) Running this command produces the result:
export PATH=/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
on my system because using $PATH within double quotes means bash will substitute it with its value. >> is then used to redirect stdout to the end of the ~/.bash_profile file. ~ is shorthand for your home directory. (NB be very careful as > will redirect to the file and overwrite it rather than appending.)
&& means run the next command is the previous is successful and
source ~/.bash_profile
simply carries out the actions contained in that file.
As per the latest documentation, you need to do this:
echo 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"' >> /home/dhruv/.bashrc
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"
Now you should be able to run brew from anywhere.
When you type in a program somewhere and click enter, it checks certain locations to see if that program exists there.
Linux brew uses locations different from the normal linux programs, so we are adding these locations to the ~/.profile file which sets the paths.
Run this in your terminal, and it will place the correct code in the .profile file, automatically.
echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.profile
Don't use .bash_profile because when you use something different from bash, like zsh, it may not work. .profile is the correct location.
I'm new to Apache SVN and I need some help to use a pre-commit script to filter which files are being upload to my repository.
I was searching a lot and found this script on another question, but it didn't work for me.
#!/bin/bash
REPOS=$1
TXN=$2
AWK=/usr/bin/awk
SVNLOOK="/usr/bin/svnlook";
#Put all the restricted formats in variable FILTER
FILTER=".(sh|xls|xlsx|exe|xlsm|XLSM|vsd|VSD|bak|BAK|class|CLASS)$"
# Figure out what directories have changed using svnlook.
FILES=`${SVNLOOK} changed -t ${REPOS} ${TXN} | ${AWK} '{ print $2 }'` > /dev/null
for FILE in $FILES; do
#Get the base Filename to extract its extension
NAME=`basename "$FILE"`
#Get the extension of the current file
EXTENSION=`echo "$NAME" | cut -d'.' -f2-`
#Checks if it contains the restricted format
if [[ "$FILTER" == *"$EXTENSION"* ]]; then
echo "Your commit has been blocked because you are trying to commit a restricted file." 1>&2
echo "Please contact SVN Admin. -- Thank you" 1>&2
exit 1
fi
done
exit 0
If I try to use svnlook changed -t repodirectory it didn't work because had a missing subcommand.
I overwrote my pre-commit.tmpl but it didn't work, can someone help me?
First - seems you incorrectly use svnlook. It should has parameters:
svnlook changed ${REPOS} -t ${TXN}
-t means 'read from transaction' and TXN - transaction name itself.
Second - not sure if I understand correctly, but hook file should has name pre-commit not pre-commit.tmpl
Third - pre-commit should has correct rights. For tests try a+rwx
update. It is not easy to obtain transaction object for tests, but you can use svnlook -r <revision> <repositiry_path> and experiment on already commited revisions.
When I run the following command in csh, I got nothing, but it works in bash.
Is there any equivalent in csh which can redirect the standard error to standard out?
somecommand 2>&1
The csh shell has never been known for its extensive ability to manipulate file handles in the redirection process.
You can redirect both standard output and error to a file with:
xxx >& filename
but that's not quite what you were after, redirecting standard error to the current standard output.
However, if your underlying operating system exposes the standard output of a process in the file system (as Linux does with /dev/stdout), you can use that method as follows:
xxx >& /dev/stdout
This will force both standard output and standard error to go to the same place as the current standard output, effectively what you have with the bash redirection, 2>&1.
Just keep in mind this isn't a csh feature. If you run on an operating system that doesn't expose standard output as a file, you can't use this method.
However, there is another method. You can combine the two streams into one if you send it to a pipeline with |&, then all you need to do is find a pipeline component that writes its standard input to its standard output. In case you're unaware of such a thing, that's exactly what cat does if you don't give it any arguments. Hence, you can achieve your ends in this specific case with:
xxx |& cat
Of course, there's also nothing stopping you from running bash (assuming it's on the system somewhere) within a csh script to give you the added capabilities. Then you can use the rich redirections of that shell for the more complex cases where csh may struggle.
Let's explore this in more detail. First, create an executable echo_err that will write a string to stderr:
#include <stdio.h>
int main (int argc, char *argv[]) {
fprintf (stderr, "stderr (%s)\n", (argc > 1) ? argv[1] : "?");
return 0;
}
Then a control script test.csh which will show it in action:
#!/usr/bin/csh
ps -ef ; echo ; echo $$ ; echo
echo 'stdout (csh)'
./echo_err csh
bash -c "( echo 'stdout (bash)' ; ./echo_err bash ) 2>&1"
The echo of the PID and ps are simply so you can ensure it's csh running this script. When you run this script with:
./test.csh >test.out 2>test.err
(the initial redirection is set up by bash before csh starts running the script), and examine the out/err files, you see:
test.out:
UID PID PPID TTY STIME COMMAND
pax 5708 5364 cons0 11:31:14 /usr/bin/ps
pax 5364 7364 cons0 11:31:13 /usr/bin/tcsh
pax 7364 1 cons0 10:44:30 /usr/bin/bash
5364
stdout (csh)
stdout (bash)
stderr (bash)
test.err:
stderr (csh)
You can see there that the test.csh process is running in the C shell, and that calling bash from within there gives you the full bash power of redirection.
The 2>&1 in the bash command quite easily lets you redirect standard error to the current standard output (as desired) without prior knowledge of where standard output is currently going.
I object the above answer and provide my own. csh DOES have this capability and here is how it's done:
xxx |& some_exec # will pipe merged output to your some_exec
or
xxx |& cat > filename
or if you just want it to merge streams (to stdout) and not redirect to a file or some_exec:
xxx |& tee /dev/null
As paxdiablo said you can use >& to redirect both stdout and stderr. However if you want them separated you can use the following:
(command > stdoutfile) >& stderrfile
...as indicated the above will redirect stdout to stdoutfile and stderr to stderrfile.
xxx >& filename
Or do this to see everything on the screen and have it go to your file:
xxx | & tee ./logfile
What about just
xxx >& /dev/stdout
???
I think this is the correct answer for csh.
xxx >/dev/stderr
Note most csh are really tcsh in modern environments:
rmockler> ls -latr /usr/bin/csh
lrwxrwxrwx 1 root root 9 2011-05-03 13:40 /usr/bin/csh -> /bin/tcsh
using a backtick embedded statement to portray this as follows:
echo "`echo 'standard out1'` `echo 'error out1' >/dev/stderr` `echo 'standard out2'`" | tee -a /tmp/test.txt ; cat /tmp/test.txt
if this works for you please bump up to 1. The other suggestions don't work for my csh environment.
I want to download a lot of urls in a script but I do not want to save the ones that lead to HTTP errors.
As far as I can tell from the man pages, neither curl or wget provide such functionality.
Does anyone know about another downloader who does?
I think the -f option to curl does what you want:
-f, --fail
(HTTP) Fail silently (no output at all) on server errors. This is mostly done to better
enable scripts etc to better deal with failed attempts. In normal cases when an HTTP
server fails to deliver a document, it returns an HTML document stating so (which often
also describes why and more). This flag will prevent curl from outputting that and
return error 22. [...]
However, if the response was actually a 301 or 302 redirect, that still gets saved, even if its destination would result in an error:
$ curl -fO http://google.com/aoeu
$ cat aoeu
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
here.
</BODY></HTML>
To follow the redirect to its dead end, also give the -L option:
-L, --location
(HTTP/HTTPS) If the server reports that the requested page has moved to a different
location (indicated with a Location: header and a 3XX response code), this option will
make curl redo the request on the new place. [...]
One liner I just setup for this very purpose:
(works only with a single file, might be useful for others)
A=$$; ( wget -q "http://foo.com/pipo.txt" -O $A.d && mv $A.d pipo.txt ) || (rm $A.d; echo "Removing temp file")
This will attempt to download the file from the remote Host. If there is an Error, the file is not kept. In all other cases, it's kept and renamed.
Ancient thread.. landed here looking for a solution... ended up writing some shell code to do it.
if [ `curl -s -w "%{http_code}" --compress -o /tmp/something \
http://example.com/my/url/` = "200" ]; then
echo "yay"; cp /tmp/something /path/to/destination/filename
fi
This will download output to a tmp file, and create/overwrite output file only if status was a 200. My usecase is slightly different.. in my case the output takes > 10 seconds to generate... and I did not want the destination file to remain blank for that duration.
NOTE: I am aware that this is an older question, but I believe I have found a better solution for those using wget than any of the above answers provide.
wget -q $URL 2>/dev/null
Will save the target file to the local directory if and only if the HTTP status code is within the 200 range (Ok).
Additionally, if you wanted to do something like print out an error whenever the request was met with an error, you could check the wget exit code for non-zero values like so:
wget -q $URL 2>/dev/null
if [ $? != 0]; then
echo "There was an error!"
fi
I hope this is helpful to someone out there facing the same issues I was.
Update:
I just put this into a more script-able form for my own project, and thought I'd share:
function dl {
pushd . > /dev/null
cd $(dirname $1)
wget -q $BASE_URL/$1 2> /dev/null
if [ $? != 0 ]; then
echo ">> ERROR could not download file \"$1\"" 1>&2
exit 1
fi
popd > /dev/null
}
I have a workaround to propose, it does download the file but it also removes it if its size is 0 (which happens if a 404 occurs).
wget -O <filename> <url/to/file>
if [[ (du <filename> | cut -f 1) == 0 ]]; then
rm <filename>;
fi;
It works for zsh but you can adapt it for other shells.
But it only saves it in first place if you provide the -O option
As alternative you can create a temporal rotational file:
wget http://example.net/myfile.json -O myfile.json.tmp -t 3 -q && mv list.json.tmp list.json
The previous command will always download the file "myfile.json.tmp" however only when the wget exit status is equal to 0 the file is rotated as "myfile.json".
This solution will prevent to overwrite the final file when a network failure occurs.
The advantage of this method is that in case that something is wrong you can inspect the temporal file and see what error message is returned.
The "-t" parameter attempt to download the file several times in case of error.
The "-q" is the quiet mode and it's important to use with cron because cron will report any output of wget.
The "-O" is the output file path and name.
Remember that for Cron schedules it's very important to provide always the full path for all the files and in this case for the "wget" program it self as well.
You can download the file without saving using "-O -" option as
wget -O - http://jagor.srce.hr/
You can get mor information at http://www.gnu.org/software/wget/manual/wget.html#Advanced-Usage