I'm trying to sanitise my apache logs of sensitive data as it's passed around in query string parameters. I'm aware this is not good, it cannot be changed.
Following from this question: https://stackoverflow.com/a/9473943/1046387
This is in my apache.conf
CustomLog "|/bin/sed -u -E s/'api_key=[^& \t\n]*'/'api_key=\[FILTERED\]'/g >> /var/log/apache2/access.log" combined
I've been getting this error:
AH00106: piped log program '/bin/sed -u -E s/'api_key=[^& \\t\\n]*'/'api_key=\\[FILTERED\\]'/g >> /var/log/apache2/access.log' failed unexpectedly
/bin/sed: -e expression #1, char 14: unterminated `s' command
Despite being able to run the command on the box directly:
$ echo "api_key=343" | /bin/sed -u -E s/'api_key=[^& \t\n]*'/'api_key=\[FILTERED\]'/g
api_key=[FILTERED]
Seems like apache isn't handing over the command to sed properly so it's missing some of the arguments. Some problem with escape sequences or something?
Related
I am trying to open a random .sql file off the internet using the following command:
psql -h localhost -d database_name -U postgres < file_name.sql
But when I run this command I just get errors like the following:
invalid command 's
invalid command 's
invalid command 'll
invalid command 'Moving
invalid command 's
invalid command "frequently
It just continuously prints out these invalid command error messages. I thought it might be an encoding problem but I confirmed the file is UTF-8 encoded.
Any suggestions on how I can open this file
To expand and clarify on a_horse_with_no_name's comment - the psql command you are running should be run directly in your shell, not inside pgadmin4.
youruser#yourmachine:~$ psql -h localhost -d database_name -U postgres < file_name.sql
That command should load the contents of file_name.sql in to database_name. Once it's complete, you can use pgadmin4 as normal to interact with the database.
One possibility is that the file contains tabulator keys, which are expanded if you read redirect standard input to the SQL script.
Try using the -f option:
psql -h localhost -d database_name -U postgres -f file_name.sql
Apparently the .sql file was generated through a MySQL dump. I thought it would not matter whether I used PostgreSQL or MySQL but it did. Once I installed MySQL my problem got resolved and I now have a Database ready :)
Using BCP from mssql-tools on CentOS7 and trying to copy some TSV data into a local MSSQL DB, BCP fails to do the copy and throws error "BCP copy in failed". The command being run is:
TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
RECOMMEDED_IMPORT_MODE='-c' # makes a big difference, see https://stackoverflow.com/a/16310219/8236733
/opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
$TO_SERVER_ODBCDSN \
-U $USER -P $PASSWORD \
-d $DB \
$RECOMMEDED_IMPORT_MODE \
-t "\t" \
-e ${filename}.bcperror.log
Yet the error logs created by the command are empty. What does this mean / imply? Anyone have any further debugging tips for resolving the "copy in failed" error?
The errorlog created when you use the -e option is meant to capture errors regarding the data itself. So, the errorlog will contain errors when there is an overflow of data (too many bytes in a field destined for a column with too few).
Execution errors, or errors with the BCP application itself are not captured in the error file created by the -e option.
In an automated environment, if you want to capture or log such errors you will need to redirect the output of the BCP command to a file for viewing later or even loading into a log table in a SQL table.
I'm running a bcp code through command line which looks similar to the below
bcp tempdb..temptable out output.txt -S Servername -i, -U username –P pword –r \n -t
each time I do I get and error saying "Syntax Error in 'úP'
If I remove everything after the username i am able to get the code to work as I am prompted for the password however it gives the table in a format which is imposssible to use.
could anyone advise where the syntax error may be occuring?
From what I can tell, it appears you have a few issues.
-i is not a valid option
-t should specify a field delimeter
You haven't specified a mode (character or native) (-c or -n)
Assuming you are trying to create a csv, here's what you may be looking for:
bcp tempdb..temptable out output.txt -S servername -U username -P password -c -t , -r \n
You may also find this page helpful from the bcp section of the Sybase ASE Utility Guide It's from the ASE 15.5 docs, but the syntax is the same for most versions 12.0 and newer.
When I run the following command in csh, I got nothing, but it works in bash.
Is there any equivalent in csh which can redirect the standard error to standard out?
somecommand 2>&1
The csh shell has never been known for its extensive ability to manipulate file handles in the redirection process.
You can redirect both standard output and error to a file with:
xxx >& filename
but that's not quite what you were after, redirecting standard error to the current standard output.
However, if your underlying operating system exposes the standard output of a process in the file system (as Linux does with /dev/stdout), you can use that method as follows:
xxx >& /dev/stdout
This will force both standard output and standard error to go to the same place as the current standard output, effectively what you have with the bash redirection, 2>&1.
Just keep in mind this isn't a csh feature. If you run on an operating system that doesn't expose standard output as a file, you can't use this method.
However, there is another method. You can combine the two streams into one if you send it to a pipeline with |&, then all you need to do is find a pipeline component that writes its standard input to its standard output. In case you're unaware of such a thing, that's exactly what cat does if you don't give it any arguments. Hence, you can achieve your ends in this specific case with:
xxx |& cat
Of course, there's also nothing stopping you from running bash (assuming it's on the system somewhere) within a csh script to give you the added capabilities. Then you can use the rich redirections of that shell for the more complex cases where csh may struggle.
Let's explore this in more detail. First, create an executable echo_err that will write a string to stderr:
#include <stdio.h>
int main (int argc, char *argv[]) {
fprintf (stderr, "stderr (%s)\n", (argc > 1) ? argv[1] : "?");
return 0;
}
Then a control script test.csh which will show it in action:
#!/usr/bin/csh
ps -ef ; echo ; echo $$ ; echo
echo 'stdout (csh)'
./echo_err csh
bash -c "( echo 'stdout (bash)' ; ./echo_err bash ) 2>&1"
The echo of the PID and ps are simply so you can ensure it's csh running this script. When you run this script with:
./test.csh >test.out 2>test.err
(the initial redirection is set up by bash before csh starts running the script), and examine the out/err files, you see:
test.out:
UID PID PPID TTY STIME COMMAND
pax 5708 5364 cons0 11:31:14 /usr/bin/ps
pax 5364 7364 cons0 11:31:13 /usr/bin/tcsh
pax 7364 1 cons0 10:44:30 /usr/bin/bash
5364
stdout (csh)
stdout (bash)
stderr (bash)
test.err:
stderr (csh)
You can see there that the test.csh process is running in the C shell, and that calling bash from within there gives you the full bash power of redirection.
The 2>&1 in the bash command quite easily lets you redirect standard error to the current standard output (as desired) without prior knowledge of where standard output is currently going.
I object the above answer and provide my own. csh DOES have this capability and here is how it's done:
xxx |& some_exec # will pipe merged output to your some_exec
or
xxx |& cat > filename
or if you just want it to merge streams (to stdout) and not redirect to a file or some_exec:
xxx |& tee /dev/null
As paxdiablo said you can use >& to redirect both stdout and stderr. However if you want them separated you can use the following:
(command > stdoutfile) >& stderrfile
...as indicated the above will redirect stdout to stdoutfile and stderr to stderrfile.
xxx >& filename
Or do this to see everything on the screen and have it go to your file:
xxx | & tee ./logfile
What about just
xxx >& /dev/stdout
???
I think this is the correct answer for csh.
xxx >/dev/stderr
Note most csh are really tcsh in modern environments:
rmockler> ls -latr /usr/bin/csh
lrwxrwxrwx 1 root root 9 2011-05-03 13:40 /usr/bin/csh -> /bin/tcsh
using a backtick embedded statement to portray this as follows:
echo "`echo 'standard out1'` `echo 'error out1' >/dev/stderr` `echo 'standard out2'`" | tee -a /tmp/test.txt ; cat /tmp/test.txt
if this works for you please bump up to 1. The other suggestions don't work for my csh environment.
I'm trying to write several commands trought ssh connection bue I got problem with escape characters. Below an example of what I'd like to do:
/usr/bin/ssh mrtg#172.20.29.40 echo -e "ciao\nprova"
I got this result:
ciaonprova
instead of:
ciao
prova
if I use -e option for ssh:
/usr/bin/ssh -e mrtg#172.20.29.40 echo -e 'ciao\nprova'
I receive this error:
Bad escape character 'mrtg#172.20.29.40'.
Can someone give me a suggestion to let remote server interpret escape characters?
The -e option has nothing to do with your command (these are SSH escape characters, not shell).
You can just put your command in quotes:
/usr/bin/ssh mrtg#172.20.29.40 'echo -e "ciao\nprova"'