Looping over grep results with Zsh - ssh

I need to loop through a directory the first time I log into a new VM and set permissions on WSL shared folders. The SSH agent will not add private keys unless they have 0700 permissions set.
How do I use grep to get a list of file names and then loop through them?
I tried:
for file $(grep -lr "-----BEGIN OPENSSH PRIVATE KEY-----")
do
echo "Working on File: $file"
chmod 700 $file
done
The results were a mess. The grep command is being resolved into a single string of file names. Any mapped directories that have spaces in them (from the Win11 side of the house) messed things up further.

Suggesting to accomplish the task in one line:
chmod -v 700 $(grep -lr "-----BEGIN OPENSSH PRIVATE KEY-----" $PWD)
Explanation:
-v : Print verbose report on each changed file.
grep -lr "-----BEGIN OPENSSH PRIVATE KEY-----" $PWD : List all matched files to be changed, from all files under current directory.
Note: also try to replace $PWD with $HOME.

Here is my solution:
grep will output filenames with a null character after each item with the "-Z" flag.
Zsh can take that null delimiter and use it in an expanded string with the "0" flag.
The final result:
for file in ${(0o)"$(grep -lrZ "-----BEGIN OPENSSH PRIVATE KEY-----" *)"}
do
echo "Working on $file"
chmod 700 $file
done
A detailed explanation for posterity:
Zsh will execute anything inside () in a subshell. so the results of grep are in a nameless variable $().
Zsh does variable/parameter expansion with ${}. Anything inside gets expanded like hitting "Tab" at the command prompt.
We know that grep is going to return file paths with spaces in them. Therefore we can't use whitespace to expand our file list. We can use flags to make it happen with grep and Zsh using null characters as delimiters instead of spaces.
Zsh flags are specified immediately after the curly braces are opened up.
${(0o)$foo} will result in $foo being expanded into an array of items, splitting $foo on nulls because of the "0" and sorting them in ascending order because of the "o" flag.
We need the extra step of enclosing our grep results in double-quotes. ${(0o)"$foo"} We are after a string that contains both spaces and nulls. We are going to split the string on the nulls and preserve the spaces in the path names.
You could sort them in descending order with a capitalized "O" if you wanted to do it that way instead.
grep -lrZ will look for the string "-----BEGIN OPENSSH PRIVATE KEY-----" in all files found with the * wildcard.
The "r" is recursively look in sub-directories
The -l outputs just the filename
And -Z outputs a null separator at the end of each result.
So you end up with grep finding the files, Zsh expands the list and then loops through each of the items in the returned array.
As a side note in WSL, you have to turn on metadata in order to mount a windows folder and set Linux permissions. It is a nice solution to keep your ssh keys in a secure, backed-up location and use them on Linux images.

Iterate over all the files, but only operate on the ones that grep succeeds on. (You probably don't have that many files; don't worry so much over how many times grep has to run.)
for f in *; do
if grep -q "-----BEGIN OPENSSH PRIVATE KEY-----" "$f"; then
echo "Working on File: $f"
chmod 700 $f
fi
done
And because this is zsh, you can embed the call to grep in the glob:
is_key () {
grep -q "-----BEGIN OPENSSH PRIVATE KEY-----" "$REPLY"
}
for f in *(+is_key); do
echo "Working on File: $f"
chmod 700 "$f"
done
or even dispense with the explicit loop
chmod 700 *(+is_key)
(The shell function isn't actually necessary, but it does help with the readability of the glob qualifier.)

Related

Secure copying files from a remote server to local machine from a list in a text file

I have about a thousand files on a remote server (all in different directories). I would like to scp them to my local machine. I would not want to run scp command a thousand times in a row, so I have created a text file with a list of file locations on the remote server. It is a simple text file with a path on each line like below:
...
/iscsi/archive/aat/2005/20050801/A/RUN0010.FTS
/iscsi/archive/aat/2006/20060201/A/RUN0062.FTS
/iscsi/archive/aat/2013/20130923/B/RUN0010.FTS
/iscsi/archive/aat/2009/20090709/A/RUN1500.FTS
...
I have searched and found someone trying to do a similar but not the same thing here. The command I would like to edit is below:
cat /location/file.txt | xargs -i scp {} user#server:/location
In my case I need something like:
cat fileList.txt | xargs -i scp user#server:{} .
To download files from a remote server using the list in fileList.txt located in the same directory I run this command from.
When I run this I get an error: xargs: illegal option -- i
How can I get this command to work?
Thanks,
Aina.
You get this error xargs: illegal option -- i because -i was deprecated. Use -I {} instead (you could also use a different replace string but {} is fine).
If the list is remote, the files are remote, you can do this to retrieve it locally and use it with xargs -I {}:
ssh user#server cat fileList.txt | xargs -I {} scp user#server:{} .
But this creates N+1 connections, and more importantly this copies all remote files (scattered in different directories you said) to the same local directory. Probably not what you want.
So, in order to recreate a similar hierarchy locally, let's say everything under /iscsi/archive/aat, you can:
use cut -d/ to extract the part you want to be identical on both sides
use a subshell to create the command that creates the target directory and copies the file there
Thus:
ssh user#server cat fileList.txt \
| cut -d/ -f4- \
| xargs -I {} sh -c 'mkdir -p $(dirname {}); scp user#server:/iscsi/archive/{} ./{}'
Should work, but that's starting to look messy, and you still have N+1 connections, so now rsync looks like a better option. If you have passwordless ssh connection, this should work:
rsync -a --files-from=<(ssh user#server cat fileList.txt) user#server:/ .
The leading / is stripped by rsync and in the end you'll get everything under ./iscsi/archive/....
You can also copy the files locally first, and then:
rsync -a --files-from=localCopyOfFileList.txt user#server:/ .
You can also manipulate that file to remove for example 2 levels:
rsync -a --files-from=localCopyOfFileList2.txt user#server:/iscsi/archive .
etc.

Move file, change permissions and rename it keeping the same extesion

Using zsh 5.2 on Fedora 24 workstation.
I want to be programatically able to:
move an image file (can have jpg/ jpeg/ png/ JPG/ PNG extensions)
from /tmp/folder1 to ~/Pictures
This file will have the same few initial characters --- prefix111.jpg OR prefix222.png, etc.
rename the file such that samefilename.JPG becomes 20161013.jpg
20161013 is today's date in yyyymmdd format
Note that the extension becomes small letters
And JPEG or jpeg becomes jpg
change the permissions of the moved file to 644
All at one go.
If there are multiple prefix* files, the command should just fail silently.
I will initially like to do it at the command prompt with an option to add a cron job later. I mean, will the same zsh command/ script work in cron?
I am sure, this is doable. However, with my limited shell knowledge, could only achieve:
mv /tmp/folder1/prefix-*.JPG ~/Pictures/$(date +'%Y%m%d').jpg
Problems with my approach are many. It does not handle capitalization, does not take care of different extensions and does not address the permission issue.
How about this:
#!/bin/sh
FILES="/tmp/folder1/prefix*.jpg /tmp/folder1/prefix*.jpeg /tmp/folder1/prefix*.png h/tmp/folder1/prefix*.JPG /tmp/folder1/prefix*.PNG"
if [ $(ls $FILES | wc -l ) -gt 1 ]; then
exit 1
fi
if [ $(ls $FILES | grep -i '\.png$') ]; then
SUFF=png
else
SUFF=jpg
fi
DEST=$HOME/Pictures/$(date +'%Y%m%d').$SUFF
mv $FILES $DEST
chmod 644 $DEST

SSH - Grep with special chars and *

I'm looking to search some files via SSH with the grep command but I have some special chars.
The string I'm looking for is:
"$GLOBALS['....'];"
I tried this one
grep -r -H "\$GLOBALS\\['*'\\]\;" /var/www/
but nothing happens. Any help will be welcome.
Your RE actually matches "$GLOBALS['''''''];" with one or more ' there.
try this one:
grep -rHP "[$]GLOBALS\['.*?']\;" file
I use [$] instead of \$, is because ESCAPE IS SOMEHOW TRICKY, some environment you need use \\\$.
Update, less than 10 chars inside the []:
grep -rHP "[$]GLOBALS\['.{0,10}']\;" file

What, exactly, does ssh-copy-id do?

What does the ssh-copy-id command do, exactly? I've used it numerous times and it works great. However, when I try to manually cut and paste my .pub keyfile to my remote authorized_keys, it doesn't work.
I've compared the contents of my authorized_keys file where I've cut and pasted the .pub into it vs subsequently using ssh-copy-id and I'm not seeing any differences between the two, including whitespace.
Is there anything that ssh-copy-id does beyond copying the public key into authorized_keys?
This little one liner script works on sh, bash, and zsh. I use it every time there is no ssh-copy-id, for example when I'm on older version of OSX.
cat ~/.ssh/id_rsa.pub | ssh <user>#<hostname> 'cat >> ~/.ssh/authorized_keys'
How it works
I am sending the public keay to the Unix standard output (STDOUT) using the cat command. I then connect the STDOUT of cat to the standard input (STDIN) of the ssh.
The ssh executes the cat command on the server. Remember that the we have our key in the STDIN now? This key gets passed from ssh to the cat command executed on a server. The >> operator redirects the STDOUT of the cat to the end of the ~/.ssh/authorized_keys file. This way the key from public keys is appended to the authorized_keys on the server.
IMO It's better than manual copying and pasting: in this case you know exactly what content will end up in the file
I usually copy-paste keys into authorized_keys as you describe (I forget about ssh-copy-id), so it can work. Note thatchmod 600 ~/.ssh/authorized_keys is required if you're creating the file.
ssh-copy-id is a shell script so you can open it in a text editor to see what it does, this looks like the relevant bit:
printf '%s\n' "$NEW_IDS" | ssh "$#" "
umask 077 ;
mkdir -p .ssh && cat >> .ssh/authorized_keys || exit 1 ;
if type restorecon >/dev/null 2>&1 ; then restorecon -F .ssh .ssh/authorized_keys ; fi"
restorecon in the last line restores default SELinux security contexts. I haven't had to run that, but it might be necessary in your case.

How to escape $ in sed over ssh command?

I am trying to create a patch that users can use to remotely edit a file in a pre-defined way using sed, and I could do this manually on each computer, but it would take a long time.
The line I am struggling with is as follows:
host=[hostname]
port=[portnum]
ssh -t $host -p $port "cp ~/file1 ~/file1.bak ; sed -i \"s/fcn1('param1', $2)\n/fcn2('param2'):$zoom\n/g\" ~/file1"
This makes a backup of file1 and then edits a line in the file. I actually want to edit more than one line, but this line demonstrates the problems:
The command works, provided no $ signs are used within the sed command.
I have tried a number of ways of escaping these $ signs but cannot seem to find one that works.
I can use a . wildcard in the find, but obviously not in the replace string.
I would use single quotes for the sed command, in order to avoid expanding the $2, but single quotes are already used inside the command.
Does anyone have any ideas of how to overcome this problem? Thanks in advance for any suggestions!
This should work as well:
ssh -t $host -p $port "cp ~/file1 ~/file1.bak && sed -i \"s/fcn1('param1', \\\$2)/fcn2('param2'):\\\$zoom/g\" file1"
You need 3 back slashes as you have to escape the $ sign in the string passed in the remote bash to sed. And you have to escape that back slash and the $ sign when sending it over via ssh.