ipython works wrongs wrong with awk? - awk

Dear all! I find a question with ipython.
When I input
!ls -l | awk '{print $$1}'
It gives me:
drwxr-xr-x 2 ckivip ckivip 4096 Oct 11 20:38 Desktop
drwxr-xr-x 6 ckivip ckivip 4096 Oct 11 22:57 Documents
drwxrwxr-x 3 ckivip ckivip 4096 Oct 11 12:53 Downloads
drwxr-xr-x 6 ckivip ckivip 4096 Sep 29 18:22 Epigenetics
drwxr-xr-x 2 ckivip ckivip 4096 Sep 20 14:59 Music
drwxr-xr-x 23 ckivip ckivip 4096 Oct 10 11:02 Pictures
drwxr-xr-x 8 ckivip ckivip 4096 Sep 20 15:21 Project
drwx------ 5 ckivip ckivip 4096 Sep 25 21:31 R
drwxr-xr-x 5 ckivip ckivip 4096 Oct 9 19:23 Share
However, when I input
!ls -l | awk '{print $1}'
It gives me:
drwxr-xr-x
drwxr-xr-x
drwxrwxr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwx------
It's so annoying about the "$" symbol. And the most ugly thing is that I also can't transport the variables in python to shell using '$' when I use 'awk' function. How can I deal with it?

I'm not familiar with ipython but to address the part about passing the values of shell variables to awk: you do that with 'awk -v variable=value', so if you have a shell variable "$1" that contains the value "3" and you want awk to print the 3rd field of your input based on that, then the syntax is:
awk -v f="$1" '{ print $f }'
so in the above you could try:
!ls -l | awk -v f="$1" '{print $f}'
or if doubling the shell "$"s is required:
!ls -l | awk -v f="$$1" '{print $f}'
Hope that helps.

Related

what is the right way to back-up a substrate basepath?

can anyone point me at best-practice guidance for substrate basepath backup and restore? specifically, i'm wondering about these things:
it's my experience that backing up the basepath while the substrate service is running, results in a corrupt database that cannot be used in a restore. is that expected? it's not all that surprising if so.
syncing while the node service is running produces errors and warnings (in the sync process) like this:
warning: Skipping file /var/lib/substrate/polkadot/chains/ksmcc3/db/full/642785.sst. File does not exist.
and
upload failed: ../../var/lib/substrate/polkadot/chains/ksmcc3/db/full/642836.sst to s3://calamari-kusama-smoothie-calamari-systems/var/lib/substrate/polkadot/chains/ksmcc3/db/full/642836.sst [Errno 2] No such file or directory: '/var/lib/substrate/polkadot/chains/ksmcc3/db/full/642836.sst'
a basepath restored from a backup taken while the node service was running, always errors (for me) like so:
Error: Service(Client(Backend("Invalid argument: Column families not opened: col11, col10, col9, col8, col7, col6, col5, col4, col3, col2, col1, col0")))
or:
Error: Service(Client(Backend("IO error: No such file or directory: While opening a file for sequentially reading: /var/lib/substrate/chains/calamari/db/full/MANIFEST-009085: No such file or directory")))
is rsyncing the basepath the right way to go about doing backups? for ec2 nodes, i currently sync to an s3 bucket like so (the exclusions omit node key files from the backup):
para=calamari
relay=ksmcc3
bucket=${para}-${relay}-${HOSTNAME//./-}
basepath=/var/lib/substrate
sudo systemctl stop ${para}.service
/usr/bin/aws s3 sync ${basepath} s3://${bucket}${basepath} \
--exclude "chains/${para}/network/*" \
--exclude "polkadot/chains/${relay}/network/*"
which files in the basepath are useful for a restore and should be backed up? as well as the block numbered *.sst and *.log files, this is what i see (on a recently started, still syncing node):
$ ls -hl --ignore *.sst --ignore *.log /var/lib/substrate/{polkadot/chains/${relay}/db/{full,full/parachains/db},chains/${para}/db/full/}
/var/lib/substrate/chains/calamari/db/full/:
total 23M
-rw-r--r-- 1 root root 16 Nov 15 17:13 CURRENT
-rw-r--r-- 1 root root 37 Nov 15 16:50 IDENTITY
-rw-r--r-- 1 root root 0 Nov 15 16:50 LOCK
-rw-r--r-- 1 root root 21M Nov 16 06:33 LOG
-rw-r--r-- 1 root root 1.5M Nov 16 06:23 MANIFEST-002019
-rw-r--r-- 1 root root 48K Nov 15 17:13 OPTIONS-001991
-rw-r--r-- 1 root root 48K Nov 15 17:13 OPTIONS-002022
-rw-r--r-- 1 root root 1 Nov 15 17:13 db_version
/var/lib/substrate/polkadot/chains/ksmcc3/db/full:
total 1.4G
-rw-r--r-- 1 root root 16 Nov 15 17:13 CURRENT
-rw-r--r-- 1 root root 37 Nov 15 16:50 IDENTITY
-rw-r--r-- 1 root root 0 Nov 15 16:50 LOCK
-rw-r--r-- 1 root root 1.4G Nov 16 06:35 LOG
-rw-r--r-- 1 root root 74M Nov 16 06:35 MANIFEST-008095
-rw-r--r-- 1 root root 48K Nov 15 17:13 OPTIONS-007788
-rw-r--r-- 1 root root 48K Nov 15 17:13 OPTIONS-008098
-rw-r--r-- 1 root root 1 Nov 15 17:13 db_version
drwxr-xr-x 3 root root 4.0K Nov 15 16:50 parachains
drwxr-xr-x 2 root root 4.0K Nov 15 17:13 pvf-artifacts
/var/lib/substrate/polkadot/chains/ksmcc3/db/full/parachains/db:
total 2.1M
-rw-r--r-- 1 root root 16 Nov 15 17:13 CURRENT
-rw-r--r-- 1 root root 37 Nov 15 16:50 IDENTITY
-rw-r--r-- 1 root root 0 Nov 15 16:50 LOCK
-rw-r--r-- 1 root root 2.1M Nov 16 06:33 LOG
-rw-r--r-- 1 root root 437 Nov 15 17:13 MANIFEST-000033
-rw-r--r-- 1 root root 24K Nov 15 17:13 OPTIONS-000031
-rw-r--r-- 1 root root 24K Nov 15 17:13 OPTIONS-000036
-rw-r--r-- 1 root root 1 Nov 15 17:13 parachain_db_version
Here are so pro tips around doing any kind of DB snapshots:
Always stop the node before backing up! Failing to do this will usually result in a corrupted DB.
Always exclude {basepath}/keystore and {basepath}/network from your backups or only ever explicitly backup {basepath}/
The entire db folder should be backed up. This can be rsynced directly but it can also be tar'd or compressed then backed up.

Perl cgi script can not read a directory that has the same permission as its parent directory

I got a strange issue with our new Apache server.
When my cgi script tries to read a directory, the system gives me permission denied:
CGI Script:
#!/usr/bin/perl
print "Content-Type: Text/html\n\n";
# print `ls -altr / 2>&1`;
print `ls -altr /opt 2>&1`;
print `ls -altr /opt/NA 2>&1`;
this same code is working on our old apache server (v.2.2.15).
I can get output for “/”, and “/opt”, but when I try to list “/opt/NA”, I got Permission Denied.
And I can not see any difference between these 2 directories:
[root#myapache ~]# curl -k https://10.14.13.1/cgi-bin/listfile.cgi
total 12
drwxr-xr-x. 2 root root 6 Aug 25 2018 rh
drwxr-xr-x. 9 root root 110 Aug 18 15:17 . ##### this is the /opt
drwxr-xr-x. 5 root root 4096 Nov 18 14:22 chef
drwxr-xr-x. 5 root root 43 Nov 18 14:28 mcafee
drwxrwxr-x. 6 root root 4096 Nov 18 14:29 unified-monitoring-agent
drwxr-xr-x. 7 root root 80 Dec 16 14:59 McAfee
dr-xr-xr-x. 20 root root 4096 Dec 20 06:05 ..
drwxr-xr-x. 4 root root 32 Dec 20 06:05 isec
drwxr-xr-x. 4 root root 40 Jan 12 18:24 NA ##### this is the /opt/NA
ls: cannot open directory /opt/NA: Permission denied
Any idea how this has become a problem ?
Thanks for the advice from Dave Mitchell, I did that and update the Security Context bit:
[root#nccm-prod-dr-mt5 opt]# ls -Z
drwxr-xr-x. root root system_u:object_r:usr_t:s0 chef
drwxr-xr-x. root root system_u:object_r:usr_t:s0 isec
drwxr-xr-x. root root system_u:object_r:usr_t:s0 mcafee
drwxr-xr-x. root root system_u:object_r:usr_t:s0 McAfee
drwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 NA
drwxr-xr-x. root root system_u:object_r:usr_t:s0 rh
drwxrwxr-x. root root system_u:object_r:usr_t:s0 unified-monitoring-agent
[root#nccm-prod-dr-mt5 opt]# chcon system_u:object_r:usr_t:s0 NA
[root#nccm-prod-dr-mt5 opt]# ls -Z
drwxr-xr-x. root root system_u:object_r:usr_t:s0 chef
drwxr-xr-x. root root system_u:object_r:usr_t:s0 isec
drwxr-xr-x. root root system_u:object_r:usr_t:s0 mcafee
drwxr-xr-x. root root system_u:object_r:usr_t:s0 McAfee
drwxr-xr-x. root root system_u:object_r:usr_t:s0 NA
drwxr-xr-x. root root system_u:object_r:usr_t:s0 rh
drwxrwxr-x. root root system_u:object_r:usr_t:s0 unified-monitoring-agent
And it is working now.

ln not always working over SSH

I am making a deployment script using GitLab's CD. I've got a script:
- ssh USER#HOST "cd domains/$DOMAIN/ && mkdir build-$CI_JOB_ID"
- rsync -ar --port=22 * USER#HOST :domains/$DOMAIN/build-$CI_JOB_ID
- ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
- ssh USER#HOST "cd domains/$DOMAIN/ && ls | grep '^build\-.*$' | grep -Ev '^build-$CI_JOB_ID$' | xargs rm -rf"
everything works fine but not ln command. It works only 50% of the time. Here are logs from ` jobs runnning one after another.
Job 1 with $CI_JOB_ID = 76337215 worked properly. Link is correct.
$ ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
public_html -> build-76337215/public
total 20
drwx--x--x 5 USER 1000 7 Jun 20 22:15 .
drwx--x--x 23 USER 1000 23 Jun 19 16:34 ..
-rw-r--r-- 1 USER 1000 39 Jun 17 22:12 .htaccess
drwxr-xr-x 12 USER 1000 20 Jun 20 22:07 build-76335972
drwxr-xr-x 12 USER 1000 20 Jun 20 22:14 build-76337215
drwxr-xr-x 2 USER 1000 4 Jun 20 11:48 logs
lrwxr-xr-x 1 USER 1000 21 Jun 20 22:15 public_html -> build-76337215/public
Job 2 with $CI_JOB_ID = 76339729 did not work. Link is still to old 76337215 from Job 1.
$ ssh USER#HOST "cd domains/$DOMAIN/ && ln -sfv build-$CI_JOB_ID/public public_html && ls -la"
public_html/public -> build-76339729/public
total 20
drwx--x--x 5 USER 1000 7 Jun 20 22:28 .
drwx--x--x 23 USER 1000 23 Jun 19 16:34 ..
-rw-r--r-- 1 USER 1000 39 Jun 17 22:12 .htaccess
drwxr-xr-x 12 USER 1000 20 Jun 20 22:14 build-76337215
drwxr-xr-x 12 USER 1000 20 Jun 20 22:28 build-76339729
drwxr-xr-x 2 USER 1000 4 Jun 20 11:48 logs
lrwxr-xr-x 1 USER 1000 21 Jun 20 22:15 public_html -> build-76337215/public
What I am doing wrong? Why it is not working 100% of the time?
The problem is that if public_html already exists and is a directory (or a symlink to a directory), then your ln command creates a new link in that directory, rather than replacing public_html.
Use the -T option to avoid this:
ln -sfTv build-$CI_JOB_ID/public public_html
alternately, you can use the -n option to not dereference a link
ln -sfnv build-$CI_JOB_ID/public public_html
this will replace public_html if it is a symlink and create the symlink in the subdirectory if it is a real directory (-T would give an error in the latter case).

Proper permissions for website directories and to enable rsync deployment

I'm setting up a new website on a new Ubuntu droplet at Digital Ocean. I set this up previously on a different droplet and rsynced the website contents to it. I followed directions I found on the web, but didn't keep detailed-enough notes. I'm just trying to replicate this on the new Droplet. But I've messed up somehow. My knowledge of permissions, etc. is rudimentary :-(
My server has: /var/www/html
This is what I did:
• sudo usermod -a -G www-data [myusername]
• sudo chown -R www-data:www-data /var/www
• sudo chmod -R g+rw /var/www
• sudo chmod -R g+rws /var/www
I then created “/var/www/howardmann.us/public_html” for my website.
(I didn't put the public_html directory in var/www/html. I didn't on my other Droplet. Perhaps I should.)
sudo ls -la /var/www
total 16
drwxrwsr-x 4 www-data www-data 4096 Feb 25 15:43 .
drwxr-xr-x 14 root root 4096 Feb 24 14:27 ..
drwxr-sr-x 3 root www-data 4096 Feb 25 15:43 howardmann.us
drwxrwsr-x 2 www-data www-data 4096 Feb 25 15:42 html
sudo ls -la /var/www/howardmann.us/public_html
total 8
drwxr-sr-x 2 root www-data 4096 Feb 25 15:43 .
drwxr-sr-x 3 root www-data 4096 Feb 25 15:43 ..
Now, excerpts of rsync script (successful to my other Droplet) result from my laptop now:
1.
howardm$ ./deploy.sh
sending incremental file list
rsync: failed to set times on "/var/www/howardmann.us/public_html/.": Operation not permitted (1)
rsync: recv_generator: mkdir "/var/www/howardmann.us/public_html/css" failed: Permission denied (13)
* Skipping any contents from this failed directory *
3.
rsync: mkstemp "/var/www/howardmann.us/public_html/.DS_Store.8zaaQg" failed: Permission denied (13)
Question: How do I fix ownership and permissions on the server directories for 1) proper permissions for a website and 2) permit a rsync deploy?
For comparison, here is the status of my current working web site on the other Droplet:
howardm#howardmann:~$ ls -la /var/www/
total 20
drwxrwxr-x 5 howardm www-data 4096 Mar 5 2016 .
drwxr-xr-x 13 root root 4096 Mar 28 2015 ..
drwxrwxr-x 4 howardm www-data 4096 Jun 8 2016 howardmann.us
drwxrwxr-x 2 howardm www-data 4096 Mar 29 2015 html
Thanks!
Howard
Well, I fixed this after reading some more about permissions for a website.
My website folder is titled howardmann.us, which contains a public_html folder with the website contents.
So: /var/www/howardmann.us/public_html. (/var/www also contains a html directory)
I achieved the relevant ownership/permissions with:
sudo chown -R howardm:www-data /var/www/
I'm the owner, and www-data the group.
sudo usermod -a -G www-data [myusersername]
I'm a member of the www-data group
sudo chmod -R 2775 /var/www/
This yields the proper permissions, I believe for a web site.
I made sure the same owner:group applied to all these directories:
ls -l /var/www
total 8
drwxr-sr-x 3 howardm www-data 4096 Feb 26 19:23 howardmann.us
drwxrwsr-x 2 howardm www-data 4096 Feb 25 15:42 html
ls -l /var/www/howardmann.us
total 4
drwxr-xr-x 11 howardm www-data 4096 Feb 5 2017 public_html
Now rsync works without errors.
If anyone believes these ownership/permissions for a website are (substantially) improper, please comment.
Howard

maxminddb module for apache 2.4 on ubuntu error: The MaxMind DB file contains invalid metadata

here is my system information:
$ apachectl -v
Server version: Apache/2.4.18 (Ubuntu)
Server built: 2016-07-14T12:32:26
$ cat /etc/*release* | grep -i dist
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
$ uname -r
4.4.0-57-generic
i wish to install mod_maxminddb. i installed geoipupdate as a prerequisites and configured it to include not only geolite database, but either commercial one (using this)
$ ls -l /etc/GeoIP.conf
-rw-r--r-- 1 root root 818 Dec 24 18:29 /etc/GeoIP.conf
$ ls -l /usr/share/GeoIP/
total 0
$ geoipupdate
$ ls -l /usr/share/GeoIP/
total 187444
-rw-r--r-- 1 root root 112192399 Dec 24 18:46 GeoIP2-City.mmdb
-rw-r--r-- 1 root root 3012279 Dec 24 18:46 GeoIP2-Country.mmdb
-rw-r--r-- 1 root root 47721533 Dec 24 18:46 GeoIPCity.dat
-rw-r--r-- 1 root root 1699494 Dec 24 18:45 GeoIP.dat
-rw-r--r-- 1 root root 4189407 Dec 24 18:45 GeoIPISP.dat
-rw-r--r-- 1 root root 4299547 Dec 24 18:45 GeoLiteASNum.dat
-rw-r--r-- 1 root root 17760694 Dec 24 18:45 GeoLiteCity.dat
-rw-r--r-- 1 root root 1054583 Dec 24 18:45 GeoLiteCountry.dat
then complied and installed the mod_maxminddb, and when apache is configured to use the commercial databases, the following error is thrown
$ apachectl -M
AH00526: Syntax error on line 12 of /etc/apache2/mods-enabled/maxminddb.conf:
MaxMindDBFile: Failed to open /usr/share/GeoIP/GeoIPCity.dat: The MaxMind DB file contains invalid metadata
but that is not the case when apache is configured to use the geolite databases.
any ideas?
You didn't include your Apache config, but you appear to be trying to use mod_maxminddb with GeoIPCity.dat (GeoIP Legacy). It only works with GeoIP2. Adjust line 12 of maxminddb.conf to use GeoIP2-City.mmdb instead.