I just created a backup with
rsync -av --delete projects/ /Volumes/daten/projects/
Afterwards I ran
diff -r projects/ /Volumes/daten/projects/
just to check if everything's fine expecting no output from diff. However, diff found a lot of differences. Does that mean rsync did not correctly sync my data?
Update: When rerunning rsync it seems fine, there is nothing to do for rsync:
$ rsync -av --delete projects/ /Volumes/daten/projects/
building file list ... done
sent 470414 bytes received 20 bytes 188173.60 bytes/sec
total size is 295619054 speedup is 628.40
However, diff doesn't stop to generate output, as if there are lots of different files. Here is a small excerpt:
$ diff -r projects/ /Volumes/daten/projects/
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_dep/debug/package_dependencies/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_dep/debug/package_dependencies/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:41 CST 2018
---
> #Sun Nov 04 19:34:13 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_1/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_1/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:41 CST 2018
---
> #Sun Nov 04 19:34:14 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_2/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_2/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:41 CST 2018
---
> #Sun Nov 04 19:34:14 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_3/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_3/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:42 CST 2018
---
> #Sun Nov 04 19:34:14 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_4/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_4/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:42 CST 2018
---
> #Sun Nov 04 19:52:00 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_5/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_5/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:42 CST 2018
---
> #Sun Nov 04 19:34:14 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_6/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_6/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:42 CST 2018
---
> #Sun Nov 04 19:34:14 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_7/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_7/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:42 CST 2018
---
> #Sun Nov 04 19:34:14 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_8/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_8/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:42 CST 2018
---
> #Sun Nov 04 19:34:14 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_9/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/ir_slices/debug/package_slice_9/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:42 CST 2018
---
> #Sun Nov 04 19:34:14 CET 2018
diff -r projects/CatClicker/app/build/intermediates/incremental/packageInstantRunResourcesDebug/tmp/debug/dex-renamer-state.txt /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental/packageInstantRunResourcesDebug/tmp/debug/dex-renamer-state.txt
1c1
< #Sat Dec 08 16:17:34 CST 2018
---
> #Sun Nov 04 19:33:52 CET 2018
Binary files projects/CatClicker/app/build/intermediates/incremental-classes/debug/instant-run-bootstrap.jar and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-classes/debug/instant-run-bootstrap.jar differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/compat/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/compat/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/coreui/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/coreui/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/coreutils/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/coreutils/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/fragment/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/fragment/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/graphics/drawable/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/graphics/drawable/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/graphics/drawable/animated/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/graphics/drawable/animated/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/mediacompat/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/mediacompat/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/v4/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/v4/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/v7/appcompat/R$anim.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/v7/appcompat/R$anim.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/v7/appcompat/R$color.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/android/support/v7/appcompat/R$color.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/cyberdynesoftware/catclicker/R$attr.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/cyberdynesoftware/catclicker/R$attr.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/cyberdynesoftware/catclicker/R$bool.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/cyberdynesoftware/catclicker/R$bool.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/cyberdynesoftware/catclicker/R$dimen.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/cyberdynesoftware/catclicker/R$dimen.class differ
Binary files projects/CatClicker/app/build/intermediates/incremental-verifier/debug/cyberdynesoftware/catclicker/R$drawable.class and /Volumes/daten/projects/CatClicker/app/build/intermediates/incremental-verifier/debug/cyberdynesoftware/catclicker/R$drawable.class differ
It seems that a lot of files are not seen as "different" by rsync. By default rsync only checks the file sizes and the last modification timestamps of the files. Please try using rsync with the -c/--checksum option to turn on checksumming and run "diff" again.
Excerpt from the rsync man page:
-c, --checksum
This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option,
rsync uses a "quick check" that (by default) checks if each file's size and time of last modification match between the
sender and receiver. This option changes this to compare a 128-bit checksum for each file that has a matching size.
Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the
transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down
significantly.
Related
I want to get the latest values of each SIZE_TYPE day wise, ordered by TIMESTAMP. So, only 1 value of each SIZE_TYPE must be present for a given day, and that is the latest value for the day.
How do I get the desired output? I'm using PostgreSQL here.
Input
|TIMESTAMP |SIZE_TYPE|SIZE|
|----------------------------------------|---------|----|
|1595833641356 [Mon Jul 27 2020 07:07:21]|0 |541 |
|1595833641356 [Mon Jul 27 2020 07:07:21]|1 |743 |
|1595833641356 [Mon Jul 27 2020 07:07:21]|2 |912 |
|1595876841356 [Mon Jul 27 2020 19:07:21]|1 |714 |
|1595876841356 [Mon Jul 27 2020 19:07:21]|2 |987 |
|1595963241356 [Tue Jul 28 2020 19:07:21]|0 |498 |
|1595920041356 [Tue Jul 28 2020 07:07:21]|2 |974 |
|1595920041356 [Tue Jul 28 2020 07:07:21]|0 |512 |
*Note: the TIMESTAMP values are in UNIX time. I have given
the date-time string for reference*
Output
|TIMESTAMP |SIZE_TYPE|SIZE|
|----------------------------------------|---------|----|
|1595833641356 [Mon Jul 27 2020 07:07:21]|0 |541 |
|1595876841356 [Mon Jul 27 2020 19:07:21]|1 |714 |
|1595876841356 [Mon Jul 27 2020 19:07:21]|2 |987 |
|1595920041356 [Tue Jul 28 2020 07:07:21]|2 |974 |
|1595963241356 [Tue Jul 28 2020 19:07:21]|0 |498 |
*Note: the TIMESTAMP values are in UNIX time. I have given
the date-time string for reference*
Explanation
For July 27, the latest values for
0: 541 (no other entries for the day)
1: 714
2: 987
For July 28, the latest values for
0: 498
1: nothing (ignore)
2: 974 (no other entries for the day)
You can use distinct on:
select distinct on (floor(timestamp / (24 * 60 * 60 * 1000)), size_type) t.*
from input
order by floor(timestamp / (24 * 60 * 60 * 1000)), size_type,
timestamp desc;
The arithmetic is just to extract the day from the timestamp.
Here is a db<>fiddle.
Hi I have a Pyramid wsgi webapp served using Apache. The webapp has an hourly job that must be run at the 0th min to fetch time-sensitive data and write to my mysql database. I note that sometimes (not all times) the data might not be written into the database for the midnight 00:00:00 run of the task. Looking at the logs, it seems that Apache has been restarted shortly after every midnight which might cause the problem.
After searching through stackoverflow it seems logrotate might be the culprit for the restart. However, I also note that logrotate is called by crontab which defaults to 6:25am so I have no idea why the restart happens at midnight instead. (My Ubuntu server does NOT have anacron installed)
here are the log files for the last few days from Apache
[Tue May 11 00:00:35.534821 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Tue May 11 00:00:35.534867 2021] [core:notice] [pid 72273:tid 140034084613184] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Wed May 12 00:00:00.029412 2021] [wsgi:error] [pid 72660:tid 140033624434432] 2021-05-12 00:00:00,029 INFO [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-12 01:00:00 HKT)" (scheduled at 2021-05-12 00:00:00+08:00)
[Wed May 12 00:00:00.621944 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00493: SIGUSR1 received. Doing graceful restart
[Wed May 12 00:00:03.614647 2021] [wsgi:error] [pid 72660:tid 140033624434432] 2021-05-12 00:00:03,614 INFO [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-12 01:00:00 HKT)" executed successfully
Interesting to note from the log above that it seems my apscheduler still completed running (with the database written into successfully) and printed to the log after Doing graceful restart and before a new log file is created (contents shown below)
[Wed May 12 00:00:03.641095 2021] [mpm_event:notice] [pid 72273:tid 140034084613184] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Wed May 12 00:00:03.641146 2021] [core:notice] [pid 72273:tid 140034084613184] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Thu May 13 00:00:00.032261 2021] [wsgi:error] [pid 95013:tid 140083656877824] 2021-05-13 00:00:00,032 INFO [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-13 01:00:00 HKT)" (scheduled at 2021-05-13 00:00:00+08:00)
[Thu May 13 00:00:03.764471 2021] [wsgi:error] [pid 95013:tid 140083656877824] 2021-05-13 00:00:03,764 INFO [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-13 01:00:00 HKT)" executed successfully
[Thu May 13 00:00:34.829438 2021] [mpm_event:notice] [pid 95012:tid 140084121332800] AH00493: SIGUSR1 received. Doing graceful restart
In the log file above, my apscheduler job completed before the restart so my database got written into properly as well.
[Thu May 13 00:00:35.588354 2021] [mpm_event:notice] [pid 95012:tid 140084121332800] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Thu May 13 00:00:35.588433 2021] [core:notice] [pid 95012:tid 140084121332800] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
[Fri May 14 00:00:00.020559 2021] [wsgi:error] [pid 2120:tid 140241617286912] 2021-05-14 00:00:00,020 INFO [apscheduler.executors.default:123][ThreadPoolExecutor-0_0] Running job "XYZ (trigger: cron[minute='0'], next run at: 2021-05-14 01:00:00 HKT)" (scheduled at 2021-05-14 00:00:00+08:00)
[Fri May 14 00:00:00.558072 2021] [mpm_event:notice] [pid 2119:tid 140242151496768] AH00493: SIGUSR1 received. Doing graceful restart
for the midnight that just passed, the job didn't complete and database has not been written into. there is also no accompanying INFO [apscheduler.executors.default:144][ThreadPoolExecutor-0_0] Job "XYZ (trigger: cron[minute='0'], next run at: xxxxxxx)" executed successfully line written into either of the logs before and after midnight since the job got terminated abruptly before completing
[Fri May 14 00:00:03.588691 2021] [mpm_event:notice] [pid 2119:tid 140242151496768] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.6.8 Python/3.8 configured -- resuming normal operations
[Fri May 14 00:00:03.588744 2021] [core:notice] [pid 2119:tid 140242151496768] AH00094: Command line: '/usr/sbin/apache2'
.
.
.
day hasn't ended yet
Here is my crontab file which I believe is standard and state that daily jobs should be run at 6:25AM not midnight.
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
again my Ubuntu server does NOT have anacron installed
ubuntu#xxx:~$ anacron --help
Command 'anacron' not found, but can be installed with:
sudo apt install anacron
logrotate and apache has cron.daily task
ubuntu#xxx:~$ ls -ln /etc/cron.daily/
total 40
-rwxr-xr-x 1 0 0 539 Apr 14 2020 apache2
-rwxr-xr-x 1 0 0 376 Dec 5 2019 apport
-rwxr-xr-x 1 0 0 1478 Apr 9 2020 apt-compat
-rwxr-xr-x 1 0 0 355 Dec 29 2017 bsdmainutils
-rwxr-xr-x 1 0 0 1187 Sep 6 2019 dpkg
-rwxr-xr-x 1 0 0 377 Jan 21 2019 logrotate
-rwxr-xr-x 1 0 0 1123 Feb 26 2020 man-db
-rwxr-xr-x 1 0 0 4574 Jul 18 2019 popularity-contest
-rwxr-xr-x 1 0 0 214 Dec 7 23:35 update-notifier-common
vi /etc/cron.daily/logrotate
#!/bin/sh
# skip in favour of systemd timer
if [ -d /run/systemd/system ]; then
exit 0
fi
# this cronjob persists removals (but not purges)
if [ ! -x /usr/sbin/logrotate ]; then
exit 0
fi
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit $EXITVALUE
vi /etc/cron.daily/apache2
#!/bin/sh
# run htcacheclean if set to 'cron' mode
set -e
set -u
type htcacheclean > /dev/null 2>&1 || exit 0
[ -e /etc/default/apache-htcacheclean ] || exit 0
# edit /etc/default/apache-htcacheclean to change this
HTCACHECLEAN_MODE=daemon
HTCACHECLEAN_RUN=auto
HTCACHECLEAN_SIZE=300M
HTCACHECLEAN_PATH=/var/cache/apache2/mod_cache_disk
HTCACHECLEAN_OPTIONS=""
. /etc/default/apache-htcacheclean
[ "$HTCACHECLEAN_MODE" = "cron" ] || exit 0
htcacheclean ${HTCACHECLEAN_OPTIONS} \
-p${HTCACHECLEAN_PATH} \
-l${HTCACHECLEAN_SIZE}
/etc/logrotate.conf is just a standard file
# see "man logrotate" for details
# rotate log files weekly
weekly
# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
#dateext
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# system-specific logs may be also be configured here.
apache2 included under logrotate.d
ubuntu#xxx:~$ ls -ln /etc/logrotate.d
total 52
-rw-r--r-- 1 0 0 120 Sep 6 2019 alternatives
-rw-r--r-- 1 0 0 442 Apr 14 2020 apache2
-rw-r--r-- 1 0 0 126 Dec 5 2019 apport
-rw-r--r-- 1 0 0 173 Apr 9 2020 apt
-rw-r--r-- 1 0 0 91 Nov 2 2020 bootlog
-rw-r--r-- 1 0 0 130 Jan 21 2019 btmp
-rw-r--r-- 1 0 0 112 Sep 6 2019 dpkg
-rw-r--r-- 1 0 0 845 Nov 7 2019 mysql-server
-rw-r--r-- 1 0 0 501 Mar 7 2019 rsyslog
-rw-r--r-- 1 0 0 119 Mar 31 2020 ubuntu-advantage-tools
-rw-r--r-- 1 0 0 178 Jan 22 2020 ufw
-rw-r--r-- 1 0 0 235 Jul 21 2020 unattended-upgrades
-rw-r--r-- 1 0 0 145 Feb 19 2018 wtmp
vi /etc/logrotate.d/apache2
/var/log/apache2/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
if invoke-rc.d apache2 status > /dev/null 2>&1; then \
invoke-rc.d apache2 reload > /dev/null 2>&1; \
fi;
endscript
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi; \
endscript
}
I just want to know why my Apache gets restarted at midnight when according to crontab it should be at 6:25AM and change the time to avoid conflict with my 0th min hourly job. thanks!
I have a Table like:
source target
jan feb
mar apr
jun
feb aug
apr jul
oct dec
aug nov
dec may
The output (where I want to create a new_target column):
source target new_target
jan feb aug
mar apr jul
jun
feb aug nov
apr jul
oct dec may
aug nov
dec may
The aim is to create new_targetcolumn based on a logic like - for example, jan in source has value feb in target. This in turn, feb in source has a value aug in target, and so on aug has nov in target column
So the new_target column will have 3rd value: i.e (trace followed between source and target jan->feb->aug->nov, since aug is 3rd value, it is the output in new_target column)
This looks like a left join:
select t.*, tnext.target
from t left join
t tnext
on t.target = t.next.source
Try this:
select m1.source,
m1.target,
m2.target as new_target
from mytable m1
left join mytable m2 on
m1.target = m2.source
The left join will maintain all rows from the original table, while adding values to the new_target column if there is a match.
I have duplicity running an incremental daily backup to S3. About 37 GiB.
On the first month or so, it went ok. It used to finish in about 1 hour. Then it started taking too long to complete the task. Right now, as I type, it is still running the daily backup that started 7 hours ago.
I'm running two commands, first to backup and then cleanup:
duplicity --full-if-older-than 1M LOCAL.SOURCE S3.DEST --volsize 666 --verbosity 8
duplicity remove-older-than 2M S3.DEST
The logs
Temp has 54774476800 available, backup will use approx 907857100.
So the temp has enough space, good. Then it starts with this...
Copying duplicity-full-signatures.20161107T090303Z.sigtar.gpg to local cache.
Deleting /tmp/duplicity-ipmrKr-tempdir/mktemp-13tylb-2
Deleting /tmp/duplicity-ipmrKr-tempdir/mktemp-NanCxQ-3
[...]
Copying duplicity-inc.20161110T095624Z.to.20161111T103456Z.manifest.gpg to local cache.
Deleting /tmp/duplicity-ipmrKr-tempdir/mktemp-VQU2zx-30
Deleting /tmp/duplicity-ipmrKr-tempdir/mktemp-4Idklo-31
[...]
This continues for each day till today, taking long minutes for each file. And continues with this...
Ignoring incremental Backupset (start_time: Thu Nov 10 09:56:24 2016; needed: Mon Nov 7 09:03:03 2016)
Ignoring incremental Backupset (start_time: Thu Nov 10 09:56:24 2016; needed: Wed Nov 9 18:09:07 2016)
Added incremental Backupset (start_time: Thu Nov 10 09:56:24 2016 / end_time: Fri Nov 11 10:34:56 2016)
After a long time...
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: Sun Mar 12 09:54:00 2017
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /home/user/.cache/duplicity/700b5f90ee4a620e649334f96747bd08
Found 6 secondary backup chains.
Secondary chain 1 of 6:
-------------------------
Chain start time: Mon Nov 7 09:03:03 2016
Chain end time: Mon Nov 7 09:03:03 2016
Number of contained backup sets: 1
Total number of contained volumes: 2
Type of backup set: Time: Num volumes:
Full Mon Nov 7 09:03:03 2016 2
-------------------------
Secondary chain 2 of 6:
-------------------------
Chain start time: Wed Nov 9 18:09:07 2016
Chain end time: Wed Nov 9 18:09:07 2016
Number of contained backup sets: 1
Total number of contained volumes: 11
Type of backup set: Time: Num volumes:
Full Wed Nov 9 18:09:07 2016 11
-------------------------
Secondary chain 3 of 6:
-------------------------
Chain start time: Thu Nov 10 09:56:24 2016
Chain end time: Sat Dec 10 09:44:31 2016
Number of contained backup sets: 31
Total number of contained volumes: 41
Type of backup set: Time: Num volumes:
Full Thu Nov 10 09:56:24 2016 11
Incremental Fri Nov 11 10:34:56 2016 1
Incremental Sat Nov 12 09:59:47 2016 1
Incremental Sun Nov 13 09:57:15 2016 1
Incremental Mon Nov 14 09:48:31 2016 1
[...]
After listing all chains:
Also found 0 backup sets not part of any chain, and 1 incomplete backup set.
These may be deleted by running duplicity with the "cleanup" command.
This was only the backup part. It takes hours doing this and only 10 minutes to upload 37 GiB to S3.
ElapsedTime 639.59 (10 minutes 39.59 seconds)
SourceFiles 288
SourceFileSize 40370795351 (37.6 GB)
Then comes the cleanup, that gives me this:
Cleaning up
Local and Remote metadata are synchronized, no sync needed.
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: Sun Mar 12 09:54:00 2017
There are backup set(s) at time(s):
Tue Jan 10 09:58:05 2017
Wed Jan 11 09:54:03 2017
Thu Jan 12 09:56:42 2017
Fri Jan 13 10:05:05 2017
Sat Jan 14 10:24:54 2017
Sun Jan 15 09:49:31 2017
Mon Jan 16 09:39:41 2017
Tue Jan 17 09:59:05 2017
Wed Jan 18 09:59:56 2017
Thu Jan 19 10:01:51 2017
Fri Jan 20 09:35:30 2017
Sat Jan 21 09:53:26 2017
Sun Jan 22 09:48:57 2017
Mon Jan 23 09:38:45 2017
Tue Jan 24 09:54:29 2017
Which can't be deleted because newer sets depend on them.
Found old backup chains at the following times:
Mon Nov 7 09:03:03 2016
Wed Nov 9 18:09:07 2016
Sat Dec 10 09:44:31 2016
Mon Jan 9 10:04:51 2017
Rerun command with --force option to actually delete.
I found the problem. Because of an issue, I followed this answer, and added this code to my script:
rm -rf ~/.cache/deja-dup/*
rm -rf ~/.cache/duplicity/*
This is supposed to be a one time thing because of random bug duplicity had. But the answer didn't mention this. So every day the script was removing the cache just after syncing the cache, and then, on the next day it had to download the whole thing again.
I have a file of below format
02 Jul 2016 00:00:00 2736359
02 Jul 2016 00:02:00 2736594
02 Jul 2016 00:04:00 2736828
02 Jul 2016 00:06:00 2737068
02 Jul 2016 00:08:00 2737303
02 Jul 2016 00:10:00 2737542
02 Jul 2016 00:12:00 2737775
02 Jul 2016 00:14:00 2738011
02 Jul 2016 00:16:00 2738251
02 Jul 2016 00:18:00 2738483
Where the first column is the time stamp and second is a number. Given an input 2737778, I want the output to be "02 Jul 2016 00:12:00 and 02 Jul 2016 00:14:00" as 2737778 falls in between 2737775 and 2738011. Can I do this in awk? Is it possible to compare a number in current line with next line?
another similar awk
awk -v n=2737778 'n<=$NF{if(p) print p; print; exit} {p=$0}' file
02 Jul 2016 00:12:00 2737775
02 Jul 2016 00:14:00 2738011
Yes, it is possible to read ahead in awk, see Peek at next line, but don't consume it. Here is another way to do it.
awk '{ if (NR == 1) { save = $0; prtsw = 1 }
else if (prtsw == 1 && 2737778 < $5) {
print save
print $0
prtsw = 0
}
else { save = $0 }
}' abetween.txt