Deleting/Shrinking HSQLDB lob files - blob

We're using HSQLDB 2.3.2 and we would like to get rid of the file containing the LOB data. As it is taking a large amount of disk space.
We already changed our LOB columns to VARCHAR. So we don't have any LOB columns anymore.
$ ./execute_query.sh "SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE, TYPE_NAME FROM INFORMATION_SCHEMA.SYSTEM_COLUMNS WHERE TYPE_NAME LIKE '%LOB%';"
TABLE_NAME COLUMN_NAME DATA_TYPE TYPE_NAME
---------- ----------- --------- ---------
And a list of our HSQLDB files:
$ ls -hal
total 88G
drwxr-xr-x 3 admin admin 4.0K Sep 23 13:58 .
drwxr-xr-x 6 admin admin 4.0K Sep 1 09:27 ..
-rw-r--r-- 1 admin admin 1.5M Apr 9 01:13 hsqldb.jar
-rw-r--r-- 1 admin admin 2.9G Sep 23 13:53 hsqldb.db.data
-rw-r--r-- 1 admin admin 16 Sep 23 14:02 hsqldb.db.lck
-rw-r--r-- 1 admin admin 84G Aug 25 01:52 hsqldb.db.lobs
-rw-r--r-- 1 admin admin 70 Sep 23 14:00 hsqldb.db.log
-rw-r--r-- 1 admin admin 85 Sep 23 13:58 hsqldb.db.properties
-rw-r--r-- 1 admin admin 1.5G Sep 23 13:53 hsqldb.db.script
drwxr-xr-x 2 admin admin 4.0K Sep 23 13:54 hsqldb.db.tmp
We would like to get rid of the large hsqldb.db.lobs file.
After removing the file, the database does start correctly and seems to run fine. And I can run queries.
But after creating a table with LOB columns and adding data to them the hsqldb.db.lobs file gets created again with the original size (although it's a sparse file).
I suspect more magic can be done by modifying certain table values in the SYSTEM_LOBS schema. But I don't think that is a very good practice.
So what's the advised way to either
Shrink the LOB file so it is much smaller (as we don't use the data)
Remove the LOB file completely

After removing the file, the database does start correctly and seems
to run fine. And I can run queries.
At this point you need to perform CHECKPOINT to reset the lob tables. This works after a workaround described in this post (ignore the compressed setting):
HSQLDB: enable LOB compression for existing database

Related

valgrind in QNX: Can't access shared library Error

I've downloaded valgrind from https://community.qnx.com/sf/frs/do/listReleases/projects.valgrind/frs.valgrind for QNX.
OS details:
#uname -a
QNX localhost 7.1.0S 2021/02/12-21:06:45EST SA8540P_v2.1_ft0_ADP_Ride_v1.0.2_UFS_NORMAL aarch64le
# ls -l
total 282423
drwxr-xr-x 3 default 1000 4096 Apr 30 2015 armle-v7
drwxr-xr-x 3 default 1000 4096 Apr 30 2015 usr
drwxr-xr-x 3 default 1000 4096 Apr 30 2015 x86
# ls -l /var/valgrind/armle-v7/usr
total 16
drwxr-xr-x 2 default 1000 4096 Apr 30 2015 bin
drwxr-xr-x 4 default 1000 4096 Apr 30 2015 lib
# ls -l
total 917
-rwxr-xr-x 1 default 1000 41112 Apr 30 2015 callgrind_annotate
-rwxr-xr-x 1 default 1000 12016 Apr 30 2015 callgrind_control
-rwxr-xr-x 1 default 1000 32170 Apr 30 2015 cg_annotate
-rwxr-xr-x 1 default 1000 10418 Apr 30 2015 cg_diff
-rwxr-xr-x 1 default 1000 71060 Apr 30 2015 cg_merge
-rwxr-xr-x 1 default 1000 24398 Apr 30 2015 ms_print
-rwxr-xr-x 1 default 1000 53202 Apr 30 2015 **valgrind**
-rwxr-xr-x 1 default 1000 96364 Apr 30 2015 valgrind-di-server
-rwxr-xr-x 1 default 1000 35454 Apr 30 2015 valgrind-listener
-rwxr-xr-x 1 default 1000 90877 Apr 30 2015 vgdb
I am entirely new to QNX.
Should I use armle-v7 or x86?
Can I directly run valgrind under bin:
valgrind <my-program> <input arg>
If i try this I get the below error:
**sh: ./valgrind: Can't access shared library**
Even version command gives the same error:
# /var/valgrind/armle-v7/usr/bin/valgrind --version
sh: /var/valgrind/armle-v7/usr/bin/valgrind: Can't access shared library
Should I copy the valgrind to /usr/bin or some other folder? I've already exported LD_LIBRARY_PATH for the libraries needed by my executable and also valgrind lib path[/var/valgrind/armle-v7/usr/lib]
Basically I need to analyze memory leak and crash in my program[Note: Don't have QNX Momentics IDE]. From QNX software centre, can we download valgrind?

First data file in Chronicle Queue is always touched

We're wondering why ChronicleQueue always seems to touch the first data file. Is there a reason for it?
The other filer don't seem to be affected, even if data is read from them. Are we doing something wrong?
Currently we're using version 5.19.2.
[root#node-000341 dataLog]# ls -al
total 16520
drwxrwxr-x. 2 tn tn 4096 Jan 13 15:17 .
drwxrwxr-x. 7 tn tn 94 Apr 14 2020 ..
-rw-r--r--. 1 tn tn 83886080 Jan 12 11:46 20200424.cq4
-rw-r--r--. 1 tn tn 83886080 May 5 2020 20200427.cq4
-rw-r--r--. 1 tn tn 131782 May 12 2020 20200505.cq4
-rw-r--r--. 1 tn tn 131574 May 13 2020 20200512.cq4
.....
-rw-r--r--. 1 tn tn 389465 Dec 16 09:26 20201210.cq4
-rw-r--r--. 1 tn tn 184090 Jan 12 12:07 20201216.cq4
-rw-r--r--. 1 tn tn 361994 Jan 13 15:17 20210112.cq4
-rw-r--r--. 1 tn tn 83886080 Jan 13 15:22 20210113.cq4
-rw-r--r--. 1 tn tn 65536 Jan 13 15:21 metadata.cq4t
When you create a tailer, by default it goes toStart(), which is the first file, which is what causes the first file to be touched (note that ls shows access time, not modification time).
BTW unless you really need that access time for any reason, we suggest using noatime mount option to speed up file access.

Backup a keyspace in scylladb

I am trying to take a backup of my data in scylladb. Currently, my Scylla is in docker.
So I am running this command:
docker exec -it saif-scylla nodetool snapshot testkeyspace
Requested creating snapshot(s) for [testkeyspace] with snapshot name [1564405495089]
Snapshot directory: 1564405495089
But I can't find any backup in /var/lib/scylla/data folder.
Also what exactly it means by "requested".
Also when I list the snapshots by running the command:
docker exec -it saif-scylla nodetool listsnapshots
What I can see is:
Snapshot name Keyspace name Column family name True size Size on disk
1564405495089 testkeyspace new_events 0 bytes 0 bytes
1564405495089 testkeyspace new_pings 0 bytes 0 bytes
1564405495089 testkeyspace test_pings 0 bytes 0 bytes
I am not getting what's wrong is happening here.
Any Idea, what I am doing wrong?
Any help will be helpful.
Thanks
The snapshot files are located in the table directory, under "snapshots".
For example, for keyspace mykeyspace, and table heartrate_ttl, after running nodetool snapshot mykeyspace;
ls -l /var/lib/scylla/data/mykeyspace/heartrate_ttl-75359ce0b22611e9b18b000000000000/snapshots/1564421433190/
total 44
-rw-r--r--. 2 root root 66 Jul 29 17:30 la-4-big-CompressionInfo.db
-rw-r--r--. 2 root root 189 Jul 29 17:30 la-4-big-Data.db
-rw-r--r--. 2 root root 10 Jul 29 17:30 la-4-big-Digest.sha1
-rw-r--r--. 2 root root 16 Jul 29 17:30 la-4-big-Filter.db
-rw-r--r--. 2 root root 30 Jul 29 17:30 la-4-big-Index.db
-rw-r--r--. 2 root root 54 Jul 29 17:30 la-4-big-Scylla.db
-rw-r--r--. 2 root root 4466 Jul 29 17:30 la-4-big-Statistics.db
-rw-r--r--. 2 root root 92 Jul 29 17:30 la-4-big-Summary.db
-rw-r--r--. 2 root root 101 Jul 29 17:30 la-4-big-TOC.txt
-rw-r--r--. 1 root root 38 Jul 29 17:30 manifest.json
nodetool listsnapshots should give you the snapshot size
nodetool listsnapshots
Snapshot Details:
Snapshot name Keyspace name Column family name True size Size on disk
1564421433190 mykeyspace heartrate_ttl 0 bytes 4.91 KB
I used Scylla 3.0.5 Docker for the above example.
Could it be that you have no data in these tables?

Pruning did not reduce the inference time or model size?

I ran the pruning demo provided by tensorflow and found that pruning did not reduce the inference time or model size.
Below is my experiment result. It can be seen that the different pruning stages have the same model size.
-rw-r--r-- 1 root root 4.1M May 27 06:15 model.ckpt-2437.data-00000-of-00002
-rw-r--r-- 1 root root 8.2M May 27 06:15 model.ckpt-2437.data-00001-of-00002
-rw-r--r-- 1 root root 1.7K May 27 06:15 model.ckpt-2437.index
-rw-r--r-- 1 root root 368K May 27 06:15 model.ckpt-2437.meta
-rw-r--r-- 1 root root 4.1M May 27 06:25 model.ckpt-4871.data-00000-of-00002
-rw-r--r-- 1 root root 8.2M May 27 06:25 model.ckpt-4871.data-00001-of-00002
-rw-r--r-- 1 root root 1.7K May 27 06:25 model.ckpt-4871.index
-rw-r--r-- 1 root root 368K May 27 06:25 model.ckpt-4871.meta
-rw-r--r-- 1 root root 4.1M May 27 06:35 model.ckpt-7329.data-00000-of-00002
-rw-r--r-- 1 root root 8.2M May 27 06:35 model.ckpt-7329.data-00001-of-00002
-rw-r--r-- 1 root root 1.7K May 27 06:35 model.ckpt-7329.index
-rw-r--r-- 1 root root 368K May 27 06:35 model.ckpt-7329.meta
The model inference time obtained in different pruning stages is basically the same.
2018-05-27 06:16:42.845279: precision # 1 = 0.697, inf time = 0.03076
2018-05-27 06:27:26.517855: precision # 1 = 0.756, inf time = 0.02980
2018-05-27 06:37:31.223502: precision # 1 = 0.783, inf time = 0.02989
I use the default pruning parameters.

mod_wsgi w/ Apache 2.2 and Python 2.7 installation issues. Miniconda as well

This problem has eluded me thus far. I have a Centos 6.7 machine running Apache 2.2 with Python 2.7 installed at /opt/home/user/miniconda2/envs/myenv/lib. Python 2.6 is obviously also installed on this system at /usr/bin/python. At first I installed mod_wsgi with pip and copied the created *.so Apache module to my modules folder. From my perspective it was created with 2.7 but I could not get the stupid ImportError: package site not found or whatever to go away. I uninstalled mod_wsgi and compiled and installed from source 4.22. I put the folder into my /home/user/ directory and started my installation process.
However easy I expected the fine-tuning available to configure to be, it quickly became apparent it was anything but. My first hurdle I surpassed, but my second has continued to stump me. After running configure:
./configure --with-python=/opt/home/user/miniconda2/envs/myenv/bin/python LD_RUN_PATH=/opt/home/user/miniconda2/envs/myenv/lib
(myenv)[user#machine2 mod_wsgi-4.4.21]$ ldd /usr/lib/httpd/modules/mod_wsgi.so
linux-gate.so.1 => (0x002a1000)
libpython2.7.so.1.0 => not found
libpthread.so.0 => /lib/libpthread.so.0 (0x00164000)
libdl.so.2 => /lib/libdl.so.2 (0x00c1f000)
libutil.so.1 => /lib/libutil.so.1 (0x00c2d000)
libm.so.6 => /lib/libm.so.6 (0x007cf000)
libc.so.6 => /lib/libc.so.6 (0x002a2000)
/lib/ld-linux.so.2 (0x00bcc000)
I think we all know that this means the shared library was not found. But I can see it in the directory!
pwd = /opt/home/user/miniconda2/envs/myenv/lib
[user#machine2 lib]$ ls -l
total 20264
drwxrwxr-x 2 user user 4096 Mar 21 13:46 engines
-rw-rw-r-- 2 user user 3066000 Mar 1 12:23 libcrypto.a
lrwxrwxrwx 1 user user 18 Mar 21 13:46 libcrypto.so -> libcrypto.so.1.0.0
-rwxrwxr-x 1 user user 1945963 Mar 21 13:46 libcrypto.so.1.0.0
-rw-r--r-- 3 user user 104318 Jan 3 2014 libhistory.a
lrwxrwxrwx 1 user user 15 Mar 21 13:46 libhistory.so -> libhistory.so.6
lrwxrwxrwx 1 user user 17 Mar 21 13:46 libhistory.so.6 -> libhistory.so.6.2
-rwxr-xr-x 3 user user 78845 Jan 3 2014 libhistory.so.6.2
lrwxrwxrwx 1 user user 19 Mar 21 13:47 libpython2.7.so -> libpython2.7.so.1.0
-rwxrwxr-x 3 user user 4979591 Dec 6 16:09 libpython2.7.so.1.0
-rw-r--r-- 3 user user 715160 Jan 3 2014 libreadline.a
lrwxrwxrwx 1 user user 16 Mar 21 13:46 libreadline.so -> libreadline.so.6
lrwxrwxrwx 1 user user 18 Mar 21 13:46 libreadline.so.6 -> libreadline.so.6.2
-rwxr-xr-x 3 user user 516418 Jan 3 2014 libreadline.so.6.2
-rw-rw-r-- 2 user user 2977926 Jan 11 11:52 libsqlite3.a
-rwxrwxr-x 1 user user 984 Mar 21 13:46 libsqlite3.la
lrwxrwxrwx 1 user user 19 Mar 21 13:46 libsqlite3.so -> libsqlite3.so.0.8.6
lrwxrwxrwx 1 user user 19 Mar 21 13:46 libsqlite3.so.0 -> libsqlite3.so.0.8.6
-rwxrwxr-x 2 user user 2573507 Jan 11 11:52 libsqlite3.so.0.8.6
-rw-rw-r-- 2 user user 613290 Mar 1 12:23 libssl.a
lrwxrwxrwx 1 user user 15 Mar 21 13:46 libssl.so -> libssl.so.1.0.0
-rwxrwxr-x 2 user user 462887 Mar 1 12:23 libssl.so.1.0.0
-rwxr-xr-x 3 user user 1154833 Mar 16 2015 libtcl8.5.so
-rwxr-xr-x 3 user user 3008 Mar 16 2015 libtclstub8.5.a
-rwxr-xr-x 3 user user 1257824 Mar 16 2015 libtk8.5.so
-rwxr-xr-x 3 user user 4446 Mar 16 2015 libtkstub8.5.a
-rw-r--r-- 3 user user 98574 Jan 5 2015 libz.a
lrwxrwxrwx 1 user user 13 Mar 21 13:46 libz.so -> libz.so.1.2.8
lrwxrwxrwx 1 user user 13 Mar 21 13:46 libz.so.1 -> libz.so.1.2.8
-rwxr-xr-x 3 user user 91730 Jan 5 2015 libz.so.1.2.8
drwxrwxr-x 2 user user 4096 Mar 21 13:47 pkgconfig
drwxrwxr-x 26 user user 20480 Mar 21 13:49 python2.7
drwxrwxr-x 4 user user 4096 Mar 21 13:46 tcl8
drwxrwxr-x 6 user user 4096 Mar 21 13:46 tcl8.5
-rw-r--r-- 1 user user 7356 Mar 21 13:46 tclConfig.sh
drwxrwxr-x 6 user user 4096 Mar 21 13:46 tk8.5
-rw-r--r-- 1 user user 4299 Mar 21 13:46 tkConfig.sh
I have been cycling between the above configure, sudo make,sudo makeinstall and sudo make distclean but to no avail, any help is appreciated.
Don't use conda with mod_wsgi and Apache. Use virtualenv. Conda's embedded Python install will be obscured from your Apache module AFAIK.