Virtuoso: What happens when I get 'Checkpoint removed ** MB of remapped pages, leaving * MB' - sparql

I am loading freebase files into virtuoso. I have 88 folders. Every folder contains 4 files. I divided the files such way because I only have 8 GB of RAM.
#!/bin/bash
for i in {1..88}
do
isql 1111 dba dba exec="ld_dir('/data/data/${i}', '*.nt', 'http://freebase.com');"
isql 1111 dba dba exec="rdf_loader_run();"
isql 1111 dba dba exec="checkpoint;"
isql 1111 dba dba exec="commit WORK;"
isql 1111 dba dba exec="checkpoint;"
isql 1111 dba dba exec="delete from DB.DBA.load_list;"
done
Virtuoso.ini
# each buffer caches a 8K page of data and occupies approx. 8700 bytes of memory
# it's suggested to set this value to 65 % of ram for a db only server
# so if you have 32 GB of ram: 32*1000^3*0.65/8700 = 2390804
# default is 2000 which will use 16 MB ram
[Database]
MaxCheckpointRemap = 150000 (I have 8GB of RAM)
[TempDatabase]
MaxCheckpointRemap = 2000
NumberOfBuffers = 170000
MaxDirtyBuffers = 130000
Message
Checkpoint removed 628 MB of remapped pages, leaving 31 MB. Duration 31.21 s. To save this time, increase MaxCheckpointRemap and/or set Unremap quota to 0 in ini file.
Question
Why am I getting this message? and does it affect the loading process and building of the database?
update
#!/bin/bash
#clear list
isql 1111 dba dba exec="delete from DB.DBA.load_list;"
#load data
isql 1111 dba dba exec="ld_dir('/data/data, '*.gz', 'http://freebase.com');"
isql 1111 dba dba exec="set isolation='uncommitted';"
isql 1111 dba dba exec="rdf_loader_run();"
#checkpoint
isql 1111 dba dba exec="checkpoint;"
isql 1111 dba dba exec="commit WORK;"
isql 1111 dba dba exec="checkpoint;"
virtuoso.ini
MaxCheckpointRemap = 150000 (8GB RAM)
# Uncomment next two lines if there is 4 GB system memory free
NumberOfBuffers = 340000
MaxDirtyBuffers = 250000
# I have 6 cores
ThreadsPerQuery = 4
AsyncQueueMaxThreads = 10
#NO THREADS LEFT IN THE QUEUE
ThreadCleanupInterval = 0
ThreadThreshold = 0
#check point every 30 minutes.
CheckpointInterval = 30

It's best explained in the virtuoso docs, but in short:
The MaxCheckpointRemap parameter in the virtuoso.ini file controls how
many pages may be stored on a page other than their logical page.
This means that increasing the parameter will save you work (and time) on a checkpoint but reduces the data locality feature (data that is inserted together is saved close together).

Related

mbr2gpt: Too many MBR partitions found, no room to create EFI system partition

Try to convert MBR to GPT with mbr2gpt introduced with Windows 10 build 1703, it failed with
mbr2gpt: Too many MBR partitions found, no room to create EFI system partition.
Full log:
2017-06-07 22:23:24, Info ESP partition size will be 104857600
2017-06-07 22:23:24, Info MBR2GPT: Validating layout, disk sector size is: 512 bytes
2017-06-07 22:23:24, Error ValidateLayout: Too many MBR partitions found, no room to create EFI system partition.
2017-06-07 22:23:24, Error Disk layout validation failed for disk 1
The mbr2gpt disk conversion tool need three conditions for the validation of disk layout:
Admin rights (what you already know)
One of physical disk (or harddrive) with boot partition (MSR) AND os partition
The validation allows normally one of additional partitions (often it's recovery partition)
If you have more than three partition then check this with diskpart:
Microsoft Windows [Version 10.0.15063]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\WINDOWS\system32>diskpart
Microsoft DiskPart-Version 10.0.15063.0
Copyright (C) Microsoft Corporation.
On computer: SILERRAS-PC
DISKPART> list disk
Dist ### Status Size Free Dyn GPT
-------- ------ ------ ------- ---- ---
Disk 0 Online 117 GB 1024 KB * *
Disk 1 Online 489 GB 455 MB *
Disk 2 Online 186 GB 0 B
Disk 3 Online 931 GB 0 B
Disk 4 Online 931 GB 1024 KB *
DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> list partition
Partition ### Typ Größe Offset
------------- ---------------- ------- -------
Partition 1 System 350 MB 1024 KB
Partition 2 Primary 487 GB 351 MB
Partition 3 Recovery 452 MB 488 GB
DISKPART>
Try to reduce the number of partitions to three partitions.
If you have more than two recovery partitions, then check this out with "ReAgentc /info". This command shows you the current recovery partitions. Often only one of those is active. You can delete the other one with diskpart. Please be careful which partition you delete. The diskpart command is "delete partition override".
I hope my guide is helpful for you.

mysqldump performance on machine with big amount of memory

I'm doing backup of innodb database with mysqldump. It takes 2 min to perform a backup.
Is there any way how to speed it up?
I have machine with 120GB of RAM and I expect that my DB should fit in memory.
Database size on hard drive is around 8 GB:
[user#host:E018 mysql]$ du -hs database
8.3G database
Biggest table has 12054861 records and data size 2991587328.
I have tried to play with innodb_buffer_pool_size but I don't see big performance increase.
If I run mysqldump for first time it takes 2 min 7 sec. If I try it second time it takes around 2 min that is to slow.
I have also tried to archive data to avoid a lot of disk writes:
mysqldump database |pigz > database-dmp.sql.gz that has no influence on performance.
Running mysqldump on different machine from mysql engine does not change anything.
Probably mysql does not cache data to the memory or it sends data to the mysqldump to slow.
Here is configuration that I use:
max_heap_table_size=10G;
innodb_file_per_table=1
innodb_file_format=barracuda
innodb_strict_mode=1
innodb_write_io_threads=4
innodb_read_io_threads=4
innodb_adaptive_hash_index=0
innodb_support_xa=0
innodb_buffer_pool_size=40G
innodb_log_file_size=256M
innodb_log_files_in_group=3
innodb_log_buffer_size=64M
What else can I try to improve mysqldump performance?

Data Base full backup size

I have a job to full backup my database daily. I have usually between 200 to 300 new records in the database daily. However, size of backup file is always the same and is 7,472 KB. Why this is always fixed while the number of records is increasing. Database Files specifications are as following:
Logical Name ------ File Type----Filegroup---Initial Size----Autogrowth/ Maxsize
DBF-----------------Rows Data----PRIMARY-----4---------------by 15 MB, Unlimited
DBF_Data------------Rows Data----FG_Data-----221-------------by 15 MB, Unlimited
DBF_Index-----------Rows Data----FG_Index----3---------------by 15 MB, Unlimited
DBF_Log-------------Log--------Not Applicable- 793--------- by 10 percent,Limited to 2097152 MB
This is the code I wrote to make daily backup
declare #Date nvarchar(10)
select #Date=cast( cast(getdate() as date) as nvarchar)
declare #Path nvarchar(100)
select #Path= 'C:\PAT Daily BK\PAT' + #Date +'.Bak'
BACKUP DATABASE PAT TO DISK =#path
Until your database expands past the initial size, all backups will be the initial size (compression aside). After it starts expanding, if the expansion amount is set to an amount greater than the space consumed by the additional records then the backup can be expected to stay the same size for multiple days, but eventually will increase when it has to expand again.
SQL Server databases occupy all space allocated whether the data warrants it or not. After it uses up that space it will follow the rules you set for expansion.
Update: The above explains the observed behavior for encrypted databases, but not for unencrypted databases, as backups of unencrypted databases will be relative to the size of the database.

Delete or shrink lobs file in a HSQLDB database

I am using HSQLDB 2.3.0. I have a database that the following schema:
CREATE TABLE MEASUREMENT (ID INTEGER NOT NULL PRIMARY KEY IDENTITY, OBJ CLOB);
When I fill this table with test data, the LOBS file in my database grows:
ls -lath
-rw-rw-r-- 1 hsqldb hsqldb 35 May 6 16:37 msdb.log
-rw-rw-r-- 1 hsqldb hsqldb 85 May 6 16:37 msdb.properties
-rw-rw-r-- 1 hsqldb hsqldb 16 May 6 16:37 msdb.lck
drwxrwxr-x 2 hsqldb hsqldb 4.0K May 6 16:37 msdb.tmp
-rw-rw-r-- 1 hsqldb hsqldb 1.6M May 6 16:37 msdb.script
-rw-rw-r-- 1 hsqldb hsqldb 625M May 6 16:35 msdb.lobs
After running the following command:
TRUNCATE SCHEMA public AND COMMIT;
CHECKPOINT DEFRAG;
SHUTDOWN COMPACT;
The lobs file is still the same size:
-rw-rw-r-- 1 hsqldb hsqldb 84 May 6 16:44 msdb.properties
-rw-rw-r-- 1 hsqldb hsqldb 1.6M May 6 16:44 msdb.script
-rw-rw-r-- 1 hsqldb hsqldb 625M May 6 16:35 msdb.lobs
What is the best way to truncate the schema and get all the disk space back?
I have an application with the same problem using hsqldb 2.3.3. The .lobs file seems to be growing indefinatly even after calling "checkpoint defrag". My scenario is that I'm inserting a 1000 blobs of 300 bytes each. I'm periodically deleting them all and inserting a 1000 new blobs about the same size. After a number of rounds of this my .lobs file is now 1,3GB in size but it is really just storing around 300kB of data. Inspite of calling checkpoint defrag the .lobs file just grows and grows. Is this behavoiur a bug?
The database engine is designed for continuous use in real applications. If you have an application that uses lobs and deletes some of them, the space will be reused for future lobs after each checkpoint.
In normal application use, the DELETE statement is used to delete rows. This statement deallocates the lob space for reuse after each checkpoint.
You can design your tests in a way that recreates the database, rather than reuse the old database after removing the data.

Why would a nightly full backup of our SQL Server database grow 30 GB over night and then shrink again the next day?

We run SQL Server 2005 and have a database that's about 100 GB (the MDF is 100GB and the LDF is 34 GB).
Our maintenance plan takes a full database back up every night. It's set up to
This backup size is usually around 95-100 GB but it all of a sudden grew to 120 GB, then 124 GB then 130 GB then back to 100 GB over 4 consecutive days.
Does anyone know what could cause this? I don't believe we added and then removed that much data in such a short period of time.
If your backup is larger than the MDF, this means you have a lot of log activity recorded too. SQL Server notes data changes that happen during a full backup and does a "mini" log backup to capture this.
I'd say that you need to change Index maintenance and backup timings