Try to convert MBR to GPT with mbr2gpt introduced with Windows 10 build 1703, it failed with
mbr2gpt: Too many MBR partitions found, no room to create EFI system partition.
Full log:
2017-06-07 22:23:24, Info ESP partition size will be 104857600
2017-06-07 22:23:24, Info MBR2GPT: Validating layout, disk sector size is: 512 bytes
2017-06-07 22:23:24, Error ValidateLayout: Too many MBR partitions found, no room to create EFI system partition.
2017-06-07 22:23:24, Error Disk layout validation failed for disk 1
The mbr2gpt disk conversion tool need three conditions for the validation of disk layout:
Admin rights (what you already know)
One of physical disk (or harddrive) with boot partition (MSR) AND os partition
The validation allows normally one of additional partitions (often it's recovery partition)
If you have more than three partition then check this with diskpart:
Microsoft Windows [Version 10.0.15063]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\WINDOWS\system32>diskpart
Microsoft DiskPart-Version 10.0.15063.0
Copyright (C) Microsoft Corporation.
On computer: SILERRAS-PC
DISKPART> list disk
Dist ### Status Size Free Dyn GPT
-------- ------ ------ ------- ---- ---
Disk 0 Online 117 GB 1024 KB * *
Disk 1 Online 489 GB 455 MB *
Disk 2 Online 186 GB 0 B
Disk 3 Online 931 GB 0 B
Disk 4 Online 931 GB 1024 KB *
DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> list partition
Partition ### Typ Größe Offset
------------- ---------------- ------- -------
Partition 1 System 350 MB 1024 KB
Partition 2 Primary 487 GB 351 MB
Partition 3 Recovery 452 MB 488 GB
DISKPART>
Try to reduce the number of partitions to three partitions.
If you have more than two recovery partitions, then check this out with "ReAgentc /info". This command shows you the current recovery partitions. Often only one of those is active. You can delete the other one with diskpart. Please be careful which partition you delete. The diskpart command is "delete partition override".
I hope my guide is helpful for you.
Related
I've been running into the following issue when running basic queries on Impala (For example: select * from table limit 100) recently. I did some online research but have not found a fix for this. any insights on how I could fix this? I use HUE for querying.
ExecQueryFInstances rpc query_id=5d4f8d25428xxxx:813cfbd30000xxxx
failed: Failed to get minimum memory reservation of 8.00 MB on daemon
ser1830.xxxx.com:22000 for query xxxxxxxxx:813cfbd30xxxxxxxxx
due to following error: Failed to increase reservation by 8.00 MB
because it would exceed the applicable reservation limit for the
"Process" ReservationTracker: reservation_limit=68.49 GB
reservation=68.49 GB used_reservation=0 child_reservations=68.49 GB
The top 5 queries that allocated memory under this tracker are:
Query(5240724f8exxxxxx:ab377425000xxxxxx): Reservation=41.81 GB
ReservationLimit=64.46 GB OtherMemory=133.44 MB Total=41.94 GB
Peak=42.62 GB Query(394dcbbaf6bxxxxx2f4760000xxxxxx0):
Reservation=26.68 GB ReservationLimit=64.46 GB OtherMemory=92.94 KB
Total=26.68 GB Peak=26.68 GB Query(5d4f8d25428xxxxx:813cfbd30000xxxxx):
Limit=100.00 GB Reservation=0 ReservationLimit=64.46 GB OtherMemory=0
Total=0 Peak=0 Memory is likely oversubscribed. Reducing query
concurrency or configuring admission control may help avoid this
error.
The problem:
Parquet read performance of Drill appears to be 5x - 10x worse when reading from azure storage and it renders it unusable for bigger data workloads.
It appears to be only a problem when reading parquets. Reading CSV, at the other hand, runs normally.
Let's have:
Azure Blob storage account with ~1GB source.csv and parquets with same data.
Azure Premium File Storage with the same files
Local disk folder containing the same files
Drill running on Azure VM in single mode
Drill configuration:
Azure blob storage plugin working as namespace blob
Azure files mounted with SMB to /data/dfs used as namespace dfs
Local disk folder used as namespace local
The VM
Standard E4s v3 (4 vcpus, 32 GiB memory)
256GB SSD
NIC 2Gbps
6400 IOPS / 96MBps
Azure Premium Files Share
1000GB
1000 IOPS base / 3000 IOPS Burst
120MB/s throughput
Storage benchmarks
Measured with dd, 1GB data, various block sizes, conv=fdatasync
FS cache dropped before each read test (sudo sh -c "echo 3 > /proc/sys/vm/drop_caches")
Local disk
+-------+------------+--------+
| Mode | Block size | Speed |
+-------+------------+--------+
| Write | 1024 | 37MB/s |
| Write | 64 | 16MBs |
| Read | 1024 | 70MB/s |
| Read | 64 | 44MB/s |
+-------+------------+--------+
Azure Premium File Storage SMB mount
+-------+------------+---------+
| Mode | Block size | Speed |
+-------+------------+---------+
| Write | 1024 | 100MB/s |
| Write | 64 | 23MBs |
| Read | 1024 | 88MB/s |
| Read | 64 | 40MB/s |
+-------+------------+---------+
Azure Blob
Max known throughput of azure blobs is 60MB/s. Upload/download speeds are clamped to target storage read/write speeds.
Drill benchmarks
The filesystem cache was purged before every read test.
IO performance observed with iotop
Queries were chosen simple only for demonstration. Execution time growth for more complex queries is linear.
Sample queries:
-- Query A: Reading parquet
select sum(`Price`) as test from namespace.`Parquet/**/*.parquet`;
-- Query B: Reading CSV
select sum(CAST(`Price` as DOUBLE)) as test from namespace.`sales.csv`;
Results
+-------------+--------------------+----------+-----------------+
| Query | Source (namespace) | Duration | Disk read usage |
+-------------+--------------------+----------+-----------------+
| A (Parquet) | dfs(smb) | 14.8s | 2.8 - 3.5 MB/s |
| A (Parquet) | blob | 24.5s | N/A |
| A (Parquet) | local | 1.7s | 40 - 80 MB/s |
| --- | --- | --- | --- |
| B (CSV) | dfs(smb) | 22s | 30 - 60 MB/s |
| B (CSV) | blob | 29s | N/A |
| B (CSV) | local | 18s | 68 MB/s |
+-------------+--------------------+----------+-----------------+
Observations
When reading parquet, more threads will spawn but only cisfd process takes the IO performance.
Trying to tune parquet reader performance as described here but without any significant results.
There is a big peak of egress data at the time of querying parquets from azure storage, that exceeds parquet data size several times. The parquets have ~300MB but the egress peak for one read query is about 2.5GB.
Conclusion
Reading parquets from Azure Files is for some reason slowed down to ridiculous speeds.
Reading parquets from Azure Blob is even a bit slower.
Reading parquets from local filesystem is nicely fast, but not suitable for real use.
Reading CSV from any source utilizes storage throughput normally, therefore I assume some problem / misconfiguration of parquet reader.
The questions
What are the reasons that parquet read performance from Azure Storage is so drastically reduced?
Is there way to optimize it?
I assume that you would have cross checked IO performance issue using Azure Monitor and if the issue still persist, I would like to work closely on this issue. This may require a deeper investigation, so If you have a support plan, I request you file a support ticket, else please do let us know, we will try and help you get a one-time free technical support. In this case, could you send an email to AzCommunity[at]Microsoft[dot]com referencing this thread. Please mention "ATTN subm" in the subject field. Thank you for your cooperation on this matter and look forward to your reply.
I am loading freebase files into virtuoso. I have 88 folders. Every folder contains 4 files. I divided the files such way because I only have 8 GB of RAM.
#!/bin/bash
for i in {1..88}
do
isql 1111 dba dba exec="ld_dir('/data/data/${i}', '*.nt', 'http://freebase.com');"
isql 1111 dba dba exec="rdf_loader_run();"
isql 1111 dba dba exec="checkpoint;"
isql 1111 dba dba exec="commit WORK;"
isql 1111 dba dba exec="checkpoint;"
isql 1111 dba dba exec="delete from DB.DBA.load_list;"
done
Virtuoso.ini
# each buffer caches a 8K page of data and occupies approx. 8700 bytes of memory
# it's suggested to set this value to 65 % of ram for a db only server
# so if you have 32 GB of ram: 32*1000^3*0.65/8700 = 2390804
# default is 2000 which will use 16 MB ram
[Database]
MaxCheckpointRemap = 150000 (I have 8GB of RAM)
[TempDatabase]
MaxCheckpointRemap = 2000
NumberOfBuffers = 170000
MaxDirtyBuffers = 130000
Message
Checkpoint removed 628 MB of remapped pages, leaving 31 MB. Duration 31.21 s. To save this time, increase MaxCheckpointRemap and/or set Unremap quota to 0 in ini file.
Question
Why am I getting this message? and does it affect the loading process and building of the database?
update
#!/bin/bash
#clear list
isql 1111 dba dba exec="delete from DB.DBA.load_list;"
#load data
isql 1111 dba dba exec="ld_dir('/data/data, '*.gz', 'http://freebase.com');"
isql 1111 dba dba exec="set isolation='uncommitted';"
isql 1111 dba dba exec="rdf_loader_run();"
#checkpoint
isql 1111 dba dba exec="checkpoint;"
isql 1111 dba dba exec="commit WORK;"
isql 1111 dba dba exec="checkpoint;"
virtuoso.ini
MaxCheckpointRemap = 150000 (8GB RAM)
# Uncomment next two lines if there is 4 GB system memory free
NumberOfBuffers = 340000
MaxDirtyBuffers = 250000
# I have 6 cores
ThreadsPerQuery = 4
AsyncQueueMaxThreads = 10
#NO THREADS LEFT IN THE QUEUE
ThreadCleanupInterval = 0
ThreadThreshold = 0
#check point every 30 minutes.
CheckpointInterval = 30
It's best explained in the virtuoso docs, but in short:
The MaxCheckpointRemap parameter in the virtuoso.ini file controls how
many pages may be stored on a page other than their logical page.
This means that increasing the parameter will save you work (and time) on a checkpoint but reduces the data locality feature (data that is inserted together is saved close together).
I have a concern regarding Tempdb growth on one of my staging servers. Current size of Tempdb has grown to nearly 42 GB in size.
Per standards we have split the data file into a primary .mdf and 7 .ndf files along a .ldf. Currently all the 7 .ndf are more than 4 GB, and primary being over 6GB.
Tried restarting but it fills up fast.
DBCC shrink file (.ndf) is also not helping as it will shrink max to some MB.
Kindly help on how to resolve this..
Thanks
Kapil..
I have a job to full backup my database daily. I have usually between 200 to 300 new records in the database daily. However, size of backup file is always the same and is 7,472 KB. Why this is always fixed while the number of records is increasing. Database Files specifications are as following:
Logical Name ------ File Type----Filegroup---Initial Size----Autogrowth/ Maxsize
DBF-----------------Rows Data----PRIMARY-----4---------------by 15 MB, Unlimited
DBF_Data------------Rows Data----FG_Data-----221-------------by 15 MB, Unlimited
DBF_Index-----------Rows Data----FG_Index----3---------------by 15 MB, Unlimited
DBF_Log-------------Log--------Not Applicable- 793--------- by 10 percent,Limited to 2097152 MB
This is the code I wrote to make daily backup
declare #Date nvarchar(10)
select #Date=cast( cast(getdate() as date) as nvarchar)
declare #Path nvarchar(100)
select #Path= 'C:\PAT Daily BK\PAT' + #Date +'.Bak'
BACKUP DATABASE PAT TO DISK =#path
Until your database expands past the initial size, all backups will be the initial size (compression aside). After it starts expanding, if the expansion amount is set to an amount greater than the space consumed by the additional records then the backup can be expected to stay the same size for multiple days, but eventually will increase when it has to expand again.
SQL Server databases occupy all space allocated whether the data warrants it or not. After it uses up that space it will follow the rules you set for expansion.
Update: The above explains the observed behavior for encrypted databases, but not for unencrypted databases, as backups of unencrypted databases will be relative to the size of the database.