I'm currently doing some cleanup on an Azure environment and just wanted to check calculations. I have about 55 Databases on an elastic server, which is currently sitting at 3.64TB of the maximum 4TB. Having a look at each of the databases within this pool, I can see that they have their own Used / Allocated / Maximum sizes. Each of these is ranging between 0.1% and 80% of their allocated 250GB. Is the allocated size of the elastic pool dependent on the maximum sizes of each of the databases within that elastic pool? IE if I took a database that is using 1GB of 250GB and reduced the maximum size of this database within the elastic pool from the default 250GB down to 20GB, would it have any positive/negative implications? If anyone can suggest any good resources for azure environment maintenance plans it would be greatly appreciated as I'm coming from an AWS background.
Your allotted area will immediately expand; you do not need to be concerned; this is typical. You'll see that occupied space is always close to available space. The key is when the total amount of allotted and utilised storage space approaches the maximum storage size.
If the database size reaches the maximum size, use the following statement to raise the maximum size or use Azure portal to alter the maximum size.
ALTER DATABASE AzureDB2 MODIFY (EDITION='STANDARD', MAXSIZE= 50 GB)
The database receives a certain amount of log space depending on the tier. When you specify the storage size in vCore, you are given a fixed amount of space for your logs. If you pick 1TB of storage, for example, 300GB is set aside for logs.
Related
How can I limit maximum size on disk when using Ignite Persistence? For example, my data set in a database is 5TB. I want to cache maximum of 50GB of data in memory with no more than 500GB on disk. Any reasonable eviction policy like LRU for on-disk data will work for me. Parameter maxSize controls in-memory size and I will set it to 50GB. What should I do to limit my disk storage to 500GB then? Looking for something like maxPersistentSize and not finding it there.
Thank you
There is no direct parameter to limit the complete disk usage occupied by the data itself. As you mentioned in the question, you can control in-memory regon allocation, but when a data region is full, data pages are going to be flushed and loaded on demand to/from the disk, this process is called page replacement.
On the other hand, page eviction works only for non-persistent cluster preventing it from running OOM. Personally, I can't see how and why that eviction might be implemented for the data stored on disk. I'm almost sure that other "normal" DBs like Postgres or MySQL do not have this option either.
I suppose you might check the following options:
You can limit WAL and WAL archive max sizes. Though these items are rather utility ones, they still might occupy a lot of space [1]
Check if it's possible to use expiry policies on your data items, in this case, data will be cleared from disk as well [2]
Use monitoring metrics and configure alerting to be aware if you are close to the disk limits.
We have a SQL External Read replica set up and it isn't pulling in any updates from the external source, however, the size of the replica has increased each day (the size of the source database).
Looking at some metrics, there have been no Log Entries and no Read/Write operations, but there are CPU ticks and memory usage has been constant.
How can I determine what's happening? It's pulling in the entire size of the database every day, but not performing any actions. There are no SQL errors in the log.
In the Operations tab, there have been no operations listed except for the initial creation and seeding
From this other question:
With binary logs active, storage of your cloud sql will expand continuously. For anyone in the same situation, you can edit the instance and uncheck binary logs, after that the current binary logs will purge.
I would suggest checking if you have binary logging enabled, and check its size as stated in the documentation.
You can see the size of binary logs by using the SHOW BINARY LOGS MySQL command.
The impact of enabling binary logging are as follows:
Performance overhead
Cloud SQL uses row-based replication with MySQL flags sync_binlog=1 and innodb_support_xa=true. Therefore, an additional disk fsync is required for each write operation, which reduces performance.
Storage overhead
Storage of the binary logs is charged at the same rate as regular data. The binary logs are automatically truncated to the age of the oldest automated backup. Cloud SQL currently retains the most recent seven automated backups, and all on-demand backups. The size of the binary logs, and therefore the amount charged, depends on the workload. For example, a write-heavy workload consumes more binary log space than a read-heavy workload.
Another thing that could increase storage is to have Point-in-time recovery (PITR) enabled, as it uses binary logs. If the size of your binary logs are causing an issue for your instance, the documentation recommends:
You can increase the instance storage size, but the binary log size increase in disk usage might be temporary.
We recommend enabling automatic storage increase to avoid unexpected storage issues.
To delete the logs and recover storage, you can disable point-in-time recovery. Note, however, that decreasing the storage used does not shrink the size of the storage provisioned for the instance.
Logs are purged once daily, not continuously. Setting log retention to two days means that at least two days of logs, and at most three days of logs, are retained. We recommend setting the number of backups to one more than the days of log retention to guarantee a minimum of specified days of log retention.
I am creating a distribution server with separate drives. One of the drives is going to be for the SNAPSHOT folder. All my other SQL DATA Drives have 64k block Allocation sizes.
Should the SNAPSHOT drive also have 64K block Allocation size? I am thinking it does not matter for the SNAPSHOT, but this will be the first time that I am building a distribution server and I could not find a recommendation for it.
I have configured replication many times and never bothered about ,changing block size of drive,snapshot is stored..
I also don't see any mention in best practice's section for snapshot mentioning block size,so i would recommend go with defaults
I would like to use an EBS volume with data on it that I've been working with in an Ubuntu AMI in a RedHat 6 AMI. The issue I'm having is that RedHat says that the volume does not have a valid partition table. This is the fdisk output for the unmounted volume.
Disk /dev/xvdk: 901.9 GB, 901875499008 bytes
255 heads, 63 sectors/track, 109646 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/xvdk doesn't contain a valid partition table
Interestingly, the volume isn't actually 901.9 GB but 300 GB.. I don't know if that means anything. I am very concerned about possibly erasing the data in the volume by accident. Can anyone give me some pointers for formatting the volume for RedHat without deleting its contents?
I also just checked that the volume works in my Ubuntu instance and it definitely does.
I'm not able to advise on the partition issue as such, other than stating that you definitely neither need nor want to format it, because formatting is indeed a (potentially) destructive operation. My best guess would be that RedHat isn't able to identify the file system currently in use on the EBS volume, which must be advertized by some means accordingly.
However, to ease with experimenting and gain some peace of mind, you should get acquainted with one of the major Amazon EBS features, namely to create point-in-time snapshots of volumes, which are persisted to Amazon S3:
These snapshots can be used as the starting point for new Amazon EBS
volumes, and protect data for long-term durability. The same snapshot
can be used to instantiate as many volumes as you wish.
This is detailed further down in section Amazon EBS Snapshots:
Snapshots can also be used to instantiate multiple new volumes, expand
the size of a volume or move volumes across Availability Zones. When a
new volume is created, there is the option to create it based on an
existing Amazon S3 snapshot. In that scenario, the new volume begins
as an exact replica of the original volume. [...] [emphasis mine]
Therefore you can (and actually should) always start experiments or configuration changes like the one you are about to perform by at least snapshotting the volume (which will allow you to create a new one from that point in time in case things go bad) or creating a new volume from that snapshot immediately for the specific task at hand.
You can create snapshots and new volumes from snapshots via the AWS Management Console, as usual there are respective APIs available as well for automation purposes (see API and Command Overview) - see Creating an Amazon EBS Snapshot for details.
Good luck!.
The reason I ask it we have a dedicated RAID10 array with ~150GB for the tempdb (the "t" drive). It is only used for storing tempdb. The t drive isn't used by by SQL Server or any other process for anything else.
Our DBA has tempdb setup with 15GB initial size and autogrow 20% increments. Everytime the server starts it resized to 15GB and then over the course of the day grows to ~80GB (on average). Now IT is looking into making initial size larger say 30 or 40GB but given the drive is ONLY used for tempdb my thinking is why not "max it" right away.
Is the any negative effect to simply create 4 data files in the primary group for tempdb give them each an initial size of 30GB (120GB total), turn autogrow off and be done with it?
Are there any limits on SQL Server ability to span multiple tempdb data files in one query? i.e. will it cause problems if the tempdb has say 70GB total free but the file used by one process is full (30 of 30GB used)?
I would size them to about 100GB and leave autogrow on, this way you don't have to wait for it to grow every time, I would also add multiple files
Is the any negative effect to simply
create 4 data files in the primary
group for tempdb give them each an
initial size of 30GB, turn autogrow
off and be done with it?
Sounds like a good plan to me, however I would leave autogrow on just in case someone decides to do a sort operation on a big table which doesn't have an index on that column
See also here: http://technet.microsoft.com/en-us/library/cc966534.aspx
It is recommended to have .25 to 1
data files (per filegroup) for each
CPU on the host server.
This is especially true for TEMPDB
where the recommendation is 1 data
file per CPU.
Dual core counts as 2 CPUs; logical
procs (hyperthreading) do not.
We have found it very useful to create large TempDB data and log files. Any actions that limit server OS activities such as resizing TempDB increase server efficiencies. We have a 16 processor machine with 113 GB dedicated to TempDB data space. This machine is dedicated to large SSIS ETL processes, thus resulting in mass data operations.
The bulk of our ETL operations spawn up to 4 SQL threads. After initially configuring a TempDB file for each processor (16), we quickly realized via performance monitoring that our configuration was forcing SQL\windows to unnecessarily span the multiple TempDB files. We settled on 5 larger TempDB data files and realized performance improvements. We have since moved on to a 24 processor box and are using 8 TempDB files.
Please note that this is a large data migration server; I’m sure transaction-oriented systems would still benefit from the recommended 1-1 processor to TempDB file configuration. It should also be noted that having a large increase % on a TempDB file may force a critical transaction to take the windows operation hit and thus may not be appropriate for your specific application.