Backdrop
I develop a forecasting engine (time series) for different purposes. Processing, modeling and forecasting modules are written in Python, and data is currently stored in an Azure SQL database. Currently the database is General Purpose (vCore-based) service tier, Provisioned compute tier and Gen5 (12 vCores) hw config. I'm approaching the limit of maximum storage (approx 3 TB), but since I read almost the entire database daily (cold start models only), I do not see many other options than increasing the storage size. Truncating parts of historical data is out of the question.
Problem
At 12 vCores, maximum storage is approx 3 TB, and increasing vCores to enable the approx 4 TB maximum storage size is not feasible in a $-perspective (especially since it is storage, not compute, that is the bottleneck). I have read a bit about the alternative services / tiers on the Azure platform, and see that Hyperscale could possibly solve my problem: I can keep vCores untouched and have up to 100 TB storage. A config with zero secondary replicas (other things equal) will end up in the same $-range as before (see "Backdrop"). I get the impression that secondary replicas (read only nodes) are central to the Hyperscale architecture, so I'm not sure if such an outlined setup with zero secondary replicas is abuse / misuse. E.g. would it give the same performance, or could I expect a performance hit (even with the same vCore config)? Will the primary read / write node basically resemble a non-Hyperscale node? Other aspects I should think about? Adding a secondary replica (or several) might be relevant in the future (e.g. in combination with decreasing vCores), but is $-wise not an option atm.
Microsoft states that "The capability to change from Hyperscale to another service tier is not supported" (really?), so I would like clarify this to avoid doing a semi manual data migration (and delta migration) and having two instances side-by-side if the shait hits the fan. Given the scope of such a reconfig and the forecasting system as a whole, I feel it is not feasible to do small / full scale testing in advance to get representtive benchmarks. If there is anything else I should think about (related or semi related), feel free to point me in the right direction.
Since Hyperscale service tier is the newly added service tier in Azure SQL Database, it would be difficult to get the solid answer for your question.
It's true that it provides upto 100 TB database size, but the beauty is it will only charge for the capacity you use.
The Hyperscale service tier removes many of the practical limits
traditionally seen in cloud databases. Where most other databases are
limited by the resources available in a single node, databases in the
Hyperscale service tier have no such limits. With its flexible storage
architecture, storage grows as needed. In fact, Hyperscale databases
aren't created with a defined max size. A Hyperscale database grows as
needed - and you're billed only for the capacity you use. For
read-intensive workloads, the Hyperscale service tier provides rapid
scale-out by provisioning additional replicas as needed for offloading
read workloads.
You can have primary and secondary replica in Hyperscale service tier.
Primary replica serves read and write operations
Secondary replica provides read scale-out, high availability, and geo-replication
Secondary replicas are always read-only, and can be of three different types:
High Availability replica (recommended)
Named replica (in Preview, no guaranteed SLA)
Geo-replica (in Preview, no guaranteed SLA)
You should consider Hyperscale service tier because:
you need more size than 4 TB
require fast vertical and horizontal compute scaling, high performance, instant backup, and fast database restore
Note: Users may adjust the total number of high-availability replicas from 0-4, depending on the need.
You can check the Hyperscale pricing model here.
Considering above points, Hyperscale is the good, if not the best, solution for your requirement.
These two links will definitely help you to take your decision. Hyperscale service tier, Hyperscale secondary replicas
I'm one of the PM on the Azure SQL DB team. I see that UtkarshPal-MT already gave you extensive answer, so I'm chimining in on to complete the picture. Azure SQL DB Hyperscale offers different type of secondary replicas. The replicas that can help to get an higher SLA are named High-Availability replica. You can use 0 replicas without any issue. What will happen is that if the primary replicas for any reason is not available, we need to spin up a new (compute) replica from scratch (as there are no HA replica available) so that can take some time (minutes, usually) which means that your service will not be available that amount time. Having an HA replica, drastically diminish the time in which the database is not available.
You can read all the details here:
https://learn.microsoft.com/en-us/azure/azure-sql/database/service-tier-hyperscale-replicas?tabs=tsql
The SLA are defined here:
https://www.azure.cn/en-us/support/sla/sql-data/
Regarding the performances: unless you are specifically using secondary replicas also to offload read-only workload, you'll not have performance hit by not having an HA replica
Related
Does Horizontal scaling(scale out) option available in AZURE SQL Managed Instance ?
Yes, Azure SQL managed instance support scale out.
You you reference the document #Perter Bons have provided in comment:
Document here:
Scale up/down: Dynamically scale database resources with minimal downtime
Azure SQL Database and SQL Managed Instance enable you to dynamically
add more resources to your database with minimal downtime; however,
there is a switch over period where connectivity is lost to the
database for a short amount of time, which can be mitigated using
retry logic.
Scale out: Use read-only replicas to offload read-only query workloads
As part of High Availability architecture, each single database,
elastic pool database, and managed instance in the Premium and
Business Critical service tier is automatically provisioned with a
primary read-write replica and several secondary read-only replicas.
The secondary replicas are provisioned with the same compute size as
the primary replica. The read scale-out feature allows you to offload
read-only workloads using the compute capacity of one of the
read-only replicas, instead of running them on the read-write
replica.
HTH.
Yes scale out option is available in Business Critical(BC) tier. The BC utilizes three nodes. One is primary and two are secondary. They use Always on on the backend. If you need to utilize for reporting, just ApplicationIntent=Readonly in the connection string and your application will be routed one of the secondary nodes.
What I'd like to achive is to be able to scale out Azure SQL Database.
Business Critical tier has this feature to enable several read-only replicas. This is a great feature that would let me offload some traffic over to those replicas
The problem for me is that I don't understand how to manage those replicas and I don't understand how load balancing works there. Basically, I should be able to manage how many replicas there are, I probably need to have around 10 of replicas and have traffic equality balanced across them
Is this something that I could do?
If you look at the note here, it says
In Premium and Business Critical service tiers, only one of the read-only replicas is accessible at any given time. Hyperscale supports multiple read-only replicas.
This means Premium and Business critical service tiers may have multiple replicas (3-4) but only 1 of them is accessible as read only. There is no control as to which one and there is no load balancing capabilities. It is only good for use if there is a separate application which require read access only (example analytical workloads).
For Hyperscale you can refer to this.
Hyperscale allows for 1-4 secondaries(1 by default). The link states
If more than one secondary replica is present, the workload is distributed across all available secondaries.
There is no additional information and it seems the the control to load balance is abstracted away from us.
You can definitely not achieve your requirement of 10 read replicas from any of these configurations.
We want to have secondary read only database for Analytics,Reporting, Monitoring, and exposing to another application. Since we are using Azure DB, DTU is increasing because of these reads. So I want to have secondary database(Read scale out), so that I can share this secondary DB credential to them, so that it will not have any impact on primary database. So Please help me to setup the secondary database(Read scale out) in Azure. I have heard about geo replication, but it is only for certain region.
The capability to use a local readable secondary is in preview. It will work on larger reservation sizes (premium and up, though perhaps some of the standards will work). This is not limited to certain regions today.
Active geo dr can also be used to do reads scale out (but please note that it costs money for the dr copy since it gives you disaster recovery not just read scale out).
Instructions for both can be found here:
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-read-scale-out
Intra-db resource governance would be the other way to split mixed workloads. However, this feature does not currently exist in sql azure (though it is a roadmap item).
As in the question, I want to increase number of queries per second on GCS. Currently, my application is on my local machine, when it runs, it repeatedly sends queries to and receives data back from the GCS server. More specifically, my location is in Vietnam, and the server (free tier though) is in Singapore. The maximum QPS I can get is ~80, which is unacceptable. I know I can get better QPS by putting my application on the cloud, same location with the SQL server, but that alone requires a lot of configuration and works. Are there any solutions for this?
Thank you in advance.
colocating your application front-end layer with the data persistence layer should be your priority: deploy your code to the cloud as well
use persistent connections/connection pooling to cut on connection establishment overhead
free tier instances for Cloud SQL do not exist. What are you referring to here? f1-micro GCE instances are not free in Singapore region either.
depending on the complexity of your queries, read/write pattern, size of your dataset, etc. performance of your DB could be I/O bound. Ensuring your instance is provisioned with SSD storage and/or increasing the data disk size can help lifting IOPS limits, further improving DB performance.
Side note: don't confuse commonly used abbreviation GCS (Google Cloud Storage) with Google Cloud SQL.
I searched online for awhile about what is "Excessive resource usage" on SQL Azure, still cannot get an idea.
Some articles suggest query takes too long, too much memory etc will cause "Excessive resource usage". But If I use simple query, simple data structure, what will happen?
For example: I get a 1G SQL Azure as session state. Since session is a very small string, and save/delete all the time, I don't think it will grow to 1G for millions of session simultaneously. You can calculate, for 1 million session, 20 char each, only take 20M space, consider 20 minutes expire etc. Cannot even close to 1G. But the queries, should be lots and lots. Each query will be very simple and fast by index.
I wanna know, if this use will be consider as "Excessive resource usage"? Is there any hard number to limit you on the usage?
Btw, as example above, if all happen in same datacenter, so all cost is 1G database which is $10 a month, right?
Unfortunately the answer is 'it depends'. I think that probably the best reference (with guidance) on the SQL Azure Query Throttle is here: TechNet Article on SQL Azure Perormance This will povide details about the metrics that are monitored and the mechanism of the throttle.
The reason that I say it depends is that the throttle is non-deterministic for any given user. This is because the throttle will be activated based on the total load on the node (physical SQL Server in Azure DC). While the subscribers who will get throttled are the subscribers delivering the greatest load the level at which the throttle kicks in will depend on the total load on the node. SO if you are on a quiet node (where other tenant DBs are relatively inactive) then you will be able to put through a bunch more throughput than if you are on a busy node.
It is very appealing to use 1GB SQL Azure DBs for session state storage; you've identified the cost benefits. You are taking a risk though. One way to mitigate this risk is to partition across at least two SQL Azure 1GB DBs and adjust the load yourself based on whether one of the DBs starts hitting the throttle.
Another option if you want determinism for throughput is to use the WIndows Azure Cache to back your sesion state store. The Cache has hard pre-defined limits for query throughput so you can plan for it more easily Azure Caching FAQ including Limits. The Cache approach is probably a bit more expensive but with a lower risk of problems.