I have a website that need a database to store some user information and a blob storage to save some files.
I want to minimize the cost as much as possible so I played around in Microsoft Azure Pricing Calculator with a Azure SQL Database. For the database I think that over it's hole lifetime 2GB of storage would be enought.
I arrived to 2 options that where dirt cheap but I don't really understand what it gives me.
First is with a serverles computer for 3600 seconds (of runtime?)
Is that time the time that my database is processing the request? For example if I have a select statement that takes 1 sec to complete I'll be left with 3599 sec for that month?
If that's the case what happens if I run out of time?
Second option is using a Hardware Type: Gen 4
but for this one I don't have any other options to configure my needs. Is this obsolete? Can I rely on it for production?
If you need a very cheap one use the Basic or S0.
Keep in mind that Basic are very slow: try to connect to it through SSMS.
Serverless is for databases that you pause for 3/4 of the day. It might be the case for you but keep in mind that when you use them they will cost a lot. I don't think this will be suitable for you.
Related
I'd like to increase the database Max Size while the application utilizing the database is running.
Will scaling it up have any noticeable impact? This is a small scale application; a minute of down time would be OK. More than that and I'd have to push off the upgrade to outside of business hours. Thank you.
As you said, a minute of down time would be OK. Scaling the database took you 20 seconds, I think this latency is within your acceptable time range.
The document Scale single database resources in Azure SQL Database only can give you the reference max time : "Typically, less than 5 minute".
The latency usually depended on the current Azure SQL database price tier and the target server price tier which you want scale up to.
We can not calculate the specific latency time for you. It's all happening in Azure background.
Hope this helps.
Is there any way to change query execution time on tableau online. I have to extract 100 M records from Google Bigquery which will take around 3 hr on tableau online but my process stops after 2 hrs with error com.tableausoftware.nativeapi.dll.TableauCancelException: Operation canceled
which is query execution time limit
My #1 suggestion is to make sure your BigQuery tables and queries are optimized as much as possible. Your question doesn't give details on your schema design, query, or indexing, but that is the first place to start. Look at the query plan and timeline to get an idea of where the bottlenecks are in the query execution.
My #2 suggestion is to look at what aggregations your visualizations are doing and see if you can offload that aggregation to the database instead. That will reduce the amount of data needed in your extract file.
Tableau has an excellent Knowledge Base article on this topic, found at https://kb.tableau.com/articles/issue/error-operation-cancelled-query-time-resource-limit-7200-refreshing-tableau-online-extract. Much of my answer below is copy/pasted from the KB article.
Here are some options that the KB article gives. Options 3 is easy to do and can make a noticeable difference. Option 2 may be easy to implement, depending on your schema design.
Option 1
Use live connections instead of extracts. See Keep Data Fresh for identifying which cloud-based data sources are capable of connecting live on Tableau Online.
Option 2
Use incremental extracts.
Option 3
Decrease the size of the data extract. Do this by hiding unused columns and adding filtering to the extract.
Option 4
Refresh the published extract using Tableau Bridge, the Command Line Utility, or manually using Tableau Desktop.
Option 5
Try scheduling extract refreshes at different times of day, or at night.
I am just starting with biquery, my DB is small (10K of rows 1 table) and my queries are simple count and group by.
Its takes and average of 3-4 sec per request but sometimes its jumps to 10 and event 15sec
I am querying from amazon linux server in Irland using the BQ tool.
Is it possible to get results faster (under 1sec) so I will be able to present my webpages faster.
1) Big Query is a highly scalable database, before being a "super fast" database. It's designed to process HUGE amount of data distributing the processing among several different machines using a technique named Dremel. Because it's designed to use several machines and parallel processing, you should expect to have super-scalability with a good performance.
2) BigQuery is an asset when you want to analyze billions of rows.
For example: analyzing all the wikipedia revisions in 5-10 seconds isn't bad, is it? But even a much smaller table would take about the same time, even if has 10k rows.
3) Under this size, you'll be better off using more traditional data storage solutions such as Cloud SQL or the App Engine Datastore. If you want to keep SQL capability, Cloud SQL is the best guess.
Sybase IQ is often installed in a single database and it doesn't use Dremel. That said, it's going to be faster than Big Query in many scenarios...as designed.
4) Certainly the performance differ from a dedicated environment. You get your dedicated environment for 20K$ a month.
That's the expected behaviour. In BigQuery you are using a shared infrastructure, so depending on the use at the moment you will get better or worse response time. Actually batch queries (those not needing interactivity) are encouraged and rewarded by not adding up to your quota.
You typically don't use BigQuery as your main database to show data in your web application. Depending on what you want to do, BigQuery can be a Big Data storage and you should have another intermediate store where you could store computed results to display to your users. Or maybe in your use case you don't really need BigQuery and there is a better solution.
In any case, you are not going to be able to avoid a few seconds wait (even if you go Premium, you get more guarantees about the service, but in no case a service fast enough as to be your main backend for a webapp)
I decided I wanted to try out Microsoft SQL Azure, as many people have talked very highly about it. It should be fast, flexible, cheap and many other things.
I got it up and running, migrated my data to Azure and hooked up the connectionstring. I tried to run some queries on the database, and was shocked about how slow even simple queries were. A "SELECT *" from a table with 700 rows took 7 seconds. My page also seems extremely slow, compared to when I used a localhost managent studio or a database on a shared hosting.
Now, when I setup my server, I couldn't pick a physical position. However, I live in Denmark, and I can see the server is the "South Central US". This might be the issue.
I don't use any stored procedures (so I guess no parameter sniffing).. I can also see my indexes is transfered succesfully.
Any ideas on what to do? Any performance things I am missing?
I ran into this very issue the last few days. Change your database tier from basic to standard and you will see a HUGE increase in performance. I am working on a query intensive dashboard at the moment, it took a 20 sec response time down to 2 sec response.
I've used Azure now for the last many years, and my original question is pretty much solved.
My main take-aways after dealing with Azure databases for a while:
It's extremely important that your application and database is placed in the same region. If not, then you will have a slow application. Recently I had an API and app running on two different regions - it took ~1 second for every response.. After moving it to same, it was instant
If your application has a high load, it's often a good idea to upgrade. This happens earlier than you might expect
Pick the nearest region - it really matters
I searched online for awhile about what is "Excessive resource usage" on SQL Azure, still cannot get an idea.
Some articles suggest query takes too long, too much memory etc will cause "Excessive resource usage". But If I use simple query, simple data structure, what will happen?
For example: I get a 1G SQL Azure as session state. Since session is a very small string, and save/delete all the time, I don't think it will grow to 1G for millions of session simultaneously. You can calculate, for 1 million session, 20 char each, only take 20M space, consider 20 minutes expire etc. Cannot even close to 1G. But the queries, should be lots and lots. Each query will be very simple and fast by index.
I wanna know, if this use will be consider as "Excessive resource usage"? Is there any hard number to limit you on the usage?
Btw, as example above, if all happen in same datacenter, so all cost is 1G database which is $10 a month, right?
Unfortunately the answer is 'it depends'. I think that probably the best reference (with guidance) on the SQL Azure Query Throttle is here: TechNet Article on SQL Azure Perormance This will povide details about the metrics that are monitored and the mechanism of the throttle.
The reason that I say it depends is that the throttle is non-deterministic for any given user. This is because the throttle will be activated based on the total load on the node (physical SQL Server in Azure DC). While the subscribers who will get throttled are the subscribers delivering the greatest load the level at which the throttle kicks in will depend on the total load on the node. SO if you are on a quiet node (where other tenant DBs are relatively inactive) then you will be able to put through a bunch more throughput than if you are on a busy node.
It is very appealing to use 1GB SQL Azure DBs for session state storage; you've identified the cost benefits. You are taking a risk though. One way to mitigate this risk is to partition across at least two SQL Azure 1GB DBs and adjust the load yourself based on whether one of the DBs starts hitting the throttle.
Another option if you want determinism for throughput is to use the WIndows Azure Cache to back your sesion state store. The Cache has hard pre-defined limits for query throughput so you can plan for it more easily Azure Caching FAQ including Limits. The Cache approach is probably a bit more expensive but with a lower risk of problems.