I have a view in my database which returns the last updated value for a number of tables. This is to prevent those tables being queried directly for changes by the application, the application is in a multi user environment and these tables may be frequently updated for short bursts, then ignored for hours at a time.
I have a view called vwLastUpated
CREATE VIEW vwLastUpdated as
SELECT Tasks, Items, ListItems FROM
(Select Max(ModifiedTime) as Tasks from tblTasks) a CROSS JOIN
(Select Max(ModifiedTime) as Items from tblItems) b CROSS JOIN
(Select Max(ModifiedTime) as ListItems from tblListItem) c
Clients are configured to call this view around every 10-30 seconds (user configurable), the trouble is, when there are a lot of clients (around 80 at one site), the view gets hit very, very frequently, and can sometimes take a few milliseconds to run, but sometimes takes 200-300 ms to run if updates are occurring, this seems to be slowing down the front end during heavy use. The tables are properly indexed on ModifiedTime DESC.
These sites are using SQL Express in some cases, at other sites they have the full version of SQL and I can design the view differently and use Agent to update a common table (tblLastUpdated) where Agent updates the table directly by essentially running the above query every 5 seconds.
What could I do to make the process more efficient and reduce the load on the database server where SQL Express is used?
The client sites are on a minimum of SQL Server 2008 (up to SQL 2012)
Do you have an index on the following fields?
tblTasks(ModifiedTime)
tblItems(ModifiedTime)
ListItems(ModifiedTime)
This should ensure pretty good performance.
If you do, and there is still the problem of interacting locks, then you might consider having another table with this information. If you do updates/inserts directly on the tables, this would require triggers. If you wrap updates and inserts in stored procedures, then you can do the changes there.
This would basically be turning the view into a table and updating that table whenever the data changes. That update should be very fast and have minimal interaction with other queries.
Related
I execute SELECT INTO query where my source data are in other database than the table I insert to (but on the same server).
When I execute the query using the same database where my source data are (USE DATABASE_MY_SOURCE_DATA), it completes in under a minute. When I change the database to the database where my target table sits, it doesn't complete in 10 minutes (I don't know the exact time because I cancelled it).
Why is that? Why is the difference so huge? I can't get my head around it.
Querying cross-database, even using a linked server connection, is always likely (at least in 2021) to present performance concerns.
The first problem is that the optimizer doesn't have access to estimate the number of rows in the remote table(s). It's also going to miss indexes on those tables, resorting to table scans (which tend to be a lot slower on large tables than index seeks).
Another issue is that there is no data caching, so the optimizer makes round-trips to the remote database for every necessary operation.
More information (from a great source):
https://www.brentozar.com/archive/2021/07/why-are-linked-server-queries-so-bad/
Assuming that you want this to be more performant, and that you are doing substantial filtering on the remote data source, you may see some performance benefit from creating - on the remote database - a view that filters to just the rows you want on the target table and query that for your results.
Alternatively (and likely more correctly) you should wrap these operations in an ETL process (such as SSIS) that better manages these connections.
For the purpose of my application I have created an Azure Function that connects with my Dataverse environment, query data with SELECT from different tables (I create new records based on records in tables A, B, C), stores the result into a list and then saves those records into same Dataverse environment but into another table (lets say D). I decided for such solution because Power Automate was creating those new records too slowly.
It works fine, however when there are too many requests (more than 2-3 users work with application and run Azure Functions) the save into Dataverse begins to be too slow too.
So I am thinking about another way to save and store those records. What important is that those records in table D are only for calculation purpose, users do not work with them or edit them. This is why I am thinking about creating SQL Datables table, store those records (only from table D) there, and just change connection in my application where needed.
Can you suggest me the most efficient way to do this? In a nutshell what I need is:
connect to Dataverse and query data from tables A, B, C. Result of this query will be records for table D
save the result of the query into SQL Database table (table D)
There are quite a few things to consider here.
If users don't use data of table D, could you maybe run this operation overnight or at a time when there is low traffic and slow performance of this operation is acceptable?
Have you considered using SQL views? Do you really need to store the computed data?
Perhaps you are inserting 1 item at a time? Are you using the Sql Bulk Copy Class?
Bulk Insert In SQL Server From C#
Observe the CPU utilisation of your server during this operation. If probably shoots to 100%. You want to hit 70% average utilisation for good trade-off between performance and cost. So another option is to scale up.
I am having a System Setup in ASP.NET Webforms and there is Acccounts Records Generation Form In Some Specific Situation I need to Fetch All Records that are near to 1 Million .
One solution could be to reduce number of records to fetch but when we need to fetch records for more than a year of 5 years that time records are half million, 1 million etc. How can I decrease its time?
What could be points that I can use to reduce its time? I can't show full query here, it's a big view that calls some other views in it
Does it take less time if I design it in as a Linq query? That's why I asked Linq vs Views
I have executed a "Select * from TableName" Query and its 40 mins and its still executing table is having 1,17,000 Records Can we decrease this timeline
I started this as a comment but ran out of room.
Use the server to do as much filtering for you as possible and return as few rows as possible. Client side filtering is always going to be much slower than server side filtering. Eg, it does not have access to the indexes & optimisation techniques that exist on the server.
Linq uses "lazy evaluation" which means that it builds up a method for filtering but does not execute it until it is forced to. I've used it and was initially impressed with the speed ... until I started to access the data it returned. When you use the data you want from Linq, this will trigger the actual selection process, which you'll find is slow.
Use the server to return a series of small resultsets and then process those. If you need to join these resultsets on a key, save them into dictionaries with that key so you can join them quickly.
Another approach is to look at Entity Framework to create a mirror of the server database structure along with indexes so that the subset of data you retrieve can be joined quickly.
I know you can run SELECT queries on top of SELECT queries in Access, but the application also provides the Make Table query type.
I'm wondering what the benefits/reasons for using Make Table might be?
You would usually use Make Table for performance reasons. If you have a fairly complex query that returns a subset of your table's data, and that you may need to retrieve multiple times, it can be expensive to re-run the query multiple times.
Using Make Table allows you to incur the cost of running the expensive query once, and make a copy of the query results into a table. Querying this copy would then be a lot less expensive than running your original expensive query.
This is usually a good option when you don't expect your original data to change frequently, or if you don't care that you are working of a copy of the data that may not be 100% up-to-date with the original data.
Notice what the following article on Create a make table query has to say:
Typically, you create make table queries when you need to copy or archive data. For example, suppose you have a table (or tables) of past sales data, and you use that data in reports. The sales figures cannot change because the transactions are at least one day old, and constantly running a query to retrieve the data can take time — especially if you run a complex query against a large data store. Loading the data into a separate table and using that table as a data source can reduce workload and provide a convenient data archive. As you proceed, remember that the data in your new table is strictly a snapshot; it has no relationship or connection to its source table or tables.
The main defense here is that a make table query creates a table. And when you done with the table then effort and time to delete that table and recover the VERY LARGE increase in the database file will have to occur. For general reports and a query of data make much more send. A comparison would be to build a NEW garage every time you want to park your car.
The database engine and query system can fetch and pull rows at a very high rate and those results are then able to be rendered into a report or form, and this occurs without having to create a temp table. It makes little sense to go through all of the trouble of having the system create a WHOLE NEW table for such results of data when they can with ease be sent to a report.
In other words creating a whole table just to display or use some data that the database engine already fetched and returned makes little sense. A table is a set of rows that holds data that can be updated and the results are permanent. A query is a “on the fly” results or sub set of data that only exists in memory and is discarded after you use the results.
So for general reporting and display of data, it makes no sense to create a temp table. MUCH WORSE of an issue is that if you have two users wanting to run a report, if they both need different results and you send the results to the SAME temp table, then you have a big mess and collision between the two users. So use of a temp table in Access for the most part makes little sense, and this is EVEN MORE so when working in a multi-user environment. And as noted, once the table is created, then after you are done you need to delete and remove the table. And with many users in a multi-user database this becomes even more of a problem and issue.
However in a multi-user environment as pointed out that if the resulting data needs additional processing, then sending the results to a temp table can be of use. This approach however suggests that EACH USER has their own front end and own copy of the application side. And better is that the temp table is created outside of the front end application that resides on each computer. Since the application part (front end) is placed on each computer, then creating of a temp table does not occur in the production database (back end) and as a result you can have multiple users function correctly without each individual user creating a temp table in the production back end database. So if one is to adopt a make table query, it likely should occur on each local workstation and not in the back end database when you have a multiple user database application.
Thus for the most part a make table and that of reports and query of data are VERY different goals and tasks. You don't want nor as a general rule create a whole brand new table for a simple query. In a multi user database system the users might run 100's of reports in a given day and FEW if any systems will send such data to a temp table in place of sending the query results directly to the report.
It creates a table - which is useful if you have a need for that table which you may have for temporary use where you have to modify the data for calculations or further processing while not disturbing the original data.
I've been put in charge of a 10-year old transactional system where the majority of the business logic is implemented at the database level (triggers, stored procedures, etc). Win2000 server, MSSQL 2000 Enterprise.
No immediate plans for replacing or updating the system are being considered.
The core process is a program that executes transactions - specifically, it executes a stored procedure with various parameters; let's call it sp_ProcessTrans. The program executes the stored procedure at asynchronous intervals.
By itself, things work fine, but there are 30 instances of this program on remotely located workstations, all of them asynchronously executing sp_ProcessTrans and then retrieving data from the SQL server. Execution is pretty regular - ranging 0 to 60 times a minute, depending on what items the program instance is responsible for.
Performance of the system has dropped considerably with 10 years of data growth: the reason is the deadlocks, specifically deadlock wait times, on the Employee table.
I have discovered:
In sp_ProcessTrans's execution, it selects from an Employee table 7 times
The select is done on a field that is NOT the primary key
No index exists on this field. Thus a table scan is performed 7 times per transaction
So the reason for deadlocks is clear. I created a non-unique ordered clustered index on the field (almost unique, NUM(7), very rarely changes). There was immediate improvement in the test environment.
The problem is that I cannot simulate the deadlocks in a test environment. I'd need 30 workstations, and I'd need to simulate 'realistic' activity on those stations, so visualization is out.
I need to know if I must schedule downtime.
Creating an index shouldn't be a risky operation for MSSQL, but is there any danger (data corruption, extra wait time, etc.) in creating this field index on the production database while the transactions are still taking place? I can select a time when transactions are fairly quiet through the 30 stations.
Are there any hidden dangers I'm not seeing? (I'm not looking forward to restoring the DB if something goes wrong. It would take a lot of time with 10 years of data.)
Data corruption shouldn't be an issue, but if you try adding an index to a live production table you are likely to experience problems as the table will not be responsive to queries during the index creation. Creating an index will apply an exclusive table lock until it is complete, and the time this takes will depend on numerous factors (especially the number of rows).
scheduled downtime is strongly recommended and also a good habit to get into. And obviously backup taken, and a plan in case you have to undo what you're intending.