Clustered Index Update Slow Update On Large Table - sql

I am having a problem updating a large table with millions of rows please advice to reduce the update time.
Table definition
CREATE TABLE [dbo].[tbl_sms_job_detail](
[JobDetailID] [int] IDENTITY(1,1) NOT NULL,
[JobID] [int] NULL,
[DistributorID] [int] NULL,
[ResellerID] [int] NULL,
[CustomerID] [int] NULL,
[SenderID] [nvarchar](50) NULL,
[PhoneNumber] [nvarchar](100) NULL,
[SMSMessage] [nvarchar](1000) NULL,
[MessageType] [nvarchar](50) NULL,
[MessageLength] [int] NULL,
[MessageParts] [int] NULL,
[ClientRate] [decimal](18, 5) NULL,
[ClientCost] [decimal](18, 5) NULL,
[ResellerRate] [decimal](18, 5) NULL,
[ResellerCost] [decimal](18, 5) NULL,
[DistributorRate] [decimal](18, 5) NULL,
[DistributorCost] [decimal](18, 5) NULL,
[RouteDetailID] [int] NULL,
[SMSID] [nvarchar](200) NULL,
[DLRStatus] [nvarchar](100) NULL,
[ErrorCode] [int] NULL,
[ErrorDescription] [nvarchar](2000) NULL,
[SentDate] [datetime] NULL,
[SentDateUTC] [datetime] NULL,
[SMSSource] [nvarchar](50) NULL,
[SMSType] [nvarchar](100) NULL,
[APISMSID] [int] NULL,
[DLRDate] [datetime] NULL,
[DLRDateUTC] [datetime] NULL,
CONSTRAINT [PK_tbl_sms_job_detail] PRIMARY KEY CLUSTERED
(
[JobDetailID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
CREATE NONCLUSTERED INDEX [NonClusteredIndex-20170919-173756] ON [dbo].[tbl_sms_job_detail] ( [JobID] ASC, [DLRStatus] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) GO
CREATE NONCLUSTERED INDEX [NonClusteredIndex-20170919-174142] ON [dbo].[tbl_sms_job_detail]
(
[SMSID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
GO
Update Procedure
CREATE Procedure [dbo].[sp_update_message_status]
#SMSID nvarchar(200),
#DLRStatus nvarchar(100),
#ErrorCode int,
#ErrorDescription nvarchar(2000)
AS
UPDATE tbl_sms_job_detail SET DLRStatus = #DLRStatus, ErrorCode = #ErrorCode, ErrorDescription = #ErrorDescription WHERE SMSID = #SMSID
Execution Plan
This Procedure is called up to 1000 times in several minutes and some of them fail to update as it takes time to update the previous record what can be done to increase the update of a record in this table.

I suspect that the issue is not being caused by the actual clustered index but by the effect the update query has.
MSSQL is a page based storage system. When adding records to the table, as the clustered index is on the field [JobDetailID] [int] IDENTITY(1,1) NOT NULL, each new record is applied to the last page currently at the table (if it will fit) or a new page is tacked onto the end of the table and the record stored in it.
Assuming that [DLRStatus] and/or [ErrorDescription] start of a null or empty strings, when the update sproc runs, it has to find some space in the page that the record is already in to store the new value in. SQL keeps a little space in each page file for such purpose, but when that space is used up, it will have to do a page split - splitting the contents of the one page file into the existing page file and a newly created blank page file. As the primary key is clustered, this new page file has to be inserted so it keeps the records stored in the table in the clustered index order. It is quite likely that this page splitting is at the root of the problem.
The amount of space that SQL keeps back before creating a new page is configurable, therefore one solution is to initially create the page files with plenty of 'expansion' room. On an index it is called the fill factor, but I am not sure what the correct term is for data pages (probably still a fill factor, but not sure).
Another alternative is to store the returned error information in a separate table and then store the primary key for the 'error information' record in table [tbl_sms_job_detail]. As long as the key is not a nvarchar / varchar (and who would do this anyway), the space required in the page file will already be reserved. Thus recording the error information requires appending the variable text information to the end of the last page file for the new table and updating a foreign key in your original table that already has space reserved for it and so doesn't trigger of a page slit.

Related

Strange Query Plan for SQL Query - Clustered Index Seek

I've got a really strange issue with a Query plan generated for a very simple SQL query. The query is searching a full text index, and returning the count of records.
For some reason, this SQL query is producing a Non Clustered Scan on an index, which I don't believe it is optimal to do. I believe that for the count, as the Primary Key is in the full text index, a clustered seek would be all that is required.
Would anyone have any suggestions on why such a query plan is being used?
Odd thing is, with slight different variants of the SQL, sometimes it uses the Clustered Index (which is really fast), sometimes it uses the Non Clustered Seek.
Here's the query:
EXEC sp_executesql N'SELECT count(T.[ID])
FROM [dbo].[Item] AS T
WHERE CONTAINS (
(
T.[Description]
)
,#P0M
)
'
,N'#P0M nvarchar(4000)'
,#P0M = N'"mouse*"'
Here's the Query Plan: https://i.stack.imgur.com/1XJcf.png As you can see, the Non Clustered Seek costs 51%, plus 8% Parallelism and 15% Hash match on the Bitmap.
The table has a lot of data. Over 3 million records.
Many thanks
Edit:
Here's the plan: https://www.brentozar.com/pastetheplan/?id=HyiABrg1K
Here's the table definition:
CREATE TABLE [dbo].[Item](
[ID] [uniqueidentifier] NOT NULL,
[Description] [nvarchar](500) NOT NULL,
[Manufacturer] [uniqueidentifier] NOT NULL,
[Manufacturer Name] [nvarchar](100) NULL,
[Manufacturer Item No.] [nvarchar](50) NOT NULL,
[BC Item No.] [varchar](20) NULL,
[CRM Item No.] [varchar](50) NULL,
[Category] [uniqueidentifier] NULL,
[Version No.] [varchar](50) NULL,
[Blocked] [bit] NULL,
[UNSPSC Code] [int] NULL,
[Barcode] [char](13) NULL,
[Last Update Date Time] [datetime] NULL,
[Weight (kg)] [decimal](18, 3) NULL,
[RRP] [decimal](18, 2) NULL,
[RRP Currency] [uniqueidentifier] NULL,
[timestamp] [timestamp] NOT NULL,
CONSTRAINT [PK_Item] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 99, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
There's also lots of foreign keys.
Here's the only other index:
CREATE UNIQUE NONCLUSTERED INDEX [Manufacturer Part No] ON [dbo].[Item]
(
[Manufacturer] ASC,
[Manufacturer Item No.] ASC
)
INCLUDE([ID]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY]
GO
After lots of digging, I found this Query Optimizer Gone Wild - Full-Text Search Query Plans.
Looks like it is by design. A ContainsTable query doesnt need to join in order to get the row count.
the CONTAINS function must also scan a index on the source table to get the count

Why columns value change into NULL when try to update only one column?

This is my table which I created in sql server...
CREATE TABLE [dbo].[Addresses_Table](
[AddressID] [int] IDENTITY(1,1) NOT NULL,
[BuildingName] [varchar](300) NULL,
[UnitNumber] [nvarchar](50) NULL,
[StreetNumber] [varchar](20) NULL,
[StreetName] [varchar](200) NULL,
[Suburb] [varchar](100) NULL,
[POBox] [varchar](20) NULL,
CONSTRAINT [PK_Addresses_Table] PRIMARY KEY CLUSTERED (
[AddressID] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = OFF, ALLOW_PAGE_LOCKS = OFF) ON [PRIMARY] ) ON [PRIMARY]
INSERT INTO [dbo].[Addresses_Table] (BuildingName,UnitNumber,StreetNumber, StreetName, Suburb) VALUES ('mybuilding', '101', '12','Street 1', 'TEST12')
When I try to update 'POBox' column value into '1234' (any value),
BuildingName, UnitNumber, StreetNumber, StreetName turn into NULL values.
I have attached my sql update query and results of it.
Please help me out to solve this problem....
It seems there is a trigger on the table that sets StreetName to NULL after the update. This is evidenced by the SSMS output that shows 2 rowcount messages, one when the row is updated directly and the other when the same row is updated by the trigger.
The trigger makes sense from a data perspective since StreetName does not apply to a PO box address.

Attempted to read or write protected memory - .NET EF + Web API

I have a relatively simple application that pulls SQL data via a .NET Web API using EF6. There are three tables in the database:
Product
CREATE TABLE [dbo].[Product]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[Name] [varchar](255) NOT NULL,
[FullName] [varchar](max) NOT NULL,
[Description] [varchar](max) NOT NULL,
[BranchID] [int] NOT NULL,
[ProgramID] [int] NOT NULL,
[TechnologyPlatformID] [int] NOT NULL,
[StatusID] [int] NOT NULL,
[FunctionID] [int] NOT NULL,
[ProgramManagerID] [int] NOT NULL,
[TypeID] [int] NOT NULL,
[Vendor] [varchar](max) NOT NULL,
[VendorPOC] [varchar](max) NOT NULL,
[URL] [varchar](255) NULL,
[Code] [varchar](50) NULL,
[CreatedBy] [int] NOT NULL,
[CreateDate] [datetime] NOT NULL,
[ModifiedBy] [int] NOT NULL,
[ModifiedDate] [datetime] NOT NULL,
CONSTRAINT [PK_Product]
PRIMARY KEY CLUSTERED ([ID] ASC)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[Product] WITH CHECK
ADD CONSTRAINT [FK_Product_Function]
FOREIGN KEY([FunctionID]) REFERENCES [dbo].[Function] ([ID])
GO
ALTER TABLE [dbo].[Product] CHECK CONSTRAINT [FK_Product_Function]
GO
ALTER TABLE [dbo].[Product] WITH CHECK
ADD CONSTRAINT [FK_Product_Program]
FOREIGN KEY([ProgramID]) REFERENCES [dbo].[Program] ([ID])
GO
ALTER TABLE [dbo].[Product] CHECK CONSTRAINT [FK_Product_Program]
GO
Program:
CREATE TABLE [dbo].[Program]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[Name] [varchar](255) NOT NULL,
CONSTRAINT [PK_Program]
PRIMARY KEY CLUSTERED ([ID] ASC)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
Function:
CREATE TABLE [dbo].[Function]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[Name] [varchar](255) NOT NULL,
[Description] [varchar](max) NOT NULL,
CONSTRAINT [PK_Function_1]
PRIMARY KEY CLUSTERED ([ID] ASC)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
My EDMX looks normal:
And here's how I'm accessing the data in my repo:
Suddenly I'm getting the following error when calling GetProducts():
An unhandled exception of type 'System.AccessViolationException' occurred in Unknown Module.
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
I can't seem to figure out what's causing this. Here's what I have tried:
Made sure "Suppress JIT optimization..." was unchecked in VS
Repaired .NET Framework on my local machine
Neither worked...however, if I delete the Function table in SQL and update my EDMX, the issue is resolved. Why would this table be causing the issue?
Appreciate any input, thank you!
Try turning proxy generation off, and see if you still get the same problem.
My initial guess would be that you are including the Program directly via .Include(x => x.Program), but Function navigation property is being proxied into a lazy property. And when WebAPI tries to serialize it, it throws an error.

converting computed column to a specific datatype

I am currently trying to execute some SQL Query in SQLSERVER 2008 R2 form my Java GUI. I am working on currency management system.
I have to store Long data type values as the figure of currency may exceed than 10 digits but the computed column dose not show any data type option in the design view of the table. I really Need help regarding this as my value exceeds than 10 digits and I need to select total value from my database. I have tried to execute the code but its showing some sort of overflow error please help
The following is my script file of the table from database name CNV
USE [CNV]
CREATE TABLE [dbo].[soil_det](
[ID] [int] IDENTITY(1,1) NOT NULL,
[rm_id] [bigint] NULL,
[box_no] [int] NULL,
[weight] [decimal](18, 2) NULL,
[note_state] [varchar](10) NULL,
[dm_state] [varchar](10) NULL,
[1] [int] NULL,
[2] [int] NULL,
[5] [int] NULL,
[10] [int] NULL,
[20] [int] NULL,
[50] [int] NULL,
[100] [int] NULL,
[500] [int] NULL,
[1000] [int] NULL,
[tp] AS (((((((([1]+[2])+[5])+[10])+[20])+[50])+[100])+[500])+[1000]),
[tv] AS (((((((([1]*(1)+[2]*(2))+[5]*(5))+[10]*(10))+[20]*(20))+[50]*(50))+[100]*(100))+[500]*(500))+[1000]*(1000)) PERSISTED,
[tp_ex1] AS ((((((([2]+[5])+[10])+[20])+[50])+[100])+[500])+[1000]),
[tv_ex1] AS ((((((([2]*(2)+[5]*(5))+[10]*(10))+[20]*(20))+[50]*(50))+[100]*(100))+[500]*(500))+[1000]*(1000)),
[val_1] AS ([1]*(1)),
CONSTRAINT [PK_mut_det] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
here is solution for this , you can do something as given in image
Check the full article over here : SQL SERVER – Puzzle – Solution – Computed Columns Datatype Explanation

How to Insert into 2 Tables ProductOrder and ProductOrderLine using VBA & Foreign Keys in Excel

I am newbiew using VBA/FK/SQL server all in one. I am creating simple purchase order workbook user interface in excel.
I have created two tables:
CREATE TABLE [dbo].[PurchaseOrder](
[PKPurchaseOrderID] [bigint] IDENTITY(1,1) NOT NULL,
[PurchaseOrderNumber] [bigint] NULL,
[PurchaseOrderDate] [date] NULL,
[PurchaseOrderTime] [int] NULL,
[PurchaseOrderSupplierID] [nvarchar](50) NULL,
[ShipToA1] [nvarchar](50) NULL,
[ShipToA2] [nvarchar](50) NULL,
[ShipToA3] [nvarchar](50) NULL,
[ShipToA4] [nvarchar](50) NULL,
CONSTRAINT [PK_PurchaseOrder] PRIMARY KEY CLUSTERED
(
[PKPurchaseOrderID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
and
CREATE TABLE [dbo].[PurchaseOrderLines](
[PKPurchaseOrderLineID] [bigint] IDENTITY(1,1) NOT NULL,
[FKPurchaseOrderID] [bigint] NULL,
[Quantity] [smallint] NULL,
[Item] [nchar](25) NULL,
[Description] [nvarchar](50) NULL,
[siteID] [nchar](10) NULL,
[UnitPrice] [money] NULL,
[LineTotal] [money] NULL,
CONSTRAINT [PK_PurchaseOrderLines] PRIMARY KEY CLUSTERED
(
[PKPurchaseOrderLineID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
ALTER TABLE [dbo].[PurchaseOrderLines] WITH CHECK ADD CONSTRAINT
[FK_PurchaseOrderLines_PurchaseOrder] FOREIGN KEY([FKPurchaseOrderID])
REFERENCES [dbo].[PurchaseOrder] ([PKPurchaseOrderID])
ALTER TABLE [dbo].[PurchaseOrderLines] CHECK CONSTRAINT
[FK_PurchaseOrderLines_PurchaseOrder]
GO
I want to achieve that in the background when user clicks the button both tables are updated.
I am not sure how I can link brand new row created in the PurchaseOrder table with FK in the PurchaseorderLine table.
What i plan to do for single user interface:
Insert New Order
Use Max(PKPurchaseOrderID) as FK for new order lines table.
How can I determine currently inserted Order ID(PKPurchaseOrderID) if multiple users are working/submitting orders at the same time. I am afraid order lines may be assigned to different orders if I use my plan. e.g.
Please advise.
I use SQL Server 2008 and Excel 2007/2010
Many thanks
I would create a stored procedure that inserts the new row and returns the ID with SCOPE_IDENTITY()
Here's a decent article on the different "identity" methods in SQL Server.
http://blog.sqlauthority.com/2007/03/25/sql-server-identity-vs-scope_identity-vs-ident_current-retrieve-last-inserted-identity-of-record/