Would someone help, please, to get rid of the error:
Violation of PRIMARY KEY constraint 'PK_stmp_tst1'. Cannot insert duplicate key in object 'dbo.stmp_tst'. The duplicate key value is (1).
which occurs after switch partition of the table.
Full SQL Script below:
I. Create 2 equal tables in different schemas:
CREATE TABLE dbo.stmp_tst(
[inn] [varchar](20) NULL,
[id] [bigint] IDENTITY(1,1) NOT NULL,
CONSTRAINT [PK_stmp_tst] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
go
CREATE TABLE stage.stmp_tst(
[inn] [varchar](20) NULL,
[id] [bigint] IDENTITY(1,1) NOT NULL,
CONSTRAINT [PK_stmp_tst] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
II. Insert data into stage table.
insert into stage.stmp_tst (inn)
select '1111'
III. Switch-partition stage to dbo.
alter table stage.stmp_tst switch partition 1 to dbo.stmp_tst partition 1;
IV. Add data to dbo table.
insert into dbo.stmp_tst (inn)
select '1111'
V. We have the error:
Violation of PRIMARY KEY constraint 'PK_stmp_tst1'. Cannot insert
duplicate key in object 'dbo.stmp_tst'. The duplicate key value is
(1).
IV. Drop temporary tables:
drop table dbo.stmp_tst
drop table stage.stmp_tst
It can be solved by the query:
DBCC CHECKIDENT ('dbo.stmp_tst', RESEED);
but reseeding takes time.
Is it possible to do a switch-partition correctly without reseed?
Thank you.
I have a very large table ~55,000,000 records.
Indexes have been added to the most commonly used columns, but the table is still very slow.
Are there any suggestions as to how the tables performance could be improved?
I have thought about partitioning the table, but was not sure it was necessary.
--Table
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[EngineRecord](
[Id] [uniqueidentifier] NOT NULL,
[CreateDate] [datetime] NOT NULL,
[ChangeDate] [datetime] NOT NULL,
[CompanyId] [uniqueidentifier] NOT NULL,
[DriverEmployeeId] [uniqueidentifier] NOT NULL,
[EobrDeviceId] [uniqueidentifier] NOT NULL,
[EobrTimestampUtc] [datetime] NOT NULL,
[EobrOverallStatus] [int] NOT NULL,
[Speedometer] [decimal](14, 4) NOT NULL,
[Odometer] [decimal](14, 4) NOT NULL,
[Tachometer] [decimal](14, 4) NOT NULL,
[GpsTimestampUtc] [datetime] NULL,
[GpsLatitude] [decimal](18, 8) NULL,
[GPSLongitude] [decimal](18, 8) NULL,
[RecordType] [int] NOT NULL,
[FuelEconomyAverage] [decimal](8, 4) NOT NULL,
[FuelEconomyInstant] [decimal](8, 4) NOT NULL,
[FuelUseTotal] [decimal](14, 4) NOT NULL,
[BrakePressure] [decimal](8, 4) NOT NULL,
[CruiseControlSet] [bit] NOT NULL,
[TransmissionAttained] [nvarchar](2) NULL,
[TransmissionSelected] [nvarchar](2) NULL,
[IsProcessed] [bit] NOT NULL,
[LastChangedByUserId] [uniqueidentifier] NOT NULL,
CONSTRAINT [PK_EngineRecord] PRIMARY KEY NONCLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY],
CONSTRAINT [NK_EngineRecord] UNIQUE CLUSTERED
(
[CompanyId] ASC,
[EobrDeviceId] ASC,
[EobrTimestampUtc] ASC
)WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[EngineRecord] WITH NOCHECK ADD CONSTRAINT [FK_EngineRecord_CompanyLevel] FOREIGN KEY([CompanyId])
REFERENCES [dbo].[CompanyLevel] ([Id])
GO
ALTER TABLE [dbo].[EngineRecord] CHECK CONSTRAINT [FK_EngineRecord_CompanyLevel]
GO
ALTER TABLE [dbo].[EngineRecord] WITH NOCHECK ADD CONSTRAINT [FK_EngineRecord_Employee] FOREIGN KEY([DriverEmployeeId])
REFERENCES [dbo].[Employee] ([Id])
ON DELETE CASCADE
GO
ALTER TABLE [dbo].[EngineRecord] CHECK CONSTRAINT [FK_EngineRecord_Employee]
GO
ALTER TABLE [dbo].[EngineRecord] WITH NOCHECK ADD CONSTRAINT [FK_EngineRecord_EobrDevice] FOREIGN KEY([EobrDeviceId])
REFERENCES [dbo].[EobrDevice] ([Id])
GO
ALTER TABLE [dbo].[EngineRecord] CHECK CONSTRAINT [FK_EngineRecord_EobrDevice]
GO
---------------------
--Indexes/Constraints
---------------------
ALTER TABLE [dbo].[EngineRecord] ADD CONSTRAINT [PK_EngineRecord] PRIMARY KEY NONCLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
GO
CREATE NONCLUSTERED INDEX [NC_EngineRecord_Employee] ON [dbo].[EngineRecord]
(
[DriverEmployeeId] ASC
)WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
GO
CREATE NONCLUSTERED INDEX [NC_RecordType] ON [dbo].[EngineRecord]
(
[RecordType] ASC
)WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
GO
ALTER TABLE [dbo].[EngineRecord] ADD CONSTRAINT [NK_EngineRecord] UNIQUE CLUSTERED
(
[CompanyId] ASC,
[EobrDeviceId] ASC,
[EobrTimestampUtc] ASC
)WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
GO
CREATE NONCLUSTERED INDEX [IX_EngineRecord_DBA] ON [dbo].[EngineRecord]
(
[CompanyId] ASC,
[GpsLatitude] ASC,
[GPSLongitude] ASC
)
INCLUDE ( [EobrDeviceId],
[EobrTimestampUtc]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]
GO
CREATE NONCLUSTERED INDEX [NC_IsProcessed] ON [dbo].[EngineRecord]
(
[IsProcessed] ASC
)WITH (PAD_INDEX = ON, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
GO
EDIT:
Here is a sproc that takes some time to run that is used often.
CREATE PROCEDURE [dbo].[EngineRecord__GetEobrListToProcessByRecordType]
#RecordTypeEnum int
AS
DECLARE #ChangeHistory bit -- dummy variable for VS 2008 database project
SET NOCOUNT ON
SELECT EobrDevice.[Id] as EobrDeviceId,
EobrDevice.[UnitId],
CompanyGroupRoot.[Id] as CGRootId,
CompanyGroup.[Id] as CompanyGroupId,
EobrDevice.[CompanyId]
FROM dbo.EobrDevice
INNER JOIN dbo.CompanyLevel ON EobrDevice.[CompanyId] = CompanyLevel.[Id]
INNER JOIN dbo.CompanyGroup ON CompanyLevel.ParentGroupId = CompanyGroup.[Id]
INNER JOIN dbo.CompanyGroupRoot ON CompanyGroup.CGRootId = CompanyGroupRoot.[Id]
WHERE EobrDevice.[Id] IN ( SELECT DISTINCT EngineRecord.EobrDeviceId FROM dbo.EngineRecord WHERE IsProcessed = 0 AND RecordType = #RecordTypeEnum )
AND EobrDevice.UnitId IS NOT NULL
EDIT 2:
This is something we run every night to purge out old records. This always takes a lot of time.
DECLARE #dt6MonthsPrior datetime
SET #dt6MonthsPrior = DATEADD(m, -6, getdate())
SELECT * FROM EngineRecord
WHERE EngineRecord.EobrTimeStampUtc < #dt6MonthsPrior
ORDER BY EobrTimestampUtc ASC
None of the fields in your WHERE criteria are contained in an index. Indexing those fields will help. The efficacy of your other indices is impossible to determine without a more thorough understanding of how the table is used.
If you really wanted this query to fly you could have a clustered index on Odometer and Tachometer, but that's probably not reasonable given the table's other uses.
Update:
Your 2nd stored proc doesn't seem like it should be terribly slow, it does seem like the only thing that would help that is an index on the date.
55 million records isn't that big these days, I'm no expert on partitioning, but I don't think you'd see much if any improvement by partitioning your table, I usually don't bother unless I expect a table to exceed a few hundred million records, but in a production environment there are other benefits of partitioning.
Are you certain hardware is not responsible for the poor performance you're seeing? There are a host of settings/features in SQL Server that affect performance as well.
An index like this might help this specific query:
CREATE INDEX x ON dbo.EngineRecord(Odometer, Tachometer) WHERE FuelUseTotal IS NOT NULL;
This will help most if you stop ordering by the timestamp.
Do you know how to get Execution Plans? You have no index on tach or odo or FuelUse, so your sample query will result in a full table scan. From Sql Management Studio, right click in the query window, select "Include Actual Execution Plan" and then run your query. You will see an output that explains to you the steps SQL server will has to perform to actually run your query. This can be very instructive once you take the time to understand an execution plan.
Also, you might want to investigate covering indexes. These can be a dramatic difference if you have some queries that you use frequently. Of course, like any index, there is more overhead when you add/delete
Indexing the fields in the WHERE like Goat CO suggests is a good start, I would also advise to move the WHERE condition to the first INNER JOIN, that way the temporary table created for further handling is already much smaller after the first INNER JOIN (I've seen it do performance wonders)
SELECT EobrDevice.[Id] as EobrDeviceId,
EobrDevice.[UnitId],
CompanyGroupRoot.[Id] as CGRootId,
CompanyGroup.[Id] as CompanyGroupId,
EobrDevice.[CompanyId]
FROM dbo.EobrDevice
INNER JOIN dbo.CompanyLevel
ON EobrDevice.UnitId IS NOT NULL
AND EobrDevice.[CompanyId] = CompanyLevel.[Id]
AND EobrDevice.[Id] IN (
SELECT DISTINCT EngineRecord.EobrDeviceId
FROM dbo.EngineRecord
WHERE IsProcessed = 0
AND RecordType = #RecordTypeEnum
)
INNER JOIN dbo.CompanyGroup ON CompanyLevel.ParentGroupId = CompanyGroup.[Id]
INNER JOIN dbo.CompanyGroupRoot ON CompanyGroup.CGRootId = CompanyGroupRoot.[Id]
I've also moved the EobrDevice.UnitId IS NOT NULL condition to be checked first so that checking with other tables and running the subquery only happens when that condition is met.
PARTITIONNING INDEXES should have an impact in your performance. But they have to be done within appropriate separate drives. You give no information of your hardwares (what are you using, NAS ? SAS Drives ? ... )
Also, Normalization is not always the best choice regarding the objectives of your process, especially for Analytics purpose. Some fields (CompanyLevel, CompanyGroup) denormalized within your main table would have a better impact in your selection -
Well, every master chef have his own Kitchen, so let's skip this discussion....
The built of your indexes do not fit the way you purge your data. You will get a better performance if you decide to change your
[EobrTimestampUtc] ASC
change to
[EobrTimestampUtc] DESC
will impact the seek index for EngineRecord.EobrTimeStampUtc < #dt6MonthsPrior
my sql table:
CREATE TABLE [dbo].[PayrollParameter]
(
[PayrollParameterID] [CHAR](36) NOT NULL,
[Description] [NVARCHAR](50) NOT NULL,
[NumberOfDaysInMonth] [INT] NOT NULL,
[IsFixedDaysInMonth] [BIT] NOT NULL,
CONSTRAINT [PK_PayrollParameter] PRIMARY KEY CLUSTERED (
[PayrollParameterID] ASC )WITH (pad_index = OFF, statistics_norecompute =
OFF, ignore_dup_key = OFF, allow_row_locks = on, allow_page_locks = on) ON
[PRIMARY]
)
ON [PRIMARY]
GO
the database already contain records; I want to alter the table to set the default value of IsFixedDaysInMonth to Checked if NumberOfDaysInMonth!=0
How can i do this?
CREATE DEFAULT needs a constant value as the default
Is an expression that contains only constant values (it cannot include
the names of any columns or other database objects).
The only ways I know to do what you want would be to use a trigger or insert using a stored procedure.
I have implemented filestream in an existing database on SQL Server 2008 r2.
Now I have a very urgent problem as my site is practically down now:
With a very simple table like this:
CREATE TABLE [dbo].[Table1](
[Id] [int] IDENTITY(1,1),
[rowguid] [uniqueidentifier] ROWGUIDCOL NOT NULL,
[Image] [varbinary](max) FILESTREAM NULL,
CONSTRAINT [PK_Table1] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY],
CONSTRAINT [Table1RowguidUnique] UNIQUE NONCLUSTERED
(
[rowguid] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
ALTER TABLE [dbo].[Table1] ADD CONSTRAINT [DF_table1_rowguid] DEFAULT (newid()) FOR [rowguid]
GO
ALTER TABLE dbo.Table1
SET ( FILESTREAM_ON = fsfg_LiveWebsite )
GO
If I run:
select * from Table1 where Id = 1
it runs very quickly and give the correct result.
If I run anything with the "Varbinary(max) FILESTREAM" field in the where clause the whole table locks down.
So for example any of those 2 queries:
select Id from Table1 where Id = 1 and [Image] is null
select Id from Table1 where Id = 1 and [Image] = convert(varbinary(max), 'a')
What could this be?
Please reply asap with any suggestion!
Thank you
First and foremost if you want to query a VARBINARY column you'll need to enable and use Full-Text Search.
Reference Articles.
http://technet.microsoft.com/en-us/library/ms142531.aspx (this article will give you an overview of Full-Text Search and the VARBINARY column)
http://technet.microsoft.com/en-us/library/ms187787.aspx (this article will show you how to use the CONTAINS T-SQL command to query the field)
I'm creating an SQL setup script and I'm using someone else's script as an example. Here's an example of the script:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[be_Categories](
[CategoryID] [uniqueidentifier] ROWGUIDCOL NOT NULL CONSTRAINT [DF_be_Categories_CategoryID] DEFAULT (newid()),
[CategoryName] [nvarchar](50) NULL,
[Description] [nvarchar](200) NULL,
[ParentID] [uniqueidentifier] NULL,
CONSTRAINT [PK_be_Categories] PRIMARY KEY CLUSTERED
(
[CategoryID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
Does anyone know what the ON [PRIMARY] command does?
When you create a database in Microsoft SQL Server you can have multiple file groups, where storage is created in multiple places, directories or disks. Each file group can be named. The PRIMARY file group is the default one, which is always created, and so the SQL you've given creates your table ON the PRIMARY file group.
See MSDN for the full syntax.
It refers to which filegroup the object you are creating resides on. So your Primary filegroup could reside on drive D:\ of your server. you could then create another filegroup called Indexes. This filegroup could reside on drive E:\ of your server.
ON [PRIMARY] will create the structures on the "Primary" filegroup. In this case the primary key index and the table will be placed on the "Primary" filegroup within the database.
Please be aware about an important behavior related to file groups.
Using OP's SQL Script you can never mention two different file groups i.e. one for storing your data rows and the other for index data structure. This is not allowed.
This is due to the fact that the index being created in this case is a clustered Index on the column which is primary key for the table. Metadata of the clustered index and data rows of a table can never be on two different file groups.
My database has two file groups namely PRIMARY and SECONDARY. Now look at the below script. It will store the table's row data as well as clustered index data both on PRIMARY file group itself. This is happening even when I've mentioned a different file group ([SECONDARY]) for storing the table's row data.
CREATE TABLE [dbo].[be_Categories](
[CategoryID] [uniqueidentifier] ROWGUIDCOL NOT NULL CONSTRAINT [DF_be_Categories_CategoryID] DEFAULT (newid()),
[CategoryName] [nvarchar](50) NULL,
[Description] [nvarchar](200) NULL,
[ParentID] [uniqueidentifier] NULL,
CONSTRAINT [PK_be_Categories] PRIMARY KEY CLUSTERED
(
[CategoryID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [SECONDARY]
GO
More interestingly, the above script runs to completion without any error(I was expecting an error as I had given two different file groups). SQL Server does the trick behind the scene silently without throwing any error.
NOTE: But yes, the index can reside on a different file group in case of non-clustered indexes.
SQL script shown below creates a non-clustered index. The non-clustered index will get created on [SECONDARY] file group while the table's data rows reside on [PRIMARY] file group:
CREATE NONCLUSTERED INDEX [IX_Categories] ON [dbo].[be_Categories]
(
[CategoryName] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [Secondary]
GO
You can get more information here about how storing non-clustered indexes on a different file group can boost query performance.