Problems understanding intermittent inconsistencies when loading data with SSIS package - sql

The problem
During the passed few months the below described procedure has worked without any problems a vast majority of the times it has run (on 2008 r2). We have, however, three instances of incorrectly connected data. The question is, what is causing this and how do I remedy it?
DATA_PreImp
sourceid col01 col02 col03 col04 col...
100001 John Smith
100002 Calvin Klein
100003 Peter Parker
100004 Moe Greene
Usually the rendered result is that the attribute is connected to the Items_Main correctly but sometimes (less than 1%) the order is scrambled so that the value of col01 is not connected to the same Items_Main as the value of the rest of the columns.
Any insights as to what is causing this would be most appreciated.
The data moving procedure
We have an SSIS package that transfers data from a flat table called DATA_PreImp to a structure consisting of three related tables (attribute based).
Items_Main should contains one row for each row in DATA_PreImp
Items_Featurevalues contains one row for each column value of a row in DATA_PreImp
Items_MainRel contains the connection between Items_Main and Items_FeatureValues
The first step in the SSIS package inserts the data from DATA_PreImp to Items_Main and inserts the generated identifier into the TARGET_ID column in the empty DATA_PreImpMappingTMP table.
insert into items_main(creationdate, status)
output inserted.itemid into DATA_PreImpMappingTMP(TARGET_ID)
select getdate(), '0' from data_preimp
order by sourceid asc;
The second step in the SSIS package fill the Items_MainRel table with TARGET_ID (Itemid originally) and an identifier for the feature (in this case a 5).
insert into items_mainrel(itemid, featureid)
output inserted.itemrelid into DATA_PreImpMapping2TMP(INDREL_ID)
select TARGET_ID, 5 from DATA_PreImpMappingTMP
order by TARGET_ID asc;
The third step is to fill the SOURCE_ID column in the DATA_PreImpMapping2TMP table with the SOURCE_ID from DATA_PreImp.
with cte as (select sourceid, row_number() over (order by sourceid asc) as row from data_preimp)
update m set m.SOURCE_ID = s.sourceid, m.FEAT_ID = 5
from DATA_PreImpMapping2TMP as m
join cte as s on s.row = m.ROW;
The last step is to fill the Items_FeatureValues table with data from DATA_PreImpMapping2TMP and DATA_PreImp.
insert into items_featurevalues(itemrelid, languageid, fnvarchar)
select DATA_PreImpMapping2TMP.INDREL_ID, 0, data_preimp.col01
from DATA_PreImpMapping2TMP
join data_preimp on (DATA_PreImpMapping2TMP.SOURCE_ID = data_preimp.sourceid)
where FEAT_ID = 5
Data table structure
Here is what is needed to create the scenario:
CREATE TABLE [dbo].[DATA_PreImp](
[sourceid] [bigint] IDENTITY(1,1) NOT NULL,
[col01] [nvarchar](500) NULL,
[col02] [nvarchar](500) NULL,
[col03] [nvarchar](500) NULL,
[col04] [nvarchar](500) NULL,
[col05] [nvarchar](500) NULL,
[col06] [nvarchar](500) NULL,
[col07] [nvarchar](500) NULL,
[col08] [nvarchar](500) NULL,
[col09] [nvarchar](500) NULL,
[col10] [nvarchar](500) NULL,
CONSTRAINT [PK_DATA_PreImp] PRIMARY KEY CLUSTERED
(
[sourceid] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[DATA_PreImpMappingTMP](
[ROW] [int] IDENTITY(1,1) NOT NULL,
[TARGET_ID] [int] NULL,
PRIMARY KEY CLUSTERED
(
[ROW] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[Items_Main](
[Itemid] [int] IDENTITY(1,1) NOT NULL,
[creationDate] [smalldatetime] NOT NULL,
[status] [int] NOT NULL,
[purchdate] [smalldatetime] NULL,
[logindate] [smalldatetime] NULL,
CONSTRAINT [PK_Items_Main] PRIMARY KEY CLUSTERED
(
[Itemid] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[DATA_PreImpMapping2TMP](
[ROW] [int] IDENTITY(1,1) NOT NULL,
[SOURCE_ID] [int] NULL,
[INDREL_ID] [int] NULL,
[FEAT_ID] [int] NULL,
PRIMARY KEY CLUSTERED
(
[ROW] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[Items_Features](
[featureId] [int] IDENTITY(1,1) NOT NULL,
[featureRef] [varchar](15) NOT NULL,
[featureName] [varchar](50) NOT NULL,
[creationDate] [smalldatetime] NOT NULL,
[status] [int] NOT NULL,
[fieldType] [varchar](50) NOT NULL,
[featureType] [int] NOT NULL,
[featureDesc] [varchar](500) NULL,
CONSTRAINT [PK_Items_Features] PRIMARY KEY CLUSTERED
(
[featureId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]) ON [PRIMARY]
CREATE TABLE [dbo].[Items_MainRel](
[ItemRelId] [int] IDENTITY(1,1) NOT NULL,
[Itemid] [int] NOT NULL,
[featureId] [int] NOT NULL,
CONSTRAINT [PK_Items_MainRel] PRIMARY KEY CLUSTERED
(
[ItemRelId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
CREATE TABLE [dbo].[Items_FeatureValues](
[valueId] [int] IDENTITY(1,1) NOT NULL,
[ItemRelId] [int] NOT NULL,
[languageId] [int] NOT NULL,
[FnVarChar] [nvarchar](250) NULL,
[FInt] [int] NULL,
[FImage] [int] NULL,
[FNText] [ntext] NULL,
[FSmallDateTime] [smalldatetime] NULL,
CONSTRAINT [PK_Items_FeatureValues] PRIMARY KEY CLUSTERED
(
[valueId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
ALTER TABLE [dbo].[Items_MainRel] WITH CHECK ADD CONSTRAINT [FK_Items_MainRel_Items_Features] FOREIGN KEY([featureId])
REFERENCES [dbo].[Items_Features] ([featureId])
GO
ALTER TABLE [dbo].[Items_MainRel] CHECK CONSTRAINT [FK_Items_MainRel_Items_Features]
GO
ALTER TABLE [dbo].[Items_MainRel] WITH CHECK ADD CONSTRAINT [FK_Items_MainRel_Items_Main] FOREIGN KEY([Itemid])
REFERENCES [dbo].[Items_Main] ([Itemid])
GO
ALTER TABLE [dbo].[Items_MainRel] CHECK CONSTRAINT [FK_Items_MainRel_Items_Main]
GO
ALTER TABLE [dbo].[Items_FeatureValues] WITH CHECK ADD CONSTRAINT [FK_Items_FeatureValues_Items_MainRel] FOREIGN KEY([ItemRelId])
REFERENCES [dbo].[Items_MainRel] ([ItemRelId])
ON DELETE CASCADE
GO
ALTER TABLE [dbo].[Items_FeatureValues] CHECK CONSTRAINT [FK_Items_FeatureValues_Items_MainRel]
GO
INSERT INTO DATA_PreImp (col01,col02,col03,col04)
VALUES('John', 'Smith', '1964', 'NewYork'),
('Calvin', 'Klein', '1960', 'Washington D. C.'),
('Peter', 'Parker', '1974', 'Losangles'),
('Moe', 'Greene', '1928', 'Lasvegas')
INSERT INTO Items_Features (featureRef, featureName, creationDate, [status], fieldType, featureType, featureDesc)
VALUES ('firstname','First_Name', GETDATE(), 0, 'FnVarChar', 3, 'FirstName'),
('lastname','Last_Name', GETDATE(), 0, 'FnVarChar', 3, 'LastName'),
('Birth','Birth', GETDATE(), 0, 'FnVarChar', 3, 'Birth'),
('City','City', GETDATE(), 0, 'FnVarChar', 3, 'City')

The problem was the computed column CUST_CD. After a lot of researching, it seems that the BULK INSERT does not like complex computed types (see Using SQL Server spatial types in SSIS data load). The solution is to removed the computed column and just make it a varchar(20) NULL. Then I created a new Execute SQL Task that updates any NULL rows with the computed value.

Related

Performance issue with geometry table in SQL Server

I am on SQL Server 11. I have the following table that stores the boundary of all LGA in NSW
CREATE TABLE [dbo].[nsw_lga_polygon_shp](
[id] [int] IDENTITY(1,1) NOT NULL,
[geom] [geometry] NULL,
[lg_ply_pid] [nvarchar](15) NULL,
[dt_create] [date] NULL,
[dt_retire] [date] NULL,
[lga_pid] [nvarchar](15) NULL,
[nsw_lga_sh] [date] NULL,
[nsw_lga__1] [date] NULL,
[nsw_lga__2] [nvarchar](100) NULL,
[nsw_lga__3] [nvarchar](100) NULL,
[nsw_lga__4] [date] NULL,
[nsw_lga__5] [nvarchar](15) NULL,
CONSTRAINT [PK_nsw_lga_polygon_shp] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
END
GO
SET IDENTITY_INSERT [dbo].[nsw_lga_polygon_shp] ON;
The table only have 198 rows
When I try to map the result against one of my table that have 36,531 rows, I notice that it have very serious performance issue, it told 5 mins to generate 500 rows.
select
* ,
(select top 1
nsw_lga__2
from
[nsw_lga_polygon_shp]
where
Geom.Filter(geometry::STGeomFromText('point(' +
convert(varchar(100),longitude_GIS )+
' ' +
cast(latitude_GIS as varchar(100))+ ')',4283)) = 1
) LGA
from
Report_A
Is there anyway I can may it run faster? Can I index the geometry column?
Thanks in advance!

Diffrence between 2 tables

I have two tables which is one of them is
AI: Ascent insensitive and other is AS: Ascent sensitive
How can I compare between them and take difference between them? I used except
select ID,FIRST,FATHER,GRAND from t1
EXCEPT
select ID,FIRST,FATHER,GRAND from t2
But I got this error:
Cannot resolve the collation conflict between "Arabic_CI_AS" and
"Arabic_100_CI_AI" in the EXCEPT operation
Choose an appropriate collation so that the comparison of the strings does match as wanted (e. G. "CI" = case sensitive/insensitive), the use your collation like this (ignoring the problem of of the ID column in EXCEPT:
select ID,
FIRST collate Arabic_CI_AS,
FATHER collate Arabic_CI_AS,
GRAND collate Arabic_CI_AS
from t1
EXCEPT
select ID,
FIRST collate Arabic_CI_AS,
FATHER collate Arabic_CI_AS,
GRAND collate Arabic_CI_AS
from t2
it working here
USE [TestDb]
GO
Table 1
CREATE TABLE [dbo].[t1](
[ID] [int] IDENTITY(1,1) NOT NULL,
[FIRST] [nchar](10) NULL,
[FATHER] [nchar](10) NULL,
[GRAND] [nchar](10) NULL,
CONSTRAINT [PK_t1] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
Table 2
CREATE TABLE [dbo].[t2](
[ID] [int] IDENTITY(1,1) NOT NULL,
[FIRST] [nchar](10) NULL,
[FATHER] [nchar](10) NULL,
[GRAND] [nchar](10) NULL,
CONSTRAINT [PK_t2] PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO

Difference with nolock and readpast in SQL server 2016

I am updating a table Hr.employees for a single record in a transaction in one query window
Begin Tran A
update hr.Employees
set lastname = 'davis'
where empid=1
In other query window i am running two select statements without a transaction
select count(*) from hr.Employees with(nolock)
select count(*) from hr.Employees with(readpast)
Both of them returning 9, whereas readpast should return 8 as it removes dirty read when returning data.
CREATE TABLE [HR].[Employees](
[empid] [int] IDENTITY(1,1) NOT NULL,
[lastname] [nvarchar](20) NOT NULL,
[firstname] [nvarchar](10) NOT NULL,
[title] [nvarchar](30) NOT NULL,
[titleofcourtesy] [nvarchar](25) NOT NULL,
[birthdate] [datetime] NOT NULL,
[hiredate] [datetime] NOT NULL,
[address] [nvarchar](60) NOT NULL,
[city] [nvarchar](15) NOT NULL,
[region] [nvarchar](15) NULL,
[postalcode] [nvarchar](10) NULL,
[country] [nvarchar](15) NOT NULL,
[phone] [nvarchar](24) NOT NULL,
[mgrid] [int] NULL,
CONSTRAINT [PK_Employees] PRIMARY KEY CLUSTERED
(
[empid] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
ALTER TABLE [HR].[Employees] WITH CHECK ADD CONSTRAINT [FK_Employees_Employees] FOREIGN KEY([mgrid])
REFERENCES [HR].[Employees] ([empid])
GO
ALTER TABLE [HR].[Employees] CHECK CONSTRAINT [FK_Employees_Employees]
GO
ALTER TABLE [HR].[Employees] WITH CHECK ADD CONSTRAINT [CHK_birthdate] CHECK (([birthdate]<=getdate()))
GO
ALTER TABLE [HR].[Employees] CHECK CONSTRAINT [CHK_birthdate]
GO
CREATE NONCLUSTERED INDEX [idx_nc_lastname] ON [HR].[Employees]
(
[lastname] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO
CREATE NONCLUSTERED INDEX [idx_nc_postalcode] ON [HR].[Employees]
(
[postalcode] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO

How I can change the incerement value of cloumn

I have a table and have a lot of records and i can't drop and recreate it
I need to change the increment value of the Id column to 2 the old value is 1
this is an example of the table I need to change the identity increment of
CREATE TABLE [dbo].[RELATED](
[Id] [int] IDENTITY(1,1) NOT NULL,
[RELATEDDESC] [nvarchar](50) NOT NULL,
[USER_ID] [int] NULL,
[ALTER_DATE] [datetime2](7) NULL,
PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
To specify that the "ID" column should start at value 1 and increment by 2, change it to IDENTITY(1,2).
EDIT*
ALTER TABLE [dbo].[RELATED]
ALTER COLUMN [Id] [int] IDENTITY(1,2) NOT NULL

Managing multiple users getting work assignments in SQL Server 2008R2

I have a SQL database that has two tables that hold a queue of work to be done and the history of that work:
CREATE TABLE [dbo].[WorkQueue](
[WorkQueueID] [int] IDENTITY(1,1) NOT NULL,
[BatchID] [int] NOT NULL,
[AdHocProjectID] [int] NOT NULL,
[TaskID] [int] NOT NULL,
[CreateDate] [datetime] NOT NULL,
[CompletedDate] [datetime] NULL,
[AlertMessage] [varchar](1024) NULL,
CONSTRAINT [PK_WorkQueue] PRIMARY KEY CLUSTERED
(
[WorkQueueID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[WorkQueueHistory](
[WorkQueueHistoryID] [int] IDENTITY(1,1) NOT NULL,
[WorkQueueID] [int] NOT NULL,
[EmployeeID] [int] NOT NULL,
[BatchBoxID] [int] NOT NULL,
[StartTime] [datetime] NOT NULL,
[EndTime] [datetime] NULL,
[PercentageCompleted] [int] NOT NULL,
[FinishedTask] [bit] NOT NULL,
[comment] [varchar](255) NULL,
CONSTRAINT [PK_WorkQueueHistory] PRIMARY KEY CLUSTERED
(
[WorkQueueHistoryID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
) ON [PRIMARY]
The WorkQueue table contains tasks from batches (Jobs) that need to be worked. Users insert into the WorkQueueHistory table when they start working on a job, and when they finish working on a job that row in the WorkQueueHistory table is updated with their end time and the FinishedTask bit is set if that task was indeed completed 100%.
My question is what is the best way to prevent two users from starting the same task? Right now we don't have enough users for this to be a problem, but as things grow I know there is going to be a concurrency issue if I don't address this.
You should use a transaction with the appropriate isolation level set.
See http://www.blackwasp.co.uk/SQLTransactions.aspx
begin tran
-- select next job to do
-- update job as started
commit