Consider the table:
CREATE TABLE [dbo].[inputdata](
[Name] [varchar](150) NULL,
[AddressStreet] [varchar](150) NULL,
[AddressStreet2] [varchar](150) NULL,
[City] [varchar](150) NULL,
[State] [varchar](2) NULL,
[Zip] [varchar](5) NULL,
[Phone] [varchar](10) NULL,
[Campus] [varchar](50) NULL,
[Access] [varchar](50) NULL,
[Type] [varchar](50) NULL,
[Degree] [varchar](50) NULL,
[Unknown1] [varchar](50) NULL,
[Unknown2] [varchar](50) NULL,
[IdentType] [varchar](50) NULL,
[Unknown3] [varchar](50) NULL
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
And the insert script:
SET ANSI_DEFAULTS ON
BULK INSERT dbo.inputdata
FROM 'C:\inputdata.csv'
WITH (
FIELDTERMINATOR = ','
,ROWTERMINATOR='\n'
)
Why is the following error output:
The bulk load failed. The column is too long in the data file for row 1, column 15. Verify that the field terminator and row terminator are specified correctly.
DataRow 1 (I manually added the \n for this question, the character exists in each row from the file):
1CRESCENT CITY BARTENDING INSTITUTE,209 N. BROAD AVE., ,NEW ORLEANS,LA,70119,.,Regular,Private,1-2 years,Diploma, , ,IPEDSUNIT,158617,\n
I tried to recreate your problem and everything worked.
Until I tried converting the file into UNIX format, where the linefeed is different. This got me the following message:
Bulk load data conversion error (truncation) for row 1, column 15 (Unknown3).
Try to look at your file with a hex editor.
There is an extra field terminator at the end of the line. An extra field terminator can cause it to throw general errors.
I hope this helps!
One way to ensure files get loaded that have strange items like an extra field terminator or text qualified fields using a bulk insert is using a format file. The link below explains how to create one of those files.
http://technet.microsoft.com/en-us/library/ms191479(v=sql.105).aspx
Related
I've been having this problem with my database where it kept on incrementing the id column even though it has been removed. To better understand what I meant, here is a screenshot of my gridview:
As you can see from the id column, everything is fine from 11 - 16. but it suddenly skipped from 25 - 27. What i want to happen is, when i remove an item, i want it to start from the last id which is 16. So the next id should be 17. I hope this makes sense for you guys.
Here is also part of the SQL script:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[guitarItems]
(
[id] [int] IDENTITY(1,1) NOT NULL,
[type] [varchar](50) NOT NULL,
[brand] [varchar](50) NOT NULL,
[model] [varchar](50) NOT NULL,
[price] [float] NOT NULL,
[itemimage1] [varchar](255) NULL,
[itemimage2] [varchar](255) NULL,
[description] [text] NOT NULL,
[necktype] [varchar](100) NOT NULL,
[body] [varchar](100) NOT NULL,
[fretboard] [varchar](100) NOT NULL,
[fret] [varchar](50) NOT NULL,
[bridge] [varchar](100) NOT NULL,
[neckpickup] [varchar](100) NOT NULL,
[bridgepickup] [varchar](100) NOT NULL,
[hardwarecolor] [varchar](50) NOT NULL,
PRIMARY KEY CLUSTERED ([id] ASC)
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
You can use:
DBCC CHECKIDENT ("YourTableNameHere", RESEED, 1);
Before using it, visit link: https://learn.microsoft.com/en-us/sql/t-sql/database-console-commands/dbcc-checkident-transact-sql
Primary autoincrement keys in the database are used to uniquely identify a given row and shouldn't be given any business meaning. So leave the primary key as it is and add another column like guitarItemsId. Then when you delete a record from the database you may want to send an additional UPDATE statement in order to decrease the guitarItemsId column of all rows that have the guitarItemsId greater than the one you are currently deleting.
Also, remember that you should never modify the value of a primary key in a relational database because there could be other tables that reference it as a foreign key and modifying it might violate the referential constraints.
I am trying to create a table that dynamically pulls the starting IDENTITY ID based on a variable from another table. The SQL executes successfully but afterwards, I am unable to find my temporary table. The DBCC CHECKIDENT brings back Invalid object name '#address_temp'.
IF OBJECT_ID('tempdb.dbo.#address_temp', 'U') IS NOT NULL
DROP TABLE #address_temp
DECLARE #address_temp_ID VARCHAR(MAX)
SET #address_temp_ID = (SELECT MAX(ID) FROM [PRIMARYDB].[dbo].[ADDRESS])
DECLARE #SQLBULK VARCHAR(MAX)
SET #SQLBULK = 'CREATE TABLE #address_temp(
[ID] [int] IDENTITY(' + #address_temp_ID + ',1) NOT NULL,
[NAME] [varchar](128) NOT NULL,
[ADDRESS1] [varchar](128) NOT NULL,
[ADDRESS2] [varchar](128) NULL,
[CITY] [varchar](128) NULL,
[STATE_ID] [smallint] NULL,
[ZIP] [varchar](10) NOT NULL,
[COUNTY] [varchar](50) NULL,
[COUNTRY] [varchar](50) NULL
CREATE CLUSTERED INDEX pk_add ON #address_temp ([NAME])'
EXEC (#SQLBULK)
DBCC CHECKIDENT('#address_temp')
Table names that start with # are temporary tables and SQL Server treats them differently. First of all they are only available to the session that created them (this is not quite true since they you can find them in the temp name space but they have a unique system generated name)
In any case they won't persist so you don't need to drop them (that happens auto-magically) and you certainly can't look at them after your session ends.... they are gone.
Don't use a temp table, take out the # in the name. Things will suddenly start working.
i'm using the following code
create TABLE #temp1(
[date] [varchar](500) NULL,
[time] [varchar](500) NULL,
[s-ip] [varchar](500) NULL,
[cs-method] [varchar](500) NULL,
[cs-uri-stem] [varchar](max) NULL,
[cs-uri-query] [varchar](max) NULL,
[s-port] [varchar](500) NULL,
[cs-username] [varchar](500) NULL,
[c-ip] [varchar](500) NULL,
[cs(User-Agent)] [varchar](2048) NULL,
[cs(Referer)] [varchar](max) NULL,
[sc-status] [varchar](max) NULL,
[sc-substatus] [varchar](500) NULL,
[sc-win32-status] [varchar](500) NULL,
[time-taken] [varchar](500) NULL
)
BULK INSERT #temp1 FROM 'C:\iislogs\u_ex150108.log'
WITH (
FIRSTROW = 5,
FIELDTERMINATOR = ' ',
ROWTERMINATOR = '\n',
MAXERRORS = 10000
)
and the file that, i'm using has 1521000 lines but i am getting only 1305226 in #temp1
Please help
Thanks in advance
Thanks all for your valuable answer
But again after long search in Google.
I found one solution ,that this is a Microsoft sql server bug
This bug can only be solved when we install latest service pack of sql server in our system
And when I have installed service pack in my system bulk insert works well.
For any further help please take help from below link
https://support.microsoft.com/kb/837401/EN-US
Thanks Again
set all varchar to max and do the bulkfiles again, and please can you paste your log file to excel and see if the total row is really 1521000?
The problem usually caused by invalid fieldterminator and rowterminator.
Recently I had an issue inserting rows with bulk insert, I added a rowcount condition till the stored procedure has to run.
select #tablerowcount = count(*) from tbl1
while #tablerowcount = 0
begin
exec sp_custombulkinsert
end
I'm developing an app which requires a user defined custom fields on a contacts table. This contact table can contain many millions of contacts.
We're looking at using a secondary metadata table which stores information about the fields, along with a tertiary value table which stores the actual data.
Here's the rough schema:
CREATE TABLE [dbo].[Contact](
[ID] [int] IDENTITY(1,1) NOT NULL,
[FirstName] [nvarchar](max) NULL,
[MiddleName] [nvarchar](max) NULL,
[LastName] [nvarchar](max) NULL,
[Email] [nvarchar](max) NULL
)
CREATE TABLE [dbo].[CustomField](
[ID] [int] IDENTITY(1,1) NOT NULL,
[FieldName] [nvarchar](50) NULL,
[Type] [varchar](50) NULL
)
CREATE TABLE [dbo].[ContactAndCustomField](
[ID] [int] IDENTITY(1,1) NOT NULL,
[ContactID] [int] NULL,
[FieldID] [int] NULL,
[FieldValue] [nvarchar](max) NULL
)
However, this approach introduces a lot of complexity, particularly with regard to importing CSV files with multiple custom fields. At the moment this requires a update/join statement and a separate insert statement for every individual custom field. Joins would also be required to return custom field data for multiple rows at once
I've argued for this structure instead:
CREATE TABLE [dbo].[Contact](
[ID] [int] IDENTITY(1,1) NOT NULL,
[FirstName] [nvarchar](max) NULL,
[MiddleName] [nvarchar](max) NULL,
[LastName] [nvarchar](max) NULL,
[Email] [nvarchar](max) NULL
[CustomField1] [nvarchar](max) NULL
[CustomField2] [nvarchar](max) NULL
[CustomField3] [nvarchar](max) NULL /* etc, adding lots of empty fields */
)
CREATE TABLE [dbo].[ContactCustomField](
[ID] [int] IDENTITY(1,1) NOT NULL,
[FieldIndex] [int] NULL,
[FieldName] [nvarchar](50) NULL,
[Type] [varchar](50) NULL
)
The downside of this second approach is that there is a finite number of custom fields that must be specified when the contacts table is created. I don't think that's a major hurdle given the performance benefits it will surely have when importing large CSV files, and returning result sets.
What approach is the most efficient for large numbers of rows? Are there any downsides to the second technique that I'm not seeing?
Microsoft introduced sparse columns exactly for this type of problems. Tha point is that in a "classic" design you end up with large number of columns, most of the NULLs for any particular row. Same here with sparse columns, but NULLs don't require any storage. Moreover, you can create sets of columns and modify sets with XML.
Performance- and storage-wise, sparse columns are the winner.
http://technet.microsoft.com/en-us/library/cc280604.aspx
uery performance. Query performance for any "property bag table" approach is funny and comically slow - but if you need flexibility you can either have a dynamic table that is changed via an editor OR you have a property bag table. So when you need it, you need it.
But expect the performance to be slow.
The best approach would likely be a ContactCustomFields table which has - fields that are determined by an editor.
In the following table:
CREATE TABLE [dbo].[GDB_ITEMS](
[ObjectID] [int] NOT NULL,
[UUID] [uniqueidentifier] NOT NULL,
[Type] [uniqueidentifier] NOT NULL,
[Name] [nvarchar](226) NULL,
[PhysicalName] [nvarchar](226) NULL,
[Path] [nvarchar](512) NULL,
[Url] [nvarchar](255) NULL,
[Properties] [int] NULL,
[Defaults] [varbinary](max) NULL,
[DatasetSubtype1] [int] NULL,
[DatasetSubtype2] [int] NULL,
[DatasetInfo1] [nvarchar](255) NULL,
[DatasetInfo2] [nvarchar](255) NULL,
[Definition] [xml] NULL,
[Documentation] [xml] NULL,
[ItemInfo] [xml] NULL,
[Shape] [geometry] NULL,
CONSTRAINT [R2_pk] PRIMARY KEY CLUSTERED
(
[ObjectID] ASC
)
ALTER TABLE [dbo].[GDB_ITEMS] WITH CHECK ADD CONSTRAINT [g1_ck] CHECK (([Shape].[STSrid]=(4326)))
GO
The [Documentation] column contains several hundred xml items and elements. I am trying to figure out to, with T-SQL, replace one series of elements:
<NPS_Info>
<MetaPurp>NPS</MetaPurp>
<NPS_Unit>
<UnitCode>MANDATORY for Data Store: NPS Alpha Unit Code (ACAD)</UnitCode>
<UnitType>MANDATORY for Data Store: NPS Unit Type (National Park, National Monument, etc)</UnitType>
</NPS_Unit>
</NPS_Info>
With this:
<NPS_Info>
<MetaPurp>NPS</MetaPurp>
<MetaPurp>CSDGM</MetaPurp>
<MetaPurp>OnlineData</MetaPurp>
<NPS_Unit>
<UnitCode>ABCD</UnitCode>
<UnitType>Park</UnitType>
</NPS_Unit>
<DatStore>
<Category>Landscape</Category>
<Category>Monitoring</Category>
<Category>Resource Management</Category>
<DSteward>
<cntinfo>
<cntperp>
<cntper>Something</cntper>
</cntperp>
<cntaddr>
<addrtype>mailing and physical</addrtype>
<address>1 Smith Lane</address>
<address></address>
<city>Anywhere</city>
<state>ST</state>
<postal>12345</postal>
</cntaddr>
<cntemail>email#email.com</cntemail>
</cntinfo>
</DSteward>
</DatStore>
</NPS_Info>
Please forgive my clumsy cut n' paste. There are several thousand rows in this table, however, not all of them have the xml element described in the first code block (this is a global table that holds descriptions of ALL tables in the DB, some [Documentation] records will contain non-pertinent xml not of interest to this operation).
You can use XML Data Modification Language (XML DML)
This code will change the content of the first node named NPS_Info with the content in variable #XML.
-- Delete everything in node NPS_Info
update YourTable
set XMLCol.modify('delete //NPS_Info[1]/*')
-- Insert #XML to node NPS_Info
update YourTable
set XMLCol.modify('insert sql:variable("#XML") into (//NPS_Info)[1]')
Working sample on SE Data