I am trying to bulk insert XML Data into SQL Server 2005 Express with the SQLXMLBulkLoad Object Model in VB.NET using Visual Studio 2010.
When the loader gets to an element in my .xml file which does not hold a value, it throws an error because the corresponding table column in SQL Server is set to not contain null values. This was by design and instead of inserting NULL I need for the Loader to insert a blank space. I do not know how to go about doing this as this is my first time working with BulkLoad. Any suggestions?
The error.xml file text:
<?xml version="1.0" ?>
<Result State="FAILED">
<Error>
<HResult>0x80004005</HResult>
<Description> <![CDATA[ No data was provided for column 'blah' on table 'blah', and this column cannot contain NULL values.]]></Description>
<Source>General operational error</Source>
<Type>FATAL</Type>
</Error>
</Result>
EDIT:
http://social.msdn.microsoft.com/Forums/en-US/sqlxml/thread/bfa31c49-6ae5-4a5d-bcde-cd520e0cdf70/
This guy had the exact same problem as I am having, and was able to solve it by using objBl.Transaction = True. However, when I try that, I get an error that "Cannot bulk load because the file "This is a Local Temp File" could not be opened."
I am answering this for future users with this same circumstance. Albeit illogical, if you set your columns in the SQL Table to accept NULL values AND set the Default Value = (''), then the blank values in your XML file will become blanks instead of NULL values and you will no longer receive this error. I couldn't use objBl.Transaction = True because my server is a different computer than the computer that would run the app to perform the BulkLoad(). This CAN be solved by setting up a shared folder, but it was not an option in my circumstance. Therefore, the next best option was to do the above.
Related
I'm new here. I have a question in regards to SSIS and dynamic file names. I've found an article that can help for incrementing file names by appending dates, but I actually need to keep a rolling filename counter of 001-999 and start over at 001 for each time the file is created. Here's the article I reviewed:
https://www.red-gate.com/simple-talk/sql/ssis/passing-variables-to-and-from-an-ssis-task/
For instance, my first file would be "P12345001" where "001" is the first instance of the file. The next time the SSIS package runs, I'd want the filename to be "P12345002" and so on, until it gets to "P12345999" and the next file would go back to "P12345001".
Is this possible in SQL or SSIS or do I need to script something via a language like VB.NET or C# (which I only have limited experience with)?
If it helps, I am running the SSIS package on MS SQL Server 2012. Please let me know if I've included all pertinent information or not. Many thanks in advance!
SQL Work
While this is syntactically correct for SQL Server, you can use the same concept with whatever RDBMS you like, just modify it as needed.
Data Storage
To solve this, you need some place of storing the previous run's value. It appears that a key value type of table would be sufficient for your needs.
-- Simple table structure
CREATE TABLE dbo.FileExecutionHistory
(
BaseFileName varchar(50) NOT NULL
, LastSequence int NOT NULL
, CONSTRAINT PK_dbo_FileExecutionHistory PRIMARY KEY
(
BaseFileName
)
);
Into that you will store the name of the file (P12345) and the last sequence number used (1).
Data Access
Within SSIS, you will pass in the file name to this query. Note the use of ? - that will be the OLE DB provider's syntax for ordinal based query parameterization.
The inner query generates at most 2 rows. The last sequence used for this file name (if it is found) and 0 (not found). The outer query uses the TOP(1) argument to only allow one row to be generated and we order by the value descending - thus ensuring we only take the largest value.
We use the modulo % operator with 999 to restrict the allowable domain of values to 0 to 998.
However, since you want 1 to 999, we then add 1 to the resultant value.
-- Access pattern
SELECT TOP(1)
(D.LastSequence % 999) + 1 AS LastSequence
FROM
(
SELECT
FEH.LastSequence
FROM
dbo.FileExecutionHistory AS FEH
WHERE
FEH.BaseFileName = ?
-- Handle the does not exist case
UNION
SELECT
CAST(0 AS int)
) D
ORDER BY
D.LastSequence DESC;
Data Storage
Again, we will see the ? in play but this time we use it for both our file name and the sequence value.
Since the OLE DB provider is ordinal based, we would need to use four place holders and map in the "same" variable twice. That's silly so I declare a SQL Server variables up front and assign into them. For consistency, you can apply this to the Data Access pattern as well.
The Merge statement is clunky and uncertain in results so I use the tested and true update/insert pattern. I update the Value in my table for the file which will match nothing if it doesn't exist. I then attempt to insert a value into our table if it doesn't already exist.
Only one of those last two statements will ever perform an action so this is good. Plus, we have our primary key constraint preventing duplicates on file name so this is double plus good.
-- Storage pattern
DECLARE
#BaseFileName varchar(50) = ?
, #LastSequence int = ?;
UPDATE
FEH
SET
LastSequence = #LastSequence
FROM
dbo.FileExecutionHistory AS FEH
WHERE
FEH.BaseFileName = #BaseFileName;
INSERT INTO
dbo.FileExecutionHistory
(
BaseFileName
, LastSequence
)
SELECT
D.BaseFileName
, D.LastSequence
FROM
(
VALUES (#BaseFileName, #LastSequence)
) D (BaseFileName, LastSequence)
LEFT OUTER JOIN
dbo.FileExecutionHistory AS FEH
ON FEH.BaseFileName = D.BaseFileName
WHERE
FEH.BaseFileName IS NULL;
SSIS work
Break the work down into small pieces and use what SSIS gives you natively. In this approach, we're going to use Variables to do most of the heavy lifting.
Variables
You're looking at a package with a few obvious Variables in it. One Variable to hold your Sequence number. One Variable to hold the base file name. I assume the full file name is probably something like P12345001.txt so you'll want to keep track of that stuff too.
User::BaseFileName String -> P12345
User::SequenceNumber Int -> 0
User::FileExtension String -> .txt
User::CurrentFolder String -> C:\ssisdata\Output
User::SequencePad String > 001
User::FullFileName String > P12345001.txt
User::CurrentFileName String > C:\ssisdata\Output\P12345001.txt
The first 4 variables will be "normal". The last 3 will have Expressions in them. Expressions are simple formulas much as you'd see in Excel. Notice that I've assigned a value of 0 for the sequence number. That's not allowed in your scenario but it will be helpful for us to ensure all of the pieces of the puzzle are working as expected.
The expression for SequencePad will use a classic technique of padding N characters to the left of our value and then slicing the right N most characters out. For values like 999, there will be no net result. For 1, we'll end up with 001. Since we're dealing with a numeric value for SequenceNumber, we must first cast it to a string before concatenating + with the triple zeroes.
RIGHT("000" + (DT_WSTR, 3) #[User::SequenceNumber], 3)
FullFileName simply concatenates 3 variables together. Two "hard coded" values and the padded sequence number we built above.
#[User::BaseFileName] + #[User::SequencePad] + #[User::FileExtension]
Finally, CurrentFileName is built much in the same manner as FullFileName. We're going to concatenate more variables together. The reason for all these little steps is that this is the only way one has to debug Expressions. There is no ability to put break points on them. Thus, make 'em small and composite them together.
The only "trick" to be aware of here is that the slash \ is also an escape character in nearly every language so if you want to use \, you really need to use \\
#[User::CurrentFolder] + "\\" + #[User::FullFileName]
At this point, you can test putting different values in for the first 4 variables and ensure the expressed values look as expected. If you don't need a file extension, just leave that value blank - the expression will work just fine.
Flat File Connection Manager
Create a Flat File Connection manager based on the structure of your file P12345001.txt
Now that you have a variable defined, you will need to use this property as part of a Flat File Connection manager's Connection String property. See here and here for setting the Property. I'll call this Output
Ole DB Connection Manager
You need an OLE DB Connection Manager pointing to the SQL server and database we created the table in (way at the top). I'll assume it's called Config
Execute SQL 1
The purpose of this task is to assign a value into #[User::SequenceNumber]. Drag an Execute SQL Task onto the canvas and configure it much as I show in this answer. For our Connection, we'll use Config. The SQL Statement is the data access query I defined above. In the Result Set tab, we'll specify User::SequenceNumber
Run your package and if it doesn't blow up, we can assume things went well.
Data Flow Task
Somewhere, somehow you're using the computed file name and I assume you're going to query some table and push rows into that file. Your Data Flow will look something like an OLE DB Source component wired into a Flat File Destination. An example, with Biml none-the-less, is here
Execute SQL 2
Now that our data flow is complete, we need to update our original table with our used sequence number.
This will be another Execute SQL Task, using the third query from above, with No resultset defined. We will map in the variables #[User::FileName] and #[User::SequenceNumber] (in that order) as parameters 0 and 1.
Biml
The Business Intelligence Markup Lanaguage, Biml, can be thought of as the operating system for business intelligence. It has pieces in there to handle all aspects of what you need to describe to get the work done. Here, I'm using it to describe what an SSIS package would look like that has an auto increment file name. To make this work for you
Download and install BimlExpress
Open an SSIS project and add a new Biml file. Paste the contents below into it
Edit line 11 and 12 to point to a valid SQL Server and folder on your computer
Right click on BimlScript.biml and select Generate SSIS Package
There should now be an SSIS package in your project called so_45355289.dtsx - run that sucker
Assuming all that works, you can either dig in an modify the supplied Biml to meet your needs (change the queries to actually extract the data you care about and make the flat file format match). Or you can just modify the existing SSIS package and forget about Biml for now. Eventually you'll probably care about it though because it's a huge time saver.
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<FileFormats>
<FlatFileFormat Name="FF Output">
<Columns>
<Column Name="DatabaseName" Length="128" Delimiter="Tab" />
<Column Name="DatabaseId" DataType="Int32" Delimiter="CRLF" />
</Columns>
</FlatFileFormat>
</FileFormats>
<Connections>
<OleDbConnection Name="Config" ConnectionString="Provider=SQLNCLI11;Server=localhost\DEV2016;Initial Catalog=tempdb;Integrated Security=SSPI;" />
<FlatFileConnection Name="Output" FileFormat="FF Output" FilePath="C:\ssisdata\output\P12345.txt" />
</Connections>
<Packages>
<Package Name="so_45355289" ConstraintMode="Linear">
<Connections>
<Connection ConnectionName="Output">
<Expressions>
<Expression ExternalProperty="ConnectionString">#[User::CurrentFileName]</Expression>
</Expressions>
</Connection>
</Connections>
<Variables>
<Variable Name="BaseFileName" DataType="String">P12345</Variable>
<Variable Name="SequenceNumber" DataType="Int32">0</Variable>
<Variable Name="FileExtension" DataType="String">.txt</Variable>
<Variable Name="CurrentFolder" DataType="String">C:\ssisdata\Output</Variable>
<Variable Name="SequencePad" DataType="String" EvaluateAsExpression="true">RIGHT("000" + (DT_WSTR, 3) #[User::SequenceNumber], 3)</Variable>
<Variable Name="FullFileName" DataType="String" EvaluateAsExpression="true">#[User::BaseFileName] + #[User::SequencePad] + #[User::FileExtension]</Variable>
<Variable Name="CurrentFileName" DataType="String" EvaluateAsExpression="true">#[User::CurrentFolder] + "\\" + #[User::FullFileName]</Variable>
</Variables>
<Tasks>
<ExecuteSQL Name="SQL Get Current Sequence" ConnectionName="Config" ResultSet="SingleRow">
<DirectInput><![CDATA[SELECT TOP(1)
(D.LastSequence % 999) + 1 AS LastSequence
FROM
(
SELECT
FEH.LastSequence
FROM
dbo.FileExecutionHistory AS FEH
WHERE
FEH.BaseFileName = ?
-- Handle the does not exist case
UNION
SELECT
CAST(0 AS int)
) D
ORDER BY
D.LastSequence DESC;]]></DirectInput>
<Parameters>
<Parameter Name="0" DataType="String" Length="128" VariableName="User.BaseFileName" />
</Parameters>
<Results>
<Result Name="0" VariableName="User.SequenceNumber" />
</Results>
</ExecuteSQL>
<Dataflow Name="DFT Generate data">
<Transformations>
<OleDbSource Name="OLESRC Query" ConnectionName="Config">
<DirectInput>SELECT name AS DatabaseName, database_id AS DatabaseId FROM sys.databases;</DirectInput>
</OleDbSource>
<FlatFileDestination Name="FFDST Output" ConnectionName="Output" Overwrite="true">
</FlatFileDestination>
</Transformations>
</Dataflow>
<ExecuteSQL Name="SQL Store Sequence" ConnectionName="Config">
<DirectInput><![CDATA[-- Storage pattern
DECLARE
#BaseFileName varchar(50) = ?
, #LastSequence int = ?;
UPDATE
FEH
SET
LastSequence = #LastSequence
FROM
dbo.FileExecutionHistory AS FEH
WHERE
FEH.BaseFileName = #BaseFileName;
INSERT INTO
dbo.FileExecutionHistory
(
BaseFileName
, LastSequence
)
SELECT
D.BaseFileName
, D.LastSequence
FROM
(
VALUES (#BaseFileName, #LastSequence)
) D (BaseFileName, LastSequence)
LEFT OUTER JOIN
dbo.FileExecutionHistory AS FEH
ON FEH.BaseFileName = D.BaseFileName
WHERE
FEH.BaseFileName IS NULL;]]></DirectInput>
<Parameters>
<Parameter Name="0" DataType="String" Length="128" VariableName="User.BaseFileName" />
<Parameter Name="1" DataType="Int32" VariableName="User.SequenceNumber" />
</Parameters>
</ExecuteSQL>
</Tasks>
</Package>
</Packages>
</Biml>
Here is the context
I'm building a library in VB.net able to quickly handle SQL Database datas, in windows form based front end. I'm using ADODB Connections and recordsets.
I managed to link front end to Access databases, MS SQL server, MySQL and, recently I'm working on SQLite databases to provide quick and portables SQL providers.
Here is the problem
As I understand it, SQLite stores single/double/float with IEEE-7 standards, and for exemple a stored value of 9.95 will be read as 9,94999980926514.
So when I load the record again, I can edit it (other fields), and update it. but if I try to edit the float value (lets say 9,94999980926514 > 10) then update it, then I've got an error see sample code
Dim LocRs as ADODB.Recordset
LocRs.Open("SELECT ID_Montant,Mont_Value,Mont_Date FROM tMontants",SQLConnection)
LocRs.addNew
LocRs.("ID_Montant").Value =666
LocRs.("Mont_Value").Value =9.95
LocRs.("Mont_Date").Value =Date.Today
LocRs.Update
LocRs.close
'No Problems'
LocRs.open("SELECT ID_Montant,Mont_Value,Mont_Date FROM tMontants WHERE ID_Montant=666",SQLConnection)
LocRs.Mont_Date.Value=Date.Today.AddDays(-2)
Console.WriteLine(LocRs.("Mont_Value").Value) 'Returns 9,94999980926514'
LocRs.Update
LocRs.close
'No Problem again'
LocRs.open("SELECT ID_Montant,Mont_Value,Mont_Date FROM tMontants WHERE ID_Montant=666",SQLConnection)
LocRs.("Mont_Value").Value=10
LocRs.Update
'Error : Row cannot be located for updating. Some values may have been changed since it was last read. Help Code -2147217864'
The error code seems to be of little help.
I'm using
locRs.LockType = LockTypeEnum.adLockOptimistic
locRs.CursorType = CursorTypeEnum.adOpenKeyset
locRs.CursorLocation = CursorLocationEnum.adUseClient
But I tried a lot of combination without success.
As for the provider I'm using Werner ODBC Provider
I Had a similar problem with Date fields.
When editing a field with an incomplete date like 2012-03-01 12:00:00 instead of 2012-03-01 12:00:00.000, But I corrected the problem by formating the date in it's complete form.
Now I'm stuck with this because the SQLite database stores approximates float value whether I rounded it or not before storing it with an update.
I'm guessing it's a provider problem while computing the update because I can do
"UPDATE tMontants SET Mont_Value = 10 WHERE ID_Montant = 666"
Without any problem
But I'd really like to force the recordset to work in order to integrate SQLlite solutions to every software I've deployed so far.
Thanks for your time.
In my report (BIRT 4.2) I am getting data from a SQL 2008 R2 data base. The client has asked to add a new field to the report. The field has xml data that is formatted as text (example data below). I can display it fine, in html, or Excel; BUT if I have a user selected parameter the report crashes (error message below). Works fine in previews (report & SQL).
I don’t think there is any xml as text in the sample data base, so not seeing how I can create an example using the sample data.
Having string parameters is fine, but as soon as I put the ‘?’ it crashes on deploy via Apache. The parameter is a date.
Works ; where OCMQM1.SUBMIT_DATE >= '2013-12-01'
Fails ; where OCMQM1.SUBMIT_DATE >= ?
I have tried casting the field varchar (1), etc no matter how many characters, or what allowable format I cast to, I get the same error. Even converting the xml to a single character of varchar and then replacing the character with a blank space in at the SQL give the same crash on deploy. I have recreated the report from scratch and modified the existing working report, with identical results.
, replace (
cast (OCMQM1.SVC_OPTIONS as varchar (1) )
, '<' , ' '
) as 'Request_Details'
Error message
The following items have errors:
Table (id = 1293):
+ Cannot execute the statement.
org.eclipse.birt.report.data.oda.jdbc.JDBCException: SQL statement does not return a ResultSet object.
SQL error #1:The value is not set for the parameter number 1.
;
com.microsoft.sqlserver.jdbc.SQLServerException: The value is not set for the parameter number 1. (Element ID:1293)
XML data example
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><form><text id="ExistingServerName" label="Server Name:" multiline="false" readonly="false">WINHPLNSPRD128</text><text id="ProjectFunding" label="How is this project funded?" button="4001" multiline="false" readonly="false">34542 - Enable HealthPlaNET Architecture to Support Multiple Customers</text><text id="ImplementationDate" label="Requested Implementation Date:" multiline="false" readonly="false">12/12/2013 00:00:00</text><text id="CPUNeeded" label="Additinal CPU Needed:" multiline="false" readonly="false">1</text><text id="MemoryNeeded" label="Additional Memory Needed (in GB):" multiline="false" readonly="false">0</text><text id="AdditionalStorage" label="How much additional disk storage is needed (in GB)?" multiline="false" readonly="false">0</text><text id="ExpandDrive" label="If this request is to expand an existing drive, what drive letter should be expanded?" multiline="false" readonly="false"></text></form>
Note The user supplied parameters work fine, when the xml is not part of the report. Adding the field to the SQL, even if not added to the report table, and regardless of placement in the SQL causes the error.
I am getting this error:
String was not recognized as a valid Boolean.Couldn't store <No> in meetsstd Column. Expected type is Boolean
When I am running this query:
SELECT * FROM work_nylustis_2013_q3.nylustis_details WHERE siteid = 'NYLUSTIS-155718' LIMIT 50
From this code:
Adapter.SelectCommand = New NpgsqlCommand(SQL, MLConnect)
Adapter.Fill(subDT) ' This line throws error
The meetsstd field is a varchar(3) and it does store either a 'Yes' or a 'No' value. How is this getting this confused with a boolean - a varchar should not care whether is holds 'Yes', or 'Si', or 'Oui'? And it only happens on 27 records out of the 28,000 in the table.
I usually blame npgsql for this kind of strangeness, but the last entry in the stack trace is: System.Data.DataColumn.set_Item(Int32 record, Object value)
Any clues?
Thanks!
Brad
To check if it is problem with database or with driver you can reduce problem to one row and column using your current environment:
SELECT meetsstd FROM work_nylustis_2013_q3.nylustis_details WHERE sitenum=1
(of course you must change sitenum into primary key)
Then try such query using psql, pgAdmin or some JDBC/ODBC based general editor.
If psql shows such record which raises error with your Npgsql based application then problem is with Npgsql driver or problem is with displaying query results.
If other tools shows such strange errors then problem is with your data.
Have you changed type of meetsstd field? Mayby you try to show it on some grid and this grid used Boolean field which was converted to Yes/No for displaying?
I have an SQL Server Compact Edition 4.0 database file which has .sdf extension. The database is created by a book cataloging application. The database has many tables, among them a table called Book which contains one row for each book in the database. The table has many columns among them a column called FrontCover which contains a string value which has 2 parts: path part and file name part of the image file for the front cover of a book, for example:
Documents and Settings\Boson\My Documents\Books\Covers\0596003978.jpg
In this example path part is:
'Documents and Settings\Boson\My Documents\Books\Covers'
while file part is:
'0596003978.jpg'
Some books do not contain any value for the column FrontCover because the front cover is not available. For such books column FrontCover is empty. However if a book has a front cover image file then the string value has the same path part but different file part. For example for another book column FrontCover has this value:
'Documents and Settings\Boson\My Documents\Books\Covers\1590596633.jpg'
As we can see the path part is the same as in the first example, namely:
'Documents and Settings\Boson\My Documents\Books\Covers'
but the file part is different:
'1590596633.jpg'
PROBLEM:
I want to change the whole table Book so that string values of the column FrontCover are modified for every book in the table in such a way that file part is kept the same but the path part is changed from:
'Documents and Settings\Boson\My Documents\Books\Covers'
to
'Books\AEM database\Covers'
The string value of the column FrontCover for the book in the first example would thus change from:
'Documents and Settings\Boson\My Documents\Books\Covers\0596003978.jpg'
to
'Books\AEM database\Covers\0596003978.jpg'
File part is the same but the path part is changed. I want to change the whole Book table so that the file part of the string value for column FrontCover is kept the same but the path part is changed as specified above.
The book cataloging application which owns the .sdf database file is stupid and cannot do the job. Therefore I have installed a simple open source SQL viewing/editing application called SQL Compact Query Analyzer (http://sqlcequery.codeplex.com/). SQL Compact Query Analyzer can open .sdf database files and accepts SQL commands in order to modify the .sdf database file.
Can you please help me with the SQL commands which can do the job?
Thank you very much in advance for your help.
best regards
UPDATE Book SET FrontCover = REPLACE(
CAST(FrontCover AS NVARCHAR(300)),
'Documents and Settings\Boson\My Documents\Books\Covers',
'Books\AEM database\Covers')
WHERE FrontCover like
'Documents and Settings%'
Note: the where clause may not be necessary but it ensures that you only replace strings that start with 'Documents and Settings...'
The answer posted by Paul Brown is OK however it produced the following error code:
ErrorCode: -2147467259 [SQL Server Compact ADO.NET Data Provider] HResult: -2147217900, NativeError: 25922 ErrorMessage: The specified argument value for the function is not valid. [ Argument # = 1,Name of function(if known) = REPLACE ]
The reason for this is because the type of date for the column FrontCover is NTEXT (I forgot to specify that in my question) and function REPLACE does not allow NTEXT variable. So we have to covert FrontCover variable with the following command before passing it to REPLACE:
CAST(FrontCover AS NVARCHAR(100))
Therefore the answer to my question is:
UPDATE Book
SET FrontCover = REPLACE (CAST(FrontCover AS NVARCHAR(100)), 'Documents and Settings\Boson\My Documents\Books\Covers\', 'Books\AEM database\Covers\')