We are doing a migration from our old system (sql server 2008) to the new system (SQL server 2012) , the data sources we are using are Remote so we have it configured as a linked servers , the data in the source we are migrating have special data (Geography type) , we are migrating the data per customer , so some customers have more data that the others, we batch the data and we are using OPENQUERY to pull the Spatial data across. For the customers with less data the migration goes smoothly and it completes successfully , but for customers with more than couple million records in one table the migration stops and gives mainly 2 errors:
This how the error comes like :
OLE DB provider "yyy" for linked server "xxx" returned message "Query timeout expired".
Msg 7399, Level 16, State 1, Server nnn, Line 1
The OLE DB provider "yyy" for linked server "xxx" reported an error. Execution terminated by the provider because a resource limit was reached.
Msg 7320, Level 16, State 2, Server ttt , Line 1
Cannot execute the query "
select top (200000)
[row] = row_number () over ( order by t.[x])
, .....
, [Spatial] = cast(ts.[Spatial] as varbinary(max))
from [..].[..].[..] t
join [...].[..].[… ] s
on t.[..] = s.[...]
where (t.[x] > '00000000-0000-0000-0000-000000000000')
and v.[x] = x
order by t.[x]
" against OLE DB provider "yyy" for linked server "xxx".Build step 'Execute Windows batch command' marked build as failure
also this problem happened with one other table that doesn't have a spatial data in it.
The approaches we tried to follow,
We have increased the timeout of the query,
We have dropped the batch size to 200,000 per batch
The Provider is “ in process mode”
we only have couple linked servers so the buffer size is more than acceptable ,
We tried to run the migration using an admin role to make sure it’s not a permissions problem
We are thinking this might be a network problem but it’s not a Load Balancer issue. maybe its something else,
The other error that comes frequently is
HResult 0x40, Level 16, State 1
TCP Provider: The specified network name is no longer available.
Any idea's for what could be a reason will be so much appreciated
Thank you,
Lsaif
I would say the "The specified network name is no longer available." error indicates no response from the remote server. Since SQL Server hasn't "heard" from the remote server in a while, it gives up. I would cut down the batch size to something really small and increase with success (rather than the other way around). That way you'll find a batch size that works. Also, this may vary between "customers" depending on your connection to them (i.e., type and size of line, traffic on the line, etc.).
Personally, I like the BCP OUT/BCP IN option as well because I know it works however, you still have to consider the transfer method of the data from the remote server. If you have a robust enterprise MFT over a dedicated T1 or better, you probably won't have an issue.
SSIS offers more of a direct transfer but I imagine you'll run into the same network issues you're having now. That said, you can create a general BCP solution within SSIS.
Related
We're running Azure SQL Single Database (Serverless tier) and are having problems with our development environment SQL servers appearing not to pause despite the DBs being out of use and autopause being correctly configured.
We've narrowed it down to SSMS running the following SQL query against the DB if it has a query window open but we have no idea how to prevent it.
(#type int)SELECT file_id, name, size AS size_8KB, max_size AS max_size_8KB, ISNULL(FILEPROPERTY(name, 'SpaceUsed'), size) AS space_used_8KB
FROM sys.database_files
WHERE type = #type ORDER BY size DESC
This query is run every 5 - 7 minutes while SSMS is open. This is causing us considerable headache and cost.
Does anyone know what feature of SSMS is calling this query and how to turn it off?
As I know about the serverless, when the database is inactive, it can be paused. But when the SSMS or query editor opened, the connection to SQL database is open which means the database is always active., then the autopause congifuration won't work.
Ref this document: https://learn.microsoft.com/en-us/azure/azure-sql/database/serverless-tier-overview#performance-configuration
HTH.
I'm running a patched SQL 2014 trying to backup a database to one of our Azure Storage BLOBs, using:
BACKUP DATABASE [DB]
TO URL = N'https://storage.blob.core.windows.net/server-mssqlserver/DB.bak'
WITH CREDENTIAL = N'AzureCredential'
,NOFORMAT
,NOINIT
,NAME = N'DBA_DB-Full Database Backup'
,NOSKIP
,NOREWIND
,NOUNLOAD
,COMPRESSION
,STATS = 5
GO
but the query throws the following error:
Msg 3292, Level 16, State 9, Line 1
A failure occurred while attempting to execute Backup or Restore with a URL device specified. Consult the Windows Event Log for details.
Msg 3013, Level 16, State 1, Line 1
BACKUP DATABASE is terminating abnormally.
Checking the server's Event Logs shows the actual error as:
SQLVDI: Loc=IdentifySQLServer. Desc=MSSQLSERVER. ErrorCode=(5)Access is denied.
. Process=4668. Thread=6596. Client. Instance=MSSQLSERVER. VD=.
I have made sure that the SQL Server Agent service's account has the Create global objects policy, and also made sure the SQL VSS Writer service is running under the Local System account. The error keeps happening!
Is there something I can do to fix it, or just log some more detailed error messages than the "SQLVDI: Loc=IdentifySQLServer" one above?
Crikey, this is one of those "no idea how I fixed it" things.
Before I went to lunch, I could reliably generate the error in question by running the provided T-SQL, yet when I came back from lunch the BACKUP command completed fine!
The main thing I remember changing was on the Azure side, where I created a SAS (Shared Access Signature).
This is supposedly not required for SQL 2014 as it uses an actual Azure Credential to connect to the storage instead. I actually created the SAS for an instance of SQL Server 2016 that I want backing up to the same container, and that may have opened the access pathway to the container for SQL Server 2014 too!
Many thanks to Sean Gallardy for recommending ProcMon, which showed a whole ton more error log information than the "ErrorCode=(5)Access is denied" message did.
We are having an issue with a server that we linked to our sql server 2012 instance.
The server that we linked through an odbc or oledb connection is Pervasive SQL.
Selecting from the linked server this way works as long as the result set is small:
select * from linked_server.database..mytable
If the result set is more than around 2mb (might be slightly more or less) then we get this error:
Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "MSDASQL" for linked server "KSLAP208" reported an error. The provider reported an unexpected catastrophic failure.
Msg 7330, Level 16, State 2, Line 1
Cannot fetch a row from OLE DB provider "MSDASQL" for linked server "KSLAP208".
However, the silly thing is that SSMS 2012 for instance displays all the columns that need to be returned and then immediatley errors out.
What would cause SSMS 2012 to produce an error on a result set that is too big? (more than 1-2mb)??
here's what i see in the odbc administrator:
Option 1, disable pre-fetch
Sometimes, disabling pre-fetch will solve problems like this.
You can disable the prefetch by clearing the "Enable pre-fetch of data
for queries" check box in the Performance tab when creating a DSN or
adding "PREFETCH=0" to the connection string when creating a DSN-less connection.
More on this:
In the ODBC Data Source Administrator panel on the server, configure
your linked server from System DSN. Under the Performance tab, uncheck
the box 'Enable pre-fetch of data for queries'. OK, OK. Delete and
re-create your linked server in SQL Management Studio.
Source
Option 2, Allow inprocess
Go to Linked Servers
Providers
MSOLAP (or similar)
Properties
Make sure "Allow inprocess" is enabled. Even if it is, turn it off. Save. And then turn it back on. Sometimes resetting this can help.
I am trying to retrieve around 200 billion rows from a remote SQL Server. To optimize this, I have limited my query to use only an indexed column as a filter and am selecting only a subset of columns to make the query look like this:
SELECT ColA, ColB, ColC FROM <Database> WHERE RecordDate BETWEEN '' AND ''
But it looks like unless I limit my query to a time window of a few hours, the query fails in all cases with the following error:
OLE DB provider "SQLNCLI10" for linked server "<>" returned message "Query timeout expired".
Msg 7399, Level 16, State 1, Server M<, Line 1
The OLE DB provider "SQLNCLI10" for linked server "<>" reported an error. Execution terminated by the provider because a resource limit was reached.
Msg 7421, Level 16, State 2, Server <>, Line 1
Cannot fetch the rowset from OLE DB provider "SQLNCLI10" for linked server "<>".
The timeout is probably an issue because of the time it takes to execute the query plan. As I do not have control over the server, I was wondering if there is a good way of retrieving this data beyond the simple SELECT I am using. Are there any SQL Server specific tricks that I can use? Perhaps tell the remote server to paginate the data instead of issuing multiple queries or something else? Any suggestions on how I could improve this?
This is more of the kind of job SSIS is suited for. Even a simple flow like ReadFromOleDbSource->WriteToOleDbSource would handle this, creating the necessary batching for you.
Why read 200 Billion rows all at once?
You should page them, reading say a few thousand rows at a time.
Even if you do genuinely need to read all 200 Billion rows you should still consider using paging to break up the read into shorter queries - that way if a failure happens you just continue reading where you left off.
See efficient way to implement paging for at least one method of implementing paging using ROW_NUMBER
If you are doing data analysis then I suspect you are either using the wrong storage (SQL Server isn't really designed for processing of large data sets), or you need to alter your queries so that the analysis is done on the Server using SQL.
Update: I think the last paragraph was somewhat misinterpreted.
Storage in SQL Server is primarily designed for online transaction processing (OLTP) - efficient querying of massive datasets in massively concurrent environments (for example reading / updating a single customer record in a database of billions, at the same time that thousands of other users are doing the same for other records). Typically the goal is to minimise the amout of data read, reducing the amount of IO needed and also reducing contention.
The analysis you are talking about is almost the exact opposite of this - a single client actively trying to read pretty much all records in order to perform some statistical analysis.
Yes SQL Server will manage this, but you have to bear in mind that it is optimised for a completely different scenario. For example data is read from disk a page (8 KB) at a time, despite the fact that your statistical processing is probably only based on 2 or 3 columns. Depending on row density and column width you may only be using a tiny fraction of the data stored on an 8 KB page - most of the data that SQL Server had to read and allocate memory for wasn't even used. (Remember that SQL Server also had to lock that page to prevent other users from messing with the data while it was being read).
If you are serious about processing / analysis of massive datasets then there are storage formats that are optimised for exactly this sort of thing - SQL Server also has an add on service called Microsoft Analysis Services that adds additional online analytical processing (OLAP) and data mining capabilities, using storage modes more suited to this sort of processing.
Personally I would use a data extraction tool such as BCP to get the data to a local file before trying to manipulate it if I was trying to pull that much data at once.
http://msdn.microsoft.com/en-us/library/ms162802.aspx
This isn't A SQL Server specific answer, but even when the rDBMS supports server side cursors, it's considered poor form to use them. Doing so means that you are consuming resources on the server even though the server is still waiting for you to request more data.
Instead you should reformulate your query usage so that the server can transmit the entire result set as soon as it can, and then completely forget about you and your query to make way for the next one. When the result set is too large for you process all in one go, you should keep track of the last row returned by the current batch so that you can fetch another batch starting at that position.
Odds are the remote server has the "Remote Query Timeout" set. How long does it take for the query to fail?
Just run into the same problem, I also had the message at 10:01 after running the query.
Check this link. There's a remote query timeout setting under Connections that's setup to 600secs by default and you need to change it to zero (unlimited) or other value you think is right.
Try to change remote server connection timeout property.
For that go to SSMS, connect to the server, right click on server's name in object explorer, further select Properties -> Connections and change value in the Remote query timeout (in seconds, 0 = no timeout) text box.
We have a database running on SQL 2005. One of the store procedure looks up a user's email address from Active Directory using a linked server. The call to the linked server occurs in a database function.
I'm able to call is successfully from my Asp.Net application the first time, but periodically after that, it fails with the following error:
{"The requested operation could not be performed because OLE DB provider \"ADsDSOObject\" for linked server \"ADSI\" does not support the required transaction interface."}
It appears that the amount of time between calling the function affects whether the linked server query will work correctly. I am not using any transactions. When I try calling the function in a quick make-shift SQL script, it runs fine everytime (even when tested in quick succession).
Is there some sort of transaction being left open that naturally dies if I don't try calling the procedure again? I'm at a loss here.
Here is the simple call in the store procedure:
DECLARE #email varchar(50)
SELECT #email = LEFT(mail, 50)
FROM OPENQUERY (
ADSI,
'SELECT mail, sAMAccountName FROM ''LDAP://DC=Katz,DC=COM'' WHERE objectCategory = ''Person'' AND objectClass = ''User'''
)
WHERE sAMAccountName = CAST(#LoginName AS varchar(35))
RETURN #email
I've worked with SQL Server linkservers often, though rarely LDAP queries... but I got curious and read the Microsoft support page linked to in Ric Tokyo's previous post. Towards the bottom it reads:
It is typical for a directory server
to enforce a server limitation on the
number of objects that will be
returned for a given query. This is to
prevent denial-of-service attacks and
network overloading. To properly query
the directory server, large queries
should be broken up into many smaller
ones. One way to do this is through a
process called paging. While paging is
available through ADSI's OLEDB
provider, there is currently no way
available to perform it from a SQL
distributed query. This means that the
total number of objects that can be
returned for a query is the server
limit. In the Windows 2000 Active
Directory, the default server limit is
1,000 objects.
I'm thinking that the reason it fails on you (or not) depending on whether call it from the app or from a "quick make-shift sql script" (as you put it) might be related to the security context under which the operation is executing. Depending on how the link server connection was set up, the operation could be being executed under a variety of possible credentials depending on how you initiate the query.
I don't know, but that's my best guess. I'd look at the linkserver configuration, in particular the linkserver settings for what set of credentials are used as the security context under which operations executed across the linkserver run.
Rather then query Active Directory through a linked server, you might be better off caching your AD data into a SQL database and then querying that instead. You could use Integration Services by creating a OLE DB connection using "OLE DB PRovider for Microsoft Directory Services" and having a DataReader source with a query like:
SELECT physicalDeliveryOfficeName, department, company, title, displayName, SN,
givenName, sAMAccountName, manager, mail, telephoneNumber, mobile
FROM 'LDAP://DC=SOMECO,DC=COM'
WHERE objectClass='User' and objectCategory = 'Person'
order by mail
Using this method you will still run into the 1000 row limit for results from an AD query (note it is NOT advisable to try and increase this limit in AD, it is there to prevent the domain controller from becoming overloaded). Sometimes its possible to use a combination of queries to return the full data set, e.g. names A - L and M - Z
Alternatively you could use the CSVDE command line utility in Windows Server to export your directory information to a CSV file and then import it into a SQL database (see http://computerperformance.co.uk/Logon/Logon_CSVDE_Export.htm for more info on exporting AD data with CSVDE).
please read the support page from Microsoft
I suspect that it might be the cached query plan due to your statement that "When I try calling the function in a quick make-shift SQL script, it runs fine everytime (even when tested in quick succession)."
Could you try executing your stored procedure like so:
EXEC usp_MyProcedure WITH RECOMPILE
This question appears in the top of the first google page when search for the error string but has not valid answer.
This error happens intermitently when isolation level is not specified on .NET code nor in Store Procedure.
This error also happens in SQL Server 2008.
The fix is force SET TRANSACTION ISOLATION LEVEL READ (UN)COMMITTED because a isolation level any higher is not supported by Active Directory and SQL Server is trying to use SERIALIZABLE.
Now, as this error is intermitent. Why is ADO.NET or SQLServer switching its default isolation to SERIALIZABLE sometimes and sometimes not? What triggers this switching?