I have to read data in batches and I want to know what is the limit of an sql-query and is there any limit in JDBC driver as well for max-sql-query size. I am using oracle 19c database.
Related
I migrated data from Oracle to SQL Server, the rows count in Oracle is around 820m rows, but when SSMA finished the migration "message said 100% successfully migrated" the rows count in SQL Server was around 745m rows.
Is there a limit on the number of rows?
There is no limit on number of rows for SSMA
Please keep in mind that Oracle has read consistency, meaning that the cursor will produce rows as they were in the beginning of select. Maybe, when SSMA started to run there were 745m in Oracle.
We have a linked server (OraOLEDB.Oracle) defined in the SQL Server environment. Oracle 12c, SQL Server 2016. There is also an Oracle client (64 bit) installed on SQL Server.
When retrieving data from Oracle (a simple query, getting all columns from a 3M row, fairly narrow table, with varchars, dates and integers), we are seeing the following performance numbers:
sqlplus: select from Oracle > OS File on the SQL Server itself
less than 2k rows/sec
SSMS: insert into a SQL Server table select from Oracle using OpenQuery (passthrough to Oracle, so remote execution)
less than 2k rows/sec
SQL Export/Import tool (in essence, SSIS): insert into a SQL Server table, using the OLEDB Oracle for source and OLEDB SQL Server for target
over 30k rows/second
Looking for ways to improve throughput using OpenQuery/OpenResultSet, to match SSIS throughput. There is probably some buffer/flag somewhere that allows to achieve the same?
Please advise...
Thank you!
--Alex
There is probably some buffer/flag somewhere that allows to achieve the same?
Probably looking for the FetchSize parameter
FetchSize - specifies the number of rows the provider will fetch at a
time (fetch array). It must be set on the basis of data size and the
response time of the network. If the value is set too high, then this
could result in more wait time during the execution of the query. If
the value is set too low, then this could result in many more round
trips to the database. Valid values are 1 to 429,496, and 296. The
default is 100.
eg
exec sp_addlinkedserver N'MyOracle', 'Oracle', 'ORAOLEDB.Oracle', N'//172.16.8.119/xe', N'FetchSize=2000', ''
See, eg https://blogs.msdn.microsoft.com/dbrowne/2013/10/02/creating-a-linked-server-for-oracle-in-64bit-sql-server/
I think there are many way to enhance the performance on the INSERT query, I suggest reading the following article to get more information about data loading performance.
The Data Loading Performance Guide
There are one method you can try which is minimizing the logging by using clustered index. check the link below for more information:
New update on minimal logging for SQL Server 2008
Is there a way to identify the number of sql running parallel in Oracle 10g database?
What would be the maximum number of SQL run in Oracle at a time?
This is really needed to identify the performance of database response time. There is a use case wherein nearly 1 million sql need to be fired and amount of data in the table also few billions.
Really much appreciated your help in this.
I am writing a small utility that index's a SQL Server 2005 table data to LUCENE data store. i am using a JDBC SQL Server driver for connectivity. The result set returned from the server has around 2 million rows. This always throws a Out of Memory Exception. I have increased the Heap size for the client machine to around 1.6 GB, but to no avail. How can i fetch such large data sets, using JDBC giving limited memory .
Kind Regards
Have you looked at LIMIT and TOP in SQL? Also, try Googling for phrases like 'SQL query pagination'.
We know the max size of SQL data files allowed in SQL express editions are 2 GB, 4 GB , 10 GB for SQL 2000, 2005, 2008 express respectively.
Is there any way we could see the max size of the database allowed via SQL query?
The Max size is exactly that. I'm not aware of any query you can run to get this information directly.
You could get the SQL Server version:
SELECT ##VERSION
and infer from that.