I'm using SQL Link server for fetching data from MariaDB.
But I fetching issue with slowness when i used MariaDB from link server.
I used below scenarios to fetch result (also describe time taken by query)
Please suggest if you have any solutions.
Total number of row in patient table : 62520
SELECT count(1) FROM [MariaDB]...[webimslt.Patient] -- 2.6 second
SELECT * FROM OPENQUERY([MariaDB], 'select count(1) from webimslt.patient') -- 47ms
SELECT * FROM OPENQUERY([MariaDB], 'select * from webimslt.patient') -- 20 second
This isn’t really a fair comparison...
SELECT COUNT(1) is only returning a single number and will probably be using an index to count rows.
SELECT * is returning ALL data from the table.
Returning data is an expensive (slow) process, so it will obviously take time to return your data. Then there is the question of data transfer, are the servers connected using a high speed connection? That is also a factor in this. It will never be as fast to query over a linked server as it is to query your database directly.
How can you improve the speed? I would start by only returning the data you need by specifying the columns and adding a where clause. After that, you can probably use indexes in Maria to try to speed things up.
Related
I am trying to fasten up a SQL Server report regarding the IBM OS/400 operating system for my sales department.
A colleague of mine (which left the company) did this report and used a ton of sub selects.
The report usually takes about 30 min to process and often just fails to be displayed. I already tried to cut out some tables/rows in hopes of fastening up the process without success (all is needed by the sales department).
It works over all relevant data (orders, customers, articles, our order at the manufacturer, the manufacturer and so on). Any ideas?
I can't index it, due to the OS/400 system; guess it would be a new programming task for our contractor which leads to costs.
Can I use some clever joins? or somehow reduce the amount of subselects?
Are you using 4 part names in your query? That's probably your problem...
From SQL server...
-- Pull all rows from the table(s) back to MS SQL server and do the where locally on the MS SQL server
select * from LINKEDSVR.MYIBMI.MYLIB.MYTBL where locnbr = '00335';
-- Sends the statement to IBM i server for processing, only results are returned..
select * from openquery(LINKEDSVR, 'select * from MYTBL where locnbr = ''00335''');
Try running the subselects first, sending the output of each to its own table.
Update statistics on the tables. Then run the rest of the SQL, replacing what were originally subselects with the tables created in the first step.
Treat multiple layers of nesting the same way: each layer is its own insert into another table.
I've found that query optimizers have a hard time with complex SQL. Breaking-out the subqueries into separate steps often resolves this.
Between runs my preference is to leave the data intact as a reference in case debugging is needed, then truncate the tables as the first step of a run.
Responding to eraser's comments
Assuming your original query takes this general form:
select [columns] from
(-- subquery
select [columns] from TableA
) as Subquery
from TableB
where mainquery_where_clause
Re-write:
-- Create a table to handle results for your subquery:
Create Table A ;
-- Update the data distribution statistics:
update stats (TableA) ;
-- Now run the subquery:
insert into SubQTable select [columns] from TableA
-- Now run the re-written main query:
Select [columns]
from TableA, TableB
where TableA.joincol = TableB.joincol
and mainquery_where_clause ;
I noticed some syntax issues with the SQL you posted. Looks like something got left out. But the principle of my answer remains the same. Please note that applying my suggestion may not help, as there are potentially many variables to your scenario; you mentioned subqueries, so I chose to address that.
Halfer's suggestion is a great one: edit your original question, adding the SQL code, and putting it in the "{}" supplied by the text editing tool.
I strongly suggest that you obtain the SQL execution plan and post the results.
UPDATE
I have noticed that when I do a simple view from this table, results come back fairly quickly, but as soon as a I make WHERE clause restriction it slows way down.
I'm running a query that runs well, about 1.5 minutes. Yet when I include a single column from a linked Oracle DB the query takes 19 minutes. Is performance degradation like this normal?
I'm also not familiar with where I should start troubleshooting this type of issue, and am new Linked Servers.
Thank you,
UPDATE
Below is the code that creates the Oracle Linked Server DB view
SELECT Patient
,Account
,MR#
,Diagnosis
,ICD9
,TransferEMTALAFormsCmpltd
,Disposition
,AdmittingDxTranscribed
,AxisIPrimaryDx
,AgeDOB
,sex
,age
,EDRecordSentToEDM
,TimeRNSignature
,timemdsignature
,RNSgntr
,TriageByNameEntered as 'Triage_End'
,TriageStartTime as 'Triage_Start'
,AddedToAdmissionsTrack
,StatusAdmitConfirmed
,SUBSTRING(statusadmitconfirmed,1,4)+'-'+SUBSTRING(statusadmitconfirmed,7,2)+'-'+SUBSTRING(statusadmitconfirmed,1,4)+' '+SUBSTRING(statusadmitconfirmed,9,2)+':'+SUBSTRING(statusadmitconfirmed,11,2)+':00.000' as 'Admit_Cnrfm_String'
,AdmittingMD
,AreaOfCare
,EDMD
,EDMDID
,Specialty
,AccessProceduresED
,MDSgntr
,Arrival
,ArrivalED
,ChiefComplaint
,TransferringFacility
,ReferMD
,PrivateName
FROM [BMH-EDIS-CL]..[WELLUSER].[Patient_Chart] a
LEFT OUTER JOIN [BMH-EDIS-CL]..[WELLUSER].[Patient_Diagnoses] b
ON a.Master_Rec_Id=b.Master_Rec_Id
AND a.Slave_Rec_Id=b.Slave_Rec_Id
WHERE TriageStartTime > '201299999999'
Just getting a single result out of this view takes some time. When I try to add EDMD to the query I am working on, this is what causes the massive slow down, I have not yet been able to review Materialized Views as mentioned in the comments.
I'm new to SQL (using postgreSQL) and I've written a java program that selects from a large table and performs a few functions. The problem is that when I run the program I get a java OutOfMemoryError because the table is simply too big. I know that I can select from the beginning of the table using the LIMIT operator, but is there a way I can start the selection from a certain index where I left off with the LIMIT command? Thanks!
There is offset option in Postgres as in:
select from table
offset 50
limit 50
For mysql you can use the follwoing approaches:
SELECT * FROM table LIMIT {offset}, row_count
SELECT * FROM table WHERE id > {max_id_from_the previous_selection} LIMIT row_count. First max_id_from_the previous_selection = 0.
This is actually something that the jdbc driver will handle for you transparently. You can actually stream the result set instead of loading it all into memory at once. To do this in MySQL, you need to follow the instructions here: http://javaquirks.blogspot.com/2007/12/mysql-streaming-result-set.html
Basically when you create you call connection.prepareStatement you need to pass ResultSet.TYPE_FORWARD_ONLY and ResultSet.CONCUR_READ_ONLY as the second and third parameters, then call setFetchSize(Integer.MIN_VALUE) on your PreparedStatement object.
There are similar instructions for doing this with other databases which I could iterate if needed.
EDIT: now we know you need instructions for PostgreSQL. Follow the instructions here: How to read all rows from huge table?
I have a database with many tables that get used, and many tables that are no longer used. While I could sort through each table manually to see if they are still in use, that would be a cumbersome task. Is there any software/hidden feature that can be used on a SQL Server/Oracle database that would return information like "Tables x,y,z have not been used in the past month" "Tables a,b,c have been used 17 times today"? Or possibly a way to sort tables by "Date Last Modified/Selected From"?
Or is there a better way to go about doing this? Thanks
edit: I found a "modify_date" column when executing "SELECT * FROM sys.tables ORDER BY modify_date desc", but this seems to only keep track of modifications to the table's structure, not its contents.
replace spt_values with the tablename you are interested in, the query will give the the last time it was used and what it was used by
From here: Finding Out How Many Times A Table Is Being Used In Ad Hoc Or Procedure Calls In SQL Server 2005 And 2008
SELECT * FROM(SELECT COALESCE(OBJECT_NAME(s2.objectid),'Ad-Hoc') AS ProcName,execution_count,
(SELECT TOP 1 SUBSTRING(s2.TEXT,statement_start_offset / 2+1 ,
( (CASE WHEN statement_end_offset = -1
THEN (LEN(CONVERT(NVARCHAR(MAX),s2.TEXT)) * 2)
ELSE statement_end_offset END) - statement_start_offset) / 2+1)) AS sql_statement,
last_execution_time
FROM sys.dm_exec_query_stats AS s1
CROSS APPLY sys.dm_exec_sql_text(sql_handle) AS s2 ) x
WHERE sql_statement like '%spt_values%' -- replace here
AND sql_statement NOT like 'SELECT * FROM(SELECT coalesce(object_name(s2.objectid)%'
ORDER BY execution_count DESC
Keep in mind that if you restart the box, this will be cleared out
In Oracle you can use the ASH (Active Session History) to find info about SQL that was used. You can also perform code coverage tests with the Hierarchical profiler, where you can find which parts of the stored procedures is used or not used.
If you wonder about the updates on table data, you can also use DBA_TAB_MODIFICATIONS. This shows how many inserts, updates, deletes are done on a table or table partition. As soon as new object statistics are generated, the row for the specified table is removed from DBA_TAB_MODIFICATIONS. You still have help here, since you could also have a peek in the table statistics history. This does not show anything about tables that are queried only. If you really need to know about this, you are to use the ASH.
Note, for both ASH and statistics history access, you do need the diagnostics or tuning pack license. (normally you would want this anyway).
If you use trigger you can detect update insert or delete on table.
Access is problably more difficult.
I use a combination of static analysis in the metadata to determine tables/columns which have no dependencies and runtime traces in SQL Server to see what activity is happening.
Some more queries that might be useful for you.
select * from sys.dm_db_index_usage_stats
select * from sys.dm_db_index_operational_stats(db_id(),NULL,NULL,NULL)
select * from sys.sql_expression_dependencies /*SQL Server 2008 only*/
The difference betweeen what the first 2 DMVs report is explained well in this blog post.
Ed Elliott's open source tool, SQL Cover, is a good bet and has built-in support for the popular unit testing tool, tSQLt.
I'm getting the following error: AeSymResolveStatement [775] ... Meta-SQL error at or near position 34338 in statement (108,512). The SQL statement itself is over 40,000 chars long, hence the question.
The DB is oracle. Running on Tools 8.49.24.
I know that there is a limit on the size of the SQL used in an Application Engine (SQL Step). I had once recieved a similar error while trying to use an exceptionally long SQL in an Application Engine.
I wouldn't be surprised if that same limit applies to SQL Objects.
To fix the problem, I was able to split the SQL into 2 (was an update statement). Hopefully that's possible in your case as well.
There is no such limit.
You can confirm this yourself by creating an SQL like:
select 'x' from PS_INSTALLATION where
1 = 1 and
1 = 1 and
1 = 1 and
1 = 1 and
/* ... copy paste '1 = 1 and' 90000 times or so times more */
1 = 1
Although it makes pside quite slow, It saves and validates just fine.
There are limits within PeopleCode, mostly due to the limits on string length, however I have never found a limit on stored SQL statements.
Personally I'd look at breaking the statement into pieces in some way.
You could:
Using the inbuilt looping mechanism of App Engines
Use a mixture of SQL and PeopleCode
Use a temporary table and perform intermediate SQLs, storing in the temp table
Apart from giving your database a heart seizure, not the mention the DBA when he sees the statement in the SQL monitor. You are saving yourself a world of pain if you ever have to look at the statement again.
I think the SQLs in App Engines are stored as longs, so it would be 4GB under Oracle, something similarly huge under DB2.