I am using IgniteCache.loadCache(null, keyClassName, sqlArray) to load RDMS data into cache by running sql query specified by sqlArray
It looks that loadCache internally will run the sqlArray with ThreadPool(each sql will be run within a task)
My question is:
Does IgniteCache internally will control the parallesm? I have the following scenario:
My datasource's max connection is set to 200.
The sqlArray's length is about 1000 since I have a large table:
select * from person where id >=0 and i <=20000
...
select * from person where id >=10000000 and i <=10020000
If all these 1000 sql runs at the same time, then the connection would be unavailable from the connection pool which will lead to error
IgniteCache.loadCache method fully relies on configured CacheStore implementation. It looks like CacheAbstractJdbcStore is support parallelism internally.
By default, pool size is equal number of available processors, but you are free to change it with CacheAbstractJdbcStore.setMaximumPoolSize(int) method.
So, you'll run out of connections, if only you have more than 200 processor available.
I am using an ORACLE db with SAS/connect.
I recently implemented a change in my libname statement (a week ago) in which I added the following (don't know if related to issue):
insertbuff=10000 updatebuff=10000 readbuff=10000
Starting yesterday, I have been having an ORACLE issue when, after doing a
proc sql;
drop table oralib.mytable;
quit;
data oralib.mytable;
set work.mytable;
run;
I get the following error:
ERROR: ERROR: ERROR: ORACLE execute error: ORA-04031: unable to
allocate 4160 bytes of shared memory ("shared pool","unknown
object","sga heap(1,0)","modification "). With the occurrence of the above ERROR, the error limit of 1 set by the
ERRLIMIT= option has been reached. ROLLBACK has been issued(Any Rows processed after the last COMMIT are lost).
Total rows processed: 1001
Rows failed : 1
It seems to happen randomly on any table of any size. Sometimes it will go through, sometimes (most of the times) it won't. Is there a shared pool release I should do from SAS?
Thanks for your help!
The shared pool is a memory structure on Oracle which keeps the following stuff:
data dictionary cache
SQL query and PL/SQL function result caches
storage for recently executed code in its parsed form
It is possible to flush the shared pool, but this is not a good idea and I would not recommend it. What you have to do is size the shared pool of the database properly. Note that the shared pool is a pool for the entire Oracle instance - it is not on a per user base. So, if there are other users of the database, they might contribute the problem. I doubt that any particular query is the cause and I guess that the problem is that the shared pool is undersized.
In case you have some DBA privileges granted for your user, you can check the current shared pool size by running the following query:
SELECT * FROM v$sgainfo;
You can increase the size of the shared pool with the following query
ALTER SYSTEM SET SHARED_POOL_SIZE = 200M;
Nevertheless, the best solution will be turn to the DBA managing the database (if there is such).
I'm not a SAS guy, so, I'll answer your question from the POV of an Oracle DBA, which is what I am.
ORA-04031 means you're running out of space in the shared pool. With a product like SAS, I'm sure they have a recommended minimum size that the shared pool should be. So, you should check the SAP installaiton documentation, and confirm whether your database has a large enough shared pool size set. (Use show parameter shared_pool_size to see what size it's set in yourr database.)
Second, I'm not familiar with the changes you made, and I'm not sure if that would have an effect on the shared pool utilization. Perhaps check the SAS documentation on that one.
Third, it could be an Oracle bug. You should check My Oracle Support for your version of Oracle, and do a search on ORA-04031, with those specific arguments you are seeing in your error message. If it's a known bug, there may be a patch already available.
If it's none of the above, you may need to open an SR with Oracle.
Hope that helps.
Has anyone ever encountered a problem where a SQL query from a classic ASP page returns partial results, but no errors?
I have a situation where a particular query on a page (one of many throughout the system) is returning a different number of rows each time it is run, and always less than the "correct" number, as determined by running the SQL against the server directly.
I think this may be related to the connection timeout (since this occurs with a long-running query and my timing shows that it is returning very close to the timeout), but I am receiving no timeout errors. Instead, as far as I can tell, it causes no errors, and returns a valid DataSet which the code then loops over to build the results table.
Since there is no indication that an error has occurred, there is no suggestion that the data is not complete which means that users can no longer trust this report. Generally with this system, where SQL timeouts do occur frequently for "large" queries, we get error messages displayed on the page.
Investigations
I've check the HTML source to make sure there are no injected error's that I'm missing and that all tags are well-formed and that expected page elements are present. This indicates it's not an error writing a particular row from the results.
** Furthermore, the number of rows returned is different each time.
I've verified that the exact same query is being run each time.
I've verified that the data in the DB is not changin underneath the report (it's historic and I've cross-checked by running the report and the query against the DB at the same time.)
I've tried to manually print any errors from the query, but get nothing.
I've tried changing the timeout (though this hasn't helped as I can only do
this on the Dev environment and there there is not sufficient data in
the DB to reach the timeout, due to this issue.).
There are only around 20 rows in total expected, so not an issue with a very large dataset.
Has anyone run into a situation where a SQL query from a classic asp page only returns partial results, and is there any way to check for or prevent this condition?
Setup:
Classic asp web application
Heavy use throughout of ADODB.Connection objects for connecting to the DB2 server backend databases.
The database is static as far as the queried data is concerned.
Shared connection initilization as follows:
connString = "Provider=MSDASQL.1;User ID=xxx;Data Source=XXX;Extended Properties=""DSN=XXX;UID=XXX;PWD=XXX;"""
Set conn = Server.CreateObject( "ADODB.Connection" )
Set cmd = Server.CreateObject("ADODB.Command")
conn.Mode = adOpenReadOnly
conn.Open connString
cmd.ActiveConnection = conn
cmd.CommandTimeout = 600
Usage as follows:
query = " SELECT blah FROM Foo WHERE ... " ' big long list of clauses defined from user selections.
cmd.CommandText = sql
Set oRs = cmd.Execute
Resposne.Write "<table>..." ' Write table headers here'
Do while (Not oRs.eof)
Response.Write "<tr>...</tr>" ' WRite details from oRs here
oRs.MoveNext
Loop
Response.Write "</table>"
Try adding an order by to the query, that way it should send all rows at once and you can rule in or out timeout issues
We got a legacy vb.net applicaction that was working for years
But all of a sudden it stops working yesterday and gives sql server timeout
Most part of application gives time out error , one part for example is below code :
command2 = New SqlCommand("select * from Acc order by AccDate,AccNo,AccSeq", SBSConnection2)
reader2 = command2.ExecuteReader()
If reader2.HasRows() Then
While reader2.Read()
If IndiAccNo <> reader2("AccNo") Then
CAccNo = CAccNo + 1
CAccSeq = 10001
IndiAccNo = reader2("AccNo")
Else
CAccSeq = CAccSeq + 1
End If
command3 = New SqlCommand("update Acc Set AccNo=#NewAccNo,AccSeq=#NewAccSeq where AccNo=#AccNo and AccSeq=#AccSeq", SBSConnection3)
command3.Parameters.Add("#AccNo", SqlDbType.Int).Value = reader2("AccNo")
command3.Parameters.Add("#AccSeq", SqlDbType.Int).Value = reader2("AccSeq")
command3.Parameters.Add("#NewAccNo", SqlDbType.Int).Value = CAccNo
command3.Parameters.Add("#NewAccSeq", SqlDbType.Int).Value = CAccSeq
command3.ExecuteNonQuery()
End While
End If
It was working and now gives time out in command3.ExecuteNonQuery()
Any ideas ?
~~~~~~~~~~~
Some information :
There isnt anything that has been changed on network and the app uses local database
The main issue is that even in development environment it donest work anymore
I'll state the obvious - something changed. It could be an upgrade that isn't having the desired effect - it could be a network component going south - it could be a flakey disk - it could be many things - but something in the access path has changed. What other problem indications are you seeing, including problems not directly related to this application? Where is the database stored (local disk, network storage box, written by angels on the head of a pin, other)? Has your system administrator "helped" or "improved" things somehow? The code has not worn out - something else has happened.
Is it possible that this query has been getting slower over time and is now just exceeded the default timeout?
How many records would be in the acc table and are there indexes on AccNo and AccSeq?
Also what version of SQL are you using?
How long since you updated statistics and rebuilt indexes?
How much has your data grown? Queries that work fine for small datasets can be bad for large ones.
Are you getting locking issues? [AMJ] Have you checked activity monitor to see if there are locks when the timeout occurs?
Have you run profiler to grab the query that is timing out and then run it directly onthe server? Is it faster then? Could also be network issues in moving the information from the database server to the application. That would at least tell you if it s SQl Server issue or a network issue.
And like Bob Jarvis said, what has recently changed on the server? Has something changed in the database structure itself? Has someone added a trigger?
I would suggest that there is a lock on one of the records that you are trying to update, or there are transactions that haven't been completed.
I know this is not part of your question, but after seeing your sample code i have to make this comment: is there any chance you could change your method of executing sql on your database? It is bad on so many levels.
Perhaps should you set the CommandTimeout property to a higher delay?
Doing so will allow your command to wait a little longer for the underlying database to respond. As I see it, perhaps are you not letting time enough for your database engine to perform all what is required before creating another command to perform your update.
Know that the SqlDataReader continues to "SELECT" while feeding the in-memory objects. Then, while reading, you require your code to update some other table, which your DBE just can't handle, by the time your SqlCommand requires, than times out.
any chances of a "quotes" as part of the strings you are passing to queries?
any chances of date dependent queries where a special condition is not working anymore?
Have you tested the obvious?
Have you run the "update Acc Set AccNo=#NewAccNo,AccSeq=#NewAccSeq where AccNo=#AccNo and AccSeq=#AccSeq" query directly on your SQL Server Management Studio? (Please replace the variables with some hard coded values)
Have you run the same test on another colleague's PC?
Can we make sure that the SQLConnection is working fine. It could be the case that SQL login criteria is changed and connection is getting a timeout. It will be probably more helpful if you post the error message here.
You can rewrite the update as a single query. This will run much faster than the original query.
UPDATE subquery
SET AccNo = NewAccNo, AccSeq = NewAccSeq
FROM
(SELECT AccNo, AccSeq,
DENSE_RANK() OVER (PARTITION BY AccNo ORDER BY AccNo) NewAccNo,
ROW_NUMBER() OVER (PARTITION BY AccNo ORDER BY AccDate, AccSeq)
+ 10000 NewAccSeq
FROM Acc) subquery
After HLGEM's suggestions, I would check the data and make sure it is okay. In cases like this, 95% of the time it is the data.
Make sure disk is defragged. Yes, I know, but it does make a difference. Not the built-in defragger. One that defrags and optimizes like PerfectDisk.
This may be a bit of a long shot, but if your entire application has stopped working, have you run out of space for the transaction log in your database? Either it's been specified to an absolute size, and that has been reached, or your disk is just full.
May be your tables include more information, and defined SqlConnection.ConnectionTimeout property value in config file with little value. And this value isn't necessary to execute your queries.
you can trying optimize your queries, and also rebuilt indexes.
I have some ASP (Classic) code that queries a SQL 2005 Express database. I'm currently handling programmatically if this DB goes down by handling the error when someone tries to connect and can't. I capture the error and bypass subsequent db queries to this database using a session variable.
My problem is that this first query takes about 20 seconds before it timeouts.
I'd like to reduce this timeout length but can't find which property either in the code or database is the right one to reduce.
I've tried following in the code;
con.CommandTimeout = 5
con.CONNECTIONTIMEOUT = 5
Any suggestions please?
Thanks,
Andy
First off you should investigate why the DB is going down at all. We manage servers for hundreds of clients and have never run into a problem with the DB going down unless it was scheduled maintenance.
Besides that, you're already onto the right properties.
"Connect Timeout" is set in the connection string and controls how long the client waits to establish a connection to the database. It should be safe to lower this value in most cases--connections should never take long to establish.
"CommandTimeout" is a property on the IDbCommand implementation and controls how long the client waits for a particular query to return. You can lower this value if you know your queries will not take longer than the value you're setting.
Ended up using the "Connect Timeout" option within ADODB.Connection string.
e.g.
Set con = Server.CreateObject( "ADODB.Connection" )
con.Open "Provider=SQLOLEDB;Server=databaseserver;User ID=databaseuser;Password=databasepassword;Initial Catalog=databasename;Connect Timeout=5;"
If Err.Number = 0 And con.Errors.Count = 0 Then
'connected to database successfully
Else
'did not connect to database successfully within timeout period specified (5 seconds in this example)
End If