I have a general question about golang and performing queries. I am using gin-gonic as my http framework and pgx driver to perform different queries. I'm running into a problem that some queries return conn busy and need to know how to solve this problem for future references. Note some of my queries use pgx.Conn and others pgx.Pool I have also configured my pgx.Pool to have a max 10 connection pools.
An example query I have is
SELECT user_id,first_name,last_name,email,users.username,dob,country,is_verified, bio,"+
"profile_json, tier, casual_games, stream_time, profile_image,"+
"is_streaming, users.created_at FROM users INNER JOIN profiles ON profiles.username = users.username WHERE users.user_id = $1
Note some of my queries use pgx.Conn and others pgx.Pool
https://pkg.go.dev/github.com/jackc/pgx#Conn
Conn is a PostgreSQL connection handle. It is not safe for concurrent usage. Use ConnPool to manage access to multiple database connections from multiple goroutines.
Seems like your code must be breaking due to use of non threadsafe Conn
Related
I want to run multiple statements on a single execution against SQL Server. I do it with Node.js but I cannot run the same query using next.jdbc
For example, if I run this:
(def db {:jdbcUrl "jdbc:jtds:sqlserver://localhost:1433/TESTDB;user=sa;password=passwd"})
(def ds (jdbc/get-datasource db))
(jdbc/execute! ds ["select * from EMPLOYEE;select FIRST_NAME from EMPLOYEE;"])
I have also tried to wrap the statement within a transaction with the same result
(jdbc/execute! ds ["BEGIN TRANSACTION select * from EMPLOYEE;select FIRST_NAME from EMPLOYEE; COMMIT"])
I always get the first query.
I have tried Microsoft's JDBC driver also.
Sean Corfield says that if the database supports it, then next.jdbc should support it. next-jdbc: execute multiple statements?
But I cannot make it work
Solution
As indicated by Sean Corfield
(jdbc/execute! ds ["BEGIN select * from EMPLOYEE;select FIRST_NAME from EMPLOYEE; END"] {:multi-rs true})
Yes, you can run multiple statements and multiple result sets back but you have to tell next.jdbc that's the behavior you want.
Take a look at the tests for MS SQL Server running multiple statements: https://github.com/seancorfield/next-jdbc/blob/develop/test/next/jdbc_test.clj#L560-L572
This is mentioned (briefly) in the Getting Started guide: "If you pass the :multi-rs true option to execute!, you will get back a vector of results sets, instead of just one result set: a vector of zero or more vectors."
As far as I know, SQL Server's JDBC driver does not support multiple statements. But, even if it did, you should probably not be using it, as it opens a potential security hole for injection type attacks. Instead, if you really need to execute multiple SQL statements, either refactor your current SQL into a single statement, or else use multiple statements wrapped in a single transaction.
For reference, some other JDBC drivers, such as MySQL, might support multiple statements.
I have a tomcat server calling a stored procedure from a jsp. In the stored procedure I have a query that fills a temp table with data. That temp table is then joined to another table over a dblink to fill another temp table using the hint - DRIVING_SITE. The last temp table is then joined to another table on our database to return a resultset to tomcat.
I'm sorry but I can't really provide a code example for all this, but the issue I'm having is this - After a while of the database link not being used, the first query made using the link will do nothing and return the error:
test.jsp caught exception, closing connection: ORA-02068: following severe error from DATABASE_LINK_NAME
ORA-03135: connection lost contact
Every subsequent query made on the database link within 10 minutes or so of the last call will be fine. The temp table can be huge or small, the amount of data queried seems to make no difference, but the first call after the idle time will get this error probably 75% of the time. Has anyone experienced this issue? If so, are there any resolutions?
The query is structured like so:
INSERT INTO temp_table_2
WITH last_submissions AS (
SELECT /*+ DRIVING_SITE(some_schema.some_table_1) */
bs.unique_id,
CASE WHEN COUNT(bs.unique_id) > 1 THEN 'Y' ELSE 'N' END some_flag,
MAX(trx.unique_id) last_submission
FROM (SELECT unique_id
FROM temp_table_1) oids,
some_schema.some_table_1#DATABASE_LINK bs,
some_schema.some_table_1#DATABASE_LINK trx
WHERE oids.unique_id = bs.unique_id
AND bs.non_unique_join_id = trx.non_unique_join_id
GROUP BY bs.unique_id),
something_relevant AS (
SELECT /*+ DRIVING_SITE(some_schema.some_table_2) */
last_value_of_something.unique_id,
last_value_of_something.some_flag,
mv.value_description status
FROM (
SELECT /*+ DRIVING_SITE(some_schema.some_table_1) */
ls.unique_id,
CASE WHEN COUNT(ls.unique_id) > 1 THEN 'Y' ELSE 'N' END some_flag,
MAX(prd.prd_some_id) last_submission
FROM last_submissions ls,
some_schema.some_table_1#DATABASE_LINK trx,
some_schema.some_table_2#DATABASE_LINK prd
WHERE ls.last_submission = trx.unique_id
AND trx.some_unique_id = prd.some_unique_id (+)
GROUP BY ls.unique_id) last_value_of_something,
some_schema.some_table_2#DATABASE_LINK prd,
some_schema.some_table_3#DATABASE_LINK cs,
some_schema.some_display_value_table#DATABASE_LINK mv
WHERE last_value_of_something.last_submission = prd.prd_some_id (+)
AND prd.some_id = cs.some_id (+)
AND cs.status_code = mv.value (+)
AND mv.value_type (+) = 'SOME_INDICATOR_FOR_DISPLAY_VALUES')
SELECT ls.unique_id unique_id,
NVL(pr.status, trx.some_code) status,
CASE WHEN ls.some_flag = 'Y' OR pr.some_flag = 'Y' THEN 'Yes' ELSE 'No' END display_the_flag
FROM /*+ DRIVING_SITE(some_schema.some_table_1) */
last_submissions ls,
some_schema.some_table_1#DATABASE_LINK trx,
something_relevant pr
WHERE ls.last_submission = trx.unique_id
AND ls.unique_id = pr.unique_id
Do you expect the network between the two database servers to be stable and to allow connections to exist for some time?
When you use a database link, the local server opens up a connection to the remote server. That connection will be kept open as long as your session is open to be used by other queries. If you are seeing connections getting dropped, that often means that there is something in the network (a firewall commonly) that is detecting and killing idle connections. It can also mean that the network between the two servers is simply unstable.
Ideally, you would resolve the problem by fixing whatever underlying network issue you have. If there is a firewall that is killing idle connections, you should be able to modify the firewall configuration to avoid killing these connections for example.
If fixing the infrastructure is not an option, you could close the connection to the remote server after every query (or at least after every query that could be followed by a long idle time)
ALTER SESSION CLOSE DATABASE LINK <<dblink name>>
That means, however, that you would be setting up and tearing down a connection to the remote server potentially on every query-- that is potentially relatively expensive and potentially causes more load on the remote server (depending, of course, on how frequently it happens and how many sessions you might have).
The whole process of pulling data over a database link into a series of temporary tables in order to serve up data to a human using a web application also strikes me as a potentially problematic architecture. Perhaps you have valid reasons for this. But I would be strongly considering using some sort of replication technology (materialized views, Streams, or GoldenGate are the built-in options) rather than pulling data at runtime over database links.
would like to ask how to retrieve all my past database sql queries within that session? thanks
I'm pretty sure Oracle doesn't keep data on all past queries (closed cursors) for each session. I can think of a couple of ways to get this data however:
If you're using PL/SQL, most of your past cursors will remain in your session cache (up to the cursor_sharing initialization parameter). You can query the view v$open_cursor:
SELECT * FROM v$open_cursor WHERE sid=to_number(sys_context('USERENV','SID'))
Join this view to v$sqltext (or v$sqltext_with_newlines) to get the full sql text:
SELECT o.saddr, s.address, o.hash_value, s.piece, s.sql_text
FROM v$open_cursor o
JOIN v$sqltext_with_newlines s ON o.address = s.address
AND o.hash_value = s.hash_value
WHERE sid = to_number(sys_context('USERENV', 'SID'))
ORDER BY o.saddr, s.address, o.hash_value, s.piece;
You could trace your session, opening the resulting trace file once the session terminates will reveal all SQL (furthermore, you can tkprof the trace file to get a summary and statistics).
As Vincent pointed out the only way would be (afaik) to trace the session at the client level.
In addition to the open cursors (which is how Toad does it), another, less precise, way would be to use ASH (Active Session History).
The problems with ASH are that
it samples every seconds for active sessions (so you are missing all the quick ones),
it's a circular buffer (backed up by the DBA_HIST_ACTIVE_SESS_HISTORY view) so that you are missing the older ones.
This is because it's only meant to "catch" long running queries for performance purpose.
It is well adapted however if one is only interested in the queries with long response time.
For what it's worth, here is a simple query returning a session's history of long queries.
select
sqla.sql_text
from
v$active_session_history hist,
v$sqlarea sqla,
v$session ss
where
sqla.sql_id = hist.sql_id and
ss.sid = hist.session_id and
ss.serial# = hist.session_serial# and
ss.audsid = sys_context('USERENV', 'SESSIONID') ;
We got a legacy vb.net applicaction that was working for years
But all of a sudden it stops working yesterday and gives sql server timeout
Most part of application gives time out error , one part for example is below code :
command2 = New SqlCommand("select * from Acc order by AccDate,AccNo,AccSeq", SBSConnection2)
reader2 = command2.ExecuteReader()
If reader2.HasRows() Then
While reader2.Read()
If IndiAccNo <> reader2("AccNo") Then
CAccNo = CAccNo + 1
CAccSeq = 10001
IndiAccNo = reader2("AccNo")
Else
CAccSeq = CAccSeq + 1
End If
command3 = New SqlCommand("update Acc Set AccNo=#NewAccNo,AccSeq=#NewAccSeq where AccNo=#AccNo and AccSeq=#AccSeq", SBSConnection3)
command3.Parameters.Add("#AccNo", SqlDbType.Int).Value = reader2("AccNo")
command3.Parameters.Add("#AccSeq", SqlDbType.Int).Value = reader2("AccSeq")
command3.Parameters.Add("#NewAccNo", SqlDbType.Int).Value = CAccNo
command3.Parameters.Add("#NewAccSeq", SqlDbType.Int).Value = CAccSeq
command3.ExecuteNonQuery()
End While
End If
It was working and now gives time out in command3.ExecuteNonQuery()
Any ideas ?
~~~~~~~~~~~
Some information :
There isnt anything that has been changed on network and the app uses local database
The main issue is that even in development environment it donest work anymore
I'll state the obvious - something changed. It could be an upgrade that isn't having the desired effect - it could be a network component going south - it could be a flakey disk - it could be many things - but something in the access path has changed. What other problem indications are you seeing, including problems not directly related to this application? Where is the database stored (local disk, network storage box, written by angels on the head of a pin, other)? Has your system administrator "helped" or "improved" things somehow? The code has not worn out - something else has happened.
Is it possible that this query has been getting slower over time and is now just exceeded the default timeout?
How many records would be in the acc table and are there indexes on AccNo and AccSeq?
Also what version of SQL are you using?
How long since you updated statistics and rebuilt indexes?
How much has your data grown? Queries that work fine for small datasets can be bad for large ones.
Are you getting locking issues? [AMJ] Have you checked activity monitor to see if there are locks when the timeout occurs?
Have you run profiler to grab the query that is timing out and then run it directly onthe server? Is it faster then? Could also be network issues in moving the information from the database server to the application. That would at least tell you if it s SQl Server issue or a network issue.
And like Bob Jarvis said, what has recently changed on the server? Has something changed in the database structure itself? Has someone added a trigger?
I would suggest that there is a lock on one of the records that you are trying to update, or there are transactions that haven't been completed.
I know this is not part of your question, but after seeing your sample code i have to make this comment: is there any chance you could change your method of executing sql on your database? It is bad on so many levels.
Perhaps should you set the CommandTimeout property to a higher delay?
Doing so will allow your command to wait a little longer for the underlying database to respond. As I see it, perhaps are you not letting time enough for your database engine to perform all what is required before creating another command to perform your update.
Know that the SqlDataReader continues to "SELECT" while feeding the in-memory objects. Then, while reading, you require your code to update some other table, which your DBE just can't handle, by the time your SqlCommand requires, than times out.
any chances of a "quotes" as part of the strings you are passing to queries?
any chances of date dependent queries where a special condition is not working anymore?
Have you tested the obvious?
Have you run the "update Acc Set AccNo=#NewAccNo,AccSeq=#NewAccSeq where AccNo=#AccNo and AccSeq=#AccSeq" query directly on your SQL Server Management Studio? (Please replace the variables with some hard coded values)
Have you run the same test on another colleague's PC?
Can we make sure that the SQLConnection is working fine. It could be the case that SQL login criteria is changed and connection is getting a timeout. It will be probably more helpful if you post the error message here.
You can rewrite the update as a single query. This will run much faster than the original query.
UPDATE subquery
SET AccNo = NewAccNo, AccSeq = NewAccSeq
FROM
(SELECT AccNo, AccSeq,
DENSE_RANK() OVER (PARTITION BY AccNo ORDER BY AccNo) NewAccNo,
ROW_NUMBER() OVER (PARTITION BY AccNo ORDER BY AccDate, AccSeq)
+ 10000 NewAccSeq
FROM Acc) subquery
After HLGEM's suggestions, I would check the data and make sure it is okay. In cases like this, 95% of the time it is the data.
Make sure disk is defragged. Yes, I know, but it does make a difference. Not the built-in defragger. One that defrags and optimizes like PerfectDisk.
This may be a bit of a long shot, but if your entire application has stopped working, have you run out of space for the transaction log in your database? Either it's been specified to an absolute size, and that has been reached, or your disk is just full.
May be your tables include more information, and defined SqlConnection.ConnectionTimeout property value in config file with little value. And this value isn't necessary to execute your queries.
you can trying optimize your queries, and also rebuilt indexes.
I'd like to "dry-run" Hibernate HQL queries. That is I'd like to know what actual SQL queries Hibernate will execute from given HQL query without actually executing the HQL query against real database.
I have access to hibernate mapping for tables, the HQL query string, the dialect for my database. I have also access to database if that is needed.
Now, how can I find out all the SQL queries Hibernate can generate from my HQL without actually executing the query against any database? Are there any tools for this?
Note, that many SQL queries can be generated from one HQL query and the set of generated SQL queries may differ based on the contents of database.
I am not asking how to log SQL queries while HQL query is executing.
Edit: I don't mind connecting to database to fetch some metadata, I just don't want to execute queries.
Edit: I also know what limits and offsets are applied to query. I also have the actual parameters that will be bind to query.
The short answer is "you can't". The long answer is below.
There are two approaches you can take:
A) Look into HQLQueryPlan class, particularly its getSqlStrings() method. It will not get you the exact SQL because further preprocessing is involved before query is actually executed (parameters are bound, limit / offset are applied, etc...) but it may be close enough to what you want.
The thing to keep in mind here is that you'll need an actual SessionFactory instance in order to construct HQLQueryPlan, which means you won't be able to do so without "connecting to any database". You can, however, use in-memory database (SqlLite and the likes) and have Hibernate auto-create necessary schema for it.
B) Start with ASTQueryTranslatorFactory and descend into AST / ANTLR madness. In theory you may be able to hack together a parser that would work without relying on metadata but I have a hardest time imagining what is it you're trying to do for this to be worth it. Perhaps you can clarify? There has to be a better approach.
Update: for an offline, dry-run of some HQL, using HQLQueryPlan directly is a good approach. If you want to intercept every query in the app, while it's running, and record the SQL, you'll have to use proxies and reflection as described below.
Take a look at this answer for Criteria Queries.
For HQL, it's the same concept - you have to cast to Hibernate implementation classes and/or access private members, so it's not a supported method, but it will work with a the 3.2-3.3 versions of Hibernate. Here is the code to access the query from HQL (query is the object returned by session.createQuery(hql_string):
Field f = AbstractQueryImpl.class.getDeclaredField("session");
f.setAccessible(true);
SessionImpl sessionImpl = (SessionImpl) f.get(query);
Method m = AbstractSessionImpl.class.getDeclaredMethod("getHQLQueryPlan", new Class[] { String.class, boolean.class });
m.setAccessible(true);
HQLQueryPlan plan = (HQLQueryPlan) m.invoke(sessionImpl, new Object[] { query.getQueryString(), Boolean.FALSE });
for (int i = 0; i < plan.getSqlStrings().length; ++i) {
sql += plan.getSqlStrings()[i];
}
I would wrap all of that in a try/catch so you can go on with the query if the logging doesn't work.
It's possible to proxy your session and then proxy your queries so that you can log the sql and the parameters of every query (hql, sql, criteria) before it runs, without the code that builds the query having to do anything (as long as the initial session is retrieved from code you control).