I create the Session factory like:
FluentConfiguration cfg =
Fluently.Configure().Database(MsSqlConfiguration.MsSql2005.ConnectionString(
c => c.Is(dbConnectionString)).**AdoNetBatchSize(100)**.ShowSql()).
Mappings(m => m.FluentMappings.AddFromAssembly(mappingAssembly)).
Mappings(m => m.HbmMappings.AddFromAssembly(mappingAssembly));
If I later set
session.SetBatchSize(someOtherSize); during the later program execution
nothing happens. it is as if this command is just a mock.
Why that?
Thanks in advance
I have no idea if and how the NHProf reports batching but using the normal SQL Profiler you cannot notice it.
To verify how it works and if it is indeed enabled as I have set it up, I had to debug the NHibernate's code.
What NHinernate does is to add each generated SQL command in a collection of SQL commands that it is flushed (send to the DB) when the defined BatchSize is reached or when there are no more SQL commands to execute.
Observing the SQL profiler this is not noticable as SQL queries appear but actually NHibernate sends the commands in bactches to the DB.
This way if you want to execute 10 SQL statements without setting the BatchSize NHinerante will talk to the DB 10 times but setting the BatchSize to 10 then it will talk to the DB only once sending the all SQL queries in one go. Unfortunately this is not noticeable in the SQL Profiler...
How are you checking that batching actually occurs and what batch size is being used? SQL profiler does not show batching, you have to use NHibernate Profiler to get a good understanding of what is being batched.
Looking at the NH source session.SetBatchSize() does what it says it does, so it should work :)
Don't forget to set the <property name="adonet.batch_size">3</property> in the config file. The max value, I think is 50. But NH doesn't throw any error if set an higher value and I don't know the default value the.
Related
I am designing a testing framework that makes extensive use of SQL Sever Database. I am using Entity Framework 6 of .NET to felicitate it. I want to log the Underlying SQL query each time when I run a test case. I am using LINQ to SQL for querying Database.
I am having a hard time logging the SQL. LINQ to SQL generates some uncooked SQL which needs to be converted into SQL by filling in the parameters which I want to avoid.
Is there a better approach which will log all the SQL which I can directly feed to my SQL Server without doing any changes in Query ?
According to Entity Framework Logging:
The DbContext.Database.Log property can be set to a delegate for any method that takes a string. Most commonly it is used with any TextWriter by setting it to the “Write” method of that TextWriter. All SQL generated by the current context will be logged to that writer. For example, the following code will log SQL to the console:
using (var context = new BlogContext())
{
context.Database.Log = Console.Write;
// Your code here...
}
in the above way you should be able to log everything.
The following gets logged:
When the Log property is set all of the following will be logged:
SQL for all different kinds of commands. For example:
Queries, including normal LINQ queries, eSQL queries, and raw queries from methods such as SqlQuery
Inserts, updates, and deletes generated as part of SaveChanges
Relationship loading queries such as those generated by lazy loading
Parameters
Whether or not the command is being executed asynchronously
A timestamp indicating when the command started executing
Whether or not the command completed successfully, failed by throwing an exception, or, for async, was canceled
Some indication of the result value
The approximate amount of time it took to execute the command. Note that this is the time from sending the command to getting the
result object back. It does not include time to read the results.
Looking at the example output above, each of the four commands logged
are:
The query resulting from the call to context.Blogs.First
Notice that the ToString method of getting the SQL would not have worked for this query since “First” does not provide an
IQueryable on which ToString could be called
The query resulting from the lazy-loading of blog.Posts
Notice the parameter details for the key value for which lazy loading is happening
Only properties of the parameter that are set to non-default values are logged. For example, the Size property is only shown if it
is non-zero.
Two commands resulting from SaveChangesAsync; one for the update to change a post title, the other for an insert to add a new post
Notice the parameter details for the FK and Title properties
Notice that these commands are being executed asynchronously
I have a Rails 4 application that I use in conjunction with sidekiq to run asynchronous jobs. One of the jobs I normally run outside of my Rails application is a large set of complex SQL queries that cannot really be modeled by ActiveRecord. The connection this set of SQL queries has with my Rails app is that it should be executed anytime one of my controller actions is invoked.
Ideally, I'd queue a job from my Rails application within the controller for Sidekiq to go ahead and run the queries. Right now they're stored in an external file, and I'm not entirely sure what the best way is to have Rails run the said SQL.
Any solutions are appreciated.
I agree with Sharagoz, if you just need to run a specific query, the best way is to send the query string directly into the connection, like:
ActiveRecord::Base.connection.execute(File.read("myquery.sql"))
If the query is not static and you have to compose it, I would use Arel, it's already present in Rails 4.x:
https://github.com/rails/arel
You didn't say what database you are using, so I'm going to assume MySQL.
You could shell out to the mysql binary to do the work:
result = `mysql -u #{user} --password #{password} #{database} < #{huge_sql_filename}`
Or use ActiveRecord::Base.connection.execute(File.read("huge.sql")), but it won't work out of the box if you have multiple SQL statements in your SQL file.
In order to run multiple statements you will need to create an initializer that monkey patches the ActiveRecord::Base.mysql2_connection to allow setting MySQL's CLIENT_MULTI_STATEMENTS and CLIENT_MULTI_RESULTS flags.
Create a new initializer config/initializers/mysql2.rb
module ActiveRecord
class Base
# Overriding ActiveRecord::Base.mysql2_connection
# method to allow passing options from database.yml
#
# Example of database.yml
#
# login: &login
# socket: /tmp/mysql.sock
# adapter: mysql2
# host: localhost
# encoding: utf8
# flags: 131072
#
# #param [Hash] config hash that you define in your
# database.yml
# #return [Mysql2Adapter] new MySQL adapter object
#
def self.mysql2_connection(config)
config[:username] = 'root' if config[:username].nil?
if Mysql2::Client.const_defined? :FOUND_ROWS
config[:flags] = config[:flags] ? config[:flags] | Mysql2::Client::FOUND_ROWS : Mysql2::Client::FOUND_ROWS
end
client = Mysql2::Client.new(config.symbolize_keys)
options = [config[:host], config[:username], config[:password], config[:database], config[:port], config[:socket], 0]
ConnectionAdapters::Mysql2Adapter.new(client, logger, options, config)
end
end
end
Then update config/database.yml to add flags:
development:
adapter: mysql2
database: app_development
username: user
password: password
flags: <%= 65536 | 131072 %>
I just tested this on Rails 4.1 and it works great.
Source: http://www.spectator.in/2011/03/12/rails2-mysql2-and-stored-procedures/
Executing one query is - as outlined by other people - quite simply done through
ActiveRecord::Base.connection.execute("SELECT COUNT(*) FROM users")
You are talking about a 20.000 line sql script of multiple queries. Assuming you have the file somewhat under control, you can extract the individual queries from it.
script = Rails.root.join("lib").join("script.sql").read # ah, Pathnames
# this needs to match the delimiter of your queries
STATEMENT_SEPARATOR = ";\n\n"
ActiveRecord::Base.transaction do
script.split(STATEMENT_SEPARATOR).each do |stmt|
ActiveRecord::Base.connection.execute(stmt)
end
end
If you're lucky, then the query delimiter could be ";\n\n", but this depends of course on your script. We had in another example "\x0" as delimiter. The point is that you split the script into queries to send them to the database. I wrapped it in a transaction, to let the database know that there is coming more than one statement. The block commits when no exception is raised while sending the script-queries.
If you do not have the script-file under control, start talking to those who control it to get a reliable delimiter. If it's not under your control and you cannot talk to the one who controls it, you wouldn't execute it, I guess :-).
UPDATE
This is a generic way to solve this. For PostgreSQL, you don't need to split the statements manually. You can just send them all at once via execute. For MySQL, there seem to be solutions to get the adapter into a CLIENT_MULTI_STATEMENTS mode.
If you want to execute raw SQL through active record you can use this API:
ActiveRecord::Base.connection.execute("SELECT COUNT(*) FROM users")
If you are running big SQL every time, i suggest you to create a sql view for it. It be boost the execution time. The other thing is, if possible try to split all those SQL query in such a way that it will be executed parallely instead of sequentially and then push it to sidekiq queue.
You have to use ActiveRecord::Base.connection.execute or ModelClass.find_by_sql to run custom SQL.
Also, put an eye on ROLLBACK transactions, you will find many places where you dont need such ROLLBACK feature. If you avoid that, the query will run faster but it is dangerous.
Thanks all i can suggest.
use available database tools to handle the complex queries, such as views, stored procedures etc and call them as other people already suggested (ActiveRecord::Base.connection.execute and ModelClass.find_by_sql for example)- it might very well cut down significantly on query preparation time in the DB and make your code easier to handle
http://dev.mysql.com/doc/refman/5.0/en/create-view.html
http://dev.mysql.com/doc/connector-cpp/en/connector-cpp-tutorials-stored-routines-statements.html
abstract your query input parameters into a hash so you can pass it on to sidekiq, don't send SQL strings as this will probably degrade performance (due to query preparation time) and make your life more complicated due to funny SQL driver parsing bugs
run your complex queries in a dedicated named queue and set concurrency to such a value that will prevent your database of getting overwhelmed by the queries as they smell like they could be pretty db heavy
https://github.com/mperham/sidekiq/wiki/API
https://github.com/mperham/sidekiq/wiki/Advanced-Options
have a look at Squeel, its a great addition to AR, it might be able to pull some of the things you are doing
https://github.com/activerecord-hackery/squeel
http://railscasts.com/episodes/354-squeel
I'll assume you use MySQL for now, but your mileage will vary depending on the DB type that you use. For example, Oracle has some good gems for handling stored procedures, views etc, for example https://github.com/rsim/ruby-plsql
Let me know if some of this stuff doesn't fit your use case and I'll expand
I see this post is kind of old. But I would like to add my solution to it. I was in a similar situation; I also needed a way to force feed "PRAGMA foreign_keys = on;" into my sqlite connection (I could not find a previous post that spelled it out how to do it.) Anywho, this worked like a charm for me. It allowed me to write "pretty" sql and still get it executed. Blank lines are ignored by the if statement.
conn = ActiveRecord::Base.establish_connection(adapter:'sqlite3',database:DB_NAME)
sqls = File.read(DDL_NAME).split(';')
sqls.each {|sql| conn.connection.execute(sql<<';') unless sql.strip.size == 0 }
conn.connection.execute('PRAGMA foreign_keys = on;')
I had the same problem with a set of sql statements that I needed to execute all in one call to the server. What worked for me was to set up an initializer for Mysql2 adapter (as explained in infused answer) but also do some extra work to process multiple results. A direct call to ActiveRecord::Base.connection.executewould only retrieve the first result and issue an Internal Error.
My solution was to get the Mysql2 adapter and work directly with it:
client = ActiveRecord::Base.connection.raw_connection
Then, as explained here, execute the query and loop through the results:
client.query(multiple_stms_query)
while client.next_result
result = client.store_result
# do something with it ...
end
I'm having insane simple query: It pulls an ID from one table. The implementation is done using EF 3.5.
This query is repeated in a loop, where I collected a ID from a file and do the search in the database. When running this program, the SQL server is stressed like crazy (the processor utilization soars to 100% for all 16 cores).
It looks like the table of this query is completely locked and nobody gets in anymore. I've read about the necessity to use DbTransaction (begin transaction, commit) or TransactionScope, but the thing is I'm only selecting/reading.
Also it's one query, which is atomic in itself, so the use of Transaction(Scope) is shady at best.
I did try an implementation, but that doesn't seem to do it.
My (LINQ) query: Image image = context.Images.First(i => i.ImageUid == identifier)
Any thoughts on why this is happening? Again I'd like to stress that I'm only selecting/reading records. I don't delete or update records in the database. This is so insanely straight forward that it is frustrating!
For sake of being complete (my attempt at a fix):
// This defaults the isolation level to 'READ COMMITTED' which
// doesn't lock the table when querying.
DbTransaction trx = context.Connection.BeginTransaction();
string isolationLevel = trx.IsolationLevel.ToString();
Image image = context.Images.First(i => i.ImageUid == identifier);
trx.Commit();
NEW: The profiler shows that the Entity framework is doing a SELECT TOP(1) in the image table. This amounts to a MASSIVE amount of reads, hundreds of thousands!
That would suggest that there is no index, but I've looked it up (see comments) and there is one! Also very weird, on the logout, again hundreds of thousands of reads.
I decided to throw out the Entity Framework and do this query using SqlConnection and SqlCommand, but the result is the same!
Next we copied the sp_executesql in the management console and found it took an amazing 4 seconds to execute. Doing the query 'direct' gives an instant result.
Something in the sp_executesql appears to slow things to a crawl. Any ideas?
I think I got it... After finding out that sp_executesql was the culprit it became clear.
See http://yasirbam.blogspot.nl/2009/06/spexecutesql-may-cause-slow-perfomance.html
Due to the stupid conversion, the index on the table is NOT used!
That explains everything visible in the SQL Profiler.
Right now the tool is being tested and it's as fast as lighting!!
I am using SQL Server database and after calling a simple SQL script I would like to know how many records were affected by last (or only) executed statement in a script.
I cannot find the reference how to achieve this in Delphi's TADOCommand and I know SQL Server gives this information to provider. I am aware of workarounds like getting ##ROWCOUNT in another query, yet this gives some overhead and unnecessary complexity.
Thanks.
Do you use the
function Execute(var RecordsAffected: Integer; const Parameters: OleVariant): _Recordset;
version of the Execute method?
From the doc:
RecordsAffected indicates the number
of records, if the command operates on
data, that are affected by the command
after execution.
So that should give you what you need.
Disclaimer: I cannot test this against SQL Server (don't have it).
We got a legacy vb.net applicaction that was working for years
But all of a sudden it stops working yesterday and gives sql server timeout
Most part of application gives time out error , one part for example is below code :
command2 = New SqlCommand("select * from Acc order by AccDate,AccNo,AccSeq", SBSConnection2)
reader2 = command2.ExecuteReader()
If reader2.HasRows() Then
While reader2.Read()
If IndiAccNo <> reader2("AccNo") Then
CAccNo = CAccNo + 1
CAccSeq = 10001
IndiAccNo = reader2("AccNo")
Else
CAccSeq = CAccSeq + 1
End If
command3 = New SqlCommand("update Acc Set AccNo=#NewAccNo,AccSeq=#NewAccSeq where AccNo=#AccNo and AccSeq=#AccSeq", SBSConnection3)
command3.Parameters.Add("#AccNo", SqlDbType.Int).Value = reader2("AccNo")
command3.Parameters.Add("#AccSeq", SqlDbType.Int).Value = reader2("AccSeq")
command3.Parameters.Add("#NewAccNo", SqlDbType.Int).Value = CAccNo
command3.Parameters.Add("#NewAccSeq", SqlDbType.Int).Value = CAccSeq
command3.ExecuteNonQuery()
End While
End If
It was working and now gives time out in command3.ExecuteNonQuery()
Any ideas ?
~~~~~~~~~~~
Some information :
There isnt anything that has been changed on network and the app uses local database
The main issue is that even in development environment it donest work anymore
I'll state the obvious - something changed. It could be an upgrade that isn't having the desired effect - it could be a network component going south - it could be a flakey disk - it could be many things - but something in the access path has changed. What other problem indications are you seeing, including problems not directly related to this application? Where is the database stored (local disk, network storage box, written by angels on the head of a pin, other)? Has your system administrator "helped" or "improved" things somehow? The code has not worn out - something else has happened.
Is it possible that this query has been getting slower over time and is now just exceeded the default timeout?
How many records would be in the acc table and are there indexes on AccNo and AccSeq?
Also what version of SQL are you using?
How long since you updated statistics and rebuilt indexes?
How much has your data grown? Queries that work fine for small datasets can be bad for large ones.
Are you getting locking issues? [AMJ] Have you checked activity monitor to see if there are locks when the timeout occurs?
Have you run profiler to grab the query that is timing out and then run it directly onthe server? Is it faster then? Could also be network issues in moving the information from the database server to the application. That would at least tell you if it s SQl Server issue or a network issue.
And like Bob Jarvis said, what has recently changed on the server? Has something changed in the database structure itself? Has someone added a trigger?
I would suggest that there is a lock on one of the records that you are trying to update, or there are transactions that haven't been completed.
I know this is not part of your question, but after seeing your sample code i have to make this comment: is there any chance you could change your method of executing sql on your database? It is bad on so many levels.
Perhaps should you set the CommandTimeout property to a higher delay?
Doing so will allow your command to wait a little longer for the underlying database to respond. As I see it, perhaps are you not letting time enough for your database engine to perform all what is required before creating another command to perform your update.
Know that the SqlDataReader continues to "SELECT" while feeding the in-memory objects. Then, while reading, you require your code to update some other table, which your DBE just can't handle, by the time your SqlCommand requires, than times out.
any chances of a "quotes" as part of the strings you are passing to queries?
any chances of date dependent queries where a special condition is not working anymore?
Have you tested the obvious?
Have you run the "update Acc Set AccNo=#NewAccNo,AccSeq=#NewAccSeq where AccNo=#AccNo and AccSeq=#AccSeq" query directly on your SQL Server Management Studio? (Please replace the variables with some hard coded values)
Have you run the same test on another colleague's PC?
Can we make sure that the SQLConnection is working fine. It could be the case that SQL login criteria is changed and connection is getting a timeout. It will be probably more helpful if you post the error message here.
You can rewrite the update as a single query. This will run much faster than the original query.
UPDATE subquery
SET AccNo = NewAccNo, AccSeq = NewAccSeq
FROM
(SELECT AccNo, AccSeq,
DENSE_RANK() OVER (PARTITION BY AccNo ORDER BY AccNo) NewAccNo,
ROW_NUMBER() OVER (PARTITION BY AccNo ORDER BY AccDate, AccSeq)
+ 10000 NewAccSeq
FROM Acc) subquery
After HLGEM's suggestions, I would check the data and make sure it is okay. In cases like this, 95% of the time it is the data.
Make sure disk is defragged. Yes, I know, but it does make a difference. Not the built-in defragger. One that defrags and optimizes like PerfectDisk.
This may be a bit of a long shot, but if your entire application has stopped working, have you run out of space for the transaction log in your database? Either it's been specified to an absolute size, and that has been reached, or your disk is just full.
May be your tables include more information, and defined SqlConnection.ConnectionTimeout property value in config file with little value. And this value isn't necessary to execute your queries.
you can trying optimize your queries, and also rebuilt indexes.