I am trying to understand some performance summary output from MSBuild. The results look like this:
Target Performance Summary:
0 ms PrepareForRun 74 calls
0 ms Compile 74 calls
...
...
...
...
15173 ms ResolveProjectReferences 74 calls
29607 ms Build 75 calls
Task Performance Summary:
2 ms CreateCSharpManifestResourceName 6 calls
2 ms ResolveKeySource 1 calls
...
...
...
...
3623 ms Copy 511 calls
7468 ms ResolveAssemblyReference 74 calls
11234 ms Exec 12 calls
48600 ms MSBuild 210 calls
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:29.60
Why does the total time for MSBuild exceed the total elapsed time? The actual stopwatch time to run this build was, in fact, ~29 seconds. Any examples of these summaries that I see online show that the total Task and Target times are same.
The MSBuild and CallTarget tasks invoke other targets (and thus other tasks). The time spent in those tasks is counted against the MSBuild and CAllTarget tasks as well. A bug, really.
So when reading this data, always ignore those two tasks. Summing the other tasks should come to a little less than the total build time.
Related
I am new to Pentaho. I currently has a job like this:
condition 1 -> condition 2 -> if successful then run this sql scripts, if failed then send email
I would like to have a loop that is more like:
(condition 1 -> condition -> 2) are run every 30 minutes
-> if successful then run the sql scripts and stop going back to conditions to check
-> if failed then loop back and run in the next interval
Is this possible to achieve?
Many thanks!
You can do it with a job. See screenshots.
You'll need to replace the simple evaluation step by your condition checking routine.
You could program a job to run every 30 minutes and add a step generating a file or inserting/updating a DB table that you could use to check if you need to run the conditions or not.
I have been following this tutorial on MSDN to access the sample Northwind Access 97 and Access 2000 databases.
The code is mostly from the tutorial and relatively complex to re-create here, but my question is easy to state. Opening and closing the database first with DAO speeds up subsequent access using ODBC and OleDB by ~20-fold...why?
Test 1: Repeatedly run (200 times) the following SQL query against the Access 97 and Access 2000 databases, reading all records in the returned data:
"SELECT ProductID, UnitPrice, ProductName FROM Products " +
"WHERE UnitPrice > ? " +
"ORDER BY UnitPrice DESC;"
Results (ODBC) 1:
Access 97: 200 cycles, reading 75 records per cycle took 22875ms
Access 2000: 200 cycles, reading 75 records per cycle took 12125ms
Results (OleDb) 1:
Access 97: 200 cycles, reading 75 records per cycle took 21656ms
Access 2000: 200 cycles, reading 75 records per cycle took 11578ms
Now make the following change to the code:
private void OpenDbWithDAO(string strDatabase)
{
// Dummy open and close the target database...
DAO.DBEngine dbEngine = new DAO.DBEngine();
DAO.Database db = dbEngine.OpenDatabase(strDatabase, false, false);
db.Close();
// This consumes ~15ms for each database opened
}
Test 2: Exactly the same as test 1, except first call OpenDbWithDAO(PATH).
Results (ODBC) 2:
Access 97: 200 cycles, reading 75 records per cycle took 922ms
Access 2000: 200 cycles, reading 75 records per cycle took 610ms
Results (OleDb) 2:
Access 97: 200 cycles, reading 75 records per cycle took 625ms
Access 2000: 200 cycles, reading 75 records per cycle took 390ms
Is this normal?
UPDATE
Added timing for the dummy open/close of the database. For both the Access 97 and 2000 databases, the time to open/close the databases is ~30ms.
We have a client side application that allows very complex reporting on large data sets. On the left hand side is a tree of possible columns to include in the report, and on the right hand side is a table where we dynamically populate the results.
When a user clicks on the columns they want to include in their report, we're building the necessary SQL (often including a lot of joins and complex sub-queries), and firing those queries off asynchronously.
Problem: Users are creating complex reports resulting in very complex and costly queries. These queries end up backing up and causing us to run out of connections.
I ended up finding sections of our logs that look like this:
169077:2019-09-11 22:14:29 UTC LOG: duration: 65018.497 ms execute <unnamed>:
169105:2019-09-11 22:14:31 UTC LOG: duration: 22494.712 ms execute <unnamed>: SELEC
169129:2019-09-11 22:14:34 UTC LOG: duration: 67866.947 ms execute <unnamed>:
169157:2019-09-11 22:14:40 UTC LOG: duration: 51176.815 ms execute <unnamed>:
169185:2019-09-11 22:14:41 UTC LOG: duration: 51836.988 ms execute <unnamed>:
169213:2019-09-11 22:14:42 UTC LOG: duration: 52655.482 ms execute <unnamed>:
169244:2019-09-11 22:14:46 UTC LOG: duration: 55871.561 ms execute <unnamed>:
Ouch! You can see by the timestamps this is a user adding more columns they want to report on (I confirmed by looking at the queries), blissfully unaware of the pain our database is going through.
Here are some solutions I thought of:
1) Remove the asynchronous querying, making the end user build the report first, then clicking a button to actually run it. This is not ideal as our current user base (which is quite large) would definitely be confused if we made this change (it's unfortunately a UX digression).
2) When the end user clicks a column and the query is fired off asynchronously, somehow trace the PID of the query that is actually running within Postgres. When the same user clicks another column, kill the previous PID, and start tracking the new PID. This would ensure only one query is running for this end user during the report building process at any given time, and would prevent the long running query buildup as seen in the example above.
Is #2 even possible? I looked at possible tracing with probes, and looked at PGBouncer briefly, but I'm not too familiar with either and wasn't able to find a definitive answer.
Any thoughts or suggestions are welcome!
Here is a generic approach to implementing #2 on the client side.
It looks like you are after the pg_backend_pid function.
Call it after establishing the connection, remember your PID, then start your async query using that connection and PID.
If it turns out that you need to stop the query later, use pg_cancel_backend with the PID remembered previously to cancel it.
Or, you can use functions specific to the library that you use to connect to Postgres.
For example, if you use libpq, you can use PQcancel function to stop the query.
I have an Excel workbook that connects to a PostgreS db via ODBC.
Using VBA it executes 27 SQL queries one by one and copies each resulting set to
a different worksheet.
I am content with the data I get, but the performance is mediocre. The database should have plenty of resources.
Can I parallelize/multi-thread the SQL queries? I have read that parallelization is not possible in VBA, per se.
Following #JNevills advice I concatenated the 27 queries in one mega-query
separated by " ; " with a total weight of 54.778 characters.
Execution time:
27 queries sequentially: 45 seconds
1 mega-query: 30 seconds
I learned, that I work on a well seasoned db-version - namely PostgreS
9.3.11.
Even if I had more cores this version coudn't be bothered to use them.
If I execute the mega-query in Squirrel it takes about 20 seconds.
The performance might be better with an up-to-date database.
Sketch of the solution
rs.Open sqlQuery, conn
for i = 1 to 27
call writeRecordSetToSheet(rs, sheetname)
Set rs = rs.NextRecordset '<- magic line for accessing the other 26 recordsets
next i
I have an ASP Classic script that has to do a lot of things (admin only) including running a stored proc that takes 4 mins to execute.
However even though I have
Server.ScriptTimeout = 0
Right at the top of the page it seems to ignore it - 2003 and 2012 servers.
I have tried indexing the proc as much as possible but its for a drop down of commonly searched for terms so I have to clean out all the hacks, sex spam and so on.
What I don't understand is why the Server.ScriptTimeout = 0 is being ignored.
The actual error is this and comes just
Active Server Pages error 'ASP 0113'
Script timed out
/admin/restricted/fix.asp
The maximum amount of time for a script to execute was exceeded. You
can change this limit by specifying a new value for the property
Server.ScriptTimeout or by changing the value in the IIS
administration tools.
It never used to do this and it should override the default set in IIS of 30 seconds.
I have ensured the command timeout of the proc is 0 so unlimited and it errors AFTER coming back from the proc.
Anybody got any ideas?
0 is an invalid value for Server.ScriptTimeout for 2 reasons:
ASP pages have to have a timeout. They cannot run indefinitely.
ScriptTimeout can never be lower than the AspScriptTimeout set in IIS according to this statement from the MSDN docs:
For example, if NumSeconds is set to 10, and the metabase setting
contains the default value of 90 seconds, scripts will time out after
90 seconds.
The best way to work around this is to set Server.ScriptTimeout to a really high value.