My Analysis Services database is responsing really slow after processing. The problem can be reproduced also by clearing cache with the ClearCache XMLA -command. I understand that after clearing the cache, the query performance is slower but I'm seeing slow performance also when using Microsoft.AnalysisServices.AdomdClient library.
I made a small timing test.
DateTime start = DateTime.Now;
int dc = cube.Dimensions.Count; // cube = Microsoft.AnalysisServices.AdomdClient.CubeDef
DateTime end = DateTime.Now;
Debug.WriteLine("Start: " + start.ToLongTimeString());
Debug.WriteLine("Dimensions count: " + dc.ToString());
Debug.WriteLine("End: " + end.ToLongTimeString());
For example this will give the following result
Start: 8:41:53
Dimensions count: 18
End: 8:43:15
So it takes almost 1.5 minutes to get the count for the dimensions. Same performance if I get the measures (which there are only few).
After the first operation all the following operations and queries are fast. My question is that how can I work around this issue? It's a real problem when the database comes almost non-responsive after every database processing. I could do something to automatically "fire up" the database after processing but wouldn't that just move the waiting time from one place to another?
Update:
I've found out the problem. The reason why the performance was different with Management Studio and with AdomdClient was that with AdomdClient I had different connection string to the Analysis Services database. I have some custom stuff at the database which fired with that connectionstring. Anyway, the problem is now solved and wasn't directly related to the actual Analysis Services.
Lesson learned: Make sure you're testing with the correct connectionstring :)
The answer is in the question update - the culprit is an incorrect connection string.
Related
I have the following script:
SELECT
DEPT.F03 AS F03, DEPT.F238 AS F238, SDP.F04 AS F04, SDP.F1022 AS F1022,
CAT.F17 AS F17, CAT.F1023 AS F1023, CAT.F1946 AS F1946
FROM
DEPT_TAB DEPT
LEFT OUTER JOIN
SDP_TAB SDP ON SDP.F03 = DEPT.F03,
CAT_TAB CAT
ORDER BY
DEPT.F03
The tables are huge, when I execute the script in SQL Server directly it takes around 4 min to execute, but when I run it in the third party program (SMS LOC based on Delphi) it gives me the error
<msg> out of memory</msg> <sql> the code </sql>
Is there anyway I can lighten the script to be executed? or did anyone had the same problem and solved it somehow?
I remember having had to resort to the ROBUST PLAN query hint once on a query where the query-optimizer kind of lost track and tried to work it out in a way that the hardware couldn't handle.
=> http://technet.microsoft.com/en-us/library/ms181714.aspx
But I'm not sure I understand why it would work for one 'technology' and not another.
Then again, the error message might not be from SQL but rather from the 3rd-party program that gathers the output and does so in a 'less than ideal' way.
Consider adding paging to the user edit screen and the underlying data call. The point being you dont need to see all the rows at one time, but they are available to the user upon request.
This will alleviate much of your performance problem.
I had a project where I had to add over 7 million individual lines of T-SQL code via batch (couldn't figure out how to programatically leverage the new SEQUENCE command). The problem was that there was limited amount of memory available on my VM (I was allocated the max amount of memory for this VM). Because of the large amount lines of T-SQL code I had to first test how many lines it could take before the server crashed. For whatever reason, SQL (2012) doesn't release the memory it uses for large batch jobs such as mine (we're talking around 12 GB of memory) so I had to reboot the server every million or so lines. This is what you may have to do if resources are limited for your project.
I’m here with another question this time.
I have an application which builds to move data from one database to another. It also deals with validation & comparison between the databases. When we start moving the data from source to destination it takes a while as it always deals with thousands of records. We use WCF service and SQL server # server side and WPF # client side to handle this.
Now I have a requirement to notify user with the time it is going to take based on the source database no: records (eventually that is what im going to create in the destination database) right before user starts this movement process.
Now my real question, which is the best way we can do this and get an estimated time out of it?
Thanks and appreciated your helps.
If your estimates are going to be updated during the upload process, you can take the time already spent, delete on number of processed records, and multiply by number of remaining records. This will give you an updating average remaining time:
TimeSpan spent = DateTime.Now - startTime;
TimeSpan remaining = (spent / numberOfProcessedRecords) * numberOfRemainingRecords;
Hi Mahout community at SO!
I have couple of questions about speeding up recommendation calculations. On my server I have Mahout installed without Hadoop. Also jRuby is used for recommendation script. In the database I have 3k users and 100k items (270k items in join table). So when user requests recommendations the simple script starts working:
First it establishes db connection using PGPoolingDataSource like this:
connection = org.postgresql.ds.PGPoolingDataSource.new()
connection.setDataSourceName("db_name");
connection.setServerName("localhost")
connection.setPortNumber(5432)
connection.setDatabaseName("db_name")
connection.setUser("mahout")
connection.setPassword("password")
connection.setMaxConnections(100)
connection
I get this warning:
WARNING: You are not using ConnectionPoolDataSource. Make sure your DataSource pools connections to the database itself, or database performance will be severely reduced.
Any ideas how to fix that?
After it I create recommendations:
model = PostgreSQLJDBCDataModel.new(
connection,
'stars',
'user_id',
'repo_id',
'preference',
'created_at'
)
similarity = TanimotoCoefficientSimilarity.new(model)
neighborhood = NearestNUserNeighborhood.new(5, similarity, model)
recommender = GenericBooleanPrefUserBasedRecommender.new(model, neighborhood, similarity)
recommendations = recommender.recommend user_id, 30
For now it takes about 5-10 seconds to generate recommendation for one user. The question is how to make recommendations faster (200ms would be nice)?
If you know you are using a pooling data source, you can ignore the warning. It means the implementation does not implement the usual interface for pooling implementations, ConnectionPoolDataSource.
You're never going to make this run fast if trying to run directly off a database. There is just too much data access. Wrap the JDBCDataModel in ReloadFromJDBCDataModel and it will be cached in memory, which should work, literally, 100x faster.
This is being done using Fluent NHibernate
I've got a NHibernate lookup that is retrieving data from one table. If i take the generated sql and run it through query analyzer, it takes ~18ms to run.
Using NHProfiler, i'm getting the duration of this query as ~1800ms - 100 times longer than sql !
Query duration
- Database only:1800ms
- Total: 1806ms
The object that is being populated contains a child class, but this child is being loaded from the NHibernate 2nd level cache
The data that is being returned is paged (50 per query) although as far as i can tell, this shouldn't make any difference
I've also got a count running, and again, this is taking ~4ms in query analyzer and ~1800ms according to NHProfiler.
Is NH Profiler displaying the query execution time, or the complete time to retrieve, map the classes and construct the object graph? And if it's the former - why's it taking so much longer than running the query directly?
EDIT: Just found this post by Ayende about the Query Duration value given in NH Profiler: http://ayende.com/Blog/archive/2009/06/28/nh-prof-query-duration.aspx - so it is definitely the query of the database that is taking a long time
Finally managed to track down the problem.
The primary key for the object is a varchar in the database. NHibernate was converting the value to an nvarchar when it ran the query. Unfortunately this wasn't obvious when looking at the generated sql in NH Profiler. The slowdown was caused by sql converting the nvarchar back to a varchar
I've specified the mapping to use a custom type
map.Id(x => x.Id).CustomType("AnsiString");
and the problem is solved
Cheers for all the help people :)
generally these problems resolve to the network between you and your data base. QA usually connects directly to the data base and all it has to send is the raw data back where its formatted. Your app is probably converting your result set into a data set or similar construct. To prove this, change a bit of code (not your entire data layer) to use a SQL Data Reader to read your data. Just read all of the records without trying to parse out all of the columns and save the data. It will likely perform as fast as your network will let it.
I am currently trying to ring more performance out of my reporting website which uses linq to sql and an sql server express 2008 database.
I am finding that as I now approach a million rows in on of my more 'ugly' tables that performance is becoming a real issue, with one report in particular taking 3 minutes to generate.
Essentially, I have a loop that, for each user, hits the database and grabs a collection of data on them. This data is then queried in various ways (and more rows loaded as needed) until I have a nice little summary object that I can fire off to a set of silverlight charts. Lazy loading comes is used and the reporting pulls into data from around 8 linked tables.
The problem is I don't know where the bottleneck now is and how to improve performance. Due to certain constraints I was forced to use uniqueidentifiers for a number of primary keys in the tables involved - could this be an issue?
Basically, I need to put time into increasing performance but don't have enough to do that with both the database or the linq to sql. Is there anyway I can see where the bottlenecks are?
As im running express I don't have access to the profiler. I am considering rewriting my queries into compiled linq to sql but fear the database may be the culprit.
I understand this question is a bit open ended and its hard to answer without knowing much more about my setup (database schema etc) but any advice on how to find out where the bottlenecks are is more appreciated!
Thanks
UPDATE:
Thanks for all the great advice guys, and some links to some great tools.
UPDATE for those interested
I have been unable to make my queries any quicker through tweaking the linq. the problem seems to be that the majority of my database access code takes place in a loop. I can't see a way around it. Basically I am building up a report by looking through a number of users data - hence the loop. Pulling all the records up front seems a bit crazy - 800,000 + rows. My gut feeling is that there is a much better way, but its a technological leap too far for me!
However, adding another index to one of the foreign keys in one of the tables boosted performance so now the report takes 20 seconds to generate as opposed to 3 minutes!
I used this excelent tool: Linq2Sql profiler. It works on the application side, so there is no need for database server profiling functionality.
You have to add one line of initialization code to your application and then in separate desktop application profiler shows you SQL query for each LINQ query with exact line of code where it was executed (cs or aspx), database time and application time of executions and it even detects some common performance problems like n+1 queries (query executed for iteration) or unbounded datasets. You have to pay for it, but the trial version is also available.
As you're using SQL Express which doesn't have Profiler, there is a free third party profiler you can download here. I've used it when running SQL Express. That will allow you to trace what's going on in the database.
Also, you can query the Dynamic Management Views to see what the costly queries are:
e.g. TOP 10 queries that have taken the most time
SELECT TOP 10 t.text, q.*, p.query_plan
FROM sys.dm_exec_query_stats q
CROSS APPLY sys.dm_exec_sql_text(q.sql_handle) t
CROSS APPLY sys.dm_exec_query_plan (q.plan_handle) AS p
ORDER BY q.total_worker_time DESC
There are 2 tools I use for this, LinqPad and the Visual Studio Debugger. First, check out LinqPad, even the free version is very powerful, showing you execution time, the SQL generated and you can use it to run any code snippet...it's tremendously useful.
Second, you can use the Visual studio debugger, this is something we use on our DataContext (note: only use this in debug, it's a performance hit and completely unnecessary outside of debugging)
#if DEBUG
private readonly Stopwatch Watch = new Stopwatch();
private static void Connection_StateChange(object sender, StateChangeEventArgs e)
{
if (e.OriginalState == ConnectionState.Closed && e.CurrentState == ConnectionState.Open)
{
Current.Watch.Start();
}
else if (e.OriginalState == ConnectionState.Open && e.CurrentState == ConnectionState.Closed)
{
Current.Watch.Stop();
string msg = string.Format("SQL took {0}ms", Current.Watch.ElapsedMilliseconds);
Trace.WriteLine(msg);
}
}
#endif
private static DataContext New
{
get
{
var dc = new DataContext(ConnectionString);
#if DEBUG
if (Debugger.IsAttached)
{
dc.Connection.StateChange += Connection_StateChange;
dc.Log = new DebugWriter();
}
#endif
return dc;
}
}
In a debug build, as an operation completes with each context, we see the timestamp in the debug window and the SQL it ran. The DebugWriter class you see can be found here (Credit: Kris Vandermotten). We can quickly see if a query's taking a while. To use it we just initiate a DataContext by:
var DB = DataContext.New;
(The profiler is not an option for me since we don't use SQL server, this answer is simply to give you some alternatives that have been very useful for me)