Hi i'm working on a large SSAS tabular model and it's so slow every time i change even the smallest thing it goes away and thinks about it for ages.
The model is massive and pretty sure that's the problem but I've inherited it like that so at the moment cant do anything about that.
Is there a way to stop SSAS loading all the data (or less data) while i'm developing the model in visual studio?
I encountered the same problem in my cube; apparently due to the hundreds of measures my model has. I tried the process clear method described by Vercelli but it did not help in my situation. To resolve this issue I did the following:
Open the project in VS
Go to Model in the Menu
Go to Calculation Options
Select Manual Calculation
This brought my time down from 7 - 10 minutes doing something as simple as hiding a measure to 3 to 5 seconds.
If you process-clear your workspace DB no data will appear on Visual Studio.
Navigate with SSMS to your workspace instance. The Database will appear as your tabular model followed by your userName and a GUID. Right-Click -> Process Database-> Process Clear.
Please try this with caution if your are not sure which DB you are processing.
PS: If your fact tables are partitioned, you can process-clear those and then process only some of the partitions in order to have some data to test.
There's nothing you can do about it for the time being - I'm in a similar situation and have searched far and wide for a solution!
What sometimes helps is if you delete the workspace files (the ones with the guid at the end) from the SSAS\data directory. Doing this deletes all the data in the local version of the model which will bring the file size a fair bit. I do it every now and then and find it does usually help - it still keeps the table structures and relationships etc
For big, mature models Visual Studio is not the best dev environment. Try to use Tabular Editor from GitHub created by Daniel Otyker. It's an editor that works in both on-line and off-line modes and is super fast. What's more, it lets you change things in bulk using either GUI or scripting via a bit of C# (nothing to be afraid of, though). There's a lot of documentation on GitHub and good examples that can very easily be customized. Also, Daniel has a video on YT that teaches how to use the editor to the best advantage. Hope this helps.
When building SSAS cubes with Visual Studio 2019 and you are having performance issues while maintaining the cube, for example, 3-8 minute long delays between modifications to the cube. The way to fix this issue is to Turn Off Automatic Calculation for the model and set the model calculation to Manual Calculation.
To change the calculation method, from the Visual Studio 2019 Enterprise SSDT main menu bar "Model", select "Calculation Options" and select "Manual".
As you can see, it allows toggling between "Automatic Calculation" and "Manual Calculation". If you select "Manual Calculation", then you have a menupad offering the option to "Calculate Now".
Related
Actually my issue is : the more measure i created, the more time it need to Visual Studio to created a new measure. (Visual Studio stay in not reponding)
The measures are not really complexe but actually it take thirty seconds~one minute to create a new measure (and i have around 50-100 measures actually).
The temporary solution i have actually is to create the measures in the code page but i was wondering about the scalabilty of ssas tabular if i have to create 100 measures again.
-My memory : 16 Go, 2,60GhZ i7.
-The data in the fact table is about ~50 thousand lines.
-I don't have other project open.
-SQL server 2016, the SQL Server has the latest build 2020-09-28 (build : 13.0.5850.14).
-i don't have build action and i don't backup the data to disk.
I was wondering if it's normal that visual studio take some times between the creation of each measure.
If you have some tips or advice don't hesitate to share :)
Maybe i should try to split my measure per Fact Tables instead of created in one measure table?
Maybe there are some parameters i can try to change?
Depending on a lot of things, like how the data looks (not only the number of rows in the fact table, but size etc), the hardware, network etc, I would say it's not very rare to experience some delays and slowness working with SSAS in visual studio when it starts loading data into the grids in the designer. As someone mentioned in the comments, there are third party alternatives for editing tabular models that doesn't have this behavior/functionality.
However, if you still want to use visual studio there are ways to possibly get around this. For example if you have a dedicated server/database for development purposes, where you have views as sources for the tables in the tabular model, you can add a TOP condition to the SELECT statement in the views to eliminate/limit the data loaded into visual studio during development.
I am working using Tableau and have to write down multiple different SQL each time, while making new data sources.
I have to save all changes on SQL for every data source.
Currently I would paste the SQL on notepad and save them on separate folder in my computer, along with description of the changes.
Is there any better way to do this?
Assuming you have permission to create objects in the database, begin by creating database views, As #Nick.McDermaid commented.
Then, instead of using Custom SQL data source in Tableau, just connect to the View as if it were a table.
If you need to track the changes to these SQL views of your data, you will need to learn how to use source control for the .sql files that can be scripted from within SQL Server Management Studio:
Your company or school may have a preferred source control system already in use, in which case you should use that. If they don't, or if you are learning at home, then Git and Subversion are popular open source choices.
There are many courses available on learning platforms like Coursera that will teach you how to learn how to use those systems.
I had similar problem as you.
We ended up writing the queries in SQL Editor SQL Work bench (https://www.sql-workbench.eu/), then managed the code history and performed code peer-review (logic, error check, etc) in team shared space (like confluence).
The reasons we did that is
1) SQL queries are much easy to write on Work Bench
2) Code review is a must! You will find through implementing a review process more mistakes than you could ever think about
3) The shared space is just really convenient as it is accessible by everyone, and all errors are documented. After sometimes you get a lot of visible knowledge accumulated.
I also totally agree with Nick as this is one step to a reporting solution. But developing a whole reporting server is heavy, costly and takes time. Unless management are really convinced of the importance of developing a reporting solution, you may have to get a workaround with queries and Tableau (at least that was the case for us)
A little late to the party, but I would suggest you simply version the tableau workbook. The contents of the workbook are XML, so perfect for versioning using file based tools (Dropbox, One Drive, etc.) or source control (git, etc.). The workbooks themselves are usually quite small, so just make sure to keep the extract data separate if you use it.
Broadly speaking, can someone tell me if I'm headed in the right direction?
I now know how to write SQL Queries pretty well.
I would like to start aggregating multiple queries onto one "form"/template (not sure if that's the correct terminology).
I have access to lots of clean data in the form of Excel Files.
I should be able to load the excel files into Visual Studio and then write reports that refer to those excel files as databases, am I right?
I haven't been able to find a great SSDT tutorial yet, but I'll keep looking. Thanks for any feedback.
First off, I apologize that I'm writing a bit of a novel here. My understanding of your question is that you're looking for architectural guidance on the best way to go, and that's not a quick answer.
If, however, I've misinterpreted your intent and you are actually just looking for how to code up an excel file as a database, there is already a lot of articles online that you can google.
Regarding your architectural question...it is really going to come down to choosing the best trade-offs for what you're building. I will give you some pointers that I have learned and hopefully it is helpful to you and others in the community.
I would be very hard pressed to advise that you use an excel file as a database.
While it might seem like the most straight forward solution, the trade-offs here are very costly in debugging file locking issues and dealing with excel specific errors, it becomes a death by a thousand cuts. It is certainly possible, but this is a trap that I personally fell into early in my career.
Here's is a link to some descriptions of the problems that you'd have with an excel file database and here is a 2nd link.
To paraphrase your question, it sounds like you're developing a personal ETL application for improving your productivity and your company's metrics. Spreadsheets come into your e-mail inbox and transformed versions of the spreadsheets go out of your e-mail outbox. You are wanting to look at the departments' data from a historical and comparative perspective. I have done this many times in the past as well and it is a very reasonable goal.
The best way that I have found to do this is to use a SQL Server database. You can start this out in phases of minimal viable product to do this in small easy chunks.
Phase I:
Download and install SQL Server 2016 Express free. Make sure
to install localdb when you install SQL Server 2016. See the localdb
instructions for more information.
Create the localdb instance on the command line.
Connect to the new localdb instance in SQL Server
Create a new Database that you'll use for importing the data. Give it a name like "ReportData"
Import the excel files received from the variety of businesses into the new database. This stackoverflow answer gives a great description of how to do it. Here is an alternate example.
If you get any error messages about drivers you may need to download the correct drivers.
Develop your SQL queries that you want to use. For simplicity, I'm just showing a basic select statement here, but you can build some sophisticated SQL queries for aggregating the data in this step.
Export the data from the excel file into a CSV file or an excel file. You do this by right clicking in the "Results" area and selecting "Save Results As..."
Manually copy and paste the resulting values into the excel templates that you would like.
Note step 9 will be automated soon, but it is better for now to understand your domain objects and be thinking about the business logic that you're building in a quick iterative manner.
Phase II:
Create a new Console application in Visual Studio that will transform the data from the database into an Excel file output. The most powerful way to do this is to use EPPlus. Here is a detailed explanation on how to do this.
Note, when you run the source code from the detailed explanation link, you need to change the output path first, for example to c:\temp. Note also that there are plenty of other Excel spreadsheet helper packages out there, so feel free to look around at other packages as well. EPPlus is simply one that I have been successful with in my projects. This Console application will be querying your SQL Server database using the queries that you built in step 7 above.
Phase III:
In time, you many find that co-workers and managers within your company want to start accessing your data directly through a web page...
At a high level, the steps you would take are:
Backup the database and restore it onto a server.
Implement a simple MVC application
Perhaps even build web pages to allow users to import excel so that they don't need to e-mail them to you any longer.
An additional note, there are Enterprise level ETL and reporting tools out there as well, such as SSIS/SSRS, etc that you could look into if you're looking for a more sophisticated tool set, but I didn't get that impression with your question.
I hope that this answer helps and isn't too long winded. Please let me know if any of the steps are unclear, I know it's a lot of information in one post.
We have been looking into possible solutions for our SQL Source control. I just came across Red Gates SQL Source control and wondered if anyone has implemented it? I am going to download the trial and give it a shot, but just wanted to see if others have real experience.
As always greatly appreciate the input
--S
I have updated my original post below to reflect changes in the latest versions of SQL Source Control (3.0) and SQL Compare (10.1).
Since this question was asked over a year ago, my response may not be that helpful to you, but for others who may currently be evaluating SSC, I thought I would throw in my two cents. We just started using SQL Source Control (SSC) and overall I am fairly satisfied with it so far. It does have some quirks though, especially if you are working in a shared database environment (as opposed to every developer working locally) and particularly working in a legacy environment where objects in the same database are divided haphazardly between development teams.
To give a brief overview of how we are using the product in our organization, we are working in a shared environment where we all make changes to the same development database, so we attached the shared database to the source control repository. Each developer is responsible for making changes to the objects in the database through SQL Server Management Studio (SSMS), and when they are finished, they can commit their changes to source control. When we are ready to deploy to staging, the build master (me) merges the development branch of the database code to the main (staging) branch and then runs SQL Compare using the main branch repository version of the database as the source and the live staging database as the target, and SQL Compare generates the necessary scripts to deploy the changes made to the staging environment. Staging to production deployments works in similar fashion. One other important point to note is that, given the fact that we are sharing the same database with other development teams, we use a built in feature of SSC that allows you to create filters on database objects by name, type, etc. We manually set up filters on our specific team's objects, excluding all other objects, so that we don't accidentally commit other development team's changes when we do our deployments.
So in general it's a fairly simple product to set up and use and it's really nice because you're always working with live objects in SSMS, as opposed to disconnected script files stored in a separate source repository that run the risk of getting out of sync. It's also nice because SQL Compare generates the deployment scripts for you so you don't have to worry about introducing errors as you would if you were creating the scripts on your own. And as SQL Compare is a very mature and stable product, you can feel pretty confident that it's going to create the proper scripts for you.
With that being said, however, here are some of the quirks that I have run into so far:
SSC is pretty chatty out of the box in terms of communicating with the db server in order to keep track of database items that are out of sync with the source control repository. It polls every few milliseconds and if you add in multiple developers all working against the same database using SSC, you can imagine that our dba's weren't very happy. Fortunately, you can easily reduce your polling frequency to something more acceptable, although at the cost of sacrificing responsive visual notifications of when objects have been changed.
Using the object filtering feature, you can't easily tell from looking at objects in SSMS which objects are included in your filter. So you don’t know for sure if an object is under source control, unlike in Visual Studio, where icons are used to indicate source controlled objects.
The object filtering GUI is very clunky. Due to the fact that we are working in a legacy database environment, there is currently not a clear separation between the objects that our team owns and those owned by other teams, so in order to prevent us from accidentally committing/deploying other teams’ changes, we have set up a filtering scheme to explicitly include each specific object that we own. As you can imagine, this becomes quite cumbersome, and as the GUI to edit the filters is set up to enter one object at a time, it could become quite painful, especially trying to set up your environment for the first time (I ended up writing an application to do this). Going forward, we are creating a new schema for our application to better facilitate object filtering (besides being a better practice anyway).
Using the shared database model, developers are allowed to commit any pending changes to a source controlled database, even if the changes are not theirs. SSC does give you a warning if you try to check in a bunch of changes that these changes might not be yours, but other than that you’re on your own. I actually find this to be one of SSC’s most dangerous “quirks”.
SQL Compare can’t currently share the object filters created by SSC, so you would have to manually create a matching filter in SQL Compare, so there is a danger that these could get out of sync. I just ended up cut-and-pasting the filters from the underlying SSC filter file into the SQL Compare project filter to avoid dealing with the clunky object filtering GUI. I believe that the next version of SQL Compare will allow it to share filters with SSC, so at least this problem is only a short term one. (NOTE: This issue has been resolved in the latest version of SQL Compare. SQL Compare can now use the object filters created by SSC.)
SQL Compare also can’t compare against a SSC database repository when launched directly. It has to be launched from within SSMS. I believe that the next version of SQL Compare will provide this functionality, so again it’s another short term problem. (NOTE: This issue has been resolved in the latest version of SQL Compare.)
Sometimes SQL Compare isn’t able to create the proper scripts to get the target database from one state to another, usually in the case where you are updating the schema of existing tables that aren’t empty, so you currently have to write manual scripts and manage the process yourself. Fortunately, this will be addressed through “migration scripts” in the next release of SSC, and from looking at the early release version of the product, it appears that the implementation of this new feature was well thought out and designed. (NOTE: Migration scripts functionality has been officially released. However, it does not currently support branching. If you want to use migration scripts, you will need to run sql compare against your original development code branch... the one where you checked in your changes... which is pretty clunky and has forced me to modify my build process in a less than ideal way in order to work around this limitation. Hopefully this will be addressed in a future release.)
Overall, I am pretty happy with the product and with Redgate’s responsiveness to user feedback and the direction that the product is taking. The product is very easy to use and well designed, and I feel that in the next release or two the product will probably give us most, if not all, of what we need.
I use SQL Compare for generating scripts when going from dev -> test -> production and it saves me tons of time.
For source control though, we use SVN and ScriptDB (http://scriptdb.codeplex.com/) though. I mainly use source control of SQL scripts for keeping track of changes. I think that rolling back a version of the database seldomly (if ever) works since data may have changed when making structure changes.
This works fine for a few of our current projects (largest is 200 tables and 2000 sprocs). The main reason for doing this though is cost since not all team members have to buy SQL Compare (I avoid adding dependencies to commercial projects unless really needed).
We performed an extensive evaluation of Red Gate's product and found a few major flaws. If you want to look at who changed an object, you can't do it without SysAdmin privileges. The product needs to look at the trace on your server, which requires those rights. I'm on a 5+ person team, and not knowing who had pending changes is what will stop us from using the product.
I just started working for a new company and they use Redgate SQL Source Control for all their projects, amonst them a large and complex one. It does the job well in tandem with TFS. The only drawback from my point of view is that the SQL Server Management Studio integration is highly unstable. Frequent crashes of SQL Server Management Studio happen when the tools are installed.
I was just about to buy redgate.com's compare package when I came to realize it seems highly unlikely a task like this isn't something that can be performed from the Management Studio.
I have a development database and I have a live database.
I develop my application adding stored procedures, correcting small things in the data structure and maybe add some new tables.
Now I want the changes added to the live server. As it is unlikely I have kept 100% track of all of my changes I would of course like if it just new itself what I have changed.
Does anyone know what's best practice in Management Studio for this task? I mean it must be a task we all gets confronted with in larger projects.
If you do it with management studio it's all restricted to your own head. YOU will have to keep track of all the changes etc. I find that Redgates SQL Compare 8 and SQL Data Compare 7 does the trick. I recently moved the structure of a production database over to something totally different within minutes thanks to Redgate.
I have earlier tried to use Management Studio but it does not perform very well at complex tasks.
You really should be keeping your database schema in version control along with your source code.
What's wrong with it not being in the IDE? It has its own interface.
This tool save me hours a day. Not only because I dont HAVE to remember everything, but I then dont have to fix the production server if I missed something. I can work on multiple systems and push updates to each of them in a 10th of the time it would take me in the Management Studio.
This tool is by far the best 3rd party tool ive used for sql server. Its a godsend. A cheap one at that.
Just dont have the arrow pointing the wrong way ;-)
Okay I just bought Redgate :)
I have worked with it before so i can only agree that it is state of the art, I was just stunned to realize that Management Studio don't have anything that corresponds to this.
Thanks for the great answers!
Stackoverflow really is the pleace to ask these kinda things.