I'm a newbie in here. I had a problem that need your help.
Here is my context: I have 1 table that contains 100 mil data rows. I need to build reports from this table. I use Power BI, using direct import, and write SQL statements in Power BI. When the data loaded to Power BI, it reduced to 20 mil rows (cause I used GROUP BY in SQL statement). But the performance of Power BI is really terrible. In my opinion, Power BI had to run the query statement, and then visualize data, so its performance is bad.
Here is my solution: I'm going to CREATE VIEW (using GROUP BY statement) in my database. So that, the run query workload is no longer belongs to Power BI. My database will take responsibility for the executive SQL statement. And Power BI just only need to visualize data, so the performance of Power BI will be better.
Here are my questions:
1 - Does my solution work? :)))
2 - IF my solution work, my database just needs 1 time in running CREATE VIEW statement, and no need to run it anymore in the future, right?
3- If my solution work, the SQL running workload will move from Power BI to my database, right?
Thank you in advance.
When you use Import mode, then your Database is query once (at refresh time), but the view still need to make aggregation and there is no difference between select from view vs select with the group by (the view is only a nice packed query, better to materialize view or populate standard Table with the daily job);
It's a good idea to remove unused columns and rows, older than X (also aggregate if possible).
Consider using Incremental refresh to shorted your load.
https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
Incremental refresh is supported for Power BI Premium, Premium per user, Power BI Pro, and Power BI Embedded datasets.
I have no idea about powerBI but if you use view it executes the query in view every call. You can try materialized view instead of view. But be aware materialized view has the snapshot of data when it created or refresed time.
as an example the view MY_MATERIALIZED_VIEW is refreshing every day.
create materialized view MY_MATERIALIZED_VIEW
build immediate
refresh force
on demand
start with sysdate next sysdate + 1
as
....query
Related
I am looking to see if the capability is there to have a custom SSMS sql query imported in SPSS (Statistical Package for the Social Sciences). I would want to build syntax that generates this query as my new dataset that I can then continue my scripted analysis. I see the basic query capability of one table from a Sql Server but I would like to create a query that joins to many tables. I anticipate the query to be a bit complex with many joins and perhaps data transformations.
Has anybody had experience or a solution to this situation?
I know I could take the query and make a table of it that SPSS can then connect to but my data changes daily and I would need a job in another application to refresh this table before my SPSS syntax would pull it and I would like to eliminate that first step by just having the query that grabs the data at the beginning of my syntax.
Ultimately I am looking to build out my SPSS syntax and schedule it in the Production Facility to run daily.
I have an Excel spreadsheet that pulls data from a view (that's pulling from another view) in SQL Server, this was created by someone other than myself. Due to recent changes in how we are storing data in SQL Server, the views no longer actively reflect the data needed.
The current Excel spreadsheet use a command to simply pull the rows from the view, would it make more sense to simply use a query to gather the data from the necessary tables? Is there any advantages to using the views over a regular query?
There is no security being enforced in regards to the view as well, the connection is using an admin user.
When we talk about regular views, the only advantage over query is that I see a shorter expression, i.e. 'select * from view' will be easier to type and edit in MS Query than 'select * from ... join ... join ... where ...' etc. If we are talking about indexed views, then they can greatly improve performance
I am using Oracle 11g EE and created one Materialized view for a complex query.
But as customer is using Standard edition and it doesn't support Materialized view.
So I need to convert materialized view into a normal view.
Along with this conversion I also need to make sure performance of a query.
When I treid to execute a query for the view independently, it takes around 20 mins which is too much time.
All my tables are indexed.
Any sort of help would be appreciated !!
I don't currently have access to an oracle database to test this, but if you get the DDL of the mview, and remove the word materialized along with the mview related options:
select dbms_metadata.get_ddl('MATERIALIZED_VIEW', 'MVIEW_NAME') from dual;
You should be able to create a normal view from the resulting modified statement.
As for performance, make sure you are indexing the right things, and using the right type of indexes. Make sure statistics are up to date, and finally run explain plan on the view query. The topic of interpreting and optimizing execution plans is deep - start here:
https://docs.oracle.com/cd/B10501_01/server.920/a96533/ex_plan.htm
RIght at the outset I'd like to say that I am NOT a Cognos Guy .So I have totally disconnected myself from developing cognos cubes / reports whatever you want to call it.
There are COGNOS queries auto generated - very badly written that will cause the Teradata ( DBS 15.1.x ) system to Hog on spool & CPU . I can tune them beautifully after I pull them out from DBQL. I want to know HOW can I implement Custom Queries that can be run periodically as batch reports instead of Cognos auto-generating these queries.
E.g. You create a cube - its writes code behind it and then you can open the code and write custom code that is equivalent to the original code but performs a lot better. Then when you open the cube again - it remembers there is a custom SQL and runs that instead of its own auto generated SQL . This is just how I imagine one way it can do it but again- I am not a cognos resource so pl dont flag me down for lack of knowledge. That is exactly what I am trying to get an idea about
Thanks for bearing with me
In Framework Manager you can create one Query Subject with complex query inside. Do not import tables etc. Just create QS in put your query inside.
You need to use stored procedure to return your expected data and add it to Model.
Then instead of using couple of tables in Cognos report studio (and joins), add one query and point it to your stored procedure. This way your Cognos report will execute the procedure instead of generating query (which may not be efficient in many cases)
I need to pull a large amount of data from various tables across a line that has very low bandwidth. I need to minimize the amount of data that gets sent too and fro.
On that side is a Sybase database, on this side SQL Server 2008.
What I need is to pull all the tables from the Sybase database that have to do with this office. Lets say I have the following tables as an example:
Farm
Tree
Branch
etc.
(one farm has many trees, one tree has many branches etc.)
Lets say the "Farm" table has a field called "CountryID", and I only want the data for where CountryID=12. The actual table structures I am looking at are very complex (and I am also not very familiar with them) so I want to try to keep the queries simple.
So I am thinking of setting up a series of views:
CREATE VIEW vw_Farm AS
SELECT * from Farm where CountryID=12
CREATE VIEW vw_Tree AS
SELECT * from Tree where FarmID in (SELECT FarmID FROM vw_Farm)
CREATE VIEW vw_Branch AS
SELECT * from Tree where BranchID in (SELECT BranchID FROM vw_Branch)
etc.
To then pull the actual data across I would then do:
SELECT * from vw_Farm into localDb.Farm
SELECT * from vw_Tree into localDb.Tree
SELECT * from vw_Branch into localDb.Branch
etc.
Simple enough to set up. I am wondering how this will perform though? Will it perform all the SELECT statements on the Sybase side and then just send back the result? Also, since this will be an iterative process, is it possible to index the views for subsequent calls?
Any other optimisation suggestions would also be welcome!
Thanks
Karl
EDIT: Just to clarify, the views will be set up in SQL Server. I am using a linked server using Sybase ASE to set up those views. What is worrying me in particular is whether the fact that the view is in SQL Server on this side and not on Sybase on that side will mean that for each iteration the data from the preceeing view will get pulled across to SQL Server first before the calculations get executed. I want Sybase to do all the calcs and just pass the results across.
It's difficult to be certain without testing, but my somewhat-relevant experience (using linked servers to platforms other than Sybase, and on SQL Server 2005) has been that using subqueries (such as your code for vw_Tree and vw_Branch) more or less guarantees that SQL Server will pull all the data for the outer table into a local temp table, then match it to the results of the inner query.
The problem is that SQL Server has no access to the linked server's table statistics, so can make no meaningful decisions about how to optimise the query.
If you want to be sure to have the work done on the Sybase server, your best bet will be to write code (could be views or stored procedures) on the Sybase side and reference them from SQL Server.
Linked server connections are, in my experience, not particularly resilient over flaky networks. If it's available, you could consider using Integration Services rather than linked-server queries - but even that may not be much better. You may need to consider falling back on moving text files with robocopy and bcp.