I have a table (tbl1) in an Excel file with about 70k rows. I have linked that table into Power Pivot. There is another table (tbl2) in SQL Server with millions of rows that I need to left join to the table in my Excel file on
tbl1.[Member Number] = tbl2.[memid]
What query should I use to do it without having to import the whole tbl2 from SQL Server (throws error on Power Pivot due to memory constraints)?
Preferred environment is Power Pivot, but I do have SQL Sever Management Studio. I don't have WRITE permission in the server where tbl2 is located. I do however have WRITE access in a different server.
Thank you!
Import the Excel file into the SQL server where you have WRITE access, do the join there and import the data from this server. Any problem you see with this approach?
A few options spring to mind:
Get more memory/64 bit Excel and use PowerPivot as you are currently trying to do. If you are working on 32bit excel then you are effectively constrained to using 1gb of RAM whereas on 64bit you can use everything you've got. 64bit Powerpivot can potentially deal with hundreds of millions of records.
Read the SQL data and your csv into R, do the join and either write the output to your WRITE db or save as CSV to feed into PowerPivot. Although R has a steep initial learning curve, doing this kind of thing with the dplyr library is straightforward.
Dump your SQL table to csv and read both that your current csv into the WRITE db and do the join in your PowerPivot SQL query.
Which works best probably depends on the skills you have and how often you are doing this. I'd probably go down the 'R' route as you can set it up as a scheduled job.
Related
I am looking to see if the capability is there to have a custom SSMS sql query imported in SPSS (Statistical Package for the Social Sciences). I would want to build syntax that generates this query as my new dataset that I can then continue my scripted analysis. I see the basic query capability of one table from a Sql Server but I would like to create a query that joins to many tables. I anticipate the query to be a bit complex with many joins and perhaps data transformations.
Has anybody had experience or a solution to this situation?
I know I could take the query and make a table of it that SPSS can then connect to but my data changes daily and I would need a job in another application to refresh this table before my SPSS syntax would pull it and I would like to eliminate that first step by just having the query that grabs the data at the beginning of my syntax.
Ultimately I am looking to build out my SPSS syntax and schedule it in the Production Facility to run daily.
I am not sure if this is possible but I have a very large data set (extracted using Microsoft SQL Server Management Studio) and would like to import this into some sort of cube (OLAP?) to be analysed in Microsoft Excel (Pivot tables etc.) which is of course limited by c.1 million lines.
Ideally I would to run a query from management studio directly into a cube which is then accessed via Excel.
Thanks in advance.
You can create a Pivot Table using an external data.
Then you can give the SQL Server as a source and it would not matter that the source is with more than 1 million rows:
I'm having trouble performing a query on a remote Oracle SQL Server via Excel VBA while trying to perform an INNER JOIN in the same query with a local table in a Excel file sheet.
Example:
Excel Sheet with local table ["LTE_Cells$LTE_Cells_Tmp"]:
Sheet "LTE_Cells"
Oracle SQL Query
SELECT a.STARTDATE, a.ENODEB, a.EUTRANCELLFDD, (a.COUNTER_1/8/1024)+
a.COUNTER2/8/1024) AS Total_Total_Traffic_TB FROM »»»LOCAL_EXCEL_TABLE««««
INNER JOIN REMOTE_DATABASE.LTE_KPI_1 a ON
((»»»LOCAL_EXCEL_TABLE««««.EUTRANCELLFDD =
REMOTE_DATABASE.LTE_KPI_1.EUTRANCELLFDD) AND
(»»»LOCAL_EXCEL_TABLE««««.ENODEB = REMOTE_DATABASE.LTE_KPI_1.ENODEB)) WHERE
(((REMOTE_DATABASE.LTE_KPI_1.STARTDATE)>=sysdate-3));`
Thanks in advance for the help!
This doesn't answer your question directly, and I may be all wet on this but I don't know that either Excel or Oracle explicitly handles what you are trying to accomplish.
However, MS Access will out of the box. Short answer: I think you are using the wrong tool for this task. You are using the proverbial hammer to saw a board in half. Link the spreadsheet and the Oracle table as linked objects in Excel, and your query should be easy-peazy.
Longer answer: while Access does this simply and easily, it can and probably will leave a path of destruction behind it on the DBMS. Specifically, you can expect to thrash the shared pool in Oracle, as Access will be issuing one query (using literals, no less) for every line in Excel. For 1,000 lines, it probably doesn't matter that much, but if you're going to do this on really large datasets, you will make a fast enemy out of your DBA.
Extended answer: really, the best way to do this is to load the contents of those Excel spreadsheets in Oracle tables and let the DBMS do the heavy lifting. This is bread and butter for the RDBMS.
I've got about 13,000 AccountIDs that I need to pull various data from several data tables using left-joins. The total # of accountIDs is in the millions. I don't have write-access to the server but I was wondering if there was a way I could maybe create a custom/temporary table anyway and do a join to that rather than writing a really, really long Where AccountID in (.....) statement. The accountIDs are currently in a single Excel column so I'd have to get them back in the server somehow.
Thoughts?
You can use OPENDATASOURCE to access your Excel file, but someone will have to push you the Excel file to the server (not ideal).
SELECT * FROM OPENDATASOURCE('Microsoft.Jet.OLEDB.4.0',
'Data Source=C:\DataFolder\Documents\TestExcel.xls;Extended Properties=EXCEL 5.0')...[Sheet1$] ;
You can ask your DBA team to install a SQl Server Client on your local machine and set up a linked server to the live server, and you are all set. Ask them to set up a view of the data you need, and give you access to the view. That's all you need, I guess.
I've got a database that resides on an SQL server box, and another in a separate mdb file.
Both contain similar data, but I'd like to run a query that checks unmatched records from a field that exists in both.
Is this something that's easy enough to do using ADO (VBA)? If so can you point me in the right direction?
The easiest would be to create a new Access Database or ADP, and link the tables from both the SQL server and the other MDB. That way you've got an interface to both from within the same instance, which allows you to query or join different tables.