I work as a consultant for a data-driven decision making consulting projects, so every day, I need to cleaning data, analyze, as well as, present our financial analysis. Most of our tasks require the same type of financial data as well as financial analysis, so instead of using Microsoft Excel to do this, which is quite slow for solving a huge amount of data, I would like to ask for your help that is there any simple model (necessary software, method of importing and exporting data among different software) for data cleaning and analysis by SQL and Excel?
If using Excel 2019 try pressing the short key
Alt + A + P + N + D + S
or go to "Data" main menu, then in that there is a button "Get Data", then there is "From Database", then take "From SQL Database"
Specify the connection details and the query. The data will be loaded on to excel. From which you can start processing.
When you connect SQL to Excel, beware of then using Excel as if it were a database itself. There are a couple of tell-tale signs that you are using Excel as a database, some are listed at https://www.crmsoftwareblog.com/2022/03/yes-you-can-connect-excel-to-sql-server-dynamics-365-and-more/
Related
Currently We have developed a system for a manual work they have been doing using many excel files.
Is there a best practice for data migration? because I wanted to use backend language like .net to do the validation and insert into tables rather than using SQL to do migration.
Total record in excel is around 12K rows but for many tables so its not needed consider a lot about performance and it is only one time.
I would add a few calculated columns in Excel that would generate SQL Insert / Update scripts. Something like ="INSERT INTO table (column) VALUES ('"&A1&"');"
Then just copy calculated column and run it through SQL client. I used to have a macros to run it directly from Excel through OLEDB that would highlight failed expressions and store SQL Exceptions next to them.
That way the data can be easily tidied, corrected and SQL re-run as needed.
I'm having trouble performing a query on a remote Oracle SQL Server via Excel VBA while trying to perform an INNER JOIN in the same query with a local table in a Excel file sheet.
Example:
Excel Sheet with local table ["LTE_Cells$LTE_Cells_Tmp"]:
Sheet "LTE_Cells"
Oracle SQL Query
SELECT a.STARTDATE, a.ENODEB, a.EUTRANCELLFDD, (a.COUNTER_1/8/1024)+
a.COUNTER2/8/1024) AS Total_Total_Traffic_TB FROM »»»LOCAL_EXCEL_TABLE««««
INNER JOIN REMOTE_DATABASE.LTE_KPI_1 a ON
((»»»LOCAL_EXCEL_TABLE««««.EUTRANCELLFDD =
REMOTE_DATABASE.LTE_KPI_1.EUTRANCELLFDD) AND
(»»»LOCAL_EXCEL_TABLE««««.ENODEB = REMOTE_DATABASE.LTE_KPI_1.ENODEB)) WHERE
(((REMOTE_DATABASE.LTE_KPI_1.STARTDATE)>=sysdate-3));`
Thanks in advance for the help!
This doesn't answer your question directly, and I may be all wet on this but I don't know that either Excel or Oracle explicitly handles what you are trying to accomplish.
However, MS Access will out of the box. Short answer: I think you are using the wrong tool for this task. You are using the proverbial hammer to saw a board in half. Link the spreadsheet and the Oracle table as linked objects in Excel, and your query should be easy-peazy.
Longer answer: while Access does this simply and easily, it can and probably will leave a path of destruction behind it on the DBMS. Specifically, you can expect to thrash the shared pool in Oracle, as Access will be issuing one query (using literals, no less) for every line in Excel. For 1,000 lines, it probably doesn't matter that much, but if you're going to do this on really large datasets, you will make a fast enemy out of your DBA.
Extended answer: really, the best way to do this is to load the contents of those Excel spreadsheets in Oracle tables and let the DBMS do the heavy lifting. This is bread and butter for the RDBMS.
RIght at the outset I'd like to say that I am NOT a Cognos Guy .So I have totally disconnected myself from developing cognos cubes / reports whatever you want to call it.
There are COGNOS queries auto generated - very badly written that will cause the Teradata ( DBS 15.1.x ) system to Hog on spool & CPU . I can tune them beautifully after I pull them out from DBQL. I want to know HOW can I implement Custom Queries that can be run periodically as batch reports instead of Cognos auto-generating these queries.
E.g. You create a cube - its writes code behind it and then you can open the code and write custom code that is equivalent to the original code but performs a lot better. Then when you open the cube again - it remembers there is a custom SQL and runs that instead of its own auto generated SQL . This is just how I imagine one way it can do it but again- I am not a cognos resource so pl dont flag me down for lack of knowledge. That is exactly what I am trying to get an idea about
Thanks for bearing with me
In Framework Manager you can create one Query Subject with complex query inside. Do not import tables etc. Just create QS in put your query inside.
You need to use stored procedure to return your expected data and add it to Model.
Then instead of using couple of tables in Cognos report studio (and joins), add one query and point it to your stored procedure. This way your Cognos report will execute the procedure instead of generating query (which may not be efficient in many cases)
I have a table (tbl1) in an Excel file with about 70k rows. I have linked that table into Power Pivot. There is another table (tbl2) in SQL Server with millions of rows that I need to left join to the table in my Excel file on
tbl1.[Member Number] = tbl2.[memid]
What query should I use to do it without having to import the whole tbl2 from SQL Server (throws error on Power Pivot due to memory constraints)?
Preferred environment is Power Pivot, but I do have SQL Sever Management Studio. I don't have WRITE permission in the server where tbl2 is located. I do however have WRITE access in a different server.
Thank you!
Import the Excel file into the SQL server where you have WRITE access, do the join there and import the data from this server. Any problem you see with this approach?
A few options spring to mind:
Get more memory/64 bit Excel and use PowerPivot as you are currently trying to do. If you are working on 32bit excel then you are effectively constrained to using 1gb of RAM whereas on 64bit you can use everything you've got. 64bit Powerpivot can potentially deal with hundreds of millions of records.
Read the SQL data and your csv into R, do the join and either write the output to your WRITE db or save as CSV to feed into PowerPivot. Although R has a steep initial learning curve, doing this kind of thing with the dplyr library is straightforward.
Dump your SQL table to csv and read both that your current csv into the WRITE db and do the join in your PowerPivot SQL query.
Which works best probably depends on the skills you have and how often you are doing this. I'd probably go down the 'R' route as you can set it up as a scheduled job.
I often get requests that look something like this:
"Hey, can you reconcile this Excel spreadsheet with our database and
fill in the additional info for the ones you find? We also need to know
which ones are missing."
ID Name Client AdditionalInfo
------------------------------------------
234 John Client1
147 Mike Client2
546 Mary Client3
...
(Several hundred or thousand more rows)
Is there some way to take an Excel spreadsheet like that and join it to a SQL database within SQL Server Management Studio? I usually copy the ID to make a massive IN list, which is obviously far from ideal. Then I still have the task of lining up the data if I don't get a 1:1 match for every row.
I'm using SQL Server 2005 and Excel 2010.
I've tried to add an Excel spreadsheet as a linked server but querying it warned me that 'Ad Hoc Distributed Queries' are blocked and must be enabled. I'm not sure what negative implications enabling that may have, so I didn't go any further down that route.
My next thought was to use Excel formulas to make a SELECT [Col_A], [Col_B], [Col_C] UNION ALL for every single row in Excel, then copy that to SSMS to make a temp table I could join... but that still seems like a fairly messy and inefficient hack
You could import your Excel spreadsheet into a table in your database created specifically for that reason. In SSMS, right-click your database, go to Tasks->Import Data and choose Excel as your Data Source.
Excel can connect to Sql Server directly. Specifically, you want to use MS Query. This will allow you to join your Excel worksheet directly against a database using SQL.