I'm currently setting up a database which has an upload process in SSDT. It pulls data over time from an Excel sheet. My issue is that the Excel sheet is appended onto as time continues but the upload process uploads the entire sheet every time the process is run. This results in exact, duplicate rows for the data which were in there previously.
I have attempted to solve this in a number of ways. I have attempted to use the Sort function included in SSDT but have not been able to get it to work for whatever reason. I am considering writing and SQL task to clean the database after each upload but am only so confident in my ability to do so. Is there some method I am not thinking of which would make this easier or a way to get the sort transformation to work? Thanks for the help in advance.
You can delete duplicate rows in SQL Server using a cte with row_number() window function.
Related
I have a fairly complex database that has worked for quite a while. Today I went to troubleshoot something and I got a non-sensical error.
In my macro, first I create detail tables. Then I number each row sequentially in the detail table with a VBA script. Then I create a summarized version of that table including the just the key info like SKU and row number. (The rest of the details are added back later using the row number). The numbering script is working. The detail tables have the row numbers. However the summary table does not have the row numbers after running the macro. If I manually click the queries to add the summary tables, the numbers magically appear.
Please help. This database has worked for over a year without this problem.
Honestly it sounds like its corrupt. MS Access uses the Jet Engine which is known for bloating dB size and quite easily becoming corrupt.
You can see by the odd behaviour, the program is having trouble with the underlying file. If you open a new Access DB it won't exhibit the same problems.
The best bet is to recreate the dB from scratch or port it to SQL Server, PostGres, MySQL, SqlLite etc and put the application logic in a program - even if its a VB6 app, though I'd recommend .Net.
I've got the code to strip out every thing and recreate the MS Access dB here:
https://stackoverflow.com/a/16158027/495455
I inherited some old records for a company I volunteer for. One of the old files is an SQL Dump from their old webpage, and I would like to get the data from one of the tables for their use into Excel.
-- MySQL dump 10.11
The dump drops the table if it exists, creates the table new, and then inserts all of the data.
Is there some easy way I can get this data into Excel on my PC? I don't have SQL Server or anything like that loaded... I assumed there was some easy way to get a CSV or Excel file out of it but I have failed to find this yet without first uploading the dump to some SQL Server.
Unfortunately I don't think that there is any way to export a dump file into an excel or .CSV file. The reason for this is that the dump file is actually a collection of Select statements instead of the actual data itself. SQL servers do this to prevent a whole list of problems that can occur when you try to manipulate raw data manually.
Lucky for you, MySQL offers a free version of their server. You can find it here: http://dev.mysql.com/downloads/
I think you are best off downloading this and restoring your file as a new database. This has the added benefit of allowing you complete control over the data from that point on. Exporting to excel would be easy at that point however, you may find it a lot more fulfilling to continue using MySQL server.
Hope this helped.
So I have an excel spreadsheet with Product and Notes. I'd like to import this information into SQL and everytime people enter more products and notes into this excel sheet, it will automatically updated to the new one whenever I run the syntax?
I finished creating Product - Notes, which I imported current data into that table.I was planning to use insert into function, and every day insert the new values in the table. But this seems too manual.
Is there a way i can do this? The excel spreadsheet is updated daily.
I'm using SQL Server 2008
I'm sure this is possible. You could have the excel connect to your database and then write some macros to save the data to the table when there are changes or new rows are written.
It would not be easy. There is a lot of complicated logic here and excel was not written to be a front end for a database.
I believe the time spent changing your spreadsheet to work this way would be better spent actually writing a client server application to modify the database using a web application or a local application. Client server front end applications are easy to write these days with lots of examples, tools and templates. For someone with experience a simple data entry / modification form is just a couple of days work for a robust application.
Changing the excel file would be much harder.
You could use SSIS to import the excel data into your database on a scheduled basis.
I am trying to automate a daily monitoring activity where there are set of scripts to be executed(all are select statements). I am in the process of creating a procedure which runs these scripts and by means of scheduler, this will be running daily once. My problem is since these activities are taking place in server side(server backbone), How do i save the results? Earlier we will run all the scripts manually and save it in a notepad. Is there any option to do the same in automation? Like saving in our PC or SQL developer? Instead of logging in to server and searching the path where the file is saved? I thought of saving the results in a table but i am looking for a better option.Please suggest...
Generally it is a good idea to save the results in a table as this gives you flexibility when querying the results or exporting them in multiple formats.
There are multiple options to get the data to the client:
Query the table with the results from the client
Generate a HTML from the results table and have make it accessible from a HTTP server.
You can also create a web PL/SQL package and generate the HTML within (http://docs.oracle.com/cd/B28359_01/appdev.111/b28424/adfns_web.htm#i1006207)
Export the data from the results table to a file and put it in a shared directory that is accessible by the client.
Email the results from the PL/SQL package.
I thought of saving the results in a table but i am looking for a better option.
What is exactly the issue with the "table" option?
Regarding "saving in our PC or SQL developer": one problem with that is that a PC/app screen is:
a PC is usually less resilient to reboots, crashes, etc.
it's intended for private use. Unless you're working alone - these logs may be of interest to other people;
..
Other options: it can be made to send e-mail; copy the file to a well known place (incl. one which is directly mounted on your PC); write to database table (as already suggested); and more.
Presently am working with an Excel spreadsheet that makes at least 10 database queries to external Microsoft Access *.mdb files. For each project my company works on, we have a specific excel file related to that project, so we have hundreds of these files. Usually when an analyst opens the Data tab and click on "Refresh All" the refresh completes in a minute or two; however, on a new project for a given excel file it is taking at least an hour to complete the refresh. Here is an example of one of the connection strings:
DSN=MS Access Database;DBQ=W:\Projects\Analysis\project.mdb;DefaultDir=W:\Projects\Analysis\Analysis;DriverId=25;FIL=MS Access;MaxBufferSize=2048;PageTimeout=5;
And here is the associated query:
SELECT Field.FieldNumber, Field.FieldName, Field.GroupMnemonic, Field.ClientFieldID
FROM Field Field
ORDER BY Field.FieldName
I have spent time studying various websites discussing slow excel issues like http://msdn.microsoft.com/en-us/library/ff700515.aspx; however, these websites deal more with calculations and VBA whereas I suspect the performance problem is somewhere in an access file. Does anyone have any suggestions on how to troubleshoot and resolve this issue? TIA.
UPDATE: As suggested in the answer below by JohnFx, I checked the queries and found that they had no definite keys, so and so added keys in the Microsoft Access database generation like this:
CREATE UNIQUE INDEX PIndex ON [myTable] ([KEY])
Run the queries individually directly in Access to rule Excel in or out as part of the problem. If the queries are still slow in Access consider adding indexes on any columns that are in being sorted or filtered on.