Okay for this I am tasked with creating an Archive database for the Primary database. They want this done to keep the one they use clean. Basically a database for a database. I have been able to set up the queries for the tables that they want to get the information from to be archived out. Then once the data is pushed to the archive they want some method to go back in to deleted the data that was just pushed from the tables, making them smaller, which I have working. Everything works fine until I run it a second time. Because The code goes back in to query from the deleted fields the queries that are run and then the information pushed no longer has information(blank table) coupled with the way the DoCmd.transferdatabase works. It is just overriding the tables of information with a blank table in the archive database. This would not be a problem except for them having to implement the archiving again later down the road with new parameters for the queries for the tables. It would then just over ride the table instead of combining them. I need a way to get the tables to combine. They are the same table just need to add different information periodically that does not delete the existing table in the Archive.
I am fairly new at this and nothing I have tried has made it work. I am wondering if this is even possible to combine table information from two separate DB's?
Instead of moving archived data back & forth, just link to the archive table instead.
The command will begin like this: DoCmd.TransferDatabase acLink
The you can run queries against the production or archive tables whenever you want.
Related
I am using a Google Sheet as the source of a table in Big Query. Since I am unable to rename fieldnames in the schema of an existing table I deleted the table and attempted to re-create it after amending the column names in the source Google Sheet. I need to keep the table name the same as I already have analysis files connecting to the table, however when I create the new table as ask Big Query to auto-detect the schema it uses the schema of the previous table. Even if I enter the new schema as text when creating the table it ignores what I enter and use the schema from the old table.
Any ideas how I get Big Query to detect the new schema from the Google Sheet whilst using the same table name as the deleted table?
Thanks in advance!
After trying this multiple times and it not working - with several tables - randomly it worked and let me create a table with the new scheme (manually). Not sure why this didn't work before as I'm pretty sure I didn't do anything differently. If anyone has any insight on what might have caused the initial errors I'd love to hear it for future reference but my current problem is solved.
I inherited an Access based dashboard which uses a series of SQL queries and Make Table actions to write the results of those queries into tables located in two other Access files.
There is a macro that runs all of the make table commands which in turn run the sql queries. All of them are working when I run this on my machine, but when I run it from our VM which handles scheduled refreshes, it creates a duplicate table for one of the queries.
If I delete that table from the Tables1 database, the Queries database will run successfully and create both the correct table, and the duplicate in Tables1. However, each subsequent time, the macro will fail with an error saying that the duplicate table already exists.
This is the make table sql:
SELECT [SQLQueryName].*
INTO [TableName]
IN 'filepath\Tables1.accdb'
FROM SQLQueryName;
This is the same structure that all the other make table queries are using and they are not having this issue. I'm not sure if this matters or will tell someone something, but the duplicate table is the same name with a 1 added to the end of it. We also recently had to get a new VM setup and there have been a lot of weird issues with things not working as they did in the previous VM where this had been running without issue for quite a long time.
So far I've tried compacting both the Query and Table1 Database files. I've tried deleting the make table query and making a new version. I've tried deleting the table and duplicate from Tables1. I've tried rolling back the database files to versions several weeks old. We also made sure that the version of Access is the same as on my PC.
I am currently attempting to change it to a delete row/append row method rather than a make table, but even if that works I would still love to know why this is happening.
Edit:
Actual Code from make table that is failing, removed filepath.
SELECT [012_HHtoERorOBS].*
INTO [HHtoER-Obs]
IN '\\filepath\MiscTablesB.accdb'
FROM 012_HHtoERorOBS;
Here is code from 2 other make table queries that are working. Each make table query follows an identical format of select * from the sql query to get the data into the table in the destination accdbs.
SELECT [010_ScheduledVisitsQuery].*
INTO ScheduledVisits
IN '\\filepath\MiscTablesB.accdb'
FROM 010_ScheduledVisitsQuery;
SELECT [020_HH_HO].*
INTO HH_Referred_to_HO
IN '\\filepath\MiscTablesB.accdb'
FROM 020_HH_HO;
All of these tables exist in the destination accdbs when the make table queries are run. The macro does not include any commands to delete tables. Here is a screenshot of the top of the macro, it repeats all the make table queries then ends with a command to quit access.
I wanted to basically copy the entire content of one table to another.
Context:
Table source is SharePoint list and triggers an email per record being queried. No way to turn it off on my end as it's being utilised by another team.
When I run my queries on a local table, it's fine.
I need to just copy the data directly. So far, the only code I found is
DoCmd.TransferDatabase but I can't seem to configure it correctly.
Simplest method is probably to run a make-table query to (re)create the local table:
Dim Sql As String
Sql = "SELECT * INTO LocalTable FROM SharePointTable;"
CurrentDb.Execute Sql
That will pop a warning, though. If that is too much, create the local table, then run two queries - the first to delete all records from the local table, the second to append all records from the SharePoint table to the local table.
Problem:
I need to get data sets from CSV files into SQL Server Express (SSMS v17.6) as efficiently as possible. The data sets update daily into the same CSV files on my local hard drive. Currently using MS Access 2010 (v14.0) as a middleman to aggregate the CSV files into linked tables.
Using the solutions below, the data transfers perfectly into SQL Server and does exactly what I want. But I cannot figure out how to refresh/update/sync the data at the end of each day with the newly added CSV data without having to re-import the entire data set each time.
Solutions:
Upsizing Wizard in MS Access - This works best in transferring all the tables perfectly to SQL Server databases. I cannot figure out how to update the tables though without deleting and repeating the same steps each day. None of the solutions or links that I have tried have panned out.
SQL Server Import/Export Wizard - This works fine also in getting the data over to SSMS one time. But I also cannot figure out how to update/sync this data with the new tables. Another issue is that choosing Microsoft Access as the data source through this method requires a .mdb file. The latest MS Access file formats are .accdb files so I have to save the database in an older .mdb version in order to export it to SQL Server.
Constraints:
I have no loyalty towards MS Access. I really am just looking for the most efficient way to get these CSV files consistently into a format where I can perform SQL queries on them. From all I have read, MS Access seems like the best way to do that.
I also have limited coding knowledge so more advanced VBA/C++ solutions will probably go over my head.
TLDR:
Trying to get several different daily updating local CSV files into a program where I can run SQL queries on them without having to do a full delete and re-import each day. Currently using MS Access 2010 to SQL Server Express (SSMS v17.6) which fulfills my needs, but does not update daily with the new data without re-importing everything.
Thank you!
You can use a staging table strategy to solve this problem.
When it's time to perform the daily update, import all of the data into one or more staging tables. Execute SQL statement to insert rows that exist in the imported data but not in the base data into the base data; similarly, delete rows from the base data that don't exist in the imported data; similarly, update base data rows that have changed values in the imported data.
Use your data dependencies to determine in which order tables should be modified.
I would run all deletes first, then inserts, and finally all updates.
This should be a fun challenge!
EDIT
You said:
I need to get data sets from CSV files into SQL Server Express (SSMS
v17.6) as efficiently as possible.
The most efficient way to put data into SQL Server tables is using SQL Bulk Copy. This can be implemented from the command line, an SSIS job, or through ADO.Net via any .Net language.
You state:
But I cannot figure out how to refresh/update/sync the data at the end
of each day with the newly added CSV data without having to re-import
the entire data set each time.
It seems you have two choices:
Toss the old data and replace it with the new data
Modify the old data so that it comes into alignment with the new data
In order to do number 1 above, you'd simply replace all the existing data with the new data, which you've already said you don't want to do, or at least you don't think you can do this efficiently. In order to do number 2 above, you have to compare the old data with the new data. In order to compare two sets of data, both sets of data have to be accessible wherever the comparison is to take place. So, you could perform the comparison in SQL Server, but the new data will need to be loaded into the database for comparison purposes. You can then purge the staging table after the process completes.
In thinking further about your issue, it seems the underlying issue is:
I really am just looking for the most efficient way to get these CSV
files consistently into a format where I can perform SQL queries on
them.
There exist applications built specifically to allow you to query this type of data.
You may want to have a look at Log Parser Lizard or Splunk. These are great tools for querying and digging into data hidden inside flat data files.
An Append Query is able to incrementally add additional new records to an existing table. However the question is whether your starting point data set (CSV) is just new records or whether that data set includes records already in the table.
This is a classic dilemma that needs to be managed in the Append Query set up.
If the CSV includes prior records - then you have to establish the 'new records' data sub set inside the CSV and append just those. For instance if you have a sequencing field then you can use a > logic from the existing table max. If that is not there then one would need to do a NOT compare of the table data with the csv data to identify which csv records are not already in the table.
You state you seek something 'more efficient' - but in truth there is nothing more efficient than a wholesale delete of all records and write of all records. Most of the time one can't do that - but if you can I would just stick with it.
I am not sure how to ask this question so please direct me in the right direction if I am not using the appropriate terminology, etc. but I can explain what I am currently doing. I would like to know if there is an easier way to update content in the database than the method I'm currently using.
(I'm using SQL Server 2008 BTW.)
I have a bunch of CSV files that I use to give to my client as a means to update content which gets imported into the DB (because the content is LARGE). The import works by running a python script that I wrote that makes use of a Jinja2 template that generates the SQL file needed to insert the CSV content into the database (if it is a from-scratch scenario). This is working fine.
Now when it comes to data migration (I need to migrate the data that exists in the DB to a new version thereof) I have a lot of manual work (I hand code it in the template, there is no SQL command or auto-generated code that I can run to do this for me) to do.
So lets say I have a list of Hospitals in a CSV file and I already have a set of hospitals in the database (which is imported from the previous version of the CSV file). I create a copy of the Hospitals table (without the data) and call it HospitalsTemp. The new CSV hospitals are inserted into the HospitalsTemp table (at least that part is generated via the template).
The Hospitals table now gets detached from all its foreign-keys and constraints. Now I go through all the tables surrounding the Hospitals (again manually!) and replace the hospitalId which pointed to the old hospitalId with the new hospitalId (as I can do a lookup from the Hospitals to the HospitalsTemp based on the hospital code to ensure that referential integrity is retained).
Then I delete the Hospitals table and rename the HospitalsTemp to Hospitals and put back the foreign-keys and constraints on the new Hospitals table.
I hope I explained it well enough for everyone to understand. I'm really hoping for a simpler way to do this.
How do you know which hospital becomes which, do the names stay the same? Is there an Id that stays the same?
Have you looked at SSIS, and the Slowly Changing Dimension component? You can use it to update existing rows and add new rows: http://blogs.msdn.com/b/karang/archive/2010/09/29/slowly-changing-dimension-using-ssis.aspx
Also SSIS would be a good tool for the import, as it handles reading CSV files well.
You could replace the current logic with simple SSIS package that's just a flat-file data source and the output of the SCD wizard by the sounds of it?