I am importing data from Excel into an existing table in Access and want to suppress the below message.
I have tried using a multi field Index to import new records into the table and have also tried importing firstly to a Temporary table and then appending new records to the existing table.
However under both scenarios it still gives the below message pop up which I want to avoid the user seeing (as they could click yes by accident).
If I try SetWarnings = No in a macro, it just reimports all entries irrespective of whether they are duplicates or not so that doesn't work.
I would appreciate any help
Thanks
Don't import the Excel data, link them.
Now you have a linked table. Use that as source in a query where you join it with the existing table.
Select only linked records that are not already present.
Change the query to an append query. This query you can run as often as you like.
When a new Excel file is received, just replace the linked file with the new file.
This message will appear when trying to import data that violates an Index in the destination MS Access table. Check that your Excel column data does not violate the corresponding MS Access field index settings.
If the MS Access field is set to "Required" = Yes, Null values (empty cells in Excel) will also cause the message to appear.
That's two possibilities...
Related
I'm doing for the first time an import of an excel file via Phython to insert the data and later maintain them with update or delete in the PostgreSQL database.
However, I am not able to insert several lines, I already checked the executemany() command, but it did not work.
How can I make this script?
I'm doing it this way above to perform the inclusion of data in the database of this worksheet below:
However, when executing the script it is inserting data that does not compose the content, I managed to do it in another way as well and the same inserted only the last line.
But I expected that the content inserted in the bank would be all the lines contained in the excel.
And I would also like to know how I can make a data change later.
SQL Server 2012: using a SSIS package, how to validate the source records for duplicate before inserting?
Our source file is a .csv. We are facing duplicate records loaded in the staging table.
At present , we are following manual process of loading data.
How to validate the source file data against the destination table before loading and load only the valid records? Possibility of loading duplicate records not only because of the source file having duplicate records in it but also reloading the same file to the staging table.
We are not Truncate the staging table. We are keeping records as is.
Second question : How to pick the name of the source file and pass it in the loading ? Possibly having a derived column as "FileName" which will get loaded along with raw data to the staging table.
The typical load pattern I use in this case is:
Prepare a staging table that matches the source file
In SSIS run a SQL Task with TRUNCATE StagingTable; (which clears it out)
Then, run a data flow task that loads the entire data file into the staging table
Lastly, merge the staging table into the final table.
I prefer to do this last step in a SQL Task also:
INSERT INTO FinalTable
(PrimaryKey,Column1,Column2,Column3)
SELECT
PrimaryKey,Column1,Column2,Column3
FROM StagingTable SRC
WHERE NOT EXISTS (
SELECT * FROM FinalTable TGT WHERE TGT.PrimaryKey=SRC.PrimaryKey
);
If you prefer a graphical UI, and you don't mind the extra network traffic, and slower processing time, you can do the same type of merge operation using lookups. You can even use the SCD component but I strongly discourage it's use.
Whether you do it in T-SQL or the UI, you need a key that can be used to uniquely identify the records (referred to as PrimaryKey in my example). If you don't have this key, there is no way to 'deduplicate'
Note in this example you have a 'real' staging table whose only purpose is to get the data file into the database. Then you have a final table that contains the final consistent result
Also note that this pattern only adds new rows - it will not update existing rows if they change in the data file.
Given your exact scenario (of loading the same file again), I would first check if the data is even loaded to the staging table. If you do that, you don't have to worry about checking the duplicates at record level.
How are you setting the connection to the file? Most of the data loads I have dealt with, I designed for-each-loop-container where the file name/path would be populated in a user variable. As you said, you could just use a derived column transform to add a new column which gets the value from a variable. If you don't have the file name in a user variable, you could use expression task in the control flow to populate it.
To cover your exact requirement, I would use the above step to populate the file name in the table. You could even normalize to a different table instead of storing long file name for every data record. Once you have all the file names in the database, you could just have an "Execute SQL" at the beginning to see if that file name is already in the database.
Two years back I have faced the same problem with importing TSV files.
I tried many other solutions but best I could design is C# code script for such validation at its best.
What I did as a solution
Create one C# DataTable object in memory with Primary Key constraints,
like:-
DataColumn[] keyColumn = new DataColumn[30];
keyColumn[intJ] = dtFilterdPK.Columns["Column name"];
Then try to add one by one row from your CSV to this DataTables.
Whenever your data will get Duplication based on Primary Key will have an error
Handle this error code in (TRY)..CATCH block and make this duplication error as per your logging requirement.
Avoid those error records importing in DataTable object.
Atlast import your CSV file into your table as BulkImport
Like:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(myConnection))
{
bulkCopy.DestinationTableName = "Your DB Table Name"; //Assign table name
bulkCopy.WriteToServer(dtToBeImport); //Write into Actual table.
}
Hope this will help you.
I have created a query in an Access database and exported the query result into an Excel file. Now, I want to connect that Excel file to the Access database (or to that query) so that whenever some fields are updated in database, those changes would be automatically updated in the exported Excel file (report). What would be the best way to do it?
Thanks
You can create a link between Excel and an Access table or query by going to the Data tab on Excel and clicking the "From Access" icon.
Alternatively if there is a trigger inside access that you can use to update data (or if you don't need real time updates just hourly or daily ones), you could have access programmatically re-export.
If you want to work with data in Access, but still maintain the data in Excel, you need to link to the data rather than import it. Follow these steps:
Create a blank database or open an existing file in Access.
Select File, Get External Data, Link Tables.
Select Microsoft Excel as the file type.
Select a worksheet or named range to import, and then
click Next. You can import only one worksheet or named range at a
time, and each one will become an Access table.
In the next dialog box, select or deselect the check box First Row
Contains Column Headings, depending on whether your worksheet has
headings. Then click Next.
Enter a name for the table (or accept the default name that Access
suggests), click Finish, and click OK.
Now you have an Access table that looks almost exactly like the imported table. The advantage is that it maintains a live link to the Excel worksheet and can be edited in either application.
I have created an SSIS package (see below) to import data from an external SQL query into a SharePoint 2007 list. The data imports fine but when the package is ran again to update the data it duplicates the records. I'm guessing that as there is no link between the SharePoint ID of the imported records and the data from my SQL query the routine has no idea what to update and just creates a new record. How do I prevent this and allow my data to be updated in the SharePoint list?
If you are setting the key Id field in your SharePoint list target it will perform an update, otherwise the default is an insert. It sounds like you have not mapped the Id
You can either
Set (map) the ID column thus forcing the SharePoint destination component to perform an update. Have a look at this example by Chris Kent
Limit your source select statement based on the last inserted record inside the SharePoint list. Prior to the data flow task, you would need to select the max(date or key?) from SharePoint and set an expression for your data source to include this value in the WHERE clause resulting in selecting only new records. This has the added benefit of limiting the amount of data traveling across the network and your existing insert setup would work.
I'm using SSIS to import data from one DB to another existing DB. Some columns in the destination tables do not exist in the source tables. Seems the Import & Export Wizard only allows me to select unmapped columns from the source and match them with these new columns in the destination. I'd like to be able to just provide one piece of data to import into all rows of these new columns.
Would like to use the GUI if possible because I'm not skilled at writing scripts. Thanks!
In SSIS, you can add a "derived column" component that will add columns to the buffer rows with the value you want (either a string or an expression).
I don't believe this is possible in the GUI. However, it would be a simple script after the data is loaded with SSIS:
UPDATE table SET newcolumn = new value
If you need to filter the rows, just add
WHERE column = value ...
You could change your source to a select query and list out the columns along with the static value you want to map.
SELECT SOURCECOLUMN_1,SOURCECOLUMN_2,....,SOURCECOLUMN_N,'VALUE' AS DESTINATIONCOLUMN FROM Source_Table
My original thought was that you could use the query right in the Import & Export wizard. you can obviously do alot more if you go in and edit the package, but it sounded like you didn't have much expereince with that. Here is how you would do this in the wizard.
After you have selected your source and destination databases you can Specify Table Copy or Query. Select the Write a query to specify the data to transfer option
On the next screen enter the query listing out all of the columns and add in your static columns.
On the Next screen You will need to select the Destination table or it will default to creating a new table named Query. You should be able to choose from the drop down. As long as you aliased your extra columns with the same names it should map correctly. You can go in and edit mappings here if needed.
You can then save off the SSIS package and it will source form the query.
Alternatively if you already have the SSIS pacakge created without the extra columns you can go in to the Data Flow and change the Data access mode in the OLE DB Source to be a SQL Command instead of a table or view. Add your query here.
You can then go into the properties of the OLE DB Desitination in the Dataflow and map the new column. You could also add in a derived column as #DominicGoulet by adding in a Dervied Column task and putting your static information here and then mapping. If you want to see that solution too let me know.