I load 10 tables from an ACCESS 2007 file database. IS their a way I can get the column names into the dataset or do I have to rename each column? I am using Visual Studio 2008 in VB.NET.
I would like to reference the columns in the code by the name and not have to use an index
I just went ahead and added the following code for each of the tables:
Table1.column(0).name = "NAME"
Table1.column(1).name = "ADDRESS"
Table1.column(2).name = "CITY"
Table1.column(3).name = "STATE"
But I do this for 10 tables so it can add up fast on the lines of code. But if their is a way for me to read it from the file I would like to replace all the lines of code with one line of code.
When you create the dataset, if you provide a connection string locally, when you generate the xsd file it should automatically type out any tables you add to the set. As long as you're using the tableadapters built into .Net, you can change the connection string of any tableadapter you use for your local config when you deploy.
When you right-click in the blank space of the new dataset you create (in the designer), you can add->tableadapter. It'll ask you for the connetion string to the database (provide it), and then you can submit a query. Start with the a general query for the basic table you're looking for. You can then add subsequent queries as you need them to the same table or create custom tables (and names) based on joins/other queries.
EDIT: I'm assuming some version of .Net is being used here.
Related
SQL Server 2012: using a SSIS package, how to validate the source records for duplicate before inserting?
Our source file is a .csv. We are facing duplicate records loaded in the staging table.
At present , we are following manual process of loading data.
How to validate the source file data against the destination table before loading and load only the valid records? Possibility of loading duplicate records not only because of the source file having duplicate records in it but also reloading the same file to the staging table.
We are not Truncate the staging table. We are keeping records as is.
Second question : How to pick the name of the source file and pass it in the loading ? Possibly having a derived column as "FileName" which will get loaded along with raw data to the staging table.
The typical load pattern I use in this case is:
Prepare a staging table that matches the source file
In SSIS run a SQL Task with TRUNCATE StagingTable; (which clears it out)
Then, run a data flow task that loads the entire data file into the staging table
Lastly, merge the staging table into the final table.
I prefer to do this last step in a SQL Task also:
INSERT INTO FinalTable
(PrimaryKey,Column1,Column2,Column3)
SELECT
PrimaryKey,Column1,Column2,Column3
FROM StagingTable SRC
WHERE NOT EXISTS (
SELECT * FROM FinalTable TGT WHERE TGT.PrimaryKey=SRC.PrimaryKey
);
If you prefer a graphical UI, and you don't mind the extra network traffic, and slower processing time, you can do the same type of merge operation using lookups. You can even use the SCD component but I strongly discourage it's use.
Whether you do it in T-SQL or the UI, you need a key that can be used to uniquely identify the records (referred to as PrimaryKey in my example). If you don't have this key, there is no way to 'deduplicate'
Note in this example you have a 'real' staging table whose only purpose is to get the data file into the database. Then you have a final table that contains the final consistent result
Also note that this pattern only adds new rows - it will not update existing rows if they change in the data file.
Given your exact scenario (of loading the same file again), I would first check if the data is even loaded to the staging table. If you do that, you don't have to worry about checking the duplicates at record level.
How are you setting the connection to the file? Most of the data loads I have dealt with, I designed for-each-loop-container where the file name/path would be populated in a user variable. As you said, you could just use a derived column transform to add a new column which gets the value from a variable. If you don't have the file name in a user variable, you could use expression task in the control flow to populate it.
To cover your exact requirement, I would use the above step to populate the file name in the table. You could even normalize to a different table instead of storing long file name for every data record. Once you have all the file names in the database, you could just have an "Execute SQL" at the beginning to see if that file name is already in the database.
Two years back I have faced the same problem with importing TSV files.
I tried many other solutions but best I could design is C# code script for such validation at its best.
What I did as a solution
Create one C# DataTable object in memory with Primary Key constraints,
like:-
DataColumn[] keyColumn = new DataColumn[30];
keyColumn[intJ] = dtFilterdPK.Columns["Column name"];
Then try to add one by one row from your CSV to this DataTables.
Whenever your data will get Duplication based on Primary Key will have an error
Handle this error code in (TRY)..CATCH block and make this duplication error as per your logging requirement.
Avoid those error records importing in DataTable object.
Atlast import your CSV file into your table as BulkImport
Like:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(myConnection))
{
bulkCopy.DestinationTableName = "Your DB Table Name"; //Assign table name
bulkCopy.WriteToServer(dtToBeImport); //Write into Actual table.
}
Hope this will help you.
Will come directly to the question.
Have 2 parameter like filename and table name. The requirement is to upload the data from the excel sheet to the database table enter in the other parameter. This should be in run time. No hardcoding of field names and that program should be flexible enough to suite any table. Please help.
I can think of two possible approaches:
Dynamic code generation -- write a program which writes a program
Use dynamic type tools
For 1. try googling
For 2. see https://wiki.scn.sap.com/wiki/display/Snippets/Example+-+create+a+dynamic+internal+table - this wiki shows a way (not sure if it is overkill as it creates the type from scratch whereas any table in your SAP system is already a defined type in the Data Dictionary).
You can do easily reference a parameterised table in Open SQL e.g. MODIFY (p_tab) ...
Perhaps you could do a generic SPLIT of a line read in from file by the delimiter into a table of fields - you can then use ASSIGN COMPONENT to match the fields you have read in to the fields in your internal type.
If you are doing this I think a white list of allowed tables would be wise - and auth checks. Otherwise someone could upload SAP standard tables with no authorisation.
I'm very new to SSIS, but have this week been completely burried in turorials of how to use it, this has been very fruitful so far but now I'm stuck with what seems a more complicated task.
I have two source SQl tables, which are dynamically created using the TSQL pivot function - so the numebr of columns in each will vary.
I feed a list of criteria into a for each loop whic his set to use an ADO enumerator to go through this data set and has a script task to set the string variable SQL_DOWNLOAD_STRING to point to one of the two poential tables using an IF statement. TABLEA or TABLEB.
I tried using the standard OLE DB source in the data flow, but the associated meta data was generated based on TABLEA at the time of build. I then set the SQL statement to be based on the variable SQL_DOWNLOAD_STRING and when I execute, the meta data of the source has changed as obviously it is a differnt table. That said even if itwas the same table there is no guarentee that the columns would be the same.
So my question is - how do I create the OLE DB Source dynamically so that it reconfigures to the new structure of the table?
Then I'm trying to export the data to an Excel workbook (which again I can do for a static source) but in this instance the mapping would need to change too & I imagine I'll need to rebuild the destination table to match the source from above.
If I can get a working example - then finishing the project should be pretty staright forward from there - examples for this sort of thing seem pretty limited though :(
Please help! (PS I'm working in VB)
Thanks
I am writing a utility for myself that needs to be able to access a pair of tables in a SQL database. I used the SqlEntityConnection Type Provider and was rewarded with the data I needed from the table as easy to use entities.
One thing I noticed though was that startup and compiling of the project increased by quite a lot. I suspect this is because the database has over a hundred tables and it's compiling and getting data from all of them as opposed to just the two I need. Is there a way to restrict the EntityTypeProvider to only referencing the needed tables in the schema?
type private EntityConnection = SqlEntityConnection<ConnectionString="Server=Server;Initial Catalog=Database;Integrated Security=SSPI;MultipleActiveResultSets=true", Pluralize = true>
let private context = EntityConnection.GetDataContext()
I have not tried this myself, but I think you could add a new "ADO.NET Entity Data Model" (edmx) file to your project, let it generate from your existing database, and then delete from the model every table you don't want accessible to your code.
The EDMX designer will generate a *.csdl file that you can then reference from the LocalSchemaFile parameter of SqlEntityConnection. You'd use this parameter instead of ConnectionString.
The end result is that the entity provider would not automatically pick up changes to your database, but compilation times will go down, and only the tables you care about would be visible to your code.
I have got a backup of a live database (A copy of an ACCDB format Access database) in which I've worked, added new fields to existing tables and whole new tables.
How do I get these changes and apply that fast in the running database?
In MS SQL Server, I'd right-click > Script Table As > Alter To, save the query and run it wherever I desire, is there an as easy way as that to do it in an Access Database ?
Details:
It's an ACCDB MS-Access database created on Access 2007, copied and edited in Access 2007, in which I need to get some "alter" scripts to run on the other database so that it has all the new columns and tables I've created on my copy.
For new tables, just import them from one database into the other. In the "External Data" section of the ribbon, choose the Access icon above "Import". That choice starts an import wizard to allow you to select which objects you want imported. You will have a choice to import just the table structure, or both structure and data.
Remou is right that you can use DDL ALTER TABLE statements to add new columns. However, DDL might not support every feature you want for your new columns. And if you want not just the empty columns added, but also also any data from those new columns, you will probably need to run UPDATE statements to get it into your new columns.
As far as "Script Table As", see if OmBelt's Export Table to SQL tool for MS Access can do what you want.
Edit: Allen Browne has sample ALTER TABLE statements. See CreateFieldDDL and the following one, CreateFieldDDL2.
You can run DDL in Access. I think it would be easiest to run the SQL with VBA, in this case.
There is a product called DbWeigher that can compare Access database schemas and synchronize them. You can get a free trial (30 days). DbWeigher will write a script of all schema differences and write it out as DDL. The script is thorough and includes relationships, indexes, validation rules, allow zero length, etc.
A free tool from the same developer, DBWConsole, will let you execute a DDL script against any Access database. If you wrote your own DDL scripts this would be an easy way to apply the changes to your live database. It even handles some DDL that I don't know how to process in VBA (so it must be magic). DBWConsole is included if you downloaded the trial version of DBWeigher. Be aware that you can't make schema changes to a table in a shared Access database if anyone has the table open.
DbWeigher creates a script of all differences between the two files. It can be a lot to manually parse through if you just want a few of the changes. I built a parser for DbWeigher script files so they could be filtered by table, to extract just the parts I wanted. I contacted the DbWeigher author about it but never heard back. It's safe to say that I have no affiliation with this developer.