I was told that there is a way to do this but I have no idea. I am hoping that someone more educated than I can help me understand the answer...
There is a table that has been imported from an external source via SSIS. The destination in SSIS needs to be updated frequently from the external source. The users are complaining about performance problems in their queries.
How would you update this destination in SSIS to achieve these goals?
Anyone have a clue? I'm "dry"...
If your users are complaining about performance then it is not an SSIS issue. You need to look at what queries are running against the table. Make sure your table has a primary key and appropriate filters based on the columns used to sort and filter the data.
Can you give us a listing of the table definition?
i'll give u some advice then maybe can improve your ssis performance
Use SQL statement rather than instant dropdown interface when you import data from db to your SSIS (using SQL Statement u can import, filter, Convert, and Sort at once)
Import & Filter only Column you needed in SSIS
Some reference say, you can customize your default buffer settings (DefaultBufferSize & DefaultBufferMaxRows). this way will improve your bandwidth data process.
Related
I have a very large data set in GPDB from which I need to extract close to 3.5 million records. I use this for a flatfile which is then used to load to different tables. I use Talend, and do a select * from table using the tgreenpluminput component and feed that to a tfileoutputdelimited. However due to the very large volume of the file, I run out of memory while executing it on the Talend server.
I lack the permissions of a super user and unable to do a \copy to output it to a csv file. I think something like a do while or a tloop with more limited number of rows might work for me. But my table doesnt have any row_id or uid to distinguish the rows.
Please help me with suggestions how to solve this. Appreciate any ideas. Thanks!
If your requirement is to load data into different tables from one table, then you do not need to go for load into file and then from file to table.
There is a component named tGreenplumRow which allows you to write direct sql queries (DDL and DML queries) in it.
Below is a sample job,
If you notice, there are three insert statements inside this component. It will be executed one by one separated by semicolon.
I have a ETL project that I need to load data from some 50K Access .MDB databases in a folder to sql server. Problem with those 50K databases files is that they have different schemas and I need the ETL process to be able to identify the differences and respond correctly.
For example, in some of the .MDB files there are table A, B and C. However in some other tables there are only table A and B (Same table A and B as compared to the other tables, just table C is missing). I need to put a check on each OLE DB source to see what tables are there to achieve logic like IF table A exists, load table A, otherwise, bypass the load.
I've done my googling and searched SO but all the error handling or check methods I could find are for the execute SQL task or data conversion task. So if anyone could shed some light on solution to my above case, I would be deeply appreciated.
Thanks.
In a nutshell - SSIS assumes that metadata does not change.
However, with some tricks, this restriction can be reduced; below is the list of suggested tricks:
Test for existence of specific table (see example here How to use SQL to query the metadata in Microsoft Office Access? Like SQL Server's sys.tables, sys.columns etc) and based on the result - do conditional execution of the following tasks.
All SQL requests to MS Access tables should have DelayValidation property set to True. Reason - postpone SQL command validation from package start to specific task execution. Some tasks (for missing tables) will not be executed; thus, it will not be validated and will not fire validation error.
I have about 30 files that will need to be exported from one sql table. What I have now is creating a sql table for each file, then exporting the contents of that individual table. This works fine but I really didn't want to have 30 tables on the server. Is there a way to export from one table using 30 different sql queries?
Thanks in advance
This really depends on your data need and how complicated the export is but... generally people do not create an export table per export. Data Transform Task's source can be a table (which you are currently using) or a stored procedure or a view.
I would need to know more about your structure to advise more accurately but... one table per export is definitely not the best solution.
I would use a parameterized stored procedure if possible.
Say the source data comes in excel format, below is how I import the data.
Converting to csv format via MS Excel
Roughly find bad rows/columns by inspecting
backup the table that needs to be updated in SQL Query Analyzer
truncate the table (may need to drop foreign key constraint as well)
import data from the revised csv file in SQL Server Enterprise Manager
If there's an error like duplicate columns, I need to check the original csv and remove them
I was wondering how to make this procedure more effecient in every step? I have some idea but not complete.
For step 2&6, using scripts that can check automatically and print out all error row/column data. So it's easier to remove all errors once.
For step 3&5, is there any way to automatically update the table without manually go through the importing steps?
Could the community advise, please? Thanks.
I believe in SQL 2000 you still have DTS (Data Transformation Services) part of Enterprise Manager. Using that you should be able to create a workflow that does all of these steps in sequence. I believe it can actually natively import Excel as well. You can run everything from SQL queries to VBScript so there's pretty much nothing you can't do.
I used to use it for these kind of bucket brigade jobs all the time.
my application has the following procedure.
a database of products (20,000 rows) exists.
our client has 'import' feature where he imports an excel file.
this is implemented by deleting all products table rows, then doing the import - which is a long thing since we programmatically performing calculations on the data.
the obvious problem is if the 'import' action fails (IO stuff), they now have none/partial/curropt data in the products table.
We wish that if the 'import' operation fails, the original data remains.
this is ASP.NET application, written in C#, using SQL Server 2005 and using XSD which we created through the VS2005 design tools.
Start transaction
delete data from table
Insert new data
if no problem happens, commit changes
I would import the data onto a table with the same structure as your products table, and then replace the data on your products table once you're happy the import has been successful.
This will mean that users can carry on using the system while the import is underway, as well as minimizing the down time, while you update the products table.
Using a transaction would be the obvious choice here.
I guess the first thing I would ask is do you really need to clear the entire table? Is there a timestamp or something you could use to limit the amount of data that needs to be refreshed? Could you re-work the logic to use updates instead of all the deletes and inserts? That way your transactions will be smaller.
-Dan
I would go with the transaction approach as outlined above. But the only problem I can see is that you might end up locking the whole table for the entire period the import process is taking place. you might need to think about it. The seperate table approach can be one of the solutions.