I want to load flat file data into SQL table on daily basic.But my table hold on 1 week data. After 7 days previous data will deleted from my table and new data will append.
Thanks
Is your flat file delimited? Commas? Tabs? How many fields are there? what types are they?
If you only want one weeks worth of data in the table at any one time, you will need to delete after you do the daily import. If you import daily but only delete once a week you will have almost two weeks worth of data in the table.
What have you tried already?
before/after you run the import, delete any data that is more than 6 days old
Related
I'm using pgAdmin in PostrgreSQL to work.
There are 10 .csv files with 10 different tables on my computer. All 10 tables represent the same type of data, it's just divided into 10 different months. Basically I need one table with all the months. But first I was thinking about creating 10 tables and import 10 .csv files and then combine them together into 1 single table.
I perform the following algorithm to create 1 table: create the table, create the column names, set the data types in each column, import the .csv file into the table. Then I repeat the same operation 9 times for the rest of the .csv files. Is there any way to upload all 10 files with one step? Thank you
I have some user records in xlsx file and I am storing those records in SQL Database by applying some groupby-transform-Sum() functionality of pandas if there multiple instances for particular combination monthly basis (Exa. Id-Location-Designation-Day-PolicySold)
Until now all of the past and newly added data used to maintained in xlsx file only, but moving on only recent 3 months data will be available and I need to store this new data into SQL Db ensuring past data which is already present in SQL DB( for several months,years) intact and ensuring no duplicate entries .
Can anyone suggest me efficient approach to handle this?
My current approach is :
1. Read the past data from SQL table before performing new write operation.
2. Read new data from xlsx.
3. Merge both
4. Apply Groupby.transformation with Sum() to convert daily to monthly data
I have a 4x4 table in SQL with 20 rows. I want to split this data into four pages. Page 1 has the first 10 rows of the first column, Page 2 has the first 10 rows of the second column, etc.
After every four pages, this patern repeats showing the next 10 rows from the first column, etc. How can I arrange this?
I could arrange the data of the 4x4 table into another temporary table with just one column in its schema. Then I could read a single column of this table into my report. But can I instead do this directly without an intermediary table?
The intermediary table sounds like the best solution to me. I'd just write a custom SQL command in Crystal's Database Expert to arrange the data as you see fit.
You could in theory pull this off with repeating subreports in some manner of repeating header, but it would be much less work to have SQL properly format the incoming data for you.
I am currently entering data into a SQL Server database using SSIS. The plan is for it to do this each week but the day that it happens may differ depending on when the data will be pushed through.
I use SSIS to grab data from an Excel worksheet and enter each row into the database (about 150 rows per week). The only common denominator is the date between all the rows. I want to add a date to each of the rows on the day that it gets pushed through. Because the push date may differ I can't use the current date I want to use a week from the previous date entered for that row.
But because there are about 150 rows I don't know how to achieve this. It would be nice if I could set this up in SQL Server where every time a new set of rows are entered it adds 7 days from the previous set of rows. But I would also be happy to do this in SSIS.
Does anyone have any clue how to achieve this? Alternatively, I don't mind doing this in C# either.
Here's one way to do what you want:
Create a column for tracking the data entry date in your target table.
Add an Execute SQL Task before the Data Flow Task. This task will retrieve the latest data entry date + 7 days. The query should be something like:
select dateadd(day,7,max(trackdate)) from targettable
Assign the SQL result to a package variable.
Add a Derived Column Transformation between your Source and Destination components in the Data Flow Task. Create a dummy column to hold the tracking date and assign the variable to it.
When you map the Excel to table in a Data Flow task, map the dummy column created earlier to the tracking date column. Now when you write the data to DB, your tracking column will have the desired date.
Derived Column Transformation
I'm hoping someone might be able to help me out with this one - I have 24 files in CSV format, they all have the same layout and need to be joined onto some pre-existing data. Each file has a single column that needs to be joined onto the rest of the data, but those columns all have the same names in the original files. I need the columns automatically renamed to the filename as part of the join.
The final name of the column needs to be: Filename - data from another column.
My current approach is to use a foreach container and use the variable generated by the container to name the column, but there's nowhere I can input that value in the join, and even if I did, it'd mess up the output mappings, because the column names would be different.
Does anyone have any thoughts about how to get around these issues? Whoever has an idea will be saving my neck!
EDIT In case some more detail helps with this... SSIS version is 2008 and there are only a few hundred rows per file. It's basically a one time task to collect a full billing history from several bills which are issued monthly.
The source data has three columns, the product number, the product type and the cost.
The destination needs to have 24*3 columns, each of which has a monthly cost for a given product category. There are three product categories, and 24 bills (in seperate files) hence 24*3.
So hopefully I'm being a bit clearer - all I really need to know how to do, is to change the name of a column using a variable passed in from the foreach file container.
I think the easiest is to create a tmp database (aka staging db)
to load data from xls file to it and to define stored procedures where you can pass paramas (ex file names etc) and to build your won logic ...
Cheers Mario