How I check existing record and avoid redundant data - sql

I have two tables, both are situated at different server and belongs to different schema.
Table 1 : SALARY_DETAIL_REPORT_[monthYearSuffix]-Same Replica of Table 2.
Table 2: XXMPCD_SALARY_DETAIL_TABLE-Contains employee's salary related data,each employee having multiple pay code like HOUSE RENT ALLOW,DEARNESS ALLOW, NPS-Company Contri(Earning), BASIC PAY, NET EARNING, GROSS DEDUCTION, GROSS EARNING, GROSS DEDUCTION, NET EARNING, BASIC PAY, GRADE PAY.
So Employee code repeated multiple times and hence we are not able to maintain primary key or unique index.
Suppose 1000 records pushed in TABLE 2 which I need to copy exactly in my TABLE 1. This is handle by my SPRING service class a GUI is available in which we simply click on migrate button and services running in background fetch the data from TABLE 2 and insert the same in the TABLE 1.
There is two column inside TABLE 2
1.PICK_DATE
2.IS_DATA_PICKED
If both of these data contain null that simply means that data is not migrated into our TABLE 1, we updated the same columns after successful migration by acknowledging, so the next time the data will not be available for migration.
PROBLEM
Now suppose we migrated the 1000 records from TABLE 2 to TABLE 1.
What I done, I go back to my SQL DEVELOPER, choose 3 random records and set their PICK_DATE & IS_DATA_PICKED to NULL.
Now I migrated the same another time and those 3 records inserted again.
That means replication of 3 records, 1000 records becomes 1003.
Now what I want to check:
If same data existed, than record should be updated not inserted, means overwrite.
TABLE 1:SALARY_DETAIL_REPORT_092018
SALARY_REPORT_ID
EMP_NAME
EMP_CODE
PAY_CODE
PAY_CODE_NAME
AMOUNT
PAY_MODE
PAY_CODE_DESC
YYYYMM
REMARK
EMP_ID
PRAN_NUMBER
PF_NUMBER
PRAN_NO
ATTOFF_EMPCODE
REFERENCE_ID
**
TABLE 2:XXMPCD_SALARY_DETAIL_TABLE
EMP_NAME
EMP_CODE
PAY_CODE
AMOUNT
PAY_MODE
PAY_CODE_NAME
YYYYMM
REMARK
PUSH_DATE
PICK_DATE
IS_DATA_PICKED
ERROR_MESG
REFERENCE_ID
PRAYAS_ERP_ORG_ID
ERP_ORG_ID
PF_NUM
PRAN_NO
VERIFIED_BY

Related

Populating history table (SCD type 4) - stored procedure

I am trying to build a stored procedure that would deal with history (SCD4) and populate history table based on that.
For example I have tables Customer and Customer_hist (same columns + ValidFrom, ValidTo) that are connected via IDCust (IDCustHist).
Each time new customer is inserted into the Customer table (after the procedure is ran), the customer should also be in also in Customer_hist table.
Each time time customer is changed in Customer table (after running procedure), a new record for this customer should appear in Customer_hist (and for the old one ValidTo should be updated).
If there are no changes to record in Customer table, nothing should change in Customer_hist
Table Customer:
id
name
1
Martin
2
Danny
Table CustomerHist:
id
name
validFrom
validTo
1
Martyn
05:05:2022
20:09:2022
1
Martin
21:09:2022
31:12:2999
2
Danny
05:05:2022
31:12:2999

Add data to new column based on a different table in SQL Server

I need to add values to a new table but I have a problem with one of the columns. The table is named Staff which shows staff information and whether they are full time or part time.
The Staff table includes their staff ID, First Name, Last Name, Department, Pay Grade, FullPartTime as a Bit where 1 is full time and 0 is part time.
For the first 5 columns I'm pulling the information from a table called Employees, the FullTimePartTime column needs to come from a table called WorkPattern. This table has a column that shows total weekly working minutes (WkMins).
If the Employee has total work minutes of 2220, then they are full time (anything from 37 hours to 45 hours is full time) anything below is part time.
How do I insert 1 or 0 into the FullPartTime column in this case?
I've tried several ways but didn't keep copies of them. I tried a case but couldn't get it to work.

Need update eligible data from one table to another

I have 2 tables. One is stage XYZ which contains raw data and the other is main table SKY in which most of the data is loaded as it is from stage.
The stage table is a truncate load table.
I want to check out records between these two which are not same??? In other words , I want to know which all attribute/column value is valid for a change from XYZ to SKY??
Be sure there are many columns in the SKY which is not available in XYZ.
XYZ has around 150 columns in total
And main table has 165( 15 are populated using some Informatica transformation from XYZ values, which i am not worried about)
In my opinion, The table XYZ should have a Auto Increment ID and the table Main will have the reference of that Auto Increment ID value. Once this structure is ready, informatica should always pick new records based on the value in Auto Increment ID column.
The select statement could be-
SELECT * FROM XYZ
WHERE Auto_ID > (SELECT MAX(Auto_ID) FROM Main)
Then you can apply further transformation on selected records for your purpose.

Looping thru one table and assign ID to another

I have table of Clients (id, clientName), and table of tasks (id, taskName).
Lets say that client table have about 200 rows and tasks about 700 000 rows.
I need to assign every client about 300 tasks every day (with Job).
My idea is to make third table ClientXrefTask (idxref, fkClient, fkTask, date), and every night to fill that table with data
(I need to have assign history so I choose third table)
Is there a better way to do this other then cursos (iterate thru clients, and insert in ClientXref table) ?
You can add fkClient and date to the tasks table if those tasks are not getting changed once assigned.

How can I separate a one-column table into a table with multiple columns?

I came across with a text file that I need to import to SQL Server 2005. The data looks like this:
A1A00001
A2Name
A3Address
A4Credit
A5ModeOfPayment
D1Invoice 1 Amount
D1Invoice 2 Amount
D1Invoice N Amount (number of invoice varies per entry)
D2Total Amount Amount
S1Total Outstanding Amount
S1
A1A00002
A2Name
A3Address
A4Credit
A5ModeOfPayment
D1Invoice 1 Amount
D1Invoice 2 Amount
D1Invoice N Amount (number of invoice varies per entry)
D2Total Amount Amount
S1Total Outstanding Amount
S1
A1A00003
A2Name
A3Address
A4Credit
A5ModeOfPayment
D1Invoice 1 Amount
D1Invoice 2 Amount
D1Invoice N Amount (number of invoice varies per entry)
D2Total Amount Amount
S1Total Outstanding Amount
S1
As you can see, there are no delimiters in the data, that's why I only managed to have a table with one column containing the information above.
I need your help on how to capture one entry from A1 to S1, put it into separate columns, then proceed to the next occurrence of A1 and S1 and so on.
Like A1A00001 A2Name A3Address A4Credit A5ModeOfPayment D1Invoice 1 Amount D1Invoice.. etc
Thanks in advance!
This is a classic example of a file I would send back to the provider and tell them to send it in an appropriate format.
Unfortunately that isn't always an option. In the past how I have handled such is to add a recordidentifer column to the one column table and then populate it to keep all the records together that should be together. I also would add an Identity column at the time the records are inserted so youhave something to order on.
Then you populate the record identifier probably using a cursor or loop to add the same record identifier for each group of related records.
Now create the normalized staging tables you actually need. Populate them with SQl code (which is possible now that you have a recordidentifier). Then popluate your real tables form these nomalized staging tables.
If you have to do this in sql then use cursor. But best would be to parse and insert data to database with some application.
With cursor it would look like this (pseudocode):
If A1 insert previous row if exist
Start prepering next row
If this was like an initial load and you expect to keep the data and maybe add/modify it I would split it into two tables. Without seeing actual data something like this might work.
Obviously I just made up data types and column lengths, adjust as necessary.
Group A1 through A5, D2 and S1 into one table. Actually D2Total Amount could probably not be stored in a table but derived from invoice amounts. I included it in the table just for the example.
CREATE TABLE baseInfo (
A1 INT NOT NULL,
Name VARCHAR(25) NOT NULL
Address VARCHAR(55),
Credit VARCHAR(12),
ModeOfPayment VARCHAR(12)
TotalAmount MONEY,
OutstandingAmount MONEY
CONSTRAINT [PK_basinfoA1] PRIMARY KEY CLUSTERED (A1)
)
For D1, the invoice amounts, just a two column table. A1 to relate back to the account ID in baseInfo and the amount of the invoice.
CREATE TABLE invoice (
A1 INT NOT NULL,
invoiceAmount MONEY
)
ALTER TABLE invoice WITH CHECK ADD CONSTRAINT FK_invoice_base FOREIGN KEY (A1) REFERENCES baseInfo (A1)
To get the data into the two tables you could use TSQL but personally I’d go back to the original text file and use Powershell to parse the text and build SQL inserts.