ERROR HANDLING and Loading into target Table using SQL - sql

I have a scenario Today where I am stucked Up little bit. I have data like this below in my Source Oracle Source Table(Sample Data)
ID,NAME,SALARY,BIRTHDAY
1,ABHIJIT,2000,17/12/1990
2,ROHIT,-2000,13/11/1988
3,MOHIT,500,2075-575-43
Now Salary in the 2nd Row is negetive and BIRTHDAY in the 3rd Row is an Invalid Format(Valid is MM/dd/yyyy). Both 2nd and 3rd Row should go to INVALID_EMPLOYEE and 1st record should go to VALID_EMPLOYEE. In the Source File Date Format is coming as dd/mm/yyyy. Which I have to convert to MM/dd/yyyy format and also have to check whether Date Format incoming in Source File is dd/mm/yyyy or not. Salary should not be less than 0. Source Table All columns are in String and in Target Table ID is Integer,NAME as VARCHAR2(255), SALARY is NUMBER and BIRTHDAY is DATE.
I have handled all this in my Project ETL Tool. So I am trying to push all this in Query to improve performance.Any help is much appreciated

Although the error records should be ideally handled by the process that's doing the insert, in many cases this may not be possible to do.
Instead what you can do now is to set up constraints on the table for the Salary column. The date checking can be implemented by simply having the column of the date datatype. Then create a error logging table to log the errors.
Now we need a table to log your invalid records. For this you have two options.
You can have the INVALID_EMPLOYEES table itself as the error table. But that would mean having additional columns in the table (ORA_ERR_MESG$ and ORA_ERR_TAG$).
If having additional columns is not an option then you can create a dedicated error table and then move the records from this table to your INVALID_EMPLOYEES table. You will have to write some code that runs after your insert code to move the records.
See the DBMS_ERRLOG package to see how to create the error logging tables.
Once you have your error logging tables created you have to instruct your INSERT code to log the errors into your error logging table. This will ensure that all the valid records get inserted in your employees table and the invalid records go to your error table with a column that stores the error that was encountered while inserting that particular record. So your insert code would look something like
INSERT INTO VALID_EMPLOYEES
SELECT ID,
NAME,
SALARY,
BIRTHDAY
FROM my_source_table
LOG ERRORS INTO INVALID_EMPLOYEES ('Oops this guy failed');
In this case 'Oops this guy failed' would go into the ORA_ERR_TAG$ column of the error table. You can use this tag column to identify different insert statements that tried to insert an invalid record.
Hope that helps!

Related

ORA-01722! Why can't I INSERT a NUMBER into a NUMBER data field without getting this error?

Can someone tell me what is going on here?
So I have a simple table location and it only has two columns; one is a number and the other varchar2.
I'm simply trying to insert some data into the locations table so I can get cracking with the other larger datasets but keep getting this damn error every time.
Error starting at line : 7 in command -
INSERT INTO location
VALUES (1, 'Head Office')
Error report -
ORA-01722: invalid number
NOTE: Before down-voting, YES - I have seen others posting about this but is usually for something less obvious than my situation where they are trying to enter a string into a number field or a number into a string field!
In my case however, the data in the INSERT statement is a number AND the data type is also NUMBER!
DATA STRUCTURE:
CREATE TABLE location(
locID NUMBER(4) NOT NULL,
locName VARCHAR2(100) NOT NULL
);
INSERT STATEMENT:
INSERT INTO location
VALUES (1, 'Head Office');
The error code can be seen above there where I first mentioned it.
Thanks in advance.
P.S. It may be worth mentioning that the ID field in 'location' table is being used as a FOREIGN KEY in a separate table 'employees'. I have however, checked that the data types matched!
EDIT #1: I'm using ORACLE SQL Developer
Always include the columns when doing an insert:
INSERT INTO location (locId, locname)
VALUES (1, 'Head Office');
From your description of the problem, this should not actually fix it. This is just a good habit.
The above is correct SQL for your table. If the error continues to happen it is probably coming from a trigger on the table.
Think like its stupid, you are getting number error from "head office" not from 1. Actually you are trying to insert string into number.
If you dont want to write column names to insert you should totally define all values in insert in place as located in table. I assume your table structure is
locId|locNumber
So your insert should be like below
insert into table values (1,'head office')
I hope you understand shortcut logic

Datastage Job terminating due to the following error

I'm running a data stage job, Input through DB2 and output to DB2. Input side has a query containing joins and functions.
I'm getting the following warning message;
TRN_HEALTH_INSURANCE_DETAIL,
2: STATEMENT
INSERT
INTO
HEALTH_INSURANCE_DETAIL
(
RISK_DETAIL_ID,
RISK_COVER_ID,
RD_POLICY_SYSTEM_NO,
RD_POLICY_END_NO_IDX,
RD_POLICY_ID,
RD_LEVEL1_ID,
RD_SUM_INSURED_AMT_LC,
RD_PREMIUM_AMT_LC,
PREMIUM_AMOUNT_FC,
SUM_INSURED_AMT_FC,
RD_REC_TYPE,
RD_EFFECT_FROM_DT,
RD_EFFECT_TO_DT,
RD_END_EFFECT_FROM_DT,
SEX_MAS_CD,
MARITAL_STATUS_CD,
EMP_CATG,
NO_OF_DEPENDENTS,
EMP_AL_NO,
DOB,
EFF_DATE,
EFF_DATE2,
NAME,
RELATIONSHIP_CD_S,
RELATIONSHIP_CD,
DESIGNATION,
BRANCH,
BANK_ACCOUNT,
BANK_BRANCH_NAME,
PRE_EXISTING_AILMENT,
AUTHORITY_LETTER_NO,
AGE,
REGION,
CNIC,
CO_CODE,
EMP_LOCATION,
SUB_LOCATION,
CLH_SYSTEM_NO,
CTH_SYS_ID,
CTH_POL_SYS_ID,
CTH_END_NO_IDX,
CTH_END_SR_NO,
CTH_CATEGORY,
CLD_SYS_ID,
CLDH_SYS_ID,
CLD_COVER_CD,
CLD_END_IDX,
CLD_COVER_DESC,
CLD_CLM_TYPE_LIMIT,
CLD_CLM_REL,
CLD_CLM_AGE_FROM,
CLD_CLM_AGE_TO,
CLD_CLM_RB_LIMIT,
CLD_CATEGORY_LIMIT_FC,
CLD_CATEGORY_PREM_FC
)
VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) failed to run.
I cant see such records in my data. The data quality is good. Then what are these ????, I search a bit and found a suggestion to keep the array size and row count to 1, instead of default 2000. But still I'm getting the same warning.
There are a lot of errors followed by this warning; The next error is also interesting.
TRN_HEALTH_INSURANCE_DETAIL,2: SQLExecute reported: SQLSTATE = 23505: Native Error Code = -803: Msg = [IBM][CLI Driver][DB2/NT64] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "1" constrains table "DB2ADMIN.HEALTH_INSURANCE_DETAIL" from having duplicate values for the index key. SQLSTATE=23505 (CC_DB2DBStatement::executeInsert, file CC_DB2DBStatement.cpp, line 1,095)
I believe the errors are due to the first warning. Kindly help me out.
Regards, Nuh
Make a copy stage before the DB2 connector and put one link to the DB2 and the other to a dataset file to see the data in a data set. But the problem seems to be in the primary key you have a duplicate primary index or a duplicate unique index. It can be either in your data that you want to insert or maybe the table already have a record that you want to insert again

How to find the error value in sql table

I am inserting values from one table to another table. There are 100000 records in the table. But when I start to insert value from one table to another table there is a problem in any row in my data. I don't know where exactly is that. So how can I know that in which row value is error because the insert statement not complete? After error this stop nothing insert in table.
This is simple statement that I use:
INSERT INTO Person (FirstName, LastName,Email)
SELECT FirstName, LastName, Email
FROM Person.Contact
Without any additional information on the constraints imposed on the target table a very general, slow solution would be
a cursor loop over the selected rows
insert any row within try/catch
on exception output row and error message
For a specific answer to your problem provide more information.
An alternative method is to use SSIS: here you can provide a separate channel for the erroneous rows. This channel can be lead to a separate table collecting all rows causing an error.
quesion is hard to understand, but the usual suspects are:
Attempting to insert Null into a non-nullable column
Foreign Key violations
Constraint violations.
Attempting to insert wrong datatype.
Please edit you question to add error message.
I guess Person is the name of your schema, so maybe you mean
INSERT INTO Person.Person (FirstName, LastName,Email)
SELECT FirstName, LastName, Email
FROM Person.Contact

Getting INSERT errors when I do UPDATE?

At work we have a SQL Server database. I don't know the db that well. I have created a new column in the table for some new functionality....straight away I have started seeing errors
My statement was this:
ALTER TABLE users
ADD locked varchar(50) NULL
GO
The error is:
Insert Error: Column name or number of supplied values does not match table definition
I have read that the error message appears when during an INSERT operation either the number of supplied column names or the number of supplied values does not match the table definition.
But I have checked so many times and i have changed the PHP code to include this columns data yet I still receive the error.
I have run the SQL query directly on the db and still get the error.
Funny enough the query which gets the error is an Update.
UPDATE "users"
SET "users"."date_last_login" = GETDATE()
WHERE id = 1
Have you considered it could be a trigger causing it? 
This is the error message you would get.
If its an Update action causing it check trigger actions that Updates on that table run.
Do it with:
#sp_helptrigger Users, 'UPDATE';
This will show triggers occuring with ‘update’ actions.
If there is a trigger, grab the triggers name and run the below (but replace TriggerNameHere with real trigger):
#sp_helptext TriggerNameHere;
This will give you any SQL that the trigger runs and could be the INSERT the error message is referring to.
Hope this helps
Aside from TRIGGERS,
the reason for that is because you are using implicit type of INSERT statement. Let's say your previous number of columns on the table is 3. You have this syntax of INSERT statement,
INSERT INTO tableName VALUES ('val1','val2','val3')
which executes normally fine. But then you have altered the table to add another column. All of your INSERT queries are inserting only three values on the table which doesn't matches to the total number of columns.
In order to fix the problem, you have to update all INSERT statements to insert 4 values on the table,
INSERT INTO tableName VALUES ('val1','val2','val3', 'val4')
and it will normally work fine.
I'll advise you to use the EXPLICIT type of INSERT wherein you have to specify the columns you want to insert values with. Eg,
INSERT INTO tableName (col1, col2, col3) VALUES ('val1','val2','val3')
in this ways, even if you have altered your tables by adding additional columns, your INSERT statement won't be affected unless the column doesn't have a default value and which is non-nullable.

can't insert records in new table from existing table in sql server 2005

I need to insert records into a new table from an existing table. I used the following query to do this:
Insert into Newtable
Select * from Oldtable where date1 = #date
This query works most of the time, but in one scenario I get 10 million records to be inserted for the date1 value. In this case I'm getting the following error message:
Error : The transaction log for database "tempDB" is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases
Should I break the query into parts and insert them sequentially, or is there a way to do this with the current query?
This is, perhaps, a distasteful suggestion. But, you can try exporting the data to a file and then inserting using bulk-insert, with database logging set to SIMPLE or BULK-LOGGED.
More information is at http://msdn.microsoft.com/en-us/library/ms190422.aspx.