I used oracle 10g and I have this file.ctl
OPTIONS (SKIP=1)
LOAD DATA
INFILE '/home/gxs/segmentation/sqlloader/datos.csv'
APPEND INTO TABLE test
(id "s_test.nextval",
name char(10) TERMINATED BY ',' ,
tel char(20) TERMINATED BY ',' ,
apellido char(10) TERMINATED BY ',' )
My csv file es
name,tel,apellido
daniel,12345,buitrago
cesar,98765,san
alex,4556,ova
but when see the table the name haven't the firs caracter:
id name apellido tel
1 aniel buitrago 12345
2 esar san 98765
3 lex ova 4556
What do?
As per the documentation, you need to use the EXPRESSION keyword to show that the value is coming purely from the specified expression and is not dependent on anything in the data file:
OPTIONS (SKIP=1)
LOAD DATA
INFILE '/home/gxs/segmentation/sqlloader/datos.csv'
APPEND INTO TABLE test
(id EXPRESSION "s_test.nextval",
name char(10) TERMINATED BY ',' ,
tel char(20) TERMINATED BY ',' ,
apellido char(10) TERMINATED BY ',' )
... which inserts:
ID NAME TEL APELLIDO
---------- ---------- ---------- ----------
1 daniel 12345 buitrago
2 cesar 98765 san
3 alex 4556 ova
At the moment it's assuming ID is a field in your data file, and since you haven't specified a data type it's defaulting to char with size 1, which is consuming the first character of your real first field.
Related
I am loading data to hive table which contains comma in the data itself.
input file:emp.csv
101,deepak,kumar,das
102,sumita,kumari,das
103,rajesh kumar das
output :
id name
101 deepak kumar das
102 sumita kumari das
103 rajesh kumar das
When I created the below hive table and loaded the data, data is not coming properly:
create external table hive_test(
id int, name string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION '/hive_demo';
load data local inpath '/home/cloudera/hadoop/hive_demo/emp.csv' overwrite into table hive_test;
hive> select * from hive_test;
101 deepak
102 sumita
103 rajesh kumar das
So I created below table, but it is giving error.
create external table hive_test1(
id int,
name string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES(
"separatorChar" = ",",
"quoteChar" = "'",
"escapeChar" = "\,")
STORED AS TEXTFILE
LOCATION '/hive_demo';
load data local inpath '/home/cloudera/hadoop/hive_demo/emp.csv' overwrite into table hive_test1;
select * from hive_test1;
Failed with exception
java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException:
java.lang.UnsupportedOperationException: The separator, quote, and escape characters must be different!
How can I load the data to the Hive table?
Providing solution below assuming:
You always need to extract only 2 cols from the csv.
The first col is a numeric, the 2nd col extends till the end of the line after the first ',' character.
You need to replace any ',' character in name column with spaces.
Use RegexSerDe to define table and load
create external table hive_test(
id int, name string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "^(\d+),(.*)$" -- 2 regex groups as per assumption
)
STORED AS TEXTFILE;
LOCATION '/path/to/table';
LOAD data local inpath '/path/to/local/csv' overwrite into table hive_test;
Replace the ',' in name column by space
create table hive_test1 as
select id, regexp_replace(name, ',', ' ') as name
from hive_test;
Then, on select * from hive_test1, you obtain below:
101 deepak kumar das
102 sumita kumari das
103 rajesh kumar das
How can I have this input in the database like 15-001 where 15 is the last two digit of the year (based on datetimeNow) and 001 is my paper number?
You can keep two fields:
1. ID field - auto incremented on each record inserted
2. Varchar ID field - computed column
Try creating table like this:
CREATE TABLE PaperTable
(
PaperID int IDENTITY (1,1) NOT NULL
, PageNumber varchar(100)
, PaperAlphaID AS Cast(Right(Year(getDate()),2) as varchar(2)) +'-'+ PageNumber
);
Result I got when I added "001" and "002" in as my paper number:
PaperID PageNumber PaperAlphaID
------- ---------- ------------
1 001 15-001
2 002 15-002
You may use PaperID if you Paper number to be autogenerated. You will just then need to type cast and concate:
PaperAlphaID AS Cast(Right(Year(getDate()),2) as varchar(2)) +'-'+ Cast(PaperID as varchar(50))
I am trying to import data from txt file to table. The TXT file is having 5 records.
'ext.txt' is my file .'IMPORT' is a directory.
Records are
7499,ALLEN,SALESMAN,30
7521,WARD,SALESMAN,30
7566,JONES,MANAGER,20
7654,MARTIN,SALESMAN,30
I tried below query but its only inserts 3rd record to the external table.
Can anyone provide me the reason for this ans solution for insert all rows.
create table ext_tab (
empno CHAR(4),
ename CHAR(20),
job1 CHAR(20),
deptno CHAR(2)
)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY IMPORT
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
BADFILE IMPORT:'test.bad'
LOGFILE IMPORT:'test.log'
FIELDS TERMINATED BY ',' (
empno char(4) ,
ename char(4),
job1 CHAR(20),
deptno CHAR(2)
)
)
LOCATION (import:'ext.txt')
)
PARALLEL 5
REJECT LIMIT UNLIMITED;
This will work for your given test data
CREATE TABLE ext_tab (
empno VARCHAR(4),
ename VARCHAR(20),
job1 VARCHAR(20),
deptno VARCHAR(2)
)
ORGANIZATION EXTERNAL (
DEFAULT DIRECTORY import
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
)
LOCATION ('ext.txt')
);
providing that you have set up the import directory correctly.
Tested on 11.2.0.4
As for the reason it was given in the comments above. Always check the .bad and .log files first (created in the directory you put the file in). These are very helpful at telling you why rows are rejected.
I expect you have errors like this in the log:-
KUP-04021: field formatting error for field ENAME
KUP-04026: field too long for datatype
KUP-04101: record 1 rejected in file /import_dir/ext.txt
KUP-04021: field formatting error for field ENAME
KUP-04026: field too long for datatype
KUP-04101: record 3 rejected in file /import_dir/ext.txt
KUP-04021: field formatting error for field ENAME
KUP-04026: field too long for datatype
KUP-04101: record 4 rejected in file /import_dir/ext.txt
and that only WARD was imported because only his name fits into CHAR(4).
Hi I'm trying to load some data in to an Oracle Table I created
Here's the table I created in Vivek schema
Desc STAR
Name NULL TYPE
-----------------------------------------------------
STAR_ID Not Null Number(4)
FIRST_NAME Varchar2(30)
LAST_NAME Varchar2(30)
DOB Date
SEX Char(1)
Nationality Varchar2(40)
Alive Char(1)
Following is the data in the STAR5.CSV file I am trying to upload using SQL Loader
10,NASEERUDDIN,SHAH,M,INDIAN,Y
11,DIMPLE,KAPADIA,F,INDIAN,Y
The control file is as follows
load data
infile '/home/oracle/host/Vivek12/STAR_DATA5.csv'
append
into table vivek.STAR
fields terminated by ","
( STAR_ID,FIRST_NAME,LAST_NAME,SEX,NATIONALITY,ALIVE )
When I run the SQL Loader with the following command
$ sqlldr vivek/password
control = /home/oracle/sqlldr_add_new.ct1
I get the message:
Commit point reached - logical record count 2
However the data is not loaded and the are put in the file STAR5.bad
Any idea why the data isn't getting loaded?
You most likely have either an "invisible" character at the end of your line. Perhaps you're executing this on Linux and the file was created on Windows so you've got an extra carriage return - Linux only uses line-feed as the line terminator.
Change your ctl file to remove terminating whitespace:
load data
infile '/home/oracle/host/Vivek12/STAR_DATA5.csv'
append
into table vivek.STAR
fields terminated by ","
( STAR_ID
, FIRST_NAME
, LAST_NAME
, SEX
, NATIONALITY
, ALIVE terminated by whitespace
)
If this doesn't work, then you're going to need to work out what characters are there and replace them.
hi friends i have tried to load data from flat file to external file using oralce_loader access driver my code is
create table test_ext (
id_1 varchar(35),
emp_name varchar(25),
e_mail varchar(25))
organization external (
type oracle_loader default directory new_dir access parameters
( records delimited by newline fields(
id_1 char(30),
e_name char(25),
mail char(25)))
location('test.csv')) reject limit unlimited;
and my data file:
"E.FIRST_NAME||','||E.EMAIL||','||MANAGER_ID"
-----------------------------------------------
"Jennifer,JWHALEN,101"
"Michael,MHARTSTE,100"
"Susan,SMAVRIS,101"
"Hermann,HBAER,101"
"Shelley,SHIGGINS,101"
"William,WGIETZ,205"
"Steven,SKING,"
"Neena,NKOCHHAR,100"
"Lex,LDEHAAN,100"
"Alexander,AHUNOLD,102"
"Bruce,BERNST,103"
"David,DAUSTIN,103"
"Valli,VPATABAL,103"
"Diana,DLORENTZ,103"
"Nancy,NGREENBE,101"
"Daniel,DFAVIET,108"
"John,JCHEN,108"
while run that above query i got
**ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-04043: table column not found in external source: EMP_NAME
29913. 00000 - "error in executing %s callout"
*Cause: The execution of the specified callout caused an error.
*Action: Examine the error messages take appropriate action.**
I tried so many things but I can't.
firstly your csv file looks wrong.
"Alexander,AHUNOLD,102"
remove all quotes, otherwise it will look like one field.
secondly your using the "fields" syntax suggesting you want fixed length csv file, but your dat file is comma seperated. so i think you want to fix your csv to remove the quotes and the two spurious header lines and change your table DDL to :
create table test_ext (
id_1 varchar(35),
emp_name varchar(25),
e_mail varchar(25))
organization external (
type oracle_loader default directory new_dir access parameters
(
records delimited by newline
fields terminated by ',' optionally enclosed by '"')
location('test.csv')) reject limit unlimited;
eg:
SQL> host cat test.csv
Jennifer,JWHALEN,101
Michael,MHARTSTE,100
Susan,SMAVRIS,101
Hermann,HBAER,101
Shelley,SHIGGINS,101
William,WGIETZ,205
Steven,SKING,
Neena,NKOCHHAR,100
Lex,LDEHAAN,100
Alexander,AHUNOLD,102
Bruce,BERNST,103
David,DAUSTIN,103
Valli,VPATABAL,103
Diana,DLORENTZ,103
Nancy,NGREENBE,101
Daniel,DFAVIET,108
John,JCHEN,108
SQL> create table test_ext (
2 id_1 varchar(35),
3 emp_name varchar(25),
4 e_mail varchar(25))
5 organization external (
6 type oracle_loader default directory new_dir access parameters
7 (
8 records delimited by newline
9 fields terminated by ',' optionally enclosed by '"')
10 location('test.csv')) reject limit unlimited;
Table created.
SQL> select * from test_ext;
ID_1 EMP_NAME E_MAIL
----------------------------------- ------------------------- -------------------------
Jennifer JWHALEN 101
Michael MHARTSTE 100
Susan SMAVRIS 101
Hermann HBAER 101
Shelley SHIGGINS 101
William WGIETZ 205
Neena NKOCHHAR 100
Lex LDEHAAN 100
...etc...