i have problem with loading rows from a file, the point is that when im using External table like this
create table table_name
(
id VARCHAR2(60)
)
organization external
(
type ORACLE_LOADER
default directory DIRECTORY
access parameters
(
RECORDS DELIMITED BY NEWLINE CHARACTERSET EE8MSWIN1250 nobadfile nodiscardfile
FIELDS TERMINATED BY ";" OPTIONALLY ENCLOSED BY '\"' LDRTRIM
REJECT ROWS WITH ALL NULL FIELDS
(
ID VARCHAR2(60)
)
)
location ('tmp.txt')
)
reject limit 0;
my all rows have the newLine byte at the end of row, only thing that works is after loading data from file is update all rows using this
update table_name
set id = translate (id, 'x'||CHR(10)||CHR(13), 'x');
How can i make it automatically?
Check exactly what newline charcters are in your file and than define the record delimiter explicitely.
Example
records delimited by '\r\n'
The probable cause of your problem is that the newline character is not compatible with your operating system - which topic you can address as well.
while may have line delimiter as either \n or \r\n..
you can check that by opening file in notepad++ or any other supporting editor and by clicking show all characters
based no how the data is in the life you may create the external table as
RECORDS DELIMITED BY '\r\n' or
RECORDS DELIMITED BY '\n' etx
Related
I am trying to load a local file with "|" delimited values into hive table, we usually create a table with option "ROW FORMAT DELIMITER "|" . But I want to create a normal table and load data . What is the right syntax I need to use, please suggest.
Working Code
CREATE TABLE IF NOT EXISTS testdb.TEST_DATA_TABLE
( column1 string,
column 2 bigint,
)ROW FORMAT DELIMITED FIELDS TERMINATED BY '|';
LOAD DATA LOCAL INPATH 'xxxxx.csv' INTO TABLE testdb.TEST_DATA_TABLE;
But I want to do :
CREATE TABLE IF NOT EXISTS testdb.TEST_DATA_TABLE
( column1 string,
column 2 bigint,
);
LOAD DATA LOCAL INPATH 'xxxxx.csv' INTO TABLE testdb.TEST_DATA_TABLE FIELDS TERMINATED BY '|';
Reason begin: If i create a table, HDFS will store the data in the table with "|" delimeter
With second DDL you have provided, Hive will create default formatted table like Textformat,orc,parquet..etc(as per your configuration) with cntrl+A delimited file(default delimiter in hive).
If you want to store the hdfs file with pipe delimited then we need to create Hive Table in Text with | delimiter.
(or)
You can also write the result of select query to local (or) HDFS path with pipe delimiter also.
I had used sqoop-import command to sqoop the data into Hive from teradata. Sqoop-import command is creating a text file with comma(,) as the delimiter.
After Sqooping, I had created an external table as shown below:
CREATE EXTERNAL TABLE IF NOT EXISTS employee ( eid int, name String,
salary String, description String)
COMMENT ‘Employee details’
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘,’
LINES TERMINATED BY ‘\n’
STORED AS TEXTFILE;
But description column has values like this:"abc,xyz,mnl". Due to this,loading of data into a hive table is not proper. Then how to create a text file with a delimiter other than comma while sqooping.
Then how to delimit the fields while creating an external table of Hive?
Use --fields-terminated-by in your Sqoop job if you want to avoid the default delimiter.
--fields-terminated-by - This parameter is used for field separator character in output.
Example: --fields-terminated-by |
and then change fields separator in create table statement by FIELDS TERMINATED BY ‘|’
Can we use quote (" or ') as delimiter in hive data files? If not why?
If we could refer to a list of characters which we can use as delimiters for hive data, that would be great.
When using the decimal notation, you can use the whole basic ascii range (decimal 0-127) - tested.
Avoid using \n or\r.
As for " and ', it can be done straightforward -
create table mytable (i int,j int) row format delimited fields terminated by '"';
create table mytable (i int,j int) row format delimited fields terminated by "'";
or
create table mytable (i int,j int) row format delimited fields terminated by '\'';
create table mytable (i int,j int) row format delimited fields terminated by "\"";
CREATE TABLE IF NOT EXISTS user.name_visits(
date1 TIMESTAMP,
MV String,
visits_by_MV int
)
COMMENT ‘visits_at_MV’
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘\t’
LINES TERMINATED BY ‘\n’
;
It is saying error near BY
Below query worked for me..
CREATE TABLE IF NOT EXISTS user.name_visits(
date1 TIMESTAMP,
MV STRING,
visits_by_MV INT
)
COMMENT 'visits_at_MV'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
;
Error you are seeing could be because of the editor you are using.
If you look at your Quotation marks.. they're LEFT SINGLE QUOTATION MARK and RIGHT SINGLE QUOTATION MARK.
Only change I made was using an APOSTROPHE.
Try this way it should work
Change single quotes with double as below:
CREATE TABLE IF NOT EXISTS user.name_visits(
date1 TIMESTAMP,
MV String,
visits_by_MV int
)
COMMENT "visits_at_MV"
ROW FORMAT DELIMITED
FIELDS TERMINATED BY "\t"
LINES TERMINATED BY "\n"
;
Please can any one tell me how to accept all records from a file so that no record should go to discard or bad file using external table concept.
Below is the code
CREATE TABLE ext_tab2 (
mprn CHAR(10))
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY IMPORT
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
BADFILE IMPORT:'test.bad'
LOGFILE IMPORT:'test.log'
FIELDS TERMINATED BY ','
(mprn char(10)))
LOCATION ('abc.txt')
)
PARALLEL 5
REJECT LIMIT UNLIMITED;
All the data from file('abc.txt') should get inserted into ext_tab2 table .