I have a flat file loaded using Sql Loader.
I need to add a control when loading to strip all LF inside the values of the column MYFIELD2 for instance.
The columns are separated using '|' and I have the following control file:
LOAD DATA TRUNCATE into table MYTABLE fields terminated by '|'
trailing nullcols
(COD,DAT DATE "YYYYMMDDHH24MISS",
DATMOD DATE "YYYYMMDDHH24MISS",MYFIELD1, MYFIELD2)
Is there a way to do that?
This would work:
SELECT REPLACE(MyColumn, CHAR(10), ' ') FROM MyTable
You may also want to replace CHAR(13).
Related
i have problem with loading rows from a file, the point is that when im using External table like this
create table table_name
(
id VARCHAR2(60)
)
organization external
(
type ORACLE_LOADER
default directory DIRECTORY
access parameters
(
RECORDS DELIMITED BY NEWLINE CHARACTERSET EE8MSWIN1250 nobadfile nodiscardfile
FIELDS TERMINATED BY ";" OPTIONALLY ENCLOSED BY '\"' LDRTRIM
REJECT ROWS WITH ALL NULL FIELDS
(
ID VARCHAR2(60)
)
)
location ('tmp.txt')
)
reject limit 0;
my all rows have the newLine byte at the end of row, only thing that works is after loading data from file is update all rows using this
update table_name
set id = translate (id, 'x'||CHR(10)||CHR(13), 'x');
How can i make it automatically?
Check exactly what newline charcters are in your file and than define the record delimiter explicitely.
Example
records delimited by '\r\n'
The probable cause of your problem is that the newline character is not compatible with your operating system - which topic you can address as well.
while may have line delimiter as either \n or \r\n..
you can check that by opening file in notepad++ or any other supporting editor and by clicking show all characters
based no how the data is in the life you may create the external table as
RECORDS DELIMITED BY '\r\n' or
RECORDS DELIMITED BY '\n' etx
Let's say I want to create a simple table with 4 columns in Hive and load some pipe-delimited data.
CREATE table TEST_1 (
COL1 string,
COL2 string,
COL3 string,
COL4 string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
;
Raw Data:
123|456|Dasani Bottled \| Water|789
What I expect for Col3 value is "Dasani Bottled \| Water", it has some special character "\|" in the middle thus cause Hive table column off position starting at COL3 because I create the table using "|" as the delimiter. The special character \| does have a pipe | character within it.
Is there any way to resolve the issue so Hive can load data correctly?
Thanks for any help.
you can add the ESCAPED BY clause to your table creation like this to allow character escaping
CREATE table TEST_1 (
COL1 string,
COL2 string,
COL3 string,
COL4 string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|' ESCAPED BY '\'
;
From the Hive documentation
Enable escaping for the delimiter characters by using the 'ESCAPED BY'
clause (such as ESCAPED BY '\') Escaping is needed if you want to
work with data that can contain these delimiter characters.
A custom NULL format can also be specified using the 'NULL DEFINED AS'
clause (default is '\N').
Can we use quote (" or ') as delimiter in hive data files? If not why?
If we could refer to a list of characters which we can use as delimiters for hive data, that would be great.
When using the decimal notation, you can use the whole basic ascii range (decimal 0-127) - tested.
Avoid using \n or\r.
As for " and ', it can be done straightforward -
create table mytable (i int,j int) row format delimited fields terminated by '"';
create table mytable (i int,j int) row format delimited fields terminated by "'";
or
create table mytable (i int,j int) row format delimited fields terminated by '\'';
create table mytable (i int,j int) row format delimited fields terminated by "\"";
I have a table, my_table, that has a field my_field. myfield is defined as VARCHAR(7). When I do:
SELECT myfield
FROM my_table;
I get what appears to be the entire 7 characters, but I only want the actual data.
I tried:
SELECT TRIM(myfield)
FROM my_table;
and several variations. But instead of getting 'abcd', I get 'abcd '.
How do I get rid of the trailing blanks?
As others have said:
trim whitespace before data enters the database ("Mop the floor...);
ensure this is not actually a column of type CHAR(7).
Additionally, add a CHECK constraint to ensure no trailing spaces ("...fix the leak.") While you are at it, also prevent leading spaces, double spaces and zero-length string e.g.
CREATE TABLE my_table
(
myfield VARCHAR(7) NOT NULL
CONSTRAINT myfield__whitespace
CHECK (
NOT (
myfield = ''
OR myfield LIKE ' %'
OR myfield LIKE '% '
OR myfield LIKE '% %'
)
)
);-
VARCHAR columns will not pad the string you insert, meaning if you are getting 'ABCD ', that's what you stored in the database. Trim your data before inserting it.
Make sure you are not using the CHAR datatype, which will pad your data in the way you suggest. In any case:
SELECT TRIM(myfield) FROM mytable;
will work.
Make sure also that you are not confusing the way the SQL interpreter adds padding chars to format the data as a table with the actual response.
Make sure that you are not inserting data in this column from a CHAR(7) field.
You need to trim your result when selecting as opposed to when inserting, eg:
SELECT TRIM(myfield) FROM my_table;
I have a bunch of emails, each separated by a comma (,) (not csv). I want to upload them to a database table (with single field email) such that each email goes into separate record entry. what could be the most easiest way to do that? I have an idea of using grep to replace commas with my sql syntax.. but searching for any other workaround.. any idea?
Perhaps something like:
LOAD DATA INFILE '/where/the/file/is'
INTO TABLE table (email)
FIELDS TERMINATED BY ','
LINES STARTING BY '';
Syntax docs here: http://dev.mysql.com/doc/refman/5.1/en/load-data.html
I'd use shell tools like sed or awk to convert the input format to something that mysqlimport can handle.
Convert the current ',' separated email list to a one line per email list
tr ',' '\n' < inputfilename > outputfilename
use load data infile after logging into mysql, make sure your table only has one column in this case
load data infile 'outputfilename' into table tablename;
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
MySQL supports multiple inserts in a single statment
INSERT INTO [Table] ([col1], [col2], ... [colN] )
VALUES ([value1], [value2], ... [valueN] )
, ([value1], [value2], ... [valueN] )
, ([value1], [value2], ... [valueN] )
;
You could pretty quickly format a comma-separated file into this format.