I want to load data to hive table created with field delimiter by ','. But my load ready file is '|' delimited. How can I specify the delimiter used in file in Load data syntax.
There are two options to manage multiple delimiters:
MultiDelimiter SerDe
Regexp SerDe
With MultoDelimSerde you can define your delimiter as
WITH SERDEPROPERTIES ("field.delim"="[,\\|]"
Related
I am faced with "Extra column(s) found" error while reading the data from S3 to Redshift.
Since my data has 863830 rows an 21 columns, ill give you a small example of how the data is.
create table test_table(
name varchar(500),
age varchar(500)
)
and my data would be
(ABC,12)
(First,Last,25)
where First,last should go into a single columns
Unfortunately, i am unable to do that with this copy command
COPY test_table from 'path'
iam_role 'credentials'
region 'us-east-1'
IGNOREHEADER 1
delimiter as ','
Is there any way to accomodate commas into a field ?
Is it a CSV file that you're trying to load? If so, try loading with CSV format parameter specified in the command, rather than using delimiter ',' parameter. Here's an example -
COPY test_table from 'path'
iam_role 'credentials'
region 'us-east-1'
IGNOREHEADER 1
CSV;
If that doesn't help, you may have to use the ESCAPE parameter. This would need modifications in your file too. Here's an example - https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#r_COPY_command_examples-copy-data-with-the-escape-option
Your data doesn't conform to the CSV specification. See RTF-4180
To store your example data the field with the comma in it needs to be enclosed in " "
ABC,12
"First,Last",25
The parentheses in the data file will also need to be removed as these will be interpreted as part of the data fields.
Alternatively you could change the delimiter of your data from "," to something else like "%". However if this character is in your data then you are right back where you started. Ad hoc delimited files only work if you use a character that will never be in your data. This is why I recommend that you use the more robust CSV specification and use the "CSV" option to COPY.
While creating table in AWS Athena i have a problem with removing footer from imported csv file stored in AWS S3.
The query is pretty simple but im stuck on footer part.
CREATE EXTERNAL TABLE csvfile
(
col1 string
col2 string
col3 string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n' STORED AS TEXTFILE
LOCATION
's3://xxxxx'
TBLPROPERTIES ("skip.header.line.count"='1',"skip.footer.line.count"='1');
The header is working perfectly fine but footer is not ;/ Any idea how to do it?
The problem is that "skip.footer.line.count"='1' is not removing the last row (summary in the csv file)
I want to implement apache hive and I want to load the data from csv file to hive table. So, here's the problem :
my csv file generated by SQL Server in it's structure have " sign, and it's become something like "04748/09","2248559","2248559","2009-12-03 00:00:00". So how can I only get the value without the " sign ?
Thanks a lot, I need your suggestions......
problem mention in your comment-how can I ignore the first line when import on hive like an mysql import?
From Hive v0.13.0, you can use skip.header.line.count. You could also specify the same while creating the table. For example:
Create external table testtable (name string, message string) row format delimited fields terminated by '\t' lines terminated by '\n' location '/testtable'
tblproperties ("skip.header.line.count"="1");
or
CREATE TABLE TEMP (f1 STRING, f2 String, f3 String, f4 String) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' TBLPROPERTIES("skip.header.line.count"="1");
load data local inpath 'test.csv'
overwrite into table TEMP;
Below is the hive table i have created:
CREATE EXTERNAL TABLE Activity (
column1 type, </br>
column2 type
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/exttable/';
In my HDFS location /exttable, i have lot of CSV files and each CSV file also contain the header row. When i am doing select queries, the result contains the header row as well.
Is there any way in HIVE where we can ignore the header row or first line ?
you can now skip the header count in hive 0.13.0.
tblproperties ("skip.header.line.count"="1");
If you are using Hive version 0.13.0 or higher you can specify "skip.header.line.count"="1" in your table properties to remove the header.
For detailed information on the patch see: https://issues.apache.org/jira/browse/HIVE-5795
Lets say you want to load csv file like below located at /home/test/que.csv
1,TAP (PORTUGAL),AIRLINE
2,ANSA INTERNATIONAL,AUTO RENTAL
3,CARLTON HOTELS,HOTEL-MOTEL
Now, we need to create a location in HDFS that holds this data.
hadoop fs -put /home/test/que.csv /user/mcc
Next step is to create a table. There are two types of them to choose from. Refer this for choosing one.
Example for External Table.
create external table industry_
(
MCC string ,
MCC_Name string,
MCC_Group string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION '/user/mcc/'
tblproperties ("skip.header.line.count"="1");
Note: When accessed via Spark SQL, the header row of the CSV will be shown as a data row.
Tested on: spark version 2.4.
There is not. However, you can pre-process your files to skip the first row before loading into HDFS -
tail -n +2 withfirstrow.csv > withoutfirstrow.csv
Alternatively, you can build it into where clause in HIVE to ignore the first row.
If your hive version doesn't support tblproperties ("skip.header.line.count"="1"), you can use below unix command to ignore the first line (column header) and then put it in HDFS.
sed -n '2,$p' File_with_header.csv > File_with_No_header.csv
To remove the header from the csv file in place use:
sed -i 1d filename.csv
Is there any way to configure the delimiter for Hive MR Jobs??
The default delimiter being used by hive internally is "hive delimiter" (/001). My usecase is to configure the delimiter so that i can use any delimiter as per the requirement. In hadoop there is a property "mapred.textoutputformatter.separator" which will set the key-value separator to the value specified for this property..Is there any such way to configure the delimiter in Hive?..I searched many but didn't get any useful links. Please help me.
As of hive-0.11.0, you can write
INSERT OVERWRITE LOCAL DIRECTORY '...'
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
SELECT ...
See HIVE-3682 for the complete syntax.
You can try that:
SELECT (rest of your query)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY 'YourChar' (example: FIELDS TERMINATED BY '\t')
You can also use this :-
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('field.delim'='-','serialization.format'='-')
This will separate columns using - delimiter but it is specific to LazSimpleSerde.
i guess you are using INSERT OVERWRITE DIRECTORY option to write to a hdfs file.
If you create a hive table on top of the hdfs file with no delimiter, it will take '\001' as delimiter, so you can read the file from a hive table without any issues
If you source table dnt not specify the delimiter in the create schema statement, then you wont be able to change that. You op will always contain the default. And yes the delimiter will be controlled by create schema for the source table. So that isnt configurable either.
I have had a similar issue and ended up modifying 001 as second step after finishing hive MR job.