I'm trying to create a new table in Apache Hive, which loads the data from a CSV file.
I wrote this script
CREATE SCHEMA IF NOT EXISTS practica2;
CREATE EXTERNAL TABLE IF NOT EXISTS practica2.station_data
(IDPROVINCIA string,
SPROVINCIA string,
IDESTACION string,
SESTACION string,
FECHA string,
DIA string,
TEMPMAX string,
HORMINTEMPMAX string,
TEMPMIN string,
HORMINTEMPMIN string,
TEMPMEDIA string,
HUMEDADMAX string,
HUMEDADMIN string,
HUMEDADMEDIA string,
VELVIENTO string,
DIRVIENTO string,
RADIACION string,
PRECIPITACION string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ';'
STORED AS TEXTFILE
LOCATION 'hdfs://0.0.0.0:9000/user/hive';
But, when I execute the script, I get this error
FAILED: ParseException line 4:1 character '' not supported here
I tested with another alternative, with the same results:
CREATE SCHEMA IF NOT EXISTS practica2;
CREATE TABLE IF NOT EXISTS practica2.station_data
(IDPROVINCIA string,
SPROVINCIA string,
IDESTACION string,
SESTACION string,
FECHA string,
DIA string,
TEMPMAX string,
HORMINTEMPMAX string,
TEMPMIN string,
HORMINTEMPMIN string,
TEMPMEDIA string,
HUMEDADMAX string,
HUMEDADMIN string,
HUMEDADMEDIA string,
VELVIENTO string,
DIRVIENTO string,
RADIACION string,
PRECIPITACION string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ';';
LOAD DATA LOCAL INPATH './RIA_exportacion_datos_diarios_Huelva_20140206.csv' INTO TABLE practica2.station_data;
The full report is this:
almu#debian:~/Practicas_BigData/Practica2/Hive$ hive -f practica2.hql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-2.3.7.jar!/hive-log4j2.properties Async: true
OK
Time taken: 3.894 seconds
FAILED: ParseException line 4:1 character '' not supported here
When I create a table from the Hive command line, It is created without problems. But, when I execute the script, It always fails.
Where is the error?
Update: About the suggestions about if the ';' could be the cause of the error, I replaced this line like this:
FIELDS TERMINATED BY ',';
But the error continues
Try removing the new lines?
CREATE SCHEMA IF NOT EXISTS practica2;
CREATE EXTERNAL TABLE IF NOT EXISTS practica2.station_data (IDPROVINCIA string, SPROVINCIA string, IDESTACION string, SESTACION string, FECHA string, DIA string, TEMPMAX string, HORMINTEMPMAX string, TEMPMIN string, HORMINTEMPMIN string, TEMPMEDIA string, HUMEDADMAX string, HUMEDADMIN string, HUMEDADMEDIA string, VELVIENTO string, DIRVIENTO string, RADIACION string, PRECIPITACION string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ';'
STORED AS TEXTFILE
LOCATION 'hdfs://0.0.0.0:9000/user/hive';
Or maybe you have full-width space in your sql file. See this post. If you want to get rid of full-width spaces, type the sql file manually, and avoid copy/paste from somewhere else.
It can be due to unnecessary space/characters that are not visible (mostly this happens when we copy and paste).
So, try to write that query again by typing, it will work.
Related
I want to show a message of type E for which I have to first create a string. The string has mixed string and integer variables to be joined.
Since only strings can be concatenated, I copy integer variable into string variable, make a whole string and concatenate.
Is there a conversion function such as to_string(integer_variable) that can convert integers to string?
PROGRAM abc.
DATA: im_acc_no TYPE i VALUE 100,
lv_acc_no TYPE string,
lv_msg TYPE string.
START-OF-SELECTION.
lv_acc_no = im_acc_no.
CONCATENATE 'Acnt# ' lv_acc_no ' does not exist' INTO lv_msg.
MESSAGE lv_msg TYPE 'E'.
There is the CONV operator (SAP help) which can do something similar to to_string but it is not allowed in the CONCATENATE, so won't help you in your scenario.
You could use the && operator (SAP help) to create the message in-place in the MESSAGE command like:
MESSAGE |Acnt# | && lv_acc_no && | does not exist| type 'E'.
Side note: do not use this variant of the MESSAGE command, it might be easy to program but it makes it hard to investigate where a message is being generated. For this reason it is better to actually create a message in SE91 and use that. Variable replacements (&) in the message also handle integers just fine.
I am testing string comparison with BRO, and got some runtime errors. Hope you guys could take a look and give me some hints.
For example i have two strings, let's say str_A and str_B, str_A is sort of a pattern, like: str_A = "\x13\x02\xf0\x80";
And str_B is a payload(contents) string from the function:
event tcp_packet(c: connection, is_orig: bool, flags: string, seq: count, ack: count, len: count, contents: string)
I compared the two of the strings with: if(str_A in str_B), which reduced the runtime errors like:
1467860547.182543 error: string with embedded NUL: "\x13\x00\xf0\x13"
1467860547.182543 error: string without NUL terminator: "\x13\x00\xf0\x13\x02\xf0\x80\x02\x00\x00\xc0\x01\x00\x00\x00\x00\x87\x02"
It looks like the 'x00' in the middle of the pattern string was considered as a terminator, and for the latter there wasn't a NUL at the end of the str_B.
So the (silly) question is how i could append a NUL at the end of str_B within BRO? and how to make BRO ignore the embeded NUL in the middle of a string when comparing? Many Thanks.
This was figured all right by translating(calling the function string_to_ascii_hex()) the hex-string into an ASCII-hex-string.
I have a massive amount of files that are all made using the same schema. They are put into a format where they are space delimited. A sample file row looks like this:
1 2 abc def "g h" 3
And when I try to use the schema INT, INT, STRING, STRING, STRING, INT, it fails for me because of the space inside the quotation marks.
I know this is where the error is because if I make a sample tab separated instead of space separated, no such error occurs, but that is not feasible for me to do with all of my data. I was wondering if there is any way to be able to indicate in a file upload that delimiters in quotes should not be treated as delimiters but rather as characters? (Rather that all quoted text should be treated as one string.)
I know this feature exists for new line characters, and so I was wondering about delimiters.
Thank you!
I figured it out. The error was there was an extra delimiter character at the end of the file. Now I just need to trim each line of the file before uploading.
I am trying to load an input data file using Hive.
Consider I have the following input in a text file:
"10"
Is it possible to load the input without quotation: as an integer?
You can use the following third party CSV Serde in the following way.
add jar path/to/csv-serde.jar;
create table table_name (a string, b string, ...)
row format serde 'com.bizo.hive.serde.csv.CSVSerde'
stored as textfile
;
Here is the link: https://github.com/ogrodnek/csv-serde.git
I Have Problem when users ask me to use - divider on parameter , here's what I've tried :
Public Function Auth(ByVal Subscriber-id As String,
ByVal Country-Code As String,
ByVal Resource-ID As String,
ByVal Action-ID As String,
ByVal IP-Address As String) As AuthOut
actually that script give me an error, to put comma , or ) .
is it possible to naming parameter like that?
No it is not. Hyphen is not an allowed character in parameter name.
See restrictions on Declared Element Names:
An element name:
Must begin with an alphabetic character or an underscore (_).
Must contain only alphabetic characters, decimal digits, and underscores.
Must contain at least one alphabetic character or decimal digit it if begins with an underscore.
Must not be more than 1023 characters long.