Hive CLI option -f not working when error occur - hive

I have one file with many load data local sqls,which may contains sqls that can cause error from hive cli.
After error the cli stop proceeding the rest sqls.
If I want to ignore these errors,and continue these sqls,How can I do?


set hive.cli.errors.ignore=true;
Demo
hive -f <(echo 'select x;select 1+1 as x')
FAILED: SemanticException [Error 10004]: Line 1:7 Invalid table alias
or column reference 'x': (possible column names are: )
hive --hiveconf hive.cli.errors.ignore=true -f <(echo 'select x;select 1+1 as x')
FAILED: SemanticException [Error 10004]: Line 1:7 Invalid table alias
or column reference 'x': (possible column names are: )
OK
2

Related

Getting sintax error while import dump into postgresql in Linux

I installed Postgresql on the Redhat Linux server and try to import the dump into the database At that time I'll get a syntax error.
sudo -u postgres -i psql db_test < data.sql
psql -h localhost -U postgres -W -d postgres < data.sql
Iam used these two commands to import data. At that time I'll get syntax error like this.
ERROR: syntax error at or near "`"
LINE 1: DROP TABLE IF EXISTS `comment_details`;
^
ERROR: syntax error at or near "`"
LINE 1: CREATE TABLE `comment_details` (
^
ERROR: syntax error at or near "`"
LINE 1: LOCK TABLES `comment_details` WRITE;
^
ERROR: syntax error at or near "`"
LINE 1: INSERT INTO `comment_details` VALUES (3,1140,'ANAKAPALLI MAN...
^
ERROR: syntax error at or near "UNLOCK"
LINE 1: UNLOCK TABLES;
^
ERROR: syntax error at or near "`"
LINE 1: DROP TABLE IF EXISTS `directory_org_details`;
^
ERROR: syntax error at or near "`"
LINE 1: CREATE TABLE `directory_org_details` (
^
ERROR: syntax error at or near "`"
LINE 1: LOCK TABLES `directory_org_details` WRITE;
^
invalid command \'s
invalid command \'s
invalid command \'s
ERROR: syntax error at or near "`"
LINE 1: INSERT INTO `directory_org_details` VALUES (1,'Daniel\'s Cri...
Please Help regarding This issue. Thanks in Advance.

FAILED: ParseException line 1:21 cannot recognize input near '<EOF>' '<EOF>' '<EOF>' in table name

Command:
hive -e "use xxx;DROP TABLE IF EXISTS `xxx.flashsaleeventproducts_hist`;CREATE EXTERNAL TABLE `xxx.flashsaleeventproducts_hist`(`event_id` string,`group_code` string,`id` string,`is_deleted` int,`price` int,`price_guide` int,`product_code` int,`product_id` string,`quantity_each_person_limit` int,`quantity_limit_plan` int,`sort_num` int,`update_time` bigint,`meta_offset` bigint,`meta_status` int,`meta_start_time` bigint)PARTITIONED BY(`cur_date` string,`cur_hour` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'LOCATION '/data/ods/xxx/flashsaleeventproducts_hist';msck repair table flashsaleevents_hist;"
Error:
xxx.flashsaleeventproducts_hist: command not found
xxx.flashsaleeventproducts_hist: command not found
event_id: command not found
group_code: command not found
is_deleted: command not found
Command 'price' not found, did you mean:
command 'rice' from deb golang-rice
Try: sudo apt install <deb name>
price_guide: command not found
product_code: command not found
product_id: command not found
quantity_each_person_limit: command not found
quantity_limit_plan: command not found
sort_num: command not found
update_time: command not found
meta_offset: command not found
meta_status: command not found
meta_start_time: command not found
cur_date: command not found
cur_hour: command not found
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 24: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-protocol.jar: Permission denied
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 25: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop-compat.jar: Permission denied
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 26: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop2-compat.jar: Permission denied
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 27: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/lib/htrace-core.jar: Permission denied
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 28: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-client.jar: Permission denied
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 29: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-server.jar: Permission denied
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/libexec/hadoop-functions.sh: line 2331: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_USER: bad substitution
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/libexec/hadoop-functions.sh: line 2426: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_OPTS: bad substitution
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hive-common-2.1.1-cdh6.2.0.jar!/hive-log4j2.properties Async: false
OK
Time taken: 2.277 seconds
NoViableAltException(-1#[199:1: tableName : (db= identifier DOT tab= identifier -> ^( TOK_TABNAME $db $tab) |tab= identifier -> ^( TOK_TABNAME $tab) );])
at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
at org.antlr.runtime.DFA.predict(DFA.java:144)
at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableName(HiveParser_FromClauseParser.java:3821)
at org.apache.hadoop.hive.ql.parse.HiveParser.tableName(HiveParser.java:40055)
at org.apache.hadoop.hive.ql.parse.HiveParser.dropTableStatement(HiveParser.java:6887)
at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3126)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2266)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1318)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:218)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:564)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1493)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:342)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:800)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
FAILED: ParseException line 1:21 cannot recognize input near '<EOF>' '<EOF>' '<EOF>' in table name
Note: The sql is correct, directly run in hive cli would not throw error.
I think the problem may be some special character in sql, but can't figure it out .
Better view for that sql
use xxx;
DROP TABLE IF EXISTS `xxx.flashsaleeventproducts`;
CREATE EXTERNAL TABLE `xxx.flashsaleeventproducts`(
`id` string,
`event_id` string,
`product_id` string,
`sort_num` int,
`price_guide` int,
`price` int,
`quantity_limit_plan` int,
`quantity_each_person_limit` int,
`is_deleted` int,
`update_time` bigint,
`group_code` string,
`product_code` int
)PARTITIONED BY(`cur_date` string,`cur_hour` string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION '/data/ods/xxx/flashsaleeventproducts';
msck repair table flashsaleeventproducts;

Please find below script to connect to hive:
import subprocess
import sys
query=""" hive -e "set hive.cli.print.header=true;use db;select * from somehivetable;" """
outresutfile=open("query_result.csv", 'w')
p=subprocess.Popen(query,shell=True,stdout=outresutfile,stderr=subprocess.PIPE)
stdout,stderr = p.communicate()
if p.returncode != 0:
print stderr
sys.exit(1)

I have known the problem came from symbol : ` .
Remove that symbol solve the problem .
Only one question left is why ` work well in hive-cli but failed in hive -e "xxx".

How to locate/export Hive query?

I am new at Hive and am attempting to export a hive query to a local file on my computer that way I can import results to excel.
When I do from inside hive;
hive -e select * from TABLE limit 10'>output.txt;
I get "FAILED: ParseException line 1:0 cannot recognize input near 'hive' '-' 'e'"
when I do
hive -S -e "USE DATABASE; select * from TABLE limit 10" > /tmp/test/test.csv;
from shell OR
insert overwrite local directory '/tmp/hello'
select * from TABLE limit 10;
It goes to the hdfs system in Hive -- how do I get this to my local machine?

You can export query to CSV file like:
hive -e 'select * from your_Table' > /home/yourfile.csv
to get this file to your local machine, you should use HDFS:
HDFS DFS -get /tmp/hello /PATHinLocalMachine
Check out this Question

You are seeing the error as you are running the hive -e commands in the hive repl as show below
hive (venkat)> hive -e 'select * from a';
NoViableAltException(26#[])
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1084)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:437)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:320)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1219)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1260)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at Sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
FAILED: ParseException line 1:0 cannot recognize input near 'hive' '-' 'e'
you have to do it in the OS shell as shown below
[venkata_udamala#gw02 ~]$ hive -e 'use database_name;select * from table_name;' > temp.txt

Hive query error

I'm trying to load a tab separated file to a HIVE text file table using hiveconf parameters as below -
load data local inpath '${hiveconf:TEXT_FILE}' into table ${hiveconf:HIVE_TABLE};
But when I run this .hql file as below
hive -hiveconf DB=$DB TEXT_FILE="$text_file_name" HIVE_TABLE=$HIVE_TABLE -f file_load.hql
I get the below error -
NoViableAltException(16#[202:1: tableName : (db= identifier DOT tab= identifier -> ^( TOK_TABNAME $db $tab) |tab= identifier -> ^( TOK_TABNAME $tab) );])
at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
......
......
FAILED: ParseException line X:YY cannot recognize input near '$' '{' 'hiveconf' in table name
I searched on google and understood that it's due to hive keyword but I have already created the table successfully and when I load the file by hardcoding the file name and table name then the data gets loaded! Please help me here!
Thank you!

You passing context variables incorrectly. it should be -hiveconf before each variable:
hive -hiveconf DB=$DB -hiveconf TEXT_FILE="$text_file_name" -hiveconf HIVE_TABLE=$HIVE_TABLE -f file_load.hql

Hive coalesce parse exception

I want to create a hive script that uses as database one of two given parameters, whichever is not null.
My hive-test.sql is this:
set db_name = coalesce(${hiveconf:dbOne}, ${hiveconf:dbTwo});
use ${hiveconf:db_name};
show tables;
and I run it with:
hive -hiveconf dbOne=my_database -f hive-test.sql
and I am getting:
FAILED: ParseException line 2:12 missing EOF at '(' near 'coalesce'
I should note that if I change the first line in script to:
set db_name = my_database;
it works.
I can't figure out what I did wrong. Your assistance is appreciated.

This feature is not available in Hive.
Do variable assignment in the shell, for example like here: setting-a-shell-variable-in-a-null-coalescing-fashion and pass it to the Hive.