Is there a way to get the values of set params on beeline shell.
e.g: we can set the value as set mapreduce.reduce.java.opts=-Xmx6553m. But how to get the value?
Related
I am using below syntax to declare and use variable in hive sql query. But it gives me an error as below
SET aa='10';
SELECT
col1 as data,
${aa} as myVar from myTable;
ERROR:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify aa at runtime. It is not in list of params that are allowed to be modified at runtime
I have also tried using hiveconf
SELECT ${hiveconf:aa} from myTable;
You can not pass variable like that. You need to use --hivevar. You can create a hql file with below script - hiveqry.hql. Pls note you can use either a normal variable or with hivevar keyword.
select * from ${hivevar:aa};
select * from ${aa};
Then call that script like below
beeline --hivevar table=myTable --hivevar aa=100 -f hiveqry.hql
Depending on Hive version, when you are setting variable without explicitly specifying the namespace (hiveconf or hivevar), it may work with hiveconf as a default namespace or may not work.
BTW this works in Hive 1.2:
SET aa='10';
SELECT ${hiveconf:aa};
If you specify the namespace explicitly when setting variable, it will work with both hivevar and hiveconf
Works:
SET hivevar:aa='10';
SELECT ${hivevar:aa};
Also works:
SET hiveconf:aa='10';
SELECT ${hiveconf:aa};
Does not work:
SET aa='10';
SELECT ${hivevar:aa} as myvar;
Also if above commands do not work, check if variable substitution is enabled (default is true):
set hive.variable.substitute=true;
If set to false, substitution does not work.
Read the documentation: LanguageManual VariableSubstitution
I want to do a insert overwrite to hdfs folder as csv /textfile.
In hite-site.xml, hive.exec.compress.output is set to true.
I cannot do a set hive.exec.compress.output=false as the code is being executed in a custom build framework.
Is there an option to turn off hive compression like an attribute of the insert overwrite statement?
If you cannot modify properties in hite-site.xml, one option would be from the hive CLI or beeline, but it would be only for the current session, if you close the session and the next day start a new session you will have to do the same.
As an example:
Log in hive CLI or beeline
$ hive
to see the value of the property:
hive> SET hive.execution.engine;
to overwrite its value for the current session
hive> SET hive.execution.engine=tez
or in your case
hive> SET hive.exec.compress.output;
hive> SET hive.exec.compress.output=false
Other commands that can be useful from the Linux shell are:
$ hive -e "SET" > hive_properties
to write a file with all hive properties, or
$ hive -e "SET;" | grep compress
to see a group of hive properties from the console
I am trying to upload the binary[] of a Zip folder to my database. I used Get-Content -Encoding Byte -ReadCount 0 to read the data into a variable. I want to use this variable in an INSERT statement. Unfortunately, sqlcmd doesn't like the size of the variable, and gives me this error:
Program 'SQLCMD.EXE' failed to run: The filename or extension is too longAt line:1 char:1.
I have tried using the -Q option to run the query, and also -i to run a sql file.
DECLARE #data varbinary(MAX)
SET #data = '$(data_stuff)'
INSERT INTO MyTable
(v1,v2,v3,v4,v5)
VALUES
(v1,v2,v3,v4,#data)
sqlcmd -S servername -E -i .\file.sql -v data = "$binarydata"
Is there a workaround for doing this?
In a SQL query/batch/.sql file, binary/varbinary/image literal data values must be in hexadecimal format with a 0x prefix:
INSERT INTO tableName ( binaryColum ) VALUES ( 0x1234567890ABCDEF )
I don't know what the maximum length of a binary literal is, but I suspect things might stop working, or be very slow, if you exceed more than a few hundred kilobytes.
I recommend using ADO.NET directly via PowerShell, which will also let you use binary parameter values (SqlParameter): How do you run a SQL Server query from PowerShell?
We have a query in which we have defined more than 50 variables.
we call this hql via shell script, most of the times i get into syntax issue where i have not defined hive variables properly in the query.
Example
set hive var0=value0;
set hive var1=value1;
set hive var2=value2;
select * from ${hiveconf:var0} where col1=${hiveconf:var1} and col2=${hiveconf:var2};
I want to to check the above query result after replacing hive variables,
So is there a way to check if the variables are parsed in the right way or are there any syntax errors.
Please let me know for any alternatives as well.
Better use hivevar namespace for the same.
You can print all variable using ! echo command:
set hivevar:var0=value0;
hive> ! echo Variable hivevar:var0 is ${hivevar:var0};
Result:
Variable hivevar:var0 is value0
Also use explain extended <query> - it will print detailed query plan with predicates and fail if it is syntax error.
Update:
Also you can use SELECT for doing the same and Hive can execute simple queries without MR started if hive.fetch.task.conversion is set to more or minimal. If you are using Qubole, add also limit 1 to the query:
set hive.fetch.task.conversion=more;
select 'Variable hivevar:var0 is', '${hivevar:var0}' limit 1;
Why you may need to do this using SELECT? For example for easy checking parameter using casting or some UDF. If you need to check if parameter is of type DATE, use
set hive.fetch.task.conversion=more;
select 'Variable hivevar:var0 is', date '${hivevar:var0}' limit 1;
In this case if ${hivevar:var0} is not date, then type cast exception will be thrown and script execution terminated.
along with hivevar namespace, we can use one more property hive.root.logger=INFO,console.
this will display the query after replacing the variable value, from which we can find out the issue.
cat test.hql
set hivevar:var1=${hivevar:var11};
set hivevar:var2=2345;
select ${hivevar:var11};
select ${hivevar:var2};
hive command - hive --hiveconf hive.root.logger=INFO,console --hivevar var11=1234 -f test.hql
output on console
select 1234
2018-10-17T08:23:31,632 INFO [main] ql.Driver: Completed executing command(queryId=-4dd6-493f-88be-03810f847fe7); Time taken: 0.003 seconds
OK
2018-10-17T08:23:31,632 INFO [main] ql.Driver: OK
2018-10-17T08:23:31,670 INFO [main] io.NullRowsInputFormat$NullRowsRecordReader: Using null rows input format
1234
#echo off
REM Build YYYYMMDD_hhmmss format date/time stamp for new file name
set Stamp=%DATE:~-4%%DATE:~-10,2%%DATE:~-7,2%_%TIME:~0,8%
set Stamp=%Stamp::=%
set Stamp=%Stamp: =0%
set data_file=D:\Oracle\XML\Dump\XMLBusiness-%Stamp%.xml
set log_file=D:\Oracle\XML\Log\XMLBusiness-%Stamp%.log
set SUBJECT_AREA='ENITITY'
set STATUS='COMPLETED'
exit | sqlplus -S xx/yy#database #"C:\Documents and Settings\Desktop\XML\insert_audit_table.sql" %SUBJECT_AREA %STATUS% "SYSTIMESTAMP" > %log_file%
I am using above code to pass value from batch file to sql but I am getting this error
old 1: INSERT INTO XML_AUDIT VALUES(&1,&2,&3)
new 1: INSERT INTO XML_AUDIT VALUES(BUSINESS_ENTITY,COMPLETED,SYSTIMESTAMP)
INSERT INTO XML_AUDIT VALUES(BUSINESS_ENTITY,COMPLETED,SYSTIMESTAMP)
*
ERROR at line 1:
ORA-00984: column not allowed here
Please help me in this. How can i enclose this in single quote
Didn't understand your question completely but as per Error message, it cannot insert text as you gave. You should use single quotes for it. Make sure the values are coming as below
INSERT INTO XML_AUDIT VALUES('BUSINESS_ENTITY','COMPLETED',SYSTIMESTAMP)