We have couple of hql files for compiling ddls.
in hive we used the following command from bash :
hive -v -f abc.hql
but, in beeline this doesn't work from bash. Any idea what can be the equivalent command for beeline.
Make sure your hiveserver2 is up & running on some port
In beeline
beeline -u "jdbc:hive2://localhost:port/database_name/" -f abc.hql
Refer this doc for more commands
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients
Refer this doc if you have not yet configured hiveserver2
https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2
Related
Currently in our dev environment we have hardcoded the beeline connect string to something like
beeline -u 'jdbc:hive2://zk0-hi-clu.3qy32mhqlj1ubaea5iyw5joamf.ax.internal.cloudapp.net:2181,zk1-hi-clu.3qy32mhqlj1ubaea5iyw5joamf.ax.internal.cloudapp.net:2181,zk6-hi-clu.3qy32mhqlj1ubaea5iyw5joamf.ax.internal.cloudapp.net:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2' --hivevar date=20190101 -f test.hql
I am trying to see if there are ways we can make the connect dynamic like it will look up a config file like odbc.ini. so when we promote the code to other environment, it will automatically connect to the correct target. Is this possible?
Not exactly your case: I need some defaults in my shell and use the alias functionality of the bash.
export BEELINE_CONNECTION_STRING='jdbc:hive2://myzookeeper1:2181,myzookeeper2:2181,myzookeeper3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;'
alias beeline='beeline -u ${BEELINE_CONNECTION_STRING}'
After this typing beeline causes this:
beeline
Connecting to jdbc:hive2://myzookeeper1:2181,myzookeeper2:2181,myzookeeper3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;
Connected to: Apache Hive (version 1.2.1000.2.6.5.0-292)
Driver: Hive JDBC (version 1.2.1000.2.6.5.0-292)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.6.5.0-292 by Apache Hive
0: jdbc:hive2://myhive>
I want to run query stored file in beeline. This code works OK in putty.
beeline -u "hiveserver" -n "username" -p "password" --outputformat=csv2 --silent=true -e "select * from table;" >output1.txt
When I save sql command to query.hql or query.sql and upload to server where hadoop is, command does not export anything. I get no error.
beeline -u "hiveserver" -n "username" -p "password" --outputformat=csv2 --silent=true -f query.hql >output1.txt
Query in file works when I run it as !run query.hql directly in beeline.
What is wrong with my query in file approach?
Make sure you have a new line character at the end of the file. Otherwise, beeline will not execute that command rather will just print onto the beeline terminal. Please let me know if that works.
Please, check if below is the case.
I want to know what is the command to execute a hive Script
Complete the Code to execute the hive script ./custexport.hql
Scripts hive>
If you are using hive then
hive -f 'your hql file'
if you are using beeline then also you can use -f option with complete beeline command.
I have a script that needs to connect to oracle db hosted on a different server .I am able to connect to this oracle db using sqldeveloper.But i am not able to configured it in my bash script .
SQLDEVELOPER 4.0 is the tool that i use to connect through gui .How can i use this in my script .Is there any other way to do it ?Do i need any other software (sqlplus)
try using following once:
sqlplus db_user_name/password_for_user#DB_schema < Input_file.sql > Output
You need sqlplus to achieve what you're trying to do. The syntax of the command you need to put in you shell script should be like:
sqlplus 'USER/PASSWORD#(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DB_HOST)(PORT=1521)))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=SERVICE_NAME_YOU_USE_IN_SQLDEVELOPER)))'
On *nix systems, this will create a csv stream of results to standard out:
java -Djava.security.egd=file///dev/urandom -jar jdbcsql.jar -d oracledb_SID -h $host -p 1521 -U some_username -m oracle -P "$PW" -f excel -s "," "$1"
Note that adding the -Djava.security.egd=file///dev/urandom increases performance greatly
Windows commands are similar: see http://jdbcsql.sourceforge.net/
I have some SQL scripts that I'm trying to automate. In the past I have used SQL*Plus, and called the sqlplus binary manually, from a bash script.
However, I'm trying to figure out if there's a way to connect to the DB, and call the script from inside of the bash script... so that I can insert date and make the queries run relative to a certain number of days in the past.
I'm slightly confused. You should be able to call sqlplus from within the bash script. This may be what you were doing with your first statement
Try Executing the following within your bash script:
#!/bin/bash
echo Start Executing SQL commands
sqlplus <user>/<password> #file-with-sql-1.sql
sqlplus <user>/<password> #file-with-sql-2.sql
If you want to be able to pass data into your scripts you can do it via SQLPlus by passing arguments into the script:
Contents of file-with-sql-1.sql
select * from users where username='&1';
Then change the bash script to call sqlplus passing in the value
#!/bin/bash
MY_USER=bob
sqlplus <user>/<password> #file-with-sql-1.sql $MY_USER
You can also use a "here document" to do the same thing:
VARIABLE=SOMEVALUE
sqlplus connectioninfo << HERE
start file1.sql
start file2.sql $VARIABLE
quit
HERE
Here is a simple way of running MySQL queries in the bash shell
mysql -u [database_username] -p [database_password] -D [database_name] -e "SELECT * FROM [table_name]"
Maybe you can pipe SQL query to sqlplus. It works for mysql:
echo "SELECT * FROM table" | mysql --user=username database
I've used the jdbcsql project on Sourceforge.
On *nix systems, this will create a csv stream of results to standard out:
java -Djava.security.egd=file///dev/urandom -jar jdbcsql.jar -d oracledb_SID -h $host -p 1521 -U some_username -m oracle -P "$PW" -f excel -s "," "$1"
Note that adding the -Djava.security.egd=file///dev/urandom increases performance greatly
Windows commands are similar: see http://jdbcsql.sourceforge.net/
If you do not want to install sqlplus on your server/machine then the following command-line tool can be your friend. It is a simple Java application, only Java 8 that you need in order to you can execute this tool.
The tool can be used to run any SQL from the Linux bash or Windows command line.
Example:
java -jar sql-runner-0.2.0-with-dependencies.jar \
-j jdbc:oracle:thin:#//oracle-db:1521/ORCLPDB1.localdomain \
-U "SYS as SYSDBA" \
-P Oradoc_db1 \
"select 1 from dual"
Documentation is here.
You can download the binary file from here.
As Bash doesn't have built in sql database connectivity... you will need to use some sort of third party tool.