In an interactive impala-shell session, is there a way to load and execute a text file containing one or more SQL statements? In Hive's beeline, for example, you can use !run <filename> to run the SQL commands in that file.
This is not currently possible. You can file a JIRA.
I believe it is possible - see impala-shell -h (version v2.1.1-cdh5):
-f QUERY_FILE, --query_file=QUERY_FILE
Execute the queries in the query file, delimited by ;
[default: none]
combine this with shell command in interactive mode:
shell impala-shell -f file;
Related
I have a folder in a directory in my PC which contains multiple SQL files. Each of the file is a Postgres function. I want to execute every SQL file situated in the folder at a time in PostgreSQL server using PgAdmin or in other way. How can I accomplish this?
I apologize if I'm oversimplifying your question, but if the main issue is how to execute all SQL files without having to call them one by one, you just need to put them in a loop, e.g. in bash calling psql
#!/bin/bash
for f in *.sql
do
psql -h dbhost -d db -U dbuser -f $f
done
Or cat them and pipe the result to psql stdin:
$ cat /path/to/files/*.sql | psql -h dbhost -d db -U dbuser
And if you need them to run in a single transaction, consider merging the SQL files, e.g. using cat - this assumes all statements in your sql file are properly terminated:
$ cat /path/to/files/*.sql > merged.sql
I have some parquet files stored in HDFS that I want to convert to csv files FIRST and export them in a remote file using ssh.
I don't know if it's possible or simple by writing a spark job (I know that we can convert parquet to csv file JUST by using spark.read.parquet then to the same DF use spark.write as a csv file). But I really wanted to do it by using a impala shell request.
So, I thought about something like this :
hdfs dfs -cat my-file.parquet | ssh myserver.com 'cat > /path/to/my-file.csv'
Can you help me PLEASE with this request ? Please.
Thank you !!
Example without kerberos:
impala-shell -i servername:portname -B -q 'select * from table' -o filename '--output_delimiter=\001'
I could explain it all, but it is late and here is a link that allows you to do that as well as the header if you want: http://beginnershadoop.com/2019/10/02/impala-export-to-csv/
You can do that by multiples ways.
One approach could be as in the example below.
With impala-shell you can run a query and pipe to ssh to write the output in a remote machine.
$ impala-shell --quiet --delimited --print_header --output_delimiter=',' -q 'USE fun; SELECT * FROM games' | ssh remoteuser#ip.address.of.remote.machine "cat > /home/..../query.csv"
This command change from default database to a fun database and run a query on it.
You can change the --output_delimiter='\t', --print_header or not along with other options.
I have a script that needs to connect to oracle db hosted on a different server .I am able to connect to this oracle db using sqldeveloper.But i am not able to configured it in my bash script .
SQLDEVELOPER 4.0 is the tool that i use to connect through gui .How can i use this in my script .Is there any other way to do it ?Do i need any other software (sqlplus)
try using following once:
sqlplus db_user_name/password_for_user#DB_schema < Input_file.sql > Output
You need sqlplus to achieve what you're trying to do. The syntax of the command you need to put in you shell script should be like:
sqlplus 'USER/PASSWORD#(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DB_HOST)(PORT=1521)))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=SERVICE_NAME_YOU_USE_IN_SQLDEVELOPER)))'
On *nix systems, this will create a csv stream of results to standard out:
java -Djava.security.egd=file///dev/urandom -jar jdbcsql.jar -d oracledb_SID -h $host -p 1521 -U some_username -m oracle -P "$PW" -f excel -s "," "$1"
Note that adding the -Djava.security.egd=file///dev/urandom increases performance greatly
Windows commands are similar: see http://jdbcsql.sourceforge.net/
I have a file that contains several SQL queries.
Can I somehow run them via isql (I'm doing the calls from Bash script, so no access to Perl DBI or JDBC)
I tried piping them into isql command via echo /my/file | isql -my-other-parameters but that didn't work.
Yes.
If you're running ISQL in interactive mode, you can load an entire contents of the file using :r my-filename command from > prompt.
From Bash script, it's also possible to do - but you need to carefully make sure that
The SQL file you are piping in has a go statement at the end. That is a VERY common cause of issues like the one you mentioned.
That statement has a newline after it.
From a script, you can do it 2 ways: Pass on STDIN via a pipe/redirect; OR, pass in the file name via isql's -i parameter
In my case it was isq -n for a piped query to work.
isql -U $DB_USR -P $DB_PWD -S $DB_PATH -D $DB_NAME -w 500 < $FILE
I have some SQL scripts that I'm trying to automate. In the past I have used SQL*Plus, and called the sqlplus binary manually, from a bash script.
However, I'm trying to figure out if there's a way to connect to the DB, and call the script from inside of the bash script... so that I can insert date and make the queries run relative to a certain number of days in the past.
I'm slightly confused. You should be able to call sqlplus from within the bash script. This may be what you were doing with your first statement
Try Executing the following within your bash script:
#!/bin/bash
echo Start Executing SQL commands
sqlplus <user>/<password> #file-with-sql-1.sql
sqlplus <user>/<password> #file-with-sql-2.sql
If you want to be able to pass data into your scripts you can do it via SQLPlus by passing arguments into the script:
Contents of file-with-sql-1.sql
select * from users where username='&1';
Then change the bash script to call sqlplus passing in the value
#!/bin/bash
MY_USER=bob
sqlplus <user>/<password> #file-with-sql-1.sql $MY_USER
You can also use a "here document" to do the same thing:
VARIABLE=SOMEVALUE
sqlplus connectioninfo << HERE
start file1.sql
start file2.sql $VARIABLE
quit
HERE
Here is a simple way of running MySQL queries in the bash shell
mysql -u [database_username] -p [database_password] -D [database_name] -e "SELECT * FROM [table_name]"
Maybe you can pipe SQL query to sqlplus. It works for mysql:
echo "SELECT * FROM table" | mysql --user=username database
I've used the jdbcsql project on Sourceforge.
On *nix systems, this will create a csv stream of results to standard out:
java -Djava.security.egd=file///dev/urandom -jar jdbcsql.jar -d oracledb_SID -h $host -p 1521 -U some_username -m oracle -P "$PW" -f excel -s "," "$1"
Note that adding the -Djava.security.egd=file///dev/urandom increases performance greatly
Windows commands are similar: see http://jdbcsql.sourceforge.net/
If you do not want to install sqlplus on your server/machine then the following command-line tool can be your friend. It is a simple Java application, only Java 8 that you need in order to you can execute this tool.
The tool can be used to run any SQL from the Linux bash or Windows command line.
Example:
java -jar sql-runner-0.2.0-with-dependencies.jar \
-j jdbc:oracle:thin:#//oracle-db:1521/ORCLPDB1.localdomain \
-U "SYS as SYSDBA" \
-P Oradoc_db1 \
"select 1 from dual"
Documentation is here.
You can download the binary file from here.
As Bash doesn't have built in sql database connectivity... you will need to use some sort of third party tool.