I am attempting to export my PSQL table to a CSV file. I have read many tips online, such as Save PL/pgSQL output from PostgreSQL to a CSV file
From these posts I have been able to figure out
\copy (Select * From foo) To '/tmp/test.csv' With CSV
However I do not want to specify the path, I would like the user to be able to enter it via the export shell script. Is it possible to pass args to a .sql script from a .sh script?
I was able to figure out the solution.
In my bash script I used something similar to the following
psql $1 -c "\copy (SELECT * FROM users) TO '"$2"'" DELIMINTER ',' CSV HEADER"
This code copies the user table from the database specified by the first argument and exports it to a csv file located at the second arguement
Related
Snapshot of the "csv" file
So I have the table created by impala saved as csv file using the following code:
impala-shell -B -o output.csv --output_delimiter=',' -q "select * from foo" which is supposed to return a csv file, but it didnt.
Anyone help will be appreciated!
The output file you have is close to a CSV file, except that it is using pipes instead of commas. To import this into Excel, just use the wizard and select pipe (|) as the delimiter.
I'm not sure what went wrong with your Impala export, but you should be able to work with what you already have.
Currently What I have is:
bcp "select * from TEST.dbo.HPA_1 WITH (NOLOCK)" queryout D:\Gift_Voucher\HPA\test1.txt -T -c -t
When I run the above code for the first time it works fine and it creates the textfile test1.txt, but When new data is added in the table I want to create new textfile something like test2.txt and test3.txt and so on without changing the code in bcp. Is there anything we can do here?
Change your BAT file to replace the output file path with a command-line parameter:
bcp "select * from TEST.dbo.HPA_1 WITH (NOLOCK)" queryout "%1" -T -c -t
In the SSIS package, create a string variable for the desired output file path. Set the Execute Process Task Executable property to "cmd.exe". On the Expressions page, set the Arguments property to an expression that builds the command with the BAT file path plus the output file argument. The example below also encloses the values in quotes to handle whitespace in the paths:
Set the variable value to the output file path variable in your package prior to executing the task. This can be done in package code or set the value via SSIS configuration.
Note that you could accomplish the same functionality in a Data Flow task instead of shelling out to BCP. That leverages the native export capability of SSIS.
Currently I'm working with Exasol database first time and came across one script which is responsible to run sql script written in .sql file.
Here is the script
C:\Program Files\EXASOL\EXASolution\EXAplus\exaplusx64.exe -configDir EXASolutionConfig -profile profile_PROD_talend -q -f D:/Data/Customer/PROD/EXASolution_SQL/EXASOL_data_script.sql -- databaseName tableName /exasolution/StageArea/fileName.csv
I want to know, how this script is working and what its doing actually ? What I understood so far is below
First "C:\Program Files\EXASOL\EXASolution\EXAplus\exaplusx64.exe " is starting a Exasol on command line and then its pointing to the script where .sql file is located.
Not getting:
1) What this part is doing "-configDir EXASolutionConfig -profile profile_PROD_talend -q -f "?
2) What are these identifiers doing "-q -f "?
3)After launching exaplusx64.exe, Is exasol going to connect with database and table name mentioned in script ? If then How cav file is paying its role in this script ? I mean in .sql there is just an sql statement, If its taking data from file then how ? I'm not getting this ..!!
Please share your comments
1) This where you say to Exasol to read the profile profile_PROD_talend in the folder EXASolutionConfig and execute the file D:/Data/Customer/PROD/EXASolution_SQL/EXASOL_data_script.sql in quiet mode (-q).
From the manual:
-configDir *This is not actually in the EXASOL manual, I assume it's the folder with the profiles, or maybe it does nothing*
-profile Name of connection profile defined in <configDir>/profiles.xml (profiles can be edited in the GUI version). You can use a profile instead of specifying all connection parameters.
-q Quiet mode which suppresses additional output from EXAplus.
-f Name of a text file containing a set of instructions that run and then stop EXAplus.
2) Quiet mode and flag for the name of the file.
3) When you run this command EXAPlus connects to the db using the information provided in the profile and it will execute the .sql file passed.
Now things become interesting, the -- allows you to pass some arguments to the .sql file. So you are passing three parameters (databaseName, tableName, and /exasolution/StageArea/fileName.csv). If you open the sql script you will find &1, &2, and &3, these are the placeholders for the parameters passed by your command.
From the manual again:
-- <args> SQL files can use arguments given over via the parameter “-- ” by evaluating the variables &1, &2 etc. .
For example, the file test.sql including the content
--test.sql
SELECT * FROM &1;
can be called in the following way:
exaplus -f test.sql -- dual
I've been writing SQL in environments where the databases and tables are all easy to pull in using simple 'FROM db.table'. Now I'm trying to do my own project on .csv files. I want to be able to write all of my queries in .sql files and execute them using command line.
I'm uncertain about the following:
What the best program to use is.
Wow to execute a .sql file from command line.
How to import a .csv file.
Yipes! I'm new to using command line and I'm new to pulling in my own tables.
I'm currently trying out SQLlite3, but from the documentation* it doesn't look like I can simply execute a .sql file using SQLlite3 in command line.
I've tried running "sqlite3 HelloWorld.sql" in command line for a file that just has "SELECT 'Hello World';" in it and this is what I get:
SQLite version 3.9.2 2015-11-02 18:31:45
Enter ".help" for usage hints.
sqlite>
Any advice would be greatly appreciated!
https://www.sqlite.org/cli.htmlb
On Windows you can execute SQL (files) via the command line:
>sqlite3 "" "SELECT 'Hello World!';"
Hello World!
>sqlite3 "" ".read HelloWorld.sql"
Hello World!
This won't create a database file because the first parameter is empty ("") and would normally yield in a database file.
For a single file, I would use (I'm using PostgreSQL):
COPY mytable FROM '/usr/info.csv' NULL '' HEADER CSV;
However, now I have a folder with possibly 60-100 files that are in the same format ending in the ext as infoPROCESSED.csv and was wondering if there is a method in postgresql to go through the directory and copy the files into mytable? or would I have to write a script to do this?
Here's a (hackish) way of loading the csv files:
for x in *.ext; do psql -d yourdb -qtAc "copy mytable from '/path/to/files/$x' csv header null ''"; done