I am trying to create a batch file that connects to my local Postgres 14 server and creates tables by opening a .sql file that contains all the table scripts.
For now, I was able to connect to the server using this code:
psql -U postgres -h localhost -d test
This command opens the command prompt and asks for the password. After this, how do I update the batch file to insert the password and automtically install the tables from my .sql file?
I have a database dump in thisdb_2022.dump binary file that I'm trying to import to dbeaver, but I haven't found a way to import the database so I can see it.
I found the below in the dbeaver forum but when I try to follow the instructions and create a new connection I don't see any option I can select that will open this document.
https://dbeaver.io/forum/viewtopic.php?f=2&t=895
Edit: The database and version is PostgreSQL 12
. I'm not trying to
dump it to an existing db rather I want to create a new one with this
dump.
the dump command looks like this: pg_dump -h blah.amazonaws.com -Fc -v --dbname="blah2" -f "/tmp/dump/20220203.dump".
And it will be the same version PostgreSQL 12
The easiest way to not use DBeaver at all.
Do:
UPDATED with correct command.
--In psql
CREATE DATABASE new_db;
--Exit psql
--At command line
pg_restore -d new_db -h <the_host> -p <the_port> -U postgres /tmp/dump/20220203.dump
To work in Dbeaver directly see Backup/Restore.
I have some parquet files stored in HDFS that I want to convert to csv files FIRST and export them in a remote file using ssh.
I don't know if it's possible or simple by writing a spark job (I know that we can convert parquet to csv file JUST by using spark.read.parquet then to the same DF use spark.write as a csv file). But I really wanted to do it by using a impala shell request.
So, I thought about something like this :
hdfs dfs -cat my-file.parquet | ssh myserver.com 'cat > /path/to/my-file.csv'
Can you help me PLEASE with this request ? Please.
Thank you !!
Example without kerberos:
impala-shell -i servername:portname -B -q 'select * from table' -o filename '--output_delimiter=\001'
I could explain it all, but it is late and here is a link that allows you to do that as well as the header if you want: http://beginnershadoop.com/2019/10/02/impala-export-to-csv/
You can do that by multiples ways.
One approach could be as in the example below.
With impala-shell you can run a query and pipe to ssh to write the output in a remote machine.
$ impala-shell --quiet --delimited --print_header --output_delimiter=',' -q 'USE fun; SELECT * FROM games' | ssh remoteuser#ip.address.of.remote.machine "cat > /home/..../query.csv"
This command change from default database to a fun database and run a query on it.
You can change the --output_delimiter='\t', --print_header or not along with other options.
I am new to postgres. I have exported a large, complex database with the following command in the terminal
pg_dump -U USERNAME DBNAME > dbexport.pgsql
Now that I have transferred this .pgsql file to a different computer, what is the right command to automatically create and restore the exact same database as was exported? Any suggestions would be appreciated
The way you dumped the database, the information about the database itself is not included in the dump (which is a plain SQL file).
You can either use the -C option to include CREATE DATABASE in the dump (the dump has to be restored with psql), or you use the custom format:
pg_dump -F c -U postgres DBNAME -f dbexport.pgsql
That can be restored with pg_restore like this:
pg_restore -C -d postgres -U postgres dbexport.pgsql
Due to version incompatibilities of my postgres database on heroku (9.1) and my local installation (8.4) I need a plain text sql database dump file so I can put a copy of my production data on my local testing environment.
It seems on heroku I can't make a dump using pg_dump but can instead only do this:
$ heroku pgbackups:capture
$ curl -o my_dump_file.dump `heroku pgbackups:url`
...and this gives me the "custom database dump format" and not "plain text format" so I am not able to do this:
$ psql -d my_local_database -f my_dump_file.sql
You could just make your own pg_dump directly from your Heroku database.
First, get your postgres string using heroku config:get DATABASE_URL.
Look for the Heroku Postgres url (example: HEROKU_POSTGRESQL_RED_URL: postgres://user3123:passkja83kd8#ec2-117-21-174-214.compute-1.amazonaws.com:6212/db982398), which format is postgres://<username>:<password>#<host_name>:<port>/<dbname>.
Next, run this on your command line:
pg_dump --host=<host_name> --port=<port> --username=<username> --password --dbname=<dbname> > output.sql
The terminal will ask for your password then run it and dump it into output.sql.
Then import it:
psql -d my_local_database -f output.sql
Assuming you have a DATABASE_URL configured in your environment, there is a far simpler method:
heroku run 'pg_dump $DATABASE_URL' > my_database.sql
This will run pg_dump in your container and pipe the contents to a local file, my_database.sql. The single quotes are important. If you use double quotes (or no quotes at all), DATABASE_URL will be evaluated locally rather than in your container.
If your whole purpose is to load the contents into a local database anyways, you might as well pipe it straight there:
createdb myapp_devel # start with an empty database
heroku run 'pg_dump -xO $DATABASE_URL' | psql myapp_devel
The addition of -xO avoids dumping GRANT, REVOKE, and ALTER OWNER statements, which probably don't apply to your local database server. If any of your COPY commands fail with the error ERROR: literal carriage return found in data (mine did), see this answer.
It's quite possible this didn't work two and a half years ago when this question was originally asked, but for those looking for a way to easily get a dump of your Heroku Postgres database, this appears to be the simplest possible way to do this today.
Heroku's PGBackups actually uses pg_dump behind the scenes, and the "custom format" is actually pg_dump's custom format (-Fc parameter), not Heroku's own custom format.
This means you can use pg_restore, which is part of Postgres, to restore your Heroku backup into another database directly:
pg_restore -d mydatabase my_dump_file.dump
In addition, if you call pg_restore without specifying a database to restore to, it'll print SQL statements to standard out, so you can turn your Heroku backup into a SQL file that way:
pg_restore my_dump_file.dump > sql_statements.sql
UPDATE: on more recent versions of postgres, the following command is required (thanks to comment from PatKilg)
pg_restore latest.dump -f - > sql_statements.sql
for people like me that stumble into this problem in 2020:
heroku pg:backups:capture -a app-name
heroku pg:backups:download -a app-name
the tool will actually tell what command to use after the capture. To get SQL from your latest.dump file:
pg_restore -f sqldump.sql latest.dump
and that's it.
pg_dump accepts a connection string so you don't need to deconstruct it manually like mentioned here: https://stackoverflow.com/a/22896985/3163631.
Let's say your connection string looks like this (I randomized the username and pass and added fillers for the remaining. The "shape" of the connection string is correct):
postgres://Nb6n8BTA4rPK5m:DzEPtwZUkJfgbMSdYFUbqupvJeEekihiJNzqGXa3wN2pmYRGcLQ8Sa69ujGn2RSkb#ec2-00-000-000-000.compute-1.amazonaws.com:5432/j4aaaaaaaaaam1
Even though it is in the postgres://<username>:<password>#<host_name>:<port>/<dbname> format, you can use it directly like so:
pg_dump postgres://Nb6n8BTA4rPK5m:DzEPtwZUkJfgbMSdYFUbqupvJeEekihiJNzqGXa3wN2pmYRGcLQ8Sa69ujGn2RSkb#ec2-00-000-000-000.compute-1.amazonaws.com:5432/j4aaaaaaaaaam1 > output.sql
Maybe this was not possible with pg_dump at the time Alex(https://stackoverflow.com/users/3457661/alex) answered in 2014.
Here's what worked for me:
heroku pg:backups:capture
heroku pg:backups:download
pg_restore latest.dump -f latest.sql
psql -f 'latest.sql' -d '<DEV_DB_NAME>'
Explanation:
First we create a snapshot of the database on Heroku
Then we download the snapshot as 'latest.dump' (the name can be changed using -o '<name>.dump')
Convert the binary dump into plain SQL, which can be imported without raising "pg_dump: error: aborting because of server version mismatch"
Import the file into the local database
Of course, if your running version of postgresql is compatible with Heroku's, heroku pg:pull DATABASE_URL <DEV_DB_NAME> is simpler to type and remember.
Heroku pg:backups:capture
Heroku pg:backups:download
Taken from https://devcenter.heroku.com/articles/heroku-postgres-import-export.
Now you have a binary file. To obtain the file in plain text format, the following worked for me. Note: You will need to install PostgreSQL.
pg_restore latest.dump > latest.sql
You could just download the Heroku dump file and convert it into plain text format.
In newer versions, directly redirecting the output of pg_restore to an SQL file won't work. Doing so will produce an error:
pg_restore my_dump_file.dump > my_dump_file.sql
pg_restore: error: one of -d/--dbname and -f/--file must be specified
Instead, to output the result in plain text format, -f should be used:
pg_restore my_dump_file.dump -f my_dump_file.sql
This will convert the heroku "custom database dump format" to "plain text format".
Then import this file:
psql -d my_local_database -f my_dump_file.sql