I am running this statement in a Django app:
c = connections['default'].cursor()
query="copy (select * from analysis.\"{0}\") to STDOUT DELIMITER ',' CSV HEADER;".format(view_name)
with open(csvFile,'w') as f:
c.copy_expert(query,f)
f.close()
It does not create the correct csv file. Some of the values appear to be in the wrong columns. I am trying to test the SQL statement by running it in POSTGRESQL:
copy (select * from analysis."S03_2005_activity_140807_153431_with_geom") to 'C:/djangoProjects/web_output/csvfiles/S03_2005_activity_140807_153431_with_geom.csv' DELIMITER ',' CSV HEADER;
It gives me: "ERROR: relative path not allowed for COPY to file". I have looked into the issue and it appears to typically be one of two issues: 1. confusing '\' and '/'. My slashes should be correct. 2. The server being on a different computer. I thought this may be my issue as the database is located on an external computer, but I have the connection in my Postgresql. It also runs from Django so I'm not sure why it isn't working from PG Admin.
If you want to store data / get data from your local machine and communicate with a Postgres server on a different, remote machine, you cannot simply use COPY.
Try the meta-command \copy in psql. It's a wrapper for the SQL COPY command and uses local files.
Your filename should work as is on a Windows machine, but Postgres interprets it as a local filename on the server, which is probably a Unix derivate. And there the filename would have to start with '/'.
Related
I want to copy a Postgres table in CSV format from a network database to my computer.
For example, here is its address
psql postgresql://login:password#192.168.00.00:5432/test_table
The problem is that I don't have superuser rights and I can't copy the table via pg_admin.
For example, if I make a request in pg_admin:
COPY test_table TO 'C:\tmp\test_table.csv' DELIMITER ',' CSV HEADER;
I get an error:
ERROR: must be superuser or a member of the pg_write_server_files role to COPY to a file
HINT: Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.
SQL state: 42501
As I understand it, it is possible to copy the table - but through the command line, right? How to do it in my case? Thank
Instead of using COPY with a path, use STDOUT. Then, redirect the output to a local path:
psql -c "COPY test_table TO STDOUT DELIMITER ',' CSV HEADER" >> C:\tmp\test_table.csv
See the documentation for COPY.
In case you need this explanation: stdout stands for standard output, it means that the result of the command should be printed on your terminal. Using >> you redirect the output of the psql command to a file.
I would just learn how to use the command line, but if you want to stick with pgAdmin4 you can right click on the table in the browser tree and then choose "Import/Export Data" and follow the dialog box. Doing that is basically equivalent to using \copy from psql.
I have an interesting challenge. I am learning how to use COPY function in SQL. I need to import a data from .CSV to the table (PostgreSQL server). But every time when I try to do this I get this message:
ERROR: could not open file "/Users/olenaskoryk/Desktop/us_counties_2010.csv" for reading: Permission denied
HINT: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
SQL state: 42501
My query is:
COPY us_counties_2010
FROM '/Users/olenaskoryk/Desktop/us_counties_2010.csv'
WITH (FORMAT CSV, HEADER);
As you see I am working on Mac.
I assume that PostgreSQL is not running locally on your computer. That's why the server can't read your local file.
You may want to do this through a psql session using the \copy command.
$ psql your_db_connection_url
psql (10.5)
Type "help" for help.
db=# \copy us_counties_2010 FROM '/Users/olenaskoryk/Desktop/us_counties_2010.csv' WITH (FORMAT CSV, HEADER);
Psql's \copy command uses COPY FROM STDIN under the hood, passing the contents of the local file through standard input to the server, circumventing the limitation of the server not being able to read the local file.
The HINT is relevant. The SQL server is not running as you, and does not have the right to read your file.
Give it permissions (which may require giving permissions to containing directories), or put in in a universally readable place. (like perhaps a TEMP directory.)
I am trying to import/ copy my csv file to PostgreSQL. However, I am encountering these errors. I don't have import/ write permissions to the file. Will stdin help and how?The Postgres docs provides no examples. I was henceforth asked to do bulk insert but since there are too many columns with mixed data types, I am not sure how to proceed with that further.
Command to copy the csv file:
COPY sales.sales_tickets
FROM 'C:/Users/Nandini/Downloads/AIG_Sales_Tickets.csv'
DELIMITER ',' CSV;
ERROR: must be superuser to COPY to or from a file
Hint: Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.
1 statement failed.
Command to do bulk insert is too time taking:
insert into sales.sales_ticket values (1,'2',3,'4','5',6,7,8,'9',10','11');
Please suggest. Thank you.
From PostgreSQL docummentation on COPY:
COPY naming a file or command is only allowed to database superusers, since it allows reading or writing any file that the server has privileges to access.
and
Files named in a COPY command are read or written directly by the server, not by the client application. Therefore, they must reside on or be accessible to the database server machine, not the client. They must be accessible to and readable or writable by the PostgreSQL user (the user ID the server runs as), not the client. Similarly, the command specified with PROGRAM is executed directly by the server, not by the client application, must be executable by the PostgreSQL user. COPY naming a file or command is only allowed to database superusers, since it allows reading or writing any file that the server has privileges to access.
You're trying to use the COPY command violating two of the requirements:
You're trying to execute the COPY command from a non-super user.
You're trying to read a file on your client machine, and have it copied to the server.
This won't work. If you need to perform such a COPY, you need to:
Copy the CSV file to the server; to a directory that can be read by the (system) user executing the PostgreSQL server process.
Execute the COPY command from a superuser account.
Alternative
If you can't do some of these, you can always use a tool such as pgAdmin 4 and use its Import/Export functionality.
See also How to import CSV file data into a PostgreSQL table?
You are an ideal case to use /COPY not COPY.
/COPY sales.sales_tickets
FROM 'C:/Users/Nandini/Downloads/AIG_Sales_Tickets.csv'
DELIMITER ',' CSV;
How can I run .SQL files from other .SQL files using MonetDB?
I went through the whole documentation but had no success finding it.
For instance, in MySql I could do something like
parent.sql
use mydatabase;
source child1.sql
source child2.sql
child1.sql
SELECT * from Products;
child2.sql
SELECT * from Orders;
How can I do that (or something similar) using MonetDB?
To dump the SQL database, start the MonetDB SQL Client program and type the command
\>...\databasedump.sql
\D
\>
The path after > should be an absolute path name (i.e. start with a drive letter) and be in a save location. By default the database is located in %APPDATA%\MonetDB5. After having made a database dump it can be removed. This folder is located inside the dbfarm\demofolder.
Restoring the SQL database can be done using the MonetDB SQL Client program with the following command
\<...\databasedump.sql
Source:
https://www.monetdb.org/Documentation/UserGuide/DumpRestore
i have created a table in Hive "sample" and loaded a csv file "sample.txt" into it.
now i need that data from "sample" into my local /opt/zxy/sample.txt.
How can i do that?
Hortonworks' Sandbox lets you do it through its HCatalog menu. Otherwise, the syntax is
INSERT OVERWRITE LOCAL DIRECTORY '/tmp/c' SELECT a.* FROM b
as per Hive language manual
Since your intention is just to copy the entire file from HDFS to your local FS, I would not suggest you to do it through a Hive query, because of the following reasons :
It'll start a Mapreduce job which will take more time than a normal copy.
It'll create file(s) with different names(000000_0, 000001_0 and so on), which will require you to rename the file manually afterwards.
You might face problem in opening these files as they are without any extension. Your OS would be unable to choose an application to open these files on its own. In such a case you either have to rename the file or manually select an application to open it.
To avoid these problems you could use HDFS get command :
bin/hadoop fs -get /user/hive/warehouse/sample/sample.txt /opt/zxy/sample.txt
Simple n easy. But if you need to copy some selected data, then you have to use a Hive query.
HTH
I usually run my query directly through Hive on the command line for this kind of thing, and pipe it into the local file like so:
hive -e 'select * from sample' > /opt/zxy/sample.txt
Hope that helps.
Readers who are accessing Hive from Windows OS can check out this script on Github.
It's a Python+paramiko script that extracts Hive data to local Windows OS file-system.