Postgres COPY to CSV - sql

I'm trying to export a query to a csv file using the COPY command in pgAdminIII.
I'm using this command:
COPY (SELECT CASE WHEN cast(click as int) = 1 THEN 1
ELSE -1
END
|| ' '
|| id
|| '|'
|| 'hr_' || substring(hour, 7,8)
--|| ' dw_x' + substring(datename(dw, substring(hour,1,6) ),1,2)
|| ' |dw_' || substring(to_char(to_date(substring(hour, 1,6),'YYMMDD'), 'dy'),1,2)
|| ' |C1_' || c1
|| ' |C21_' || c21
|| ' |C22_' || substring(to_char(to_date(substring(hour, 1,6),'YYMMDD'), 'dy'),1,2) || '_' || substring(hour, 7,8)
AS COL1
FROM clickthru.train limit 10)
TO 'C:\me\train.csv' with csv;
When I run it I get:
ERROR: could not open file "C:\me\train.csv" for writing: Permission denied
SQL state: 42501
I then tried using the following in psql:
grant all privileges on train to public
and then look at access privileges using \z which returns :
but am still getting the same error. I'm using postgresql 9.4 on a Windows 7 box. Any other suggestions?

copy writes the file under the user account of the Postgrs server process (it is a server side operation).
From the manual:
The file must be accessible by the PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server
Under Windows this is the system's "Network Account" by default.
Apparently that account does not have write privileges on that directory.
You have two possible ways of solving that:
change the privileges on the directory to allow everybody full access
use the client side \copy command (in psql) instead

Related

Sessions are going Inactive status

I am running one sql query to fetch records from a table with multiple threads. Initially all the threads go in INACTIVE status and slowly it starts coming to ACTIVE status.
I have created HASH partition on my table based on the loc.
Also I have created an index over Item, loc columns.
select item || ',' || loc || ',' || stock_on_hand || ',' || on_order_qty || ',' || 0 || ',' || in_transit_qtyAS csv
from makro_m_oplcom_stg
where loc = ${loc} and ITEM_INV_IND = 0;
When I run this query in my DB for one loc it takes only one sec to fetch the records but when I am running it from shell script in multi threaded mode I am facing the session going into INACTIVE mode.
This is my wrapper script. Inside this I am calling another shell script that has the above mentioned query.
while [ $thread -le $SLOTS ] do
sleep 1
LOG_MESSAGE "$pgm_name Started by batch_makro_oplcom_process for thread $thread out of $SLOTS by ${USER}"
( ksh $pgm_name.ksh $UP $SLOTS $thread
if [ $? -ne 0 ]; then
LOG_MESSAGE "$pgm_name failed for thread: $thread, check the $pgm_name error file."
exit 1
else
LOG_MESSAGE "$pgm_name executed successfully for thread: $thread." fi )
& let thread=thread+1
done

Record not insert in the end of access file with delphi

Im reading one Access database with ADO and copying the data to another with the same structure. My problem is that sometime, it dont insert at the end of the file.
DM1.ADOConnectionReadAccess.ConnectionString :=
'Provider=Microsoft.Jet.OLEDB.4.0;' + 'User ID=Admin;' +
'Data Source=' + Directory + 'Courbe\Courbe Actuel.mdb' + ';' +
'Mode=Share Deny None;Extended Properties="";' + 'Jet OLEDB:Engine Type=5;';
DM1.ADOConnectionReadAccess.Connected := true;
DM1.ADODataSetReadAccess.CommandText := 'select * from Courbe_Reservoir order by DateHeure';
DM1.ADODataSetReadAccess.Active := true;
So i read my database and order it by DateTime. Now, i have to do some change on the data when i copy it to the new database.
DM1.ADOConnectionCopyExcel.ConnectionString :=
'Provider=Microsoft.Jet.OLEDB.4.0;' +
'User ID=Admin;' +
'Data Source=' + Directory + 'Courbe\Part of curve.mdb' + ';' +
'Mode=Share Deny None;Extended Properties="";' +
'Jet OLEDB:Engine Type=5;';
DM1.ADOConnectionCopyExcel.Connected := true;
while not DM1.ADODataSetReadAccess.Eof do
begin
Date2 := DM1.ADODataSetReadAccess.FieldByName('DateHeure').AsDateTime;
DernierDebit := DernierDebit + DM1.ADODataSetReadAccess.FieldByName('DebitRes').AsFloat / 1000;
DM1.ADOCommandCopyExcel.CommandText :=
'INSERT INTO Courbe_Reservoir' + ' VALUES (:DateHeure1, ' +
':Niveau, ' + //niveau
':DebitRes, ' + //débit entrée
':PH, ' + //ph
':Chlore, ' + //Chlore
':Turbidite, ' + //Turbidité
':Temp, ' + //température
'0, ' + //Consommation l/min
'0, ' + //Log
':DernierDebit, ' + //Consommation journalière
'0, 0, 0, '''', '''', ''''' + ')';
DM1.ADOCommandCopyExcel.Parameters.ParamByName('DateHeure1').Value := Date1;
DM1.ADOCommandCopyExcel.Parameters.ParamByName('Niveau').Value := DM1.ADODataSetReadAccess.FieldByName('Niveau').AsFloat;
DM1.ADOCommandCopyExcel.Parameters.ParamByName('DebitRes').Value := DM1.ADODataSetReadAccess.FieldByName('DebitRes').AsFloat;
DM1.ADOCommandCopyExcel.Parameters.ParamByName('PH').Value := DM1.ADODataSetReadAccess.FieldByName('PH').AsFloat;
DM1.ADOCommandCopyExcel.Parameters.ParamByName('Chlore').Value := DM1.ADODataSetReadAccess.FieldByName('Chlore').AsFloat;
DM1.ADOCommandCopyExcel.Parameters.ParamByName('Turbidite').Value := DM1.ADODataSetReadAccess.FieldByName('Turbidite').AsFloat;
DM1.ADOCommandCopyExcel.Parameters.ParamByName('Temp').Value := DM1.ADODataSetReadAccess.FieldByName('Temp').AsFloat;
DM1.ADOCommandCopyExcel.Parameters.ParamByName('DernierDebit').Value := DernierDebit;
DM1.ADOCommandCopyExcel.Execute;
end;
When i check the new database, i got the following result in the image :
As you can see, i got a gap in time because i got a record for each minute. The missing record in the image are located elsewhere in the database.
In the end, i want to know why its not show in the same order i insert them. The bottom line is i want to export the new access database (.mdb) to an excel file using an export command but the export command seem to take the access file and copy it to excel as is, so no way to order it because the export. I dont want to go throught all the record again with a select and insert it in excel one after the other.
Right now, i got a file with almost 180 000 record and i need to split it per month, so the new database will have around 44 000 record.
Thanks for your help
Insertion location is not defined in most databases — the database engine has the liberty to store data wherever it deems most appropriate. It might choose to add records to the end of the table, but it might also choose to reuse space vacated by previously deleted records.
If you want results in a particular order, then indicate that in your selection query, typically by including an ORDER BY clause.
I believe the records should appear in the "order" you inserted them due to the nature of .mdb files (in a multi-user db this would be chaos to be anything else!). Try adding an autonumber if you don't already have one -- you can then double check after what the order the autonum's were assigned in.
I put order in quotes because order is implied by some field (eg autonum). If there is no autonum, we cannot expect the results in any particular way

Error with single quotes inside text in select statement

Getting the error using Postgresql 9.3:
select 'hjhjjjhjh'mnmnmnm'mn'
Error:
ERRO:syntax error in or next to "'mn'"
SQL state: 42601
Character: 26
I tried replace single quote inside text with:
select REGEXP_REPLACE('hjhjjjhjh'mnmnmnm'mn', '\\''+', '''', 'g')
and
select '$$hjhjjjhjh'mnmnmnm'mn$$'
but it did not work.
Below is the real code:
CREATE OR REPLACE FUNCTION generate_mallet_input2() RETURNS VOID AS $$
DECLARE
sch name;
r record;
BEGIN
FOR sch IN
select schema_name from information_schema.schemata where schema_name not in ('test','summary','public','pg_toast','pg_temp_1','pg_toast_temp_1','pg_catalog','information_schema')
LOOP
FOR r IN EXECUTE 'SELECT rp.id as id,g.classified as classif, concat(rp.summary,rp.description,string_agg(c.message, ''. '')) as mess
FROM ' || sch || '.report rp
INNER JOIN ' || sch || '.report_comment rc ON rp.id=rc.report_id
INNER JOIN ' || sch || '.comment c ON rc.comments_generatedid=c.generatedid
INNER JOIN ' || sch || '.gold_set g ON rp.id=g.key
WHERE g.classified = any (values(''BUG''),(''IMPROVEMENT''),(''REFACTORING''))
GROUP BY g.classified,rp.summary,rp.description,rp.id'
LOOP
IF r.classif = 'BUG' THEN
EXECUTE format('Copy( select REPLACE(''%s'', '''', '''''''') as m ) To ''/tmp/csv-temp/BUG/'|| quote_ident(sch) || '-' || r.id::text || '.txt ''',r.mess);
ELSIF r.classif = 'IMPROVEMENT' THEN
EXECUTE format('Copy( select REPLACE(''%s'', '''', '''''''') as m ) To ''/tmp/csv-temp/IMPROVEMENT/'|| quote_ident(sch) || '-' || r.id || '.txt '' ',r.mess);
ELSIF r.classif = 'REFACTORING' THEN
EXECUTE format('Copy( select REPLACE(''%s'', '''', '''''''') as m ) To ''/tmp/csv-temp/REFACTORING/'|| quote_ident(sch) || '-' || r.id || '.txt '' ',r.mess);
END IF;
END LOOP;
END LOOP;
RETURN;
END;
$$ LANGUAGE plpgsql STRICT;
select * FROM generate_mallet_input2();
Error:
ERRO: erro de sintaxe em ou próximo a "mailto"
LINHA 1: ...e.http.impl.conn.SingleClientConnManager$HTTPCLIENT-803).The new SSLSocketFactory.connectSocket method calls the X509HostnameVerifier with InetSocketAddress.getHostName() parameter. When the selected IP address has a reverse lookup name, the verifier is called with the resolved name, and so the IP check fails.4.0 release checked for original ip/hostname, but this cannot be done with the new connectSocket() method. The TestHostnameVerifier.java only checks 127.0.0.1/.2 and so masked the issue, because the matching certificate has both "localhost" and "127.0.0.1", but actually only "localhost" is matched. A test case with 8.8.8.8 would be better.I committed a slightly better workaround for the problem that does not require reverse DNS lookups.Oleg. I had to resort to a fairly ugly hack in order to fix the problem. A better solution would require changes to the X509HostnameVerifier API. I felt that deprecation of the X509HostnameVerifier interface was not warranted, as the use of an IP address for CN in a certificate was a hack by itself.Please review.Oleg . Even the second one requires the server presenting a trusted certificate. I don't see much difference beetween the two cases.. Wrong test. Try to connect to https://93.62.162.60:8443/. The certificate has CN=93.62.162.60, but the check is done for 93-62-162-60.ip23.fastwebnet.it. Hmm, my comment was not meant to revert the patch. The first scenario was already exploitable and still is. Your patch is the "correct" solution without breaking the API.But to avoid any security issue (including the ones already present) the API have to be changed.. I am not able to reproduce the problem. SSL connections to remote peers pass the default host name verification.---executing requestGET https://www.verisign.com/ HTTP/1.1[DEBUG] SingleClientConnManager - Get connection for route HttpRoute[{s}->https://www.verisign.com][DEBUG] DefaultClientConnectionOperator - Connecting to www.verisign.com/69.58.181.89:443[DEBUG] RequestAddCookies - CookieSpec selected: best-match[DEBUG] DefaultHttpClient - Attempt 1 to execute request[DEBUG] DefaultClientConnection - Sending request: GET / HTTP/1.1[DEBUG] headers - >> GET / HTTP/1.1[DEBUG] headers - >> Host: www.verisign.com[DEBUG] headers - >> Connection: Keep-Alive[DEBUG] headers - >> User-Agent: Apache-HttpClient/4.1 (java 1.5)[DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 OK[DEBUG] headers - << HTTP/1.1 200 OK[DEBUG] headers - << Date: Thu, 03 Feb 2011 20:14:35 GMT[DEBUG] headers - << Server: Apache[DEBUG] headers - << Set-Cookie: v1st=D732270AE4FC9F76; path=/; expires=Wed, 19 Feb 2020 14:28:00 GMT; domain=.verisign.com[DEBUG] headers - << Set-Cookie: v1st=D732270AE4FC9F76; path=/; expires=Wed, 19 Feb 2020 14:28:00 GMT; domain=.verisign.com[DEBUG] headers - << X-Powered-By: PHP/5.2.13[DEBUG] headers - << Keep-Alive: timeout=5, max=100[DEBUG] headers - << Connection: Keep-Alive[DEBUG] headers - << Transfer-Encoding: chunked[DEBUG] headers - << Content-Type: text/html[DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name: v1st][value: D732270AE4FC9F76][domain: .verisign.com][path: /][expiry: Wed Feb 19 15:28:00 GMT+01:00 2020]". [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name: v1st][value: D732270AE4FC9F76][domain: .verisign.com][path: /][expiry: Wed Feb 19 15:28:00 GMT+01:00 2020]". [DEBUG] DefaultHttpClient - Connection can be kept alive for 5000 MILLISECONDS----------------------------------------HTTP/1.1 200 OKResponse content length: -1[DEBUG] SingleClientConnManager - Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter#15ad5c6[DEBUG] DefaultClientConnection - Connection shut down---Are you using a custom SSL socket factory by any chance? Does it implement the LayeredSchemeSocketFactory interface?Oleg. Great work, good patch, thanks!. Well, I looked at the patch. It should fix the issue (though not completely, since the reverse lookup could give a wrong/unresolvable hostname), but as you said it's a crude hack, and this opens to other security issues. Unfortunately the clean fix requires API modification.You say using an IP address as CN is a hack, but actually using it as an ipAddress SubjectAlternativeName is perfectly valid.The security issues arise from the fact that httpclient tries to match dns generated data (reverse lookups and now also resolved hostnames) instead of what the user actually typed, opening to DNS poisoning or connection redirect attacks.First scenario:- user wants to connect to 1.2.3.4- DNS reverse lookup is xxx.yyy.zzz- a malicious proxy redirects the connection to server 4.3.2.1- server certificate contains CN or SAN set to xxx.yyy.zzz- All OK (but shouldn't)Second scenario:- user wants to connect to xxx.yyy.zzz- hacked DNS incorrectly resolve it to 1.2.3.4- server certificate has CN or SAN set to 1.2.3.4- The connection is established OK (but clearly shouldn't). Fair enough. I'll revert the patch and close the issue as WONTFIXOleg. The first scenario you are describing would also require involvement of green men from Mars and the malicious 4.3.2.1 server sending a certificate trusted by the client to be practical. Oleg', '', '''') as m ) To '/tmp/csv-temp/BUG/httpclient-HTTPCLIENT-1051.txt '
CONTEXTO: função PL/pgSQL generate_mallet_input2() linha 31 em comando EXECUTE
********** Error **********
ERRO: erro de sintaxe em ou próximo a "mailto"
SQL state: 42601
Context: função PL/pgSQL generate_mallet_input2() linha 31 em comando EXECUTE
The retrieved content is a long text on project issues in software repositories and can have html in this text. Html quotes are causing the problem.
It is not the content of the string that needs to be escaped, but its representation within the SQL you are sending to the server.
In order to represent a single ', you need to write two in the SQL syntax: ''. So, 'IMSoP''s answer' represents the string IMSoP's answer, '''' represents ', and '''''' represents ''.
But the crucial thing is you need to do this before trying to run the SQL. You can't paste an invalid SQL command into a query window and tell it to heal itself.
Automation of the escaping therefore depends entirely how you are creating that SQL. Based on your updated question, we now know that you are creating the SQL statement using pl/pgsql, in this format() call:
format('Copy( select REPLACE(''%s'', '''', '''''''') as m ) To ''/tmp/csv-temp/BUG/'|| quote_ident(sch) || '-' || r.id::text || '.txt ''',r.mess)
Let's simplify that a bit to make the example clearer:
format('select REPLACE(''%s'', '''', '''''''') as m', r.mess)
If r.mess was foo, the result would look like this:
select REPLACE('foo', '', ''''') as m
This replace won't do anything useful, because the first argument is an empty string, and the second has 3 ' marks in; but even if you fixed the number of ' marks, it won't work. If the value of r.mess was instead bad'stuff, you'd get this:
select REPLACE('bad'stuff', '', ''''') as m
That's invalid SQL; no matter where you try to run it, it won't work, because Postgres thinks the 'bad' is a string, and the stuff that comes next is invalid syntax.
Think about how it will look if r.mess is SQL injection'); DROP TABLE users --:
select REPLACE('SQL injection'); DROP TABLE users; --', '', ''''') as m
Now we've got valid SQL, but it's probably not what you wanted!
So what you need to do is escape the ' marks in r.mess before you mix it into the string:
format('select '%s' as m', REPLACE(r.mess, '''', ''''''))
Now we're changing bad'stuff to bad''stuff before it goes into the SQL, and ending up with this:
select 'bad''stuff' as m
This is what we wanted.
There's actually a few better ways to do this, though:
Use the %L modifier to the format function, which outputs an escaped and quoted string literal:
format('select %L as m', r.mess)
Use the quote_literal() or quote_nullable() string functions instead of replace(), and concatenate the string together like you do with the filename:
'select ' || quote_literal(r.mess) || ' as m'
Finally, if the function really looks like it does in your question, you can avoid the whole problem by not using a loop at all; just copy each set of rows into a file using an appropriate WHERE clause:
EXECUTE 'Copy
SELECT concat(rp.summary,rp.description,string_agg(c.message, ''. '')) as mess
FROM ' || sch || '.report rp
INNER JOIN ' || sch || '.report_comment rc ON rp.id=rc.report_id
INNER JOIN ' || sch || '.comment c ON rc.comments_generatedid=c.generatedid
INNER JOIN ' || sch || '.gold_set g ON rp.id=g.key
WHERE g.classified = ''BUG'' -- <-- Note changed WHERE clause
GROUP BY g.classified,rp.summary,rp.description,rp.id
) To ''/tmp/csv-temp/BUG/'|| quote_ident(sch) || '-' || r.id::text || '.txt '''
';
Repeat for IMPROVEMENT and REFACTORING. I can't be sure, but in general, acting on a set of rows at once is more efficient than looping over them. Here, you'll have to do 3 queries, but the = any() in your original version is probably fairly inefficient anyway.
I'm taking a stab at this now that I think I know what you are asking.
You have a field in a table, that when you run SELECT <field> from <table> you are returned the result:
'This'is'a'test'
You want, intead, this result to look like:
'This''is''a''test'
So:
CREATE Table test( testfield varchar(30));
INSERT INTO test VALUES ('''This''is''a''test''');
You can run:
SELECT
'''' || REPLACE(Substring(testfield FROM 2 FOR LENGTH(testfield) - 2),'''', '''''') || ''''
FROM Test;
This will get only the bits inside the first and last single-quote, then it will replace the inner single-quotes with double quotes. Finally it concats back on single-quotes to the beginning and end.
SQL Fiddle:
http://sqlfiddle.com/#!15/a99e6/4
If it's not double single-quotes that you are looking for in the interior of your string result, then you can change the REPLACE() function to the appropriate character(s). Also, if it's not single-quotes you are looking for to encapsulate the string, then you can change those with the concatenation.
You need to escape your quoted ' marks:
select 'hjhjjjhjh''mnmnmnm''mn'

Correctly inserting literals in PL/PgSQL EXECUTE dynamic queries

The following is part of a plpgsql function. The problem is that the result of source_geom and target_geom is a character varying data type, and therefore I need to surround the both source_geom and target_geom in quotes(' '). The thing is that in plpgsql language how I don't know I can do it.
Here's what I have at the moment:
EXECUTE 'update ' || quote_ident(geom_table) ||
' SET source = ' || source_geom ||
', target = ' || target_geom ||
' WHERE ' || quote_ident(gid_cname) || ' = ' || _r.id;
The error that I am having is the following;
ERROR: syntax error at or near "C03B9E3B66052D400DDEFC2BD0F24140"
LINE 1: ...pdate track_points SET source = 0101000020E6100000C03B9E3B66...
^
QUERY: update track_points SET source = 0101000020E6100000C03B9E3B66052D400DDEFC2BD0F24140, target = 0101000020E610000075690DEF83052D40F88E75CCD4F24140 WHERE ogc_fid = 2
CONTEXT: PL/pgSQL function "create_network" line 26 at EXECUTE statement
Please any suggestions how I can solve this problem.?
Using EXECUTE ... USING with the format() function and its format specifiers will make your code much safer, simpler, easier to read and probably faster.
SQL INJECTION WARNING: If you ever accept source_geom or target_geom from the end user, your code is potentially vulnerable to SQL injection. It is important to use parameterized statements (like EXECUTE ... USING) or failing that, paranoid quoting to prevent SQL injection attacks. Even if you don't think your function takes user input you should still harden it against SQL injection, because you don't know how your app will evolve.
If you're on a newer PostgreSQL with the format function your code can be significantly simplified into:
EXECUTE format('update %I SET source = %L, target = %L WHERE %I = %L',
geom_table, source_geom, target_geom, gid_cname, _r.id);
... which handles identifier (%I) and literal (%L) quoting for you using format specifiers so you don't have to write all that awful || concatenation and quote_literal/quote_ident stuff.
Then, as per the documentation on EXECUTE ... USING you can further refine the query into:
EXECUTE format(
'update %I SET source = $1, target = $2 WHERE %I = $3',
geom_table, gid_cname
) USING source_geom, target_geom, _r.id;
which turns the query into a parameterised statement, clearly separating parameters from identifiers and reducing string processing costs for a more efficient query.
Use extra quotes:
EXECUTE 'update ' || quote_ident(geom_table) ||
' SET source = ''' || source_geom || '''
, target = ''' || target_geom || '''
WHERE ' || quote_ident(gid_cname) || ' = ' || _r.id;

Selenium IDE - Can I select a row based on column content to verify other columns?

I am testing an application using Selenium IDE. There is a table with unstable row ordering - which means the elements I need to verify are in different rows on each test run.
I'd like to use text known to be in the first column to find the row; and then verify the other columns in the same row.
The test looks like this:
store || //form/table || tableXpath
store || 3 || initialsRow
verifyTable || ${tableXpath}.${initialsRow}.0 || Initials
verifyTable || ${tableXpath}.${initialsRow}.1 || MJ
verifyTable || ${tableXpath}.${initialsRow}.2 || MH
Instead of hard-coding the "initialsRow" value; is it not possible to find the row index dynamically?
The solution I found was to use the Selenium's storeElementIndex command. It gets the index of an HTML element relative to its parent.
See http://release.seleniumhq.org/selenium-core/1.0.1/reference.html
I changed the test as follows:
store || //form/table || tableXpath
storeElementIndex || ${tableXpath}//tr/td[text() = "Initials"]/.. || initialsRow
verifyTable || ${tableXpath}.${initialsRow}.1 || MJ
verifyTable || ${tableXpath}.${initialsRow}.2 || MH
The XPath query //form/table//tr/td[text() = "Initials"]/.. finds the 'tr' element above the 'td' element containing the text "Initials". Selenium stores the index of this 'tr' element relative to whatever its parent element is.
Well, now I found, selenium CAN calculate. Unfortunately not implicitly like ${tableXpath}.${initialsRow + 1}.1 .
So I added an additional command:
storeEval || ${ORBInitialPort} + 1 || ORBInitialPortRow
and used ORBInitialPortRow instead of ORBInitialPort as index.