${beeline_url} --silent=true --showHeader=false --outputformat=csv2 --showWarnings=false -e "select concat('invalidate metadata ', trim(table_name) , '; refresh ', trim(table_name) ,';') from my_Table " > /home/table_list.csv
I'm trying to run this query ends up with error. The same query runs fine in hive, hue and even with beeline.
while using beeline, the below query gave results
0: jdbc:hive2://host> select concat("invalidate metadata ", trim(table_name)) from my_Table;
I tried storing the query in a file but it ends up in error.
${beeline_url} --silent=true --showHeader=false --outputformat=csv2 --verbose=false --showWarnings=false -f get_table_list.hql > /home/table_list.csv
where get_table_list.hql has
SELECT (CONCAT('invalidate metadata ', trim(table_name) , '; refresh ', trim(table_name) ,';')) from my_table;
Error:
Error: Error while compiling statement: FAILED: ParseException line
1:59 cannot recognize input near '' '' '' in select
expression (state=42000,code=40000)
Semicolons need to be shielded using \\:
SELECT (CONCAT('invalidate metadata ', trim(table_name) , '\\; refresh ', trim(table_name) ,'\\;')) from my_table;
Or replace them with \073:
SELECT (CONCAT('invalidate metadata ', trim(table_name) , '\073 refresh ', trim(table_name) ,'\073')) from my_table;
One of these workarounds should work.
Related
I am trying to string concatenate xml using below oracle sql query. its throwing error when there are invalid characters in the data. Kindly suggest if any improvement can be done.
SELECT
'Outstanding Conditions:' || outstanding_conditions "OUTSTANDING_CONDITIONS"
FROM
(
SELECT
rtrim(
XMLAGG(xmlelement(
e,
chr(13)
|| rownum
|| '. '
|| replace(
replace(
condition_desc,
'PBK''S',
'PBK'
),
'PBK',
'PBK'
)
).extract('//text()')
ORDER BY
application_id
).getclobval()
) "OUTSTANDING_CONDITIONS"
FROM
tbl_assessment_cond
WHERE
condition_status = 'U'
AND application_id = 'Z01234567'
AND instr(
'~S~', '~'
|| is_acceptance
|| '~'
) > 0
)
WHERE
outstanding_conditions IS NOT NULL;
Error:
ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00216: invalid character 160 (0xA0)
Error at line 2
31011. 00000 - "XML parsing failed"
*Cause: XML parser returned an error while trying to parse the document.
*Action: Check if the document to be parsed is valid.
I have the below select statement in hive .It executes perfectely fine.
In Hive
select
COALESCE(product_name,CONCAT(CONCAT(CONCAT(TRIM(product_id),' -
'),trim(plan_code)),' - UNKNOWN')) as product_name
from table name;
I am trying to use the same select statement in POSTGRESQL and it throw me error saying "
Query execution failed
Reason:
SQL Error [42883]: ERROR: function concat(text, unknown) does not exist
Hint: No function matches the given name and argument types. You might need to add explicit type casts.
In postgresql:
select
COALESCE(product_name,CONCAT(CONCAT(CONCAT(TRIM(product_id),' -
'),trim(plan_code)),' - UNKNOWN')) as product_name
from table name;
Could some one throw some light on this ?
Instead of concat try with ||:
SELECT COALESCE(product_name,
(TRIM(product_id) || ' - ' || TRIM(plan_code) || ' - UNKNOWN')
) AS product_name
FROM tablename;
or simply a single CONCAT as:
SELECT COALESCE(product_name,
CONCAT(TRIM(product_id)::text, ' - ', TRIM(plan_code)::text, ' - UNKNOWN')
) AS product_name
FROM tablename;
You can also consider using format function:
SELECT coalesce(product_name, format('%s - %s - UNKNOWN', trim(product_id), trim(plan_code)))
PostgreSQL version : 11.1
Platform : OSX Mojave 10.14.1
That's my SQL code:
COPY (select nom,prenom,num_carte,pdv_carte,email,date_naissance from compte where num_carte != '' order by id_compte) TO :export_file WITH DELIMITER AS ';' CSV FORCE QUOTE * ENCODING 'UTF-8';
That line is in a .sql file called by shell script like this :
psql --dbname=test -U postgres --set adresses=$DATA_ADRESSE --set export_file="$EXPORT_FILE" --file=$ANO_SQL 1>>$ANO_LOG
With EXPORT_FILE variable declared like that :
export EXPORT_FILE="'export_for_fid.csv'"
Tried many solutions but none worked, always the same syntax error:
ERROR: syntax error at or near ""
LINE 1: ...where num_carte != '' order by id_compte) TO 'export_for_fid.csv' WITH D...
^
Instead of using --file and --set arguments, you could use a here-document in your shell script: (NOTE: I replaced the COPY by a \COPY )
#!/bin/sh
EXPORT_FILE="export_for_fid.csv"
DB_NAME="test"
psql --dbname=${DB_NAME} -U postgres <<OMG
\COPY (select nom,prenom,num_carte,pdv_carte,email,date_naissance from compte where num_carte <> '' order by id_compte) TO '${EXPORT_FILE}' WITH DELIMITER AS ';' CSV FORCE QUOTE * ENCODING 'UTF-8';
OMG
#eof
I'm trying to export a query to a csv file using the COPY command in pgAdminIII.
I'm using this command:
COPY (SELECT CASE WHEN cast(click as int) = 1 THEN 1
ELSE -1
END
|| ' '
|| id
|| '|'
|| 'hr_' || substring(hour, 7,8)
--|| ' dw_x' + substring(datename(dw, substring(hour,1,6) ),1,2)
|| ' |dw_' || substring(to_char(to_date(substring(hour, 1,6),'YYMMDD'), 'dy'),1,2)
|| ' |C1_' || c1
|| ' |C21_' || c21
|| ' |C22_' || substring(to_char(to_date(substring(hour, 1,6),'YYMMDD'), 'dy'),1,2) || '_' || substring(hour, 7,8)
AS COL1
FROM clickthru.train limit 10)
TO 'C:\me\train.csv' with csv;
When I run it I get:
ERROR: could not open file "C:\me\train.csv" for writing: Permission denied
SQL state: 42501
I then tried using the following in psql:
grant all privileges on train to public
and then look at access privileges using \z which returns :
but am still getting the same error. I'm using postgresql 9.4 on a Windows 7 box. Any other suggestions?
copy writes the file under the user account of the Postgrs server process (it is a server side operation).
From the manual:
The file must be accessible by the PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server
Under Windows this is the system's "Network Account" by default.
Apparently that account does not have write privileges on that directory.
You have two possible ways of solving that:
change the privileges on the directory to allow everybody full access
use the client side \copy command (in psql) instead
This O'Reilly article gives an example of a PostgreSQL statement that parses an Apache log line:
INSERT INTO http_log(log_date,ip_addr,record)
SELECT CAST(substr(record,strpos(record,'[')+1,20) AS date),
CAST(substr(record,0,strpos(record,' ')) AS cidr),
record
FROM tmp_apache;
Obviously this only extracts the IP and timestamp fields. Is there a canonical statement for extracting all fields from a typical combined log format record? If there isn't, I will write one and I promise to post the result here!
OK, here is my solution:
insert into accesslog
select m[1], m[2], m[3],
(to_char(to_timestamp(m[4], 'DD/Mon/YYYY:HH24:MI:SS'), 'YYYY-MM-DD HH24:MI:SS ')
|| split_part(m[4], ' ',2))::timestamp with time zone,
m[5], m[6]::smallint, (case m[7] when '-' then '0' else m[7] end)::integer, m[8], m[9] from (
select regexp_matches(record,
E'(.*) (.*) (.*) \\[(.*)\\] "(.*)" (\\d+) (.*) "(.*)" "(.*)"')
as m from tmp_apache) s;
It takes raw log lines from the table tmp_apache and extracts the fields (using the regexp) into an array.
Here is my somewhat more complete solution.
The apache log file should not contain invalid characters or backslashes. If necessary, you can remove these from the log file with:
cat logfile | strings | grep -v '\\' > cleanedlogfile
Then copy and parse the log file into postgres (m[1] to m[7] correspond to the regex groups in regexp_matches function):
-- sql for postgres:
drop table if exists rawlog;
create table rawlog (record varchar);
-- import data from log file
copy rawlog from '/path/to/your/apache/cleaned/log/file';
-- parse the rawlog into table accesslog
drop table if exists accesslog;
create table accesslog as
(select m[1] as clientip,
(to_char(to_timestamp(m[4], 'DD/Mon/YYYY:HH24:MI:SS'), 'YYYY-MM-DD HH24:MI:SS ')
|| split_part(m[4], ' ',2))::timestamp with time zone as "time",
split_part(m[5], ' ', 1) as method,
split_part(split_part(m[5], ' ', 2), '?', 1) as uri,
split_part(split_part(m[5], ' ', 2), '?', 2) as query,
m[6]::smallint as status,
m[7]::bigint bytes
from
(select
regexp_matches(record, E'(.*) (.*) (.*) \\[(.*)\\] "(.*)" (\\d+) (\\d+)') as m
from rawlog) s);
-- optionally create indexes
create index accesslogclientipidx on accesslog(clientip);
create index accesslogtimeidx on accesslog(time);
create index accessloguriidx on accesslog(uri);