Related
I'm trying to write a function that takes a minimum and a maximum input and returns a double.
Inputs:
high (integer)
low (integer)
Output:
val (double)
My SQL code is:
CREATE OR REPLACE FUNCTION random_between(low INT ,high INT)
RETURNS DOUBLE AS
BEGIN
RETURN floor(random()* (high-low + 1) + low);
END;
The error:
ERROR: syntax error at or near "BEGIN"
You could write this as pure SQL function, like so:
create or replace function random_between(low int ,high int)
returns double precision as $$
select floor(random()* (high-low + 1) + low);
$$ language sql;
Problems with your code:
the body of the function needs to be surrounded with single quotes (or something equivalent, such as $$)
there is no double datatype in Postgres; maybe you meant double precision; note, however, that this is an inexact datatype: this might, or might not be what you want, but make sure that you understand the implications
you need to specify the language of the function
In table I have string column which contains a hex value. For example value '000000000000000a' means 10. Now I need to process bitand operation: bitand(tableName.hexColumn, ?). When I read the Informix specification of this function it needs 2 int. So my question is: what is the simpler way to process this operation?
PS: Probably there is no solution in Informix so I will have to create my own bitandhexstring function where input will be 2 string and hex form but I have no idea where to start.
There are a variety of issues to be dealt with:
Your hex string has 16 digits, so the values are presumably (in general) 64-bit quantities. That means you need to be sure that the BITAND function has a variant that handles BIGINT (or perhaps INT8 — I'm not going to mention INT8 again, but it is nominally an option when BIGINT is mentioned) data.
You need to convert your hex string to a BIGINT.
It is not clear whether you'll need to convert the result BIGINT back to a hex string.
Some testing with Informix 11.70.FC6 on Mac OS X 10.10.4 shows that BITAND is safe with 64-bit numbers. That's good news!
The HEX function, when passed a BIGINT, returns a CHAR(20) string that starts with 0x and contains a hex representation of the number, so that more or less addresses point 3. The residual issue is 'how to convert 16-byte strings of hex digits to a BIGINT value'. Nominally, a cast operation like:
CAST('0xde3962e8c68a8001' AS BIGINT)
should do the job (but see below). There may be a better way of doing it than a brute-force and ignorance stored procedure, but I'm not immediately sure what it is.
Caveat Lector.
While testing this, I tried two queries:
SELECT bi, HEX(bi) FROM Test_BigInt;
SELECT bi, HEX(bi), SUBSTR(HEX(bi), 3, 16) FROM Test_BigInt;
on a table Test_BigInt with a single column bi of type BIGINT (not null, as it happened, but that's not material).
The first query worked fine. The type of the HEX(bi) expression was CHAR(20) and the values were like
0 0x0000000000000000
6898532535585831936 0x5fbc82ca87117c00
-2300268458811555839 0xe013ce0628808001
The second query sort of worked for small values of bi (0, 1, 2), but generated an error -1215: Value exceeds limit of INTEGER precision when the values got large. The problem is not the SUBSTR function directly. This was testing with Informix 11.70.FC6 on Mac OS X 10.10.4 — tested on 2015-07-08. The following pair of queries worked as expected (which is my justification for claiming that the problem is not in the SUBSTR function per se).
SELECT bi, HEX(bi) AS hex_bi FROM Test_BigInt INTO TEMP t;
SELECT bi, hex_bi, SUBSTR(hex_bi, 3, 16) FROM t;
It seems to be an interaction problem when the result of HEX is used in a string operation context. I first got the problem when trying to concatenate an empty string to the result of HEX: HEX(bi) || ''. That turns out to be unnecessary given that the result of HEX is reported as CHAR(20), but also indicates SUBSTR is not directly at fault.
I also tried CAST to get the hex string converted to BIGINT:
SELECT CAST('0xde3962e8c68a8001' AS BIGINT) FROM dual;
BIGINT
-964001791
SELECT HEX(CAST('0xde3962e8c68a8001' AS BIGINT)) FROM dual;
CHAR(18)
0xffffffffc68a8001
Grrr! Something is mishandling the conversion. This is not new software (well over 2 years old), but the chances are that unless someone else has spotted the bug, it has not yet been fixed, even in the latest version.
I've reported this through back-channels to IBM/Informix.
Stored procedures to convert hex string to BIGINT
CREATE PROCEDURE hexval(c CHAR(1)) RETURNING INTEGER;
RETURN INSTR("0123456789abcdef", lower(c)) - 1;
END PROCEDURE;
CREATE PROCEDURE hexstr_to_bigint(ival VARCHAR(18)) RETURNING bigint;
DEFINE oval DECIMAL(20,0);
DEFINE i,j,len INTEGER;
LET ival = LOWER(ival);
IF (ival[1,2] = '0x') THEN LET ival = ival[3,18]; END IF;
LET len = LENGTH(ival);
LET oval = 0;
FOR i = 1 TO len
LET j = hexval(SUBSTR(ival, i, 1));
LET oval = oval * 16 + j;
END FOR;
IF (oval > 9223372036854775807) THEN
LET oval = oval - 18446744073709551616;
END IF;
RETURN oval;
END PROCEDURE;
Casual testing:
execute procedure hexstr_to_bigint('000A');
10
execute procedure hexstr_to_bigint('FFff');
65535
execute procedure hexstr_to_bigint('FFFFffffFFFFffff');
-1
execute procedure hexstr_to_bigint('0XFFFFffffFFFFffff');
-1
execute procedure hexstr_to_bigint('000000000000000A');
10
Those values are correct.
Amazon Redshift is based on ParAccel which is based on Postgres. From my research it seems that the preferred way to perform hexadecimal string to integer conversion in Postgres is via a bit field, as outlined in this answer.
In the case of bigint, this would be:
select ('x'||lpad('123456789abcdef',16,'0'))::bit(64)::bigint
Unfortunately, this fails on Redshift with:
ERROR: cannot cast type text to bit [SQL State=42846]
What other ways are there to perform this conversion in Postgres 8.1ish (that's close to the Redshift level of compatibility)? UDFs are not supported in Redshift and neither are array, regex functions or set generating functions...
It looks like they added a function for this at some point: STRTOL
Syntax
STRTOL(num_string, base)
Return type
BIGINT. If num_string is null, returns NULL.
For example
SELECT strtol('deadbeef', 16);
Returns: 3735928559
Assuming that you want a simple digit-by-digit ordinal position conversion (i.e. you're not worried about two's compliment negatives, etc) I think this should work on an 8.1-equivalent DB:
CREATE OR REPLACE FUNCTION hex2dec(text) RETURNS bigint AS $$
SELECT sum(CASE WHEN v >= ascii('a') THEN v - ascii('a') + 10 ELSE v - ascii('0') END * 16^ordpos)::bigint
FROM (
SELECT n-1, ascii(substring(reverse($1), n, 1))
FROM generate_series(1, length($1)) n
) AS x(ordpos, v);
$$ LANGUAGE sql IMMUTABLE;
The function form is optional, it just makes it easier to avoid repeating the argument a bunch of times. It should get inlined anyway. Efficiency will probably be awful, but most of the tools available to do this smarter don't seem to be available on versions that old, and this at least works:
regress=> CREATE TABLE t AS VALUES ('c13b'), ('a'), ('f');
regress=> SELECT hex2dec(column1) FROM t;
hex2dec
---------
49467
10
15
(3 rows)
If you can use regexp_split_to_array and generate_subscripts it might be faster. Or slower. I haven't tried. Another possible trick is to use a digit mapping array instead of the CASE, like:
'[48:102]={0,1,2,3,4,5,6,7,8,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,11,12,13,14,15}'::integer[]
which you can use with:
CREATE OR REPLACE FUNCTION hex2dec(text) RETURNS bigint AS $$
SELECT sum(
('[48:102]={0,1,2,3,4,5,6,7,8,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,11,12,13,14,15}'::integer[])[ v ]
* 16^ordpos
)::bigint
FROM (
SELECT n-1, ascii(substring(reverse($1), n, 1))
FROM generate_series(1, length($1)) n
) AS x(ordpos, v);
$$ LANGUAGE sql IMMUTABLE;
Personally, I'd do it client-side instead, rather than wrangling the limited capabilities of an old PostgreSQL fork, especially one you can't load your own sensible user-defined C functions on, or use PL/Perl, etc.
In real PostgreSQL I'd just use this:
hex2dec.c:
#include "postgres.h"
#include "fmgr.h"
#include "utils/builtins.h"
#include "errno.h"
#include "limits.h"
#include <stdlib.h>
PG_MODULE_MAGIC;
Datum from_hex(PG_FUNCTION_ARGS);
PG_FUNCTION_INFO_V1(hex2dec);
Datum
hex2dec(PG_FUNCTION_ARGS)
{
char *endpos;
const char *hexstr = text_to_cstring(PG_GETARG_TEXT_PP(0));
long decval = strtol(hexstr, &endpos, 16);
if (endpos[0] != '\0')
{
ereport(ERROR, (ERRCODE_INVALID_PARAMETER_VALUE, errmsg("Could not decode input string %s as hex", hexstr)));
}
if (decval == LONG_MAX && errno == ERANGE)
{
ereport(ERROR, (ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE, errmsg("Input hex string %s overflows int64", hexstr)));
}
PG_RETURN_INT64(decval);
}
Makefile:
MODULES = hex2dec
DATA = hex2dec--1.0.sql
EXTENSION = hex2dec
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
hex2dec.control:
comment = 'Utility function to convert hex strings to decimal'
default_version = '1.0'
module_pathname = '$libdir/hex2dec'
relocatable = true
hex2dec--1.0.sql:
CREATE OR REPLACE FUNCTION hex2dec(hexstr text) RETURNS bigint
AS 'hex2dec','hex2dec'
LANGUAGE c IMMUTABLE STRICT;
COMMENT ON FUNCTION hex2dec(hexstr text)
IS 'Decode the hex string passed, which may optionally have a leading 0x, as a bigint. Does not attempt to consider negative hex values.';
Usage:
CREATE EXTENSION hex2dec;
postgres=# SELECT hex2dec('7fffffffffffffff');
hex2dec
---------------------
9223372036854775807
(1 row)
postgres=# SELECT hex2dec('deadbeef');
hex2dec
------------
3735928559
(1 row)
postgres=# SELECT hex2dec('12345');
hex2dec
---------
74565
(1 row)
postgres=# select hex2dec(to_hex(-1));
hex2dec
------------
4294967295
(1 row)
postgres=# SELECT hex2dec('8fffffffffffffff');
ERROR: Input hex string 8fffffffffffffff overflows int64
postgres=# SELECT hex2dec('0x7abcz123');
ERROR: Could not decode input string 0x7abcz123 as hex
The performance difference is ... noteworthy. Given sample data:
CREATE TABLE randhex AS
SELECT '0x'||to_hex( abs(random() * (10^((random()-.5)*10)) * 10000000)::bigint) AS h
FROM generate_series(1,1000000);
conversion from hex to decimal takes about 1.3 from a warm cache using the C extension, which isn't great for a million rows. Reading them without any transformation takes 0.95s. It took 36 seconds for the SQL based hex2dec approach to process the same rows. Frankly I'm really impressed that the SQL approach was as fast as that, and surprised the C ext was that slow.
A likely explanation is that the cast from text to bit(n) relies on undocumented behavior, I repeat the quote from Tom Lane:
This is relying on some undocumented behavior of the bit-type input
converter, but I see no reason to expect that would break. A possibly
bigger issue is that it requires PG >= 8.3 since there wasn't a text
to bit cast before that.
And Amazon derivate is obviously not allowing this undocumented feature. Not surprising, since it is based off of Postgres 8.1 where there was no cast at all.
Previously quoted in this closely related answer:
Convert hex in text representation to decimal number
I want to call a function by passing multiple values on single parameter, like this:
SELECT * FROM jobTitle('270,378');
Here is my function.
CREATE OR REPLACE FUNCTION test(int)
RETURNS TABLE (job_id int, job_reference int, job_job_title text
, job_status text) AS
$$
BEGIN
RETURN QUERY
select jobs.id,jobs.reference, jobs.job_title,
ltrim(substring(jobs.status,3,char_length(jobs.status))) as status
FROM jobs ,company c
WHERE jobs."DeleteFlag" = '0'
and c.id= jobs.id and c.DeleteFlag = '0' and c.active = '1'
and (jobs.id = $1 or -1 = $1)
order by jobs.job_title;
END;
$$ LANGUAGE plpgsql;
Can someone help with the syntax? Or even provide sample code?
VARIADIC
Like #mu provided, VARIADIC is your friend. One more important detail:
You can also call a function using a VARIADIC parameter with an array type directly. Add the key word VARIADIC in the function call:
SELECT * FROM f_test(VARIADIC '{1, 2, 3}'::int[]);
is equivalent to:
SELECT * FROM f_test(1, 2, 3);
Other advice
In Postgres 9.1 or later right() with a negative length is faster and simpler to trim leading characters from a string:
right(j.status, -2)
is equivalent to:
substring(j.status, 3, char_length(jobs.status))
You have j."DeleteFlag" as well as j.DeleteFlag (without double quotes) in your query. This is probably incorrect. See:
PostgreSQL Error: Relation already exists
"DeleteFlag" = '0' indicates another problem. Unlike other RDBMS, Postgres properly supports the boolean data type. If the flag holds boolean data (true / false / NULL) use the boolean type. A character type like text would be inappropriate / inefficient.
Proper function
You don't need PL/pgSQL here. You can use a simpler SQL function:
CREATE OR REPLACE FUNCTION f_test(VARIADIC int[])
RETURNS TABLE (id int, reference int, job_title text, status text)
LANGUAGE sql AS
$func$
SELECT j.id, j.reference, j.job_title
, ltrim(right(j.status, -2)) AS status
FROM company c
JOIN job j USING (id)
WHERE c.active
AND NOT c.delete_flag
AND NOT j.delete_flag
AND (j.id = ANY($1) OR '{-1}'::int[] = $1)
ORDER BY j.job_title
$func$;
db<>fiddle here
Old sqlfiddle
Don't do strange and horrible things like converting a list of integers to a CSV string, this:
jobTitle('270,378')
is not what you want. You want to say things like this:
jobTitle(270, 378)
jobTitle(array[270, 378])
If you're going to be calling jobTitle by hand then a variadic function would probably be easiest to work with:
create or replace function jobTitle(variadic int[])
returns table (...) as $$
-- $1 will be an array if integers in here so UNNEST, IN, ANY, ... as needed
Then you can jobTitle(6), jobTitle(6, 11), jobTitle(6, 11, 23, 42), ... as needed.
If you're going to be building the jobTitle arguments in SQL then the explicit-array version would probably be easier to work with:
create or replace function jobTitle(int[])
returns table (...) as $$
-- $1 will be an array if integers in here so UNNEST, IN, ANY, ... as needed
Then you could jobTitle(array[6]), jobTitle(array[6, 11]), ... as needed and you could use all the usual array operators and functions to build argument lists for jobTitle.
I'll leave the function's internals as an exercise for the reader.
if I write a query as such:
with WordBreakDown (idx, word, wordlength) as (
select
row_number() over () as idx,
word,
character_length(word) as wordlength
from
unnest(string_to_array('yo momma so fat', ' ')) as word
)
select
cast(wbd.idx + (
select SUM(wbd2.wordlength)
from WordBreakDown wbd2
where wbd2.idx <= wbd.idx
) - wbd.wordlength as integer) as position,
cast(wbd.word as character varying(512)) as part
from
WordBreakDown wbd;
... I get a table of 4 rows like so:
1;"yo"
4;"momma"
10;"so"
13;"fat"
... this is what I want. HOWEVER, if I wrap this into a function like so:
drop type if exists split_result cascade;
create type split_result as(
position integer,
part character varying(512)
);
drop function if exists split(character varying(512), character(1));
create function split(
_s character varying(512),
_sep character(1)
) returns setof split_result as $$
begin
return query
with WordBreakDown (idx, word, wordlength) as (
select
row_number() over () as idx,
word,
character_length(word) as wordlength
from
unnest(string_to_array(_s, _sep)) as word
)
select
cast(wbd.idx + (
select SUM(wbd2.wordlength)
from WordBreakDown wbd2
where wbd2.idx <= wbd.idx
) - wbd.wordlength as integer) as position,
cast(wbd.word as character varying(512)) as part
from
WordBreakDown wbd;
end;
$$ language plpgsql;
select * from split('yo momma so fat', ' ');
... I get:
1;"yo momma so fat"
I'm scratching my head on this. What am I screwing up?
UPDATE
Per the suggestions below, I have replaced the function as such:
CREATE OR REPLACE FUNCTION split(_string character varying(512), _sep character(1))
RETURNS TABLE (postition int, part character varying(512)) AS
$BODY$
BEGIN
RETURN QUERY
WITH wbd AS (
SELECT (row_number() OVER ())::int AS idx
,word
,length(word) AS wordlength
FROM unnest(string_to_array(_string, rpad(_sep, 1))) AS word
)
SELECT (sum(wordlength) OVER (ORDER BY idx))::int + idx - wordlength
,word::character varying(512) -- AS part
FROM wbd;
END;
$BODY$ LANGUAGE plpgsql;
... which keeps my original function signature for maximum compatibility, and the lion's share of the performance gains. Thanks to the answerers, I found this to be a multifaceted learning experience. Your explanations really helped me understand what was going on.
Observe this:
select length(' '::character(1));
length
--------
0
(1 row)
A cause of this confusion is a bizarre definition of character type in SQL standard. From Postgres documentation for character types:
Values of type character are physically padded with spaces to the specified width n, and are stored and displayed that way. However, the padding spaces are treated as semantically insignificant. Trailing spaces are disregarded when comparing two values of type character, and they will be removed when converting a character value to one of the other string types.
So you should use string_to_array(_s, rpad(_sep,1)).
You had several constructs that probably did not do what you think they would.
Here is a largely simplified version of your function, that is also quite a bit faster:
CREATE OR REPLACE FUNCTION split(_string text, _sep text)
RETURNS TABLE (postition int, part text) AS
$BODY$
BEGIN
RETURN QUERY
WITH wbd AS (
SELECT (row_number() OVER ())::int AS idx
,word
,length(word) AS wordlength
FROM unnest(string_to_array(_string, _sep)) AS word
)
SELECT (sum(wordlength) OVER (ORDER BY idx))::int + idx - wordlength
,word -- AS part
FROM wbd;
END;
$BODY$ LANGUAGE plpgsql;
Explanation
Use another window function to sum up the word lengths. Faster, simpler and cleaner. This makes for most of the performance gain. A lot of sub-queries slow you down.
Use the data type text instead of character varying or even character(). character varying and character are awful types, mostly just there for compatibility with the SQL standard and historical reasons. There is hardly anything you can do with those that could not better be done with text. In the meantime #Tometzky has explained why character(1) was a particularly bad choice for the parameter type. I fixed that by using text instead.
As #Tometzky demonstrated, unnest(string_to_array(..)) is faster than regexp_split_to_table(..) - even if just a tiny bit for small strings like we use here (max. 512 characters). So I switched back to your original expression.
length() does the same as character_length().
In a query with only one table source (and no other possible naming conflicts) you might as well not table-qualify column names. Simplifies the code.
We need an integer value in the end, so I cast all numerical values (bigint in this case) to integer right away, so additions and subtractions are done with integer arithmetic which is generally fastest.
'value'::int is just shorter syntax for cast('value' as integer) and otherwise equivalent.
I found the answer, but I don't understand it.
The string_to_array(_s, _sep) function does not split with a non-varying character; even if I wrote it like so it would not work:
string_to_array(_s, cast(_sep as character_varying(1)))
BUT if I redefined the parameters as such:
drop function if exists split(character varying(512), character(1));
create function split(
_s character varying(512),
_sep character varying(1)
... all of a sudden it works as I expected. Dunno what to make of this, and really not the answer I wanted... now I have changed the signature of the function, which is not what I wanted to do.