Related
I have a column of values where some values contain brackets with text which I would like to remove. This is an example of what I have and what I want:
CREATE TABLE test
(column_i_have varchar(50),
column_i_want varchar(50))
INSERT INTO test (column_i_have, column_i_want)
VALUES ('hospital (PWD)', 'hopistal'),
('nursing (LLC)','nursing'),
('longterm (AT)', 'longterm'),
('inpatient', 'inpatient')
I have only come across approaches that use the number of characters or the position to trim the string, but these values have varying lengths. One way I was thinking was something like:
TRIM('(*',col1)
Doesn't work. Is there a way to do this in postgres SQL without using the position? THANK YOU!
If all the values contain "valid" brackets, then you may use split_part function without any regular expressions:
select
test.*,
trim(split_part(column_i_have, '(', 1)) as res
from test
column_i_have | column_i_want | res
:------------- | :------------ | :--------
hospital (PWD) | hopistal | hospital
nursing (LLC) | nursing | nursing
longterm (AT) | longterm | longterm
inpatient | inpatient | inpatient
db<>fiddle here
You can replace partial patterns using regular expressions. For example:
select *, regexp_replace(v, '\([^\)]*\)', '', 'g') as r
from (
select '''hospital (PWD)'', ''nursing (LLC)'', ''longterm (AT)'', ''inpatient''' as v
) x
Result:
r
-------------------------------------------------
'hospital ', 'nursing ', 'longterm ', 'inpatient'
See example at db<>fiddle.
Could it be as easy as:
SELECT SUBSTRING(column_i_have, '\w+') AS column_i_want FROM test
See demo
If not, and you still want to use SUBSTRING() to get upto but exclude paranthesis, then maybe:
SELECT SUBSTRING(column_i_have, '^(.+?)(?:\s*\(.*)?$') AS column_i_want FROM test
See demo
But if you really are looking upto the opening paranthesis, then maybe just use SPLIT_PART():
SELECT SPLIT_PART(column_i_have, ' (', 1) AS column_i_want FROM test
See demo
I have a column "A" which contains numbers for example- 0001, 0002, 0003
the same column "A" also contains some alphabets and special characters in some of the rows for example - connn, cco*jjj, hhhhhh11111 etc.
I want to replace these alphabets and special characters rows with blank values and only want to keep the rows containing the number.
which regex expression I can use here?
If you want to extract numbers from these values (even if they end or start with non digits), you may use something like this:
create table testing ( A varchar ) as select *
from values ('123'),('aaa123'),('3343'),('aaaa');
select REGEXP_SUBSTR( A, '\\D*(\\d+)\\D*', 1, 1, 'e', 1 ) res
from testing;
+------+
| RES |
+------+
| 123 |
| 123 |
| 3343 |
| NULL |
+------+
I understand that you want to set to null all values that do not contain digits only.
If so, you can use try_to_decimal():
update mytable
set a = null
where a try_to_decimal(a) is null
Or a regexp match:
update mytable
set a = null
where a rlike '[^0-9]'
This question already has answers here:
Why doesn't a non-greedy quantifier sometimes work in Oracle regex?
(4 answers)
Closed 5 years ago.
I'm having some issues dealing with the non-greedy regex operator in Oracle.
This seems to work:
select regexp_replace('abcc', '^ab.*?c', 'Z') from dual;
-- output: Zc (does not show greedy behavior)
while this does not:
select regexp_replace('abc:"123", def:"456", hji="789", dasdjaoijdsa', '(^.*def:")(.*?)(".*$)', '\2') from dual;
-- output: 456", hji="789 (shows greedy behavior)
-- I would expect 456 as output.
Is there something glaringly obvious that I may be missing here?
Thanks
You can use a non-greedy regular expression in REGEXP_SUBSTR:
SELECT REGEXP_SUBSTR(
'abc:"123", def:"456", hji="789", dasdjaoijdsa', -- input
'def:"(.*?)"', -- pattern
1, -- start character
1, -- occurrence
NULL, -- flags
1 -- capture group
) AS def
FROM DUAL;
Results:
| DEF |
|-----|
| 456 |
If you want to skip escaped quotation marks then you can use:
SELECT REGEXP_SUBSTR(
'abc:"123", def:"456\"Test\"", hji="789", dasdjaoijdsa',
'def:"((\\"|[^"])*)"',
1,
1,
NULL,
1
) AS def
FROM DUAL;
Results:
| DEF |
|-------------|
| 456\"Test\" |
Update:
You can get your query to work by making the first wild-card match non-greedy:
select regexp_replace(
'abc:"123", def:"456", hji="789", dasdjaoijdsa',
'(^.*?def:")(.*?)(".*$)',
'\2'
) AS def
FROM DUAL;
Results:
| DEF |
|-----|
| 456 |
I don't know exactly why your regex replace is failing, but I can offer a version of your query which is working:
select
regexp_replace('abc:"123", def:"456", hji="789", dasdjaoijdsa',
'^(.*def:")([^"]*).*',
'\2') from dual
The only explanation I have is that lazy dot isn't working, at least not in the context of the capture group. When I switch ([^"]*) above to (.*?), the query will fail.
Demo
I have a requirement to format phone numbers in the following way:
No spaces
No special characters
Remove preceding zero - if area code exists
Remove country code if present e.g. +44
For instance this: (03069) 990927 would become: 3069990927.
So far I have come up this this:
replace(replace(replace(replace(replace(replace(substr(replace(ltrim([VALUE],0), ' ', ''),nvl(length(substr(replace(ltrim([VALUE],0), ' ', ''),11)),0)+1), '-', ''), '(', ''), ')', ''),'/', ''), '.', ''), '+', '')
Is there a shorter version of this, maybe using a regular expression?
The final version of this snippet will become a column in a view that will return the following columns:
Customer Number
Customer Name
Country
Formatted Phone Number
The formatted phone number will be concatenated with the international dial code (e.g. +44) that are saved in the database in a table - DIALCODE_TAB(COUNTRY_CODE, CODE). Below is an example using the replace syntax above:
CREATE OR REPLACE FORCE VIEW "CUST_PHONE" ("CUSTOMER_ID", "NAME", "COUNTRY", "PHONE_NUMBER") AS
select
cicm.customer_id,
cicm.name,
dct.country,
dct.code || replace(replace(replace(replace(replace(replace(substr(replace(ltrim(cicm.value,0), ' ', ''),nvl(length(substr(replace(ltrim(cicm.value,0), ' ', ''),11)),0)+1), '-', ''), '(', ''), ')', ''),'/', ''), '.', ''), '+', '') phone_number
from customer_info_comm_method cicm
join dialcode_tab dct
on dct.country_code = customer_info_api.get_country_code(cicm.customer_id)
where cicm.method_id_db = 'PHONE'
--and dct.code || replace(replace(replace(replace(replace(replace(substr(replace(ltrim(cicm.value,0), ' ', ''),nvl(length(substr(replace(ltrim(cicm.value,0), ' ', ''),11)),0)+1), '-', ''), '(', ''), ')', ''),'/', ''), '.', ''), '+', '') = [phone_number]
--in terms of performance this SQL has to be written so that it returns all the records or a specific record when searching for the phone number - very quickly (<10s).
WITH read only;
N.B. A customer record can have more than 1 phone number and the same phone number can exist on more than 1 customer record.
To begin with a remark: This only works if the country is stored elsewhere for the record and there are no telephone numbers without an area code. Otherwise one would not be able to reconstruct the complete phone number again.
Then: How are country codes represented in your data? Is it always +44 or can it be 0044? Be careful here. Especially don't remove a single zero (assuming it's an area code), when it's actually the first of two zeros representing the country code :-)
Then: You need a list of all country codes. Let's take for example +1441441441. Where does the country code end? (Solution: +1441 is Bermudas.)
As to "no spaces" and "no special characters" you can solve this best with regexp_replace.
So all in all not so simple a task as you obviously expected it to be. (But not too hard to do either.)
I would use PL/SQL for this.
Hope my hints help you. Good luck.
EDIT: Here is what is needed. I still think a PL/SQL function will be best here.
Make sure your DIALCODE_TAB contains all country codes necessary.
1. Trim the phone number.
2. Then check if its starts with a country identifyer (+, 00).
2.1. If so: remove that. Remove all non-digits. Look up the country code in your table and remove it.
2.2. If not so: check if it starts with an area identifyer (0).
2.2.1. If so: remove it.
2.2.2. In any case: remove all non-digits.
That should do it, provided the numbers are valid. In Germany sometimes people write +49(0)40-123456, which is not valid, because one either uses a country code or an area code, not both in the same number. The (0) would have to be removed to make the number valid.
SELECT LTRIM(REGEXP_REPLACE(
REGEXP_REPLACE('+44(03069) 990927',
'(\+).([[:digit:]])+'), -- to strip off country code
'[^[:alnum:]]'),-- Strip off non-aplanumeric [:digit] if only digit
'0') -- Remove preceding Zero
FROM DUAL;
Wont work for +44990927 (If country code ends without any space or something or country didnt start with +)
SQL Fiddle
Oracle 11g R2 Schema Setup:
CREATE TABLE phone_numbers ( phone_number ) AS
SELECT '(03069) 990927' FROM DUAL
UNION ALL SELECT '+44 1234 567890' FROM DUAL
UNION ALL SELECT '+44(0)1234 567890' FROM DUAL
UNION ALL SELECT '+44(012) 34-567-890' FROM DUAL
UNION ALL SELECT '+44-1234-567-890' FROM DUAL
UNION ALL SELECT '+358-1234567890' FROM DUAL;
Query 1:
If you are just dealing with +44 international dialling codes then you could:
use ^\+44|\D to strip the +44 international code and all non-digit characters; then
use ^0 to strip a leading zero if its present.
Like this:
SELECT REGEXP_REPLACE(
REGEXP_REPLACE(
phone_number,
'^\+44|\D',
''
),
'^0', '' ) AS phone_number
FROM phone_numbers
Results:
| PHONE_NUMBER |
|---------------|
| 3069990927 |
| 1234567890 |
| 1234567890 |
| 1234567890 |
| 1234567890 |
| 3581234567890 |
(You can see it doesn't work for the final number with a +358 international code.)
Query 2:
This can be simplified into a single regular expression (that's slightly less readable):
SELECT REGEXP_REPLACE(
phone_number,
'^(\+44)?\D*0?|\D',
''
) AS phone_number
FROM phone_numbers
Results:
| PHONE_NUMBER |
|---------------|
| 3069990927 |
| 1234567890 |
| 1234567890 |
| 1234567890 |
| 1234567890 |
| 3581234567890 |
Query 3:
If you want to deal with multiple international dialling codes then you will need to know which ones are valid (see http://en.wikipedia.org/wiki/List_of_country_calling_codes for a list).
This is an example of a regular expression which will strip out valid international dialling codes beginning with +3, +4 or +5 (I'll leave all the other dialling codes for you to code up):
SELECT REGEXP_REPLACE(
phone_number,
'^(\+(3[0123469]|3[57]\d|38[01256789]|4[013456789]|42[013]|5[09]\d|5[12345678]))?\D*0?|\D',
''
) AS phone_number
FROM phone_numbers
Results:
| PHONE_NUMBER |
|--------------|
| 3069990927 |
| 1234567890 |
| 1234567890 |
| 1234567890 |
| 1234567890 |
| 1234567890 |
If the + at the start of the international dialling code is optional then just replace \+ (near the start of the regular expression) with \+?.
Following up on this question by Sivaram Chintalapudi, I'm interested in whether it's practical in PostgreSQL to do natural - or "humanized" - sorting " of strings that contain a mixture of multi-digit numbers and words/letters. There is no fixed pattern of words and numbers in the strings, and there may be more than one multi-digit number in a string.
The only place I've seen this done routinely is in the Mac OS's Finder, which sorts filenames containing mixed numbers and words naturally, placing "20" after "3", not before it.
The collation order desired would be produced by an algorithm that split each string into blocks at letter-number boundaries, then ordered each part, treating letter-blocks with normal collation and number-blocks as integers for collation purposes. So:
'AAA2fred' would become ('AAA',2,'fred') and 'AAA10bob' would become ('AAA',10,'bob'). These can then be sorted as desired:
regress=# WITH dat AS ( VALUES ('AAA',2,'fred'), ('AAA',10,'bob') )
regress-# SELECT dat FROM dat ORDER BY dat;
dat
--------------
(AAA,2,fred)
(AAA,10,bob)
(2 rows)
as compared to the usual string collation ordering:
regress=# WITH dat AS ( VALUES ('AAA2fred'), ('AAA10bob') )
regress-# SELECT dat FROM dat ORDER BY dat;
dat
------------
(AAA10bob)
(AAA2fred)
(2 rows)
However, the record comparison approach doesn't generalize because Pg won't compare ROW(..) constructs or records of unequal numbers of entries.
Given the sample data in this SQLFiddle the default en_AU.UTF-8 collation produces the ordering:
1A, 10A, 2A, AAA10B, AAA11B, AAA1BB, AAA20B, AAA21B, X10C10, X10C2, X1C1, X1C10, X1C3, X1C30, X1C4, X2C1
but I want:
1A, 2A, 10A, AAA1BB, AAA10B, AAA11B, AAA20B, AAA21B, X1C1, X1C3, X1C4, X1C10, X1C30, X2C1, X10C10, X10C2
I'm working with PostgreSQL 9.1 at the moment, but 9.2-only suggestions would be fine. I'm interested in advice on how to achieve an efficient string-splitting method, and how to then compare the resulting split data in the alternating string-then-number collation described. Or, of course, on entirely different and better approaches that don't require splitting strings.
PostgreSQL doesn't seem to support comparator functions, otherwise this could be done fairly easily with a recursive comparator and something like ORDER USING comparator_fn and a comparator(text,text) function. Alas, that syntax is imaginary.
Update: Blog post on the topic.
Building on your test data, but this works with arbitrary data. This works with any number of elements in the string.
Register a composite type made up of one text and one integer value once per database. I call it ai:
CREATE TYPE ai AS (a text, i int);
The trick is to form an array of ai from each value in the column.
regexp_matches() with the pattern (\D*)(\d*) and the g option returns one row for every combination of letters and numbers. Plus one irrelevant dangling row with two empty strings '{"",""}' Filtering or suppressing it would just add cost. Aggregate this into an array, after replacing empty strings ('') with 0 in the integer component (as '' cannot be cast to integer).
NULL values sort first - or you have to special case them - or use the whole shebang in a STRICT function like #Craig proposes.
Postgres 9.4 or later
SELECT data
FROM alnum
ORDER BY ARRAY(SELECT ROW(x[1], CASE x[2] WHEN '' THEN '0' ELSE x[2] END)::ai
FROM regexp_matches(data, '(\D*)(\d*)', 'g') x)
, data;
db<>fiddle here
Postgres 9.1 (original answer)
Tested with PostgreSQL 9.1.5, where regexp_replace() had a slightly different behavior.
SELECT data
FROM (
SELECT ctid, data, regexp_matches(data, '(\D*)(\d*)', 'g') AS x
FROM alnum
) x
GROUP BY ctid, data -- ctid as stand-in for a missing pk
ORDER BY regexp_replace (left(data, 1), '[0-9]', '0')
, array_agg(ROW(x[1], CASE x[2] WHEN '' THEN '0' ELSE x[2] END)::ai)
, data -- for special case of trailing 0
Add regexp_replace (left(data, 1), '[1-9]', '0') as first ORDER BY item to take care of leading digits and empty strings.
If special characters like {}()"', can occur, you'd have to escape those accordingly.
#Craig's suggestion to use a ROW expression takes care of that.
BTW, this won't execute in sqlfiddle, but it does in my db cluster. JDBC is not up to it. sqlfiddle complains:
Method org.postgresql.jdbc3.Jdbc3Array.getArrayImpl(long,int,Map) is
not yet implemented.
This has since been fixed: http://sqlfiddle.com/#!17/fad6e/1
I faced this same problem, and I wanted to wrap the solution in a function so I could re-use it easily. I created the following function to achieve a 'human style' sort order in Postgres.
CREATE OR REPLACE FUNCTION human_sort(text)
RETURNS text[] AS
$BODY$
/* Split the input text into contiguous chunks where no numbers appear,
and contiguous chunks of only numbers. For the numbers, add leading
zeros to 20 digits, so we can use one text array, but sort the
numbers as if they were big integers.
For example, human_sort('Run 12 Miles') gives
{'Run ', '00000000000000000012', ' Miles'}
*/
select array_agg(
case
when a.match_array[1]::text is not null
then a.match_array[1]::text
else lpad(a.match_array[2]::text, 20::int, '0'::text)::text
end::text)
from (
select regexp_matches(
case when $1 = '' then null else $1 end, E'(\\D+)|(\\d+)', 'g'
) AS match_array
) AS a
$BODY$
LANGUAGE sql IMMUTABLE;
tested to work on Postgres 8.3.18 and 9.3.5
No recursion, should be faster than recursive solutions
Can be used in just the order by clause, don't have to deal with primary key or ctid
Works for any select (don't even need a PK or ctid)
Simpler than some other solutions, should be easier to extend and maintain
Suitable for use in a functional index to improve performance
Works on Postgres v8.3 or higher
Allows an unlimited number of text/number alternations in the input
Uses just one regex, should be faster than versions with multiple regexes
Numbers longer than 20 digits are ordered by their first 20 digits
Here's an example usage:
select * from (values
('Books 1', 9),
('Book 20 Chapter 1', 8),
('Book 3 Suffix 1', 7),
('Book 3 Chapter 20', 6),
('Book 3 Chapter 2', 5),
('Book 3 Chapter 1', 4),
('Book 1 Chapter 20', 3),
('Book 1 Chapter 3', 2),
('Book 1 Chapter 1', 1),
('', 0),
(null::text, 0)
) as a(name, sort)
order by human_sort(a.name)
-----------------------------
|name | sort |
-----------------------------
| | 0 |
| | 0 |
|Book 1 Chapter 1 | 1 |
|Book 1 Chapter 3 | 2 |
|Book 1 Chapter 20 | 3 |
|Book 3 Chapter 1 | 4 |
|Book 3 Chapter 2 | 5 |
|Book 3 Chapter 20 | 6 |
|Book 3 Suffix 1 | 7 |
|Book 20 Chapter 1 | 8 |
|Books 1 | 9 |
-----------------------------
Adding this answer late because it looked like everyone else was unwrapping into arrays or some such. Seemed excessive.
CREATE FUNCTION rr(text,int) RETURNS text AS $$
SELECT regexp_replace(
regexp_replace($1, '[0-9]+', repeat('0',$2) || '\&', 'g'),
'[0-9]*([0-9]{' || $2 || '})',
'\1',
'g'
)
$$ LANGUAGE sql;
SELECT t,rr(t,9) FROM mixed ORDER BY t;
t | rr
--------------+-----------------------------
AAA02free | AAA000000002free
AAA10bob | AAA000000010bob
AAA2bbb03boo | AAA000000002bbb000000003boo
AAA2bbb3baa | AAA000000002bbb000000003baa
AAA2fred | AAA000000002fred
(5 rows)
(reverse-i-search)`OD': SELECT crypt('richpass','$2$08$aJ9ko0uKa^C1krIbdValZ.dUH8D0R0dj8mqte0Xw2FjImP5B86ugC');
richardh=>
richardh=> SELECT t,rr(t,9) FROM mixed ORDER BY rr(t,9);
t | rr
--------------+-----------------------------
AAA2bbb3baa | AAA000000002bbb000000003baa
AAA2bbb03boo | AAA000000002bbb000000003boo
AAA2fred | AAA000000002fred
AAA02free | AAA000000002free
AAA10bob | AAA000000010bob
(5 rows)
I'm not claiming two regexps are the most efficient way to do this, but rr() is immutable (for fixed length) so you can index it. Oh - this is 9.1
Of course, with plperl you could just evaluate the replacement to pad/trim it in one go. But then with perl you've always got just-one-more-option (TM) than any other approach :-)
The following function splits a string into an array of (word,number) pairs of arbitrary length. If the string begins with a number then the first entry will have a NULL word.
CREATE TYPE alnumpair AS (wordpart text,numpart integer);
CREATE OR REPLACE FUNCTION regexp_split_numstring_depth_pairs(instr text)
RETURNS alnumpair[] AS $$
WITH x(match) AS (SELECT regexp_matches($1, '(\D*)(\d+)(.*)'))
SELECT
ARRAY[(CASE WHEN match[1] = '' THEN '0' ELSE match[1] END, match[2])::alnumpair] || (CASE
WHEN match[3] = '' THEN
ARRAY[]::alnumpair[]
ELSE
regexp_split_numstring_depth_pairs(match[3])
END)
FROM x;$$ LANGUAGE 'sql' IMMUTABLE;
allowing PostgreSQL's composite type sorting to come into play:
SELECT data FROM alnum ORDER BY regexp_split_numstring_depth_pairs(data);
and producing the expected result, as per this SQLFiddle. I've adopted Erwin's substitution of 0 for the empty string in all strings beginning with a number so that numbers sort first; it's cleaner than using ORDER BY left(data,1), regexp_split_numstring_depth_pairs(data).
While the function is probably horrifically slow it can at least be used in an expression index.
That was fun!
create table dat(val text)
insert into dat ( VALUES ('BBB0adam'), ('AAA10fred'), ('AAA2fred'), ('AAA2bob') );
select
array_agg( case when z.x[1] ~ E'\\d' then lpad(z.x[1],10,'0') else z.x[1] end ) alnum_key
from (
SELECT ctid, regexp_matches(dat.val, E'(\\D+|\\d+)','g') as x
from dat
) z
group by z.ctid
order by alnum_key;
alnum_key
-----------------------
{AAA,0000000002,bob}
{AAA,0000000002,fred}
{AAA,0000000010,fred}
{BBB,0000000000,adam}
Worked on this for almost an hour and posted without looking -- I see Erwin arrived at a similar place. Ran into the same "could not find array type for data type text[]" trouble as #Clodoaldo. Had a lot of trouble getting the cleanup exercise to not agg all the rows until I thought of grouping by the ctid (which feels like cheating really -- and doesn't work on a psuedo table as in the OP example WITH dat AS ( VALUES ('AAA2fred'), ('AAA10bob') )
...). It would be nicer if array_agg could accept a set-producing subselect.
I'm not a RegEx guru, but I can work it to some extent. Enough to produce this answer.
It will handle up to 2 numeric values within the content. I don't think OSX goes further than that, if it even handles 2.
WITH parted AS (
select data,
substring(data from '([A-Za-z]+).*') part1,
substring('a'||data from '[A-Za-z]+([0-9]+).*') part2,
substring('a'||data from '[A-Za-z]+[0-9]+([A-Za-z]+).*') part3,
substring('a'||data from '[A-Za-z]+[0-9]+[A-Za-z]+([0-9]+).*') part4
from alnum
)
select data
from parted
order by part1,
cast(part2 as int),
part3,
cast(part4 as int),
data;
SQLFiddle
The following solution is a combination of various ideas presented in other answers, as well as some ideas from the classic solution:
create function natsort(s text) returns text immutable language sql as $$
select string_agg(r[1] || E'\x01' || lpad(r[2], 20, '0'), '')
from regexp_matches(s, '(\D*)(\d*)', 'g') r;
$$;
The design goals of this function were simplicity and pure string operations (no custom types and no arrays), so it can easily be used as a drop-in solution, and is trivial to be indexed over.
Note: If you expect numbers with more than 20 digits, you'll have to replace the hard-coded maximum length 20 in the function with a suitable larger length. Note that this will directly affect the length of the resulting strings, so don't make that value larger than needed.