Validating Phone Numbers in Batch with PostgreSQL - sql

This is my SQL:
SELECT
countries.locl_ctry_id,
countries.icc,
countries.active,
networks.locl_ntwrk_id,
networks.locl_ctry_id,
numberings.locl_ntwrk_id,
numberings.ndc,
numberings.size
FROM countries
LEFT JOIN networks
ON networks.locl_ctry_id = countries.locl_ctry_id
LEFT JOIN numberings
ON numberings.locl_ntwrk_id = networks.locl_ntwrk_id
WHERE
countries.active = 'true'
AND numberings.locl_ntwrk_id NOTNULL
AND CONCAT(countries.icc, numberings.ndc)
LIKE LEFT('381645554330', CHAR_LENGTH(CONCAT(countries.icc, numberings.ndc)))
AND LENGTH('381645554330') = numberings.size
I would like to run this script for a batch of numbers, for example:
381645554330 ‭
381629000814‬‬
381644446555‬
‭38975300155‬
‭38975604099 ‭
38976330923‬‬ ‭
38977772090‬ ‭
38978250177‬ ‭
38970333730‬
‭38971388262‬
‭38972228855‬
Take a look at the database structure here: http://sqlfiddle.com/#!17/13ce29/27
It needs to validate the Prefix as well as the Length of the number.
Any suggestions how to achieve this?

Put the batch of numbers in a union all subquery.
SELECT
countries.locl_ctry_id,
countries.icc,
countries.active,
networks.locl_ntwrk_id,
networks.locl_ctry_id,
numberings.locl_ntwrk_id,
numberings.ndc,
numberings.size
FROM countries
LEFT JOIN networks
ON networks.locl_ctry_id = countries.locl_ctry_id
LEFT JOIN numberings
ON numberings.locl_ntwrk_id = networks.locl_ntwrk_id
JOIN ( select '381645554330' as num
union all
select '38976330923‬‬‬‬'
union all
select '38975300155‬‬‬' ) batch_numbers
ON CONCAT(countries.icc, numberings.ndc)
LIKE LEFT(batch_numbers.num, CHAR_LENGTH(CONCAT(countries.icc, numberings.ndc)))
AND LENGTH(batch_numbers.num) = numberings.size
WHERE
countries.active = 'true'
AND numberings.locl_ntwrk_id NOTNULL

It seems the objective is not ability to return the set of values currently returned by the single, but to make an evaluation of the of multiple values. The issue with the above it requires an a-priori knowledge of and a modification to the query for each set to evaluate. The follow will attempt to remove that requirement.
Let's begin by developing a base line query as an extension to Jakup's "union" solution.
--- create a baseline solution
with to_be_validated (test_num) as -- CTE used strictly as data generator fir query
( values ('381645554330')
, ('381629000814')
, ('381644446555')
, ('38975300155')
, ('38975604099')
, ('38976330923')
, ('38977772090')
, ('38978250177')
, ('38970333730')
, ('38971388262')
, ('38972228855')
, ('81771388262')
, ('55572228855')
)
--- base query
select test_num
, case when icc is not null then 'Valid' else 'Invalid' end validation
from to_be_validated
left join(
select countries.icc, numberings.ndc, numberings.size
from countries
join networks on networks.locl_ctry_id = countries.locl_ctry_id
join numberings on numberings.locl_ntwrk_id = networks.locl_ntwrk_id
) base on ( concat(base.icc, base.ndc) = left( test_num, char_length(concat(base.icc, base.ndc)))
and length(test_num) = base.size
)
;
Notes on Query and Modifications:
1. the column countries.active is defined as binary, thus already providing a True/False value. Thus checking for "= 'true' is unnecessary. Altered to just contries.active.
2. The column numberings.locl_ntwrk_id is restricted to being NOT NULL, so the Predicate "nullnumberings.locl_ntwrk_id NOTNULL" is always true. Removed predicate.
3. The LEFT JOIN on networks and numberings will generate a result set with all countries, all networks, all numberings, even when the combination is itself invalid. This results in validating each phone number against every combination of the 3 base tables. Alter these inner joins.
4. Finally, I added a couple extra numbers to your test data. These are intended to fail the desired validation. You should always test with considerable invalid data, otherwise you cannot know if procedure/query/whatever gracefully and properly handles it.
Now with a base query in hand it's possible to just end here. But to be generally useful you cannot edit the query each time it wanted. Therefore lets wrap a function definition around it. We'll do this by wrapping a function definition around that base query, and provide either an array or a delimited string containing the phone numbers to b evaluated.
In each the base query remains the same, and we keep the CTE, but the CTE is modified to build a row for each phone number provided.
-- SQL Function with and Array input
create or replace function validate_phone_numbers( phone_numbers text[])
returns table ( phone_number text
, validation_status text
)
language sql
as $$
with to_be_validated as
( select unnest (phone_numbers) test_num )
-- Insert base query here --
$$
-- Test with Array
select phone_number, validation_status
from validate_phone_numbers (ARRAY
[ ('381629000814')
, ('381644446555')
, ('38975300155')
, ('38975604099')
, ('38976330923')
, ('38977772090')
, ('38978250177')
, ('38970333730')
, ('38971388262')
, ('38972228855')
, ('81771388262')
, ('55572228855')
]
) ;
With a minor extension we a delimited string version.
create or replace function validate_phone_numbers_with_string( phone_numbers text, delimiter text default ',')
returns table ( phone_number text
, validation_status text
)
language sql
as $$
with to_be_validated as
( select unnest (string_to_array (phone_numbers, delimiter)) test_num)
-- Insert base query here --
$$ ;
-- test with string
select phone_number, validation_status
from validate_phone_numbers_with_string('381629000814,381644446555,38975300155,38975604099,38976330923,38977772090,38978250177,38970333730,38971388262,38972228855,81771388262,55572228855');

Related

mismatched input 'from'. Expecting: ',', <expression>

I have a query that I am running on AWS athena that should return all the filenames that are not contained in the second table. I am basically trying to find all the filename that are not in ejpos landing table.
The one table looks like this (item sales):
origin_file
run_id
/datarite/ejpos/8023/20220706/filename1
8035
/datarite/ejpos/8023/20220706/filename2
8035
/datarite/ejpos/8023/20220706/filename3
8035
The other table looks like this (ejpos_files_landing):
filename
filename1
filename2
filename3
filename4
They don't have the same number of rows, hence I am trying to find the file names that are in ejpos_pos_landing but not in item sales table.
I get this error when I run:
mismatched input 'from'. Expecting: ',', <expression>
The query is here:
SELECT trim("/datarite/ejpos/8023/20220706/" from "validated"."datarite_ejpos_itemsale" where
run_id = '8035') as origin_file,
FROM "validated"."datarite_ejpos_itemsale"
LEFT JOIN "landing"."ejpos_landing_files" ON "landing"."ejpos_landing_files".filename =
"validated"."datarite_ejpos_itemsale".origin_file
WHERE "landing"."ejpos_landing_files".filename IS NULL;
The expected result would be:
|filename4|
Because it is not in the other table
Can anyone assist?
There is a lot of wrong stuff in your query based on the example data and declared goals.
trim("/datarite/ejpos/8023/20220706/" from "validated"."datarite_ejpos_itemsale" where run_id = '8035') as origin_file is not a valid sql.
ON "landing"."ejpos_landing_files".filename = "validated"."datarite_ejpos_itemsale".origin_file will not work cause origin_file is prefixed. You can use strpos if there should be only one instance of filename in the origin_file.
your join and filtering condition are build to find items present in datarite_ejpos_itemsale and missing in ejpos_landing_files while you state the vise versa is needed.
the mentioned in the comments extra comma
Try next:
-- sample data
WITH item_sales(origin_file, run_id) AS (
VALUES ('/datarite/ejpos/8023/20220706/filename1', 8035),
('/datarite/ejpos/8023/20220706/filename2', 8035),
('/datarite/ejpos/8023/20220706/filename3', 8035),
('/datarite/ejpos/8023/20220706/filename4', 8036)
),
ejpos_files_landing(filename) as(
VALUES ('filename1'),
('filename2'),
('filename3'),
('filename4')
)
-- query
select filename
from ejpos_files_landing l
left outer join item_sales s -- reverse the join
on strpos(s.origin_file, l.filename) >= 1 -- assuming that filename should be present only one time in the string
and s.run_id = 8035 -- if you need to filter out run id
where s.origin_file is null
Output:
filename
filename4
Alternative approach you can try:
-- query
select filename
from ejpos_files_landing l
where filename not in (
select element_at(split(origin_file, '/'), -1) -- split by '/' and get last
from item_sales
where run_id = 8035
)

Issue With SQL Pivot Function

I have a SQL query where I am trying to replace null results with zero. My code is producing an error
[1]: ORA-00923: FROM keyword not found where expected
I am using an Oracle Database.
Select service_sub_type_descr,
nvl('Single-occupancy',0) as 'Single-occupancy',
nvl('Multi-occupancy',0) as 'Multi-occupancy'
From
(select s.service_sub_type_descr as service_sub_type_descr, ch.claim_id,nvl(ci.item_paid_amt,0) as item_paid_amt
from table_1 ch, table_" ci, table_3 s, table_4 ppd
where ch.claim_id = ci.claim_id and ci.service_type_id = s.service_type_id
and ci.service_sub_type_id = s.service_sub_type_id and ch.policy_no = ppd.policy_no)
Pivot (
count(distinct claim_id), sum(item_paid_amt) as paid_amount For service_sub_type_descr IN ('Single-occupancy', 'Multi-occupancy')
)
This expression:
nvl('Single-occupancy',0) as 'Single-occupancy',
is using an Oracle bespoke function to say: If the value of the string Single-occupancy' is not null then return the number 0.
That logic doesn't really make sense. The string value is never null. And, the return value is sometimes a string and sometimes a number. This should generate a type-conversion error, because the first value cannot be converted to a number.
I think you intend:
coalesce("Single-occupancy", 0) as "Single-occupancy",
The double quotes are used to quote identifiers, so this refers to the column called Single-occupancy.
All that said, fix your data model. Don't have identifiers that need to be quoted. You might not have control in the source data but you definitely have control within your query:
coalesce("Single-occupancy", 0) as Single_occupancy,
EDIT:
Just write the query using conditional aggregation and proper JOINs:
select s.service_sub_type_descr, ch.claim_id,
sum(case when service_sub_type_descr = 'Single-occupancy' then item_paid_amt else 0 end) as single_occupancy,
sum(case when service_sub_type_descr = 'Multi-occupancy' then item_paid_amt else 0 end) as multi_occupancy
from table_1 ch join
table_" ci
on ch.claim_id = ci.claim_id join
table_3 s
on ci.service_type_id = s.service_type_id join
table_4 ppd
on ch.policy_no = ppd.policy_no
group by s.service_sub_type_descr, ch.claim_id;
Much simpler in my opinion.
for column aliases, you have to use double quotes !
don't use
as 'Single-occupancy'
but :
as "Single-occupancy",

Select rows from table with jsonb column based on arbitrary jsonb filter expression

Test data
DROP TABLE t;
CREATE TABLE t(_id serial PRIMARY KEY, data jsonb);
INSERT INTO t(data) VALUES
('{"a":1,"b":2, "c":3}')
, ('{"a":11,"b":12, "c":13}')
, ('{"a":21,"b":22, "c":23}')
Problem statement: I want to receive an arbitrary JSONB parameter which acts as a filter on column t.data, such as
{ "b":{ "from":0, "to":20 }, "c":13 }
and use this to select matching rows from my test table t.
In this example, I want rows where b is between 0 and 20 and c = 13.
No error is required if the filter specifies a "column" (or "tag") which does not exist in t.data - it just fails to find a match.
I've used numeric values for simplicity but would like an approach which generalises to text as well.
What I have tried so far. I looked at the containment approach, which works for equality conditions, but am stumped on a generic way of handling range conditions:
select * from t
where t.data#> '{"c":13}'::jsonb;
Background: This problem arose when building a generic table-preview page on a website (for Admin users).
The page displays a filter based on various columns in whichever table is selected for preview.
The filter is then passed to a function in Postgres DB which applies this dynamic filter condition to the table.
It returns a jsonb array of the rows matching the filter specified by the user.
This jsonb array is then used to populate the Preview resultset.
The columns which make up the filter may change.
My Postgres version is 9.6 - thanks.
if you want to parse { "b":{ "from":0, "to":20 }, "c":13 } you need a parser. It is out of scope of json functions, but you can write "generic" query using AND and OR to filter by such json, eg:
https://www.db-fiddle.com/f/jAPBQggG3p7CxqbKLMbPKw/0
with filt(f) as (values('{ "b":{ "from":0, "to":20 }, "c":13 }'::json))
select *
from t
join filt on
(f->'b'->>'from')::int < (data->>'b')::int
and
(f->'b'->>'to')::int > (data->>'b')::int
and
(data->>'c')::int = (f->>'c')::int
;
Thanks for the comments/suggestions.
I will definitely look at GraphQL when I have more time - I'm working under a tight deadline at the moment.
It seems the consensus is that a fully generic solution is not achievable without a parser.
However, I got a workable first draft - it's far from ideal but we can work with it. Any comments/improvements are welcome ...
Test data (expanded to include dates & text fields)
DROP TABLE t;
CREATE TABLE t(_id serial PRIMARY KEY, data jsonb);
INSERT INTO t(data) VALUES
('{"a":1,"b":2, "c":3, "d":"2018-03-10", "e":"2018-03-10", "f":"Blah blah" }')
, ('{"a":11,"b":12, "c":13, "d":"2018-03-14", "e":"2018-03-14", "f":"Howzat!"}')
, ('{"a":21,"b":22, "c":23, "d":"2018-03-14", "e":"2018-03-14", "f":"Blah blah"}')
First draft of code to apply a jsonb filter dynamically, but with restrictions on what syntax is supported.
Also, it just fails silently if the syntax supplied does not match what it expects.
Timestamp handling a bit kludgey, too.
-- Handle timestamp & text types as well as int
-- See is_timestamp(text) function at bottom
with cte as (
select t.data, f.filt, fk.key
from t
, ( values ('{ "a":11, "b":{ "from":0, "to":20 }, "c":13, "d":"2018-03-14", "e":{ "from":"2018-03-11", "to": "2018-03-14" }, "f":"Howzat!" }'::jsonb ) ) as f(filt) -- equiv to cross join
, lateral (select * from jsonb_each(f.filt)) as fk
)
select data, filt --, key, jsonb_typeof(filt->key), jsonb_typeof(filt->key->'from'), is_timestamp((filt->key)::text), is_timestamp((filt->key->'from')::text)
from cte
where
case when (filt->key->>'from') is null then
case jsonb_typeof(filt->key)
when 'number' then (data->>key)::numeric = (filt->>key)::numeric
when 'string' then
case is_timestamp( (filt->key)::text )
when true then (data->>key)::timestamp = (filt->>key)::timestamp
else (data->>key)::text = (filt->>key)::text
end
when 'boolean' then (data->>key)::boolean = (filt->>key)::boolean
else false
end
else
case jsonb_typeof(filt->key->'from')
when 'number' then (data->>key)::numeric between (filt->key->>'from')::numeric and (filt->key->>'to')::numeric
when 'string' then
case is_timestamp( (filt->key->'from')::text )
when true then (data->>key)::timestamp between (filt->key->>'from')::timestamp and (filt->key->>'to')::timestamp
else (data->>key)::text between (filt->key->>'from')::text and (filt->key->>'to')::text
end
when 'boolean' then false
else false
end
end
group by data, filt
having count(*) = ( select count(distinct key) from cte ) -- must match on all filter elements
;
create or replace function is_timestamp(s text) returns boolean as $$
begin
perform s::timestamp;
return true;
exception when others then
return false;
end;
$$ strict language plpgsql immutable;

Error in select statement, with union all in a subquery

In Oracle 11g, I came across an error for a query and cannot figure why it is erroring on me. Here is the query:
select
main_data.issue_number,
main_data.transaction_number
from
(
select
p1.payment_date,
p1.media_number,
p1.payment_amount,
p1.issue_number,
p1.advice_na_number,
name.name_address_line_1,
name.name_address_line_2,
name.name_address_line_3,
name.name_address_line_4,
name.name_address_line_5,
name.name_address_line_6,
name.name_address_line_7,
name.name_address_city,
name.state_code,
name.address_country_code,
name.zip_code,
name.tax_id_number,
p1.output_tx_number_prin,
p1.output_tx_number_int,
'' as "transaction_number",
p1header.check_account_number
from
p1
left join name on p1.name_address_number = name.name_address_number
left join p1header on p1.issue_number = p1header.issue_number
UNION ALL
select
check.date_of_payment,
check.media_number,
check.payment_amount,
check.issue_number,
check.payee_na_number,
name.name_address_line_1,
name.name_address_line_2,
name.name_address_line_3,
name.name_address_line_4,
name.name_address_line_5,
name.name_address_line_6,
name.name_address_line_7,
name.name_address_city,
name.state_code,
name.address_country_code,
name.zip_code,
name.tax_id_number,
'' as "output_tx_number_prin",
'' as "output_tx_number_int",
check.transaction_number,
check.dda_number as "check_account_number"
from check
left join name on check.payee_na_number = name.name_address_number
) main_data
Selecting individual fields like above will give me an "invalid identifier error". If I do select * then it gives me back the data without any error. What am I doing wrong here? Thank you.
The old quoted identifier problem... see point 9 in the database object naming documentation, and note that Oracle does not recommend using quoted identifiers.
You've put your column alias as lower case inside double-quotes. That means that any references to it also have to be quoted and exactly match the case. So this would work:
select
main_data.issue_number,
main_data."transaction_number"
from
...
But unless you have a burning need to have that alias like that - and I doubt you do as all the identifier names from the actual table columns are not quoted - it would be simpler to remove the double quotes from the inner selects:
select
main_data.issue_number,
main_data.transaction_number
from
(
select
...
'' as transaction_number,
p1header.check_account_number
...
UNION ALL
select
...
'' as output_tx_number_prin,
'' as output_tx_number_int,
check.transaction_number,
check.dda_number as check_account_number
...
You don't actually need to alias the columns in the second branch of the union; the column identifiers will all be taken from the first branch.

PLSQL - Select works but Select within Where Clause returns no data

This is driving me crazy. I want to do simple comparison of a column and a variable but it just doesn't work. The QUERY 1 in following code returns me my value when i do a simple select, but i use the resulting variable in my 2nd query it just doesn't work..
It looks sooooo simple but I've been working on this for hours. The complete sql proc is
The big confusing thing is that if I replace v_bbg_symbol with some hard coded 'Value' (like 'FEDL01') it gives a correct answer for Query 2, but when I use the variable v_bbg_symbol it just doesn't work any more
Declare
v_bbg_symbol VARCHAR2(50);
V_OLD_INS_NAME Varchar2(50);
Begin
--QUERY 1
SELECT BBG_SYMBOL into v_bbg_symbol FROM quotes_external WHERE ID = 1;
--Gives output - 'FEDL01'
DBMS_OUTPUT.PUT_LINE('I got here:'||v_bbg_symbol||' is my value');
-QUERY 2
SELECT NAME INTO V_OLD_INS_NAME FROM INSTRUMENT
JOIN CURVE_INSTRUMENT ON
INSTRUMENT.INSTRUMENT_ID = CURVE_INSTRUMENT.INSTRUMENT_ID
JOIN GENERIC_INSTRUMENT ON
CURVE_INSTRUMENT.GENERIC_INSTRUMENT_ID = GENERIC_INSTRUMENT.GENERIC_INSTRUMENT_ID
WHERE CURVE_INSTRUMENT.CURVE_SNAPSHOT_ID =
(SELECT MAX(CURVE_INSTRUMENT.CURVE_SNAPSHOT_ID) FROM CURVE_INSTRUMENT)
AND GENERIC_INSTRUMENT.INSTRUMENT_NAME = v_bbg_symbol;
--ORACLE ERROR 'No Data Found'
DBMS_OUTPUT.PUT_LINE('I got here:'||V_OLD_INS_NAME||' is the new value');
END;
The first 'SELECT' gives me value which i select INTO a variable 'v_bbg_symbol', but when I use the same variable 'v_bbg_symbol' in my 2nd QUERY it pretends as if there is no value passed and does not return any result. If I give static value of 'v_bbg_symbol' i.e. ('FEDL01' in this case) in my 2nd QUERY, the results come as expected.
Please help..
Here is your query, with table aliases to facilitate following it:
SELECT NAME INTO V_OLD_INS_NAME
FROM INSTRUMENT i JOIN
CURVE_INSTRUMENT ci
ON i.INSTRUMENT_ID = ci.INSTRUMENT_ID JOIN
GENERIC_INSTRUMENT gi
ON ci.GENERIC_INSTRUMENT_ID = gi.GENERIC_INSTRUMENT_ID
WHERE ci.CURVE_SNAPSHOT_ID = (SELECT MAX(ci.CURVE_SNAPSHOT_ID) FROM CURVE_INSTRUMENT ci) and
gi.INSTRUMENT_NAME = v_bbg_symbol;
What this says is that the maximum ci.curve_snapshot_id is not for the instrument that is associated with v_bbg_symbol. I think you want a correlated subquery:
SELECT NAME INTO V_OLD_INS_NAME
FROM INSTRUMENT i JOIN
CURVE_INSTRUMENT ci
ON i.INSTRUMENT_ID = ci.INSTRUMENT_ID JOIN
GENERIC_INSTRUMENT gi
ON ci.GENERIC_INSTRUMENT_ID = gi.GENERIC_INSTRUMENT_ID
WHERE ci.CURVE_SNAPSHOT_ID = (SELECT MAX(ci2.CURVE_SNAPSHOT_ID)
FROM CURVE_INSTRUMENT ci2
WHERE ci2.instrument_id = i.instrument_id
) and
gi.INSTRUMENT_NAME = v_bbg_symbol;