I try until couple of hours to fetch all the row from a view, created from multiple tables.
I have two tables (Position, and Vector) to which correspond respectively two custom composite types , type_position and type_vector
Position (id, sys_time, lat, lon)
Vector (id, sys_time, speed)
I want to create a temporary VIEW in plpgsql procedure to bring together all Position and Vector and order them by sys_time so i wrote :
CREATE TEMP VIEW posAndVectView AS
SELECT * from position
UNION ALL
SELECT * from vector;
I need to iterate over this view and made some work depends on Position and Vector attributes.
So i think i should use a cursor :
DECLARE
manyRows refcursor;
BEGIN
OPEN manyRows FOR
SELECT * FROM posAndVectView ORDER BY sys_time ASC;
LOOP
FETCH manyRows INTO posOrVect;
EXIT WHEN NOT FOUND;
...
END LOOP;
CLOSE manyRows;
So my question is what should the type of the posOrVect variable be in the DECLARE section ?
It seems to be sometimes a type_vector, sometimes a type_position...
I found the answer it is pretty simple.
I need to declare the posOrVect variable with the record type as its mentionned in the PostgreSQL documentation :
Record variables are similar to row-type variables, but they have no
predefined structure. They take on the actual row structure of the row
they are assigned during a SELECT or FOR command. The substructure of
a record variable can change each time it is assigned to. A
consequence of this is that until a record variable is first assigned
to, it has no substructure, and any attempt to access a field in it
will draw a run-time error.
Simple. The type should be posAndVectView.
Like for any other table, there is also a composite type with the same name as a temporary table.
But do you need a view? You could just open the cursor for the query with the UNION. In that case, you'd use type position because it is the first table.
Related
I have two tables with data. I need to set data building_h from one table to another with respect to the spatial relationships between data in these tables. Spatial data in the source table is polygon type and in destination table - points. I made a query for points and polygons which intersect. It works nice. But there are some points that do not intersect with polygons. My idea is to define for each of these points the closest polygon and take the building_h value from it. For this purpose, I wrote the function which takes the id of the point and returns the building_h value. And it works ok in the test.
CREATE OR REPLACE FUNCTION closest_pol(int4)
RETURNS NUMERIC
AS
$$
DECLARE
retVal NUMERIC;
BEGIN
SELECT bgs.building_h INTO retVal
FROM buildings_geoalert_spgg bgs, building_from_landuse_spgg bfl
WHERE ST_INTERSECTS(bgs.geom, ST_BUFFER(bfl.geom_centr, 0.0006,'quad_segs=8')) AND bfl.id = $1
ORDER BY ST_INTERSECTION(bgs.geom, ST_BUFFER(bfl.geom_centr, 0.0006,'quad_segs=8')) ASC
LIMIT 1;
RETURN retVal;
END;
$$
LANGUAGE plpgsql
STABLE
RETURNS NULL ON NULL INPUT;
But when I run the query for the whole table, it executes forever.. (for ~3000 rows with NULL value). It lasted several hours before I stopped it.
UPDATE building_from_landuse_spgg AS bfl SET
building_h = (SELECT closest_pol(bfl.id))
WHERE bfl.building_h IS NULL;
Do you have an idea what I did wrong?
In your WHERE clause you're using ST_Intersects with 0.006 buffer that is created in query time. Consider using a partial gist index with this buffer of yours (if for whatever reason you must use them):
CREATE INDEX idx_buffer ON building_from_landuse_spgg USING gist (ST_Buffer(geom_centr, 0.0006,'quad_segs=8'));
I'd argue that you do not need a function for that, since you could use the query inside of the function as a subquery in your UPDATE statement. But if you have your reasons to stick to the function, you can get rid of the SELECT to call it:
UPDATE building_from_landuse_spgg
SET building_h = closest_pol(id)
WHERE building_h IS NULL;
EDIT: As correctly mentioned by #JGH (see comments), using buffers is less efficient than simply using ST_DWithin. So, if you can afford it, create an index in the geometries used in the WHERE clause ..
CREATE INDEX idx_landuse_geom_centr ON building_from_landuse_spgg USING gist (geom_centr);
CREATE INDEX idx_geoalert_geom ON buildings_geoalert_spgg USING gist (geom);
.. and also partially indexing building_h would speed up things a bit, since you're only interested in the NULL records:
CREATE INDEX idx_landuse_building_h ON building_from_landuse_spgg (building_h)
WHERE building_h IS NULL;
Or if you prefer a broader index but still putting NULL values in the first class ..
CREATE INDEX idx_landuse_building_h ON building_from_landuse_spgg
(building_h NULLS FIRST);
And perhaps consider putting the code of your function in a subquery, e.g.
UPDATE building_from_landuse_spgg AS bfl
SET building_h = (
SELECT bgs.building_h
FROM buildings_geoalert_spgg bgs
WHERE ST_DWithin(bgs.geom, bfl.geom_centr, 0.0006)
ORDER BY ST_Distance(bgs.geom, bfl.geom_centr) ASC LIMIT 1)
WHERE bfl.building_h IS NULL;
Further reading:
Getting all Buildings in range of 5 miles from specified coordinates
ST_Distance
My SQL is a bit rusty, so I don't know whether the following is even possible:
I have multiple tables t_a, t_b, t_c with the same column layout and I want to apply the same operation to them, namely output some aggregation into another table. For a table t_x this would look like this:
CREATE TABLE t_x_aggregate (
<here the col definitions which are the same for all new tables t_[abc]_aggregate>
);
INSERT INTO t_x_aggregate(id, ...)
SELECT id, SUM(factor*amount)
FROM t_x
WHERE some fixed condition
GROUP BY id;
I now want to execute something like a FOR loop around this:
for t_x in t_a, t_b, t_c
CREATE TABLE ...
INSERT INTO ...
end for
Is this possible in SQL? Or would I need to build a wrapper in another language for this?
So, the result of that operation would be 3 new tables? T_A_AGGREGATE, T_B_AGGREGATE and T_C_AGGREGATE?
I think that the fastest way is to write 3 separate CREATE TABLE statements, e.g.
create table t_a_aggregate as
select id, sum(factor * amount) suma
from t_a
where some_condition
group by id;
create table t_b_aggregate as
select id, sum(factor * amount) suma
from t_b
where some_condition
group by id;
create table t_c_aggregate as
select id, sum(factor * amount) suma
from t_c
where some_condition
group by id;
OK; I understand that queries aren't that simple, but nothing much changes - only table names in CREATE and FROM (perhaps somewhere else, but that's more or less "it"). Any decent text editor's search/replace capabilities should be able to do it quickly.
If you want to do it dynamically in a loop (read: PL/SQL), you can - but dynamic SQL doesn't scale, is difficult to maintain, is painful to debug. Therefore, if you're doing it only once, consider running 3 separate statements.
How to do it dynamically?
You'd have to create a string (we usually put them into a locally declared variable) which contains the whole DDL statement. Why? Because you can't execute DDL from a PL/SQL otherwise.
If there are multiple tables and/or columns involved, you'll have to combine "fixed" parts of the statement (like create table, select, from, order by) concatenated with "dynamic" parts - such as column names. Note that in between you have to concatenate commas as separators. Pay attention to usage of multiple single quotes as you have to escape them (or use the q-quoting mechanism).
Also, for multiple columns you'll probably have to do it in a loop, concatenating each new column to previously composed string.
It (the statement stored into the varirable) is the executed by EXECUTE IMMEDIATE. If it is correctly written, it'll succeed. Otherwise, it'll fail, but it won't tell you why it failed (that's why I said difficult debugging").
So, instead of executing it, we usually display that string (using dbms_output.put_line) so that we see how it looks like and - using copy/paste - try to execute it.
Basically, it can be quite complex and - as I said - difficult to maintain and debug.
For the FOR loop you need to use PL/SQL like this:(*)
declare
type array_t is table of varchar2(10);
array array_t := array_t('a', 'b', 'c');
lo_stmt varchar2(2000);
begin
lo_stmt :=
'CREATE TABLE t_'||array(i)||'_aggregate ('||
' <here the col definitions which are the same for all new tables t_[abc]_aggregate>'||
');'||
''||
'INSERT INTO t_'||array(i)||'_aggregate(id, ...)'||
'SELECT id, SUM(factor*amount)'||
'FROM t_'||array(i)||
'WHERE some fixed condition'||
'GROUP BY id;'||
execute immediate lo_stmt;
end loop;
end;
/
Look also at this SO question: How to use Oracle PL/SQL to create...
(*) #Littlefoot describes in the 2nd part of his answer valuable background to this program.
I'm building a kind of dictionary app and I have a table for storing words like below:
id | surface_form | examples
-----------------------------------------------------------------------
1 | sounds | {"It sounds as though you really do believe that",
| | "A different bell begins to sound midnight"}
Where surface_form is of type CHARACTER VARYING and examples is an array field of CHARACTER VARYING
Since the examples are generated automatically from another API, it might not contain the exact "surface_form". Now I want to keep in examples only sentences that contain the exact surface_form. For instance, in the given example, only the first sentence is kept as it contain sounds, the second should be omitted as it only contain sound.
The problem is I got stuck in how to write a query and/or plSQL stored procedure to update the examples column so that it only has the desired sentences.
This query skips unwanted array elements:
select id, array_agg(example) new_examples
from a_table, unnest(examples) example
where surface_form = any(string_to_array(example, ' '))
group by id;
id | new_examples
----+----------------------------------------------------
1 | {"It sounds as though you really do believe that"}
(1 row)
Use it in update:
with corrected as (
select id, array_agg(example) new_examples
from a_table, unnest(examples) example
where surface_form = any(string_to_array(example, ' '))
group by id
)
update a_table
set examples = new_examples
from corrected
where examples <> new_examples
and a_table.id = corrected.id;
Test it in rextester.
Maybe you have to change the table design. This is what PostgreSQL's documentation says about the use of arrays:
Arrays are not sets; searching for specific array elements can be a sign of database misdesign. Consider using a separate table with a row for each item that would be an array element. This will be easier to search, and is likely to scale better for a large number of elements.
Documentation:
https://www.postgresql.org/docs/current/static/arrays.html
The most compact solution (but not necessarily the fastest) is to write a function that you pass a regular expression and an array and which then returns a new array that only contains the items matching the regex.
create function get_matching(p_values text[], p_pattern text)
returns text[]
as
$$
declare
l_result text[] := '{}'; -- make sure it's not null
l_element text;
begin
foreach l_element in array p_values loop
-- adjust this condition to whatever you want
if l_element ~ p_pattern then
l_result := l_result || l_element;
end if;
end loop;
return l_result;
end;
$$
language plpgsql;
The if condition is only an example. You need to adjust that to whatever you exactly store in the surface_form column. Maybe you need to test on word boundaries for the regex or a simple instr() would do - your question is unclear about that.
Cleaning up the table then becomes as simple as:
update the_table
set examples = get_matching(examples, surface_form);
But the whole approach seems flawed to me. It would be a lot more efficient if you stored the examples in a properly normalized data model.
In SQL, you have to remember two things.
Tuple elements are immutable but rows are mutable via updates.
SQL is declarative, not procedural
So you cannot "conditionally" "delete" a value from an array. You have to think about the question differently. You have to create a new array following a specification. That specification can conditionally include values (using case statements). Then you can overwrite the tuple with the new array.
Looks like one way could to update the array with array elements that are valid by doing a select using like or some regular expression.
https://www.postgresql.org/docs/current/static/arrays.html
If you want to hold elements from array that have "surface_form" in it you have to use that entries with substring(....,...) is not null
First you unnest the array, hold only items that match, and then array_agg the stored items
Here is a little query you can run to test without any table.
SELECT
id,
surface_form,
(SELECT array_agg(examples_matching)
FROM unnest(surfaces.examples) AS examples_matching
WHERE substring(examples_matching, surfaces.surface_form) IS NOT NULL)
FROM
(SELECT
1 AS id,
'example' :: TEXT AS surface_form,
ARRAY ['example form', 'test test','second example form'] :: TEXT [] AS examples
) surfaces;
You can select data in temp table using
Then update temp table using update query on row number
Merge value using
This merge value you can update in original table
For Example
Suppose you create temp table
Temp (id int, element character varying)
Then update Temp table and nest it.
Finally update original table
Here is the query you can directly try to execute in editor
CREATE TEMP TABLE IF NOT EXISTS temp_element (
id bigint,
element character varying)WITH (OIDS);
TRUNCATE TABLE temp_element;
insert into temp_element select row_number() over (order by p),p from (
select unnest(ARRAY['It sounds as though you really do believe that',
'A different bell begins to sound midnight']) as P)t;
update temp_element set element = 'It sounds as though you really'
where element = 'It sounds as though you really do believe that';
--update table
select array_agg(r) from ( select element from temp_element)r
I am using a simple cursor in a procedure that receives a couple of parameters.
I then make a cursor on a select query with a where clause with multiple conditions, which are equal to the received parameters. This cursor should only return 1 row, instead it returns multiple rows. I found this out because I'm using a for loop to go through this cursor and insert something into another table based on the values of this cursor.
When I run the query on the database statically(as in without pl/sql) I get what I expect, but when I do it from a cursor which should return only one row, and run it in a for loop, the loop does multiple iterations. How is this possible?
Thank you!
EDIT:
ID kind kolo kolo1 mjt salesman money date done
1 001 001 002 00013 00056 100,00 21-feb-12 N
I run a cursor like this:
Cursor linija IS
SELECT *
FROM table_x X
where x.mjt = mjt
and x.salesman = salesman
and x.kind = kind
and x.kolo1 = kolo1
and x.done = 'N';
This should return only one row, but instead my cursor returns %rowcount is 10.
You have a name conflict. You have called your local variables the same as your column names, and the column names are taking precedence, as noted in the documentation:
If a SQL statement references a name that belongs to both a column and either a local variable or formal parameter, then the column name takes precedence.
Caution:
When a variable or parameter name is interpreted as a column name, data can be deleted, changed, or inserted unintentionally.
The first four checks are always going to be true (unless you have null values), so you'll get every row that has done = 'N'.
Change your local variable names to something else; it's fairly common to use a prefix to distinguish between local variables, parameters, and columns, something like:
Cursor linija IS
SELECT *
FROM table_x X
where x.mjt = l_mjt
and x.salesman = l_salesman
and x.kind = l_kind
and x.kolo1 = l_kolo1
and x.done = 'N';
If this is in a stored procedure, rather than an anonymous block, you could use the procedure/function name as a prefix, which some people prefer. If your procedure was called myproc, for example, you could do:
Cursor linija IS
SELECT *
FROM table_x X
where x.mjt = myproc.mjt
and x.salesman = myproc.salesman
and x.kind = myproc.kind
and x.kolo1 = myproc.kolo1
and x.done = 'N';
In addition to what Alex has said (and I can't second his advice to distinguish variable names from column names enough!), why are you using a cursor for loop to do the insert?
You could just do your insert in one SQL statement, eg:
insert into your_table (col1, col2, ...)
select col1, col2, ...
from your_table
where ...
That will perform much better than going through the whole dataset and inserting each row one at a time. When it comes to databases, think set-based as much as you can, not procedurally!
If I have two queries, which I will call horrible_query_1 and ugly_query_2, and I want to perform the following two minus operations on them:
(horrible_query_1) minus (ugly_query_2)
(ugly_query_2) minus (horrible_query_1)
Or maybe I have a terribly_large_and_useful_query, and the result set it produces I want to use as part of several future queries.
How can I avoid copying and pasting the same queries in multiple places? How can I "not repeat myself," and follow DRY principles. Is this possible in SQL?
I'm using Oracle SQL. Portable SQL solutions are preferable, but if I have to use an Oracle specific feature (including PL/SQL) that's OK.
create view horrible_query_1_VIEW as
select .. ...
from .. .. ..
create view ugly_query_2_VIEW as
select .. ...
from .. .. ..
Then
(horrible_query_1_VIEW) minus (ugly_query_2_VIEW)
(ugly_query_2_VIEW) minus (horrible_query_1_VIEW)
Or, maybe, with a with clause:
with horrible_query_1 as (
select .. .. ..
from .. .. ..
) ,
ugly_query_2 as (
select .. .. ..
.. .. ..
)
(select * from horrible_query_1 minus select * from ugly_query_2 ) union all
(select * from ugly_query_2 minus select * from horrible_query_1)
If you want to reuse the SQL text of the queries, then defining views is the best way, as described earlier.
If you want to reuse the result of the queries, then you should consider global temporary tables. These temporary tables store data for the duration of session or transaction (whichever you choose). These are really useful in case you need to reuse calculated data many times over, especially if your queries are indeed "ugly" and "horrible" (meaning long running). See Temporary tables for more information.
If you need to keep the data longer than a session, you can consider materialized views.
Since you're using Oracle, I'd create Pipelined TABLE functions.
The function takes parameters and returns an object (which you have to create)
and then you SELECT * or even specific columns from it using the TABLE() function and can use it with a WHERE clause or with JOINs. If you want a unit of reuse (a function) you're not restricted to just returning values (i.e a scalar function) you can write a function that returns rows or recordsets.
something like this:
FUNCTION RETURN_MY_ROWS(Param1 IN type...ParamX IN Type)
RETURN PARENT_OBJECT PIPELINED
IS
local_curs cursor_alias; --you need a cursor alias if this function is in a Package
out_rec ROW_RECORD_OF_CUSTOM_OBJECT:=ROW_RECORD_OF_CUSTOM_OBJECT(NULL, NULL,NULL) --one NULL for each field in the record sub-object
BEGIN
OPEN local_curs FOR
--the SELECT query that you're trying to encapsulate goes here
-- and it can be very detailed/complex and even have WITH () etc..
SELECT * FROM baseTable WHERE col1 = x;
-- now that you have captured the SELECT into a Cursor
-- here you put a LOOP to take what's in the cursor and put it in the
-- child object (that holds the individual records)
LOOP
FETCH local_curs --opening the ref-cursor
INTO out_rec.COL1,
out_rec.COL2,
out_rec.COL3;
EXIT WHEN local_curs%NOTFOUND;
PIPE ROW(out_rec); --piping out the Object
END LOOP;
CLOSE local_curs; -- always do this
RETURN; -- we're now done
END RETURN_MY_ROWS;
after you've done that, you can use it like so
SELECT * FROM TABLE(RETURN_MY_ROWS(val1, val2));
you can INSERT SELECT or even CREATE TABLE out of it , you can have it in joins.
two more things to mention:
--ROW_RECORD_OF_CUSTOM_OBJECT is something along these lines
CREATE or REPLACE TYPE ROW_RECORD_OF_CUSTOM_OBJECT AS OBJECT
(
col1 type;
col2 type;
...
colx type;
);
and PARENT_OBJECT is a table of the other object (with the field definitions) we just made
create or replace TYPE PARENT_OBJECT IS TABLE OF ROW_RECORD_OF_CUSTOM_OBJECT;
so this function needs two OBJECTs to support it, but one is a record, the other is a table of that record (you have to create the record first).
In a nutshell, the function is easy to write, you need a child object (with fields), and a parent object that will house that child object that is of type TABLE of the child object, and you open the original base-table fetching SQL into a SYS_REFCURSOR (which you may need to alias) if you're in a package and you read from that cursor from a loop into the individual records.
The function returns a type of PARENT_OBJECT but inside it packs the records sub-object with values from the cursor.
I hope this works for you (there may be permissioning issues with your DBA if you want to create OBJECTs and Table functions)*/
If you operate with values, you could write functions.
Here you find infos on how to do it. It basically works like writing a function in any language. You can define parameters and return values.
Which gives you the cool possibility to write code just once. Here is how you do it:
http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_5009.htm
Have you tried using RESULT_CACHE hint in your queries? Also, you could
ALTER SESSION SET RESULT_CACHE_MODE=FORCE
and see if it helps.