PostgreSQL Mathematical Function - sql
I have a table aps_sections with many integer fields (such as bare_width and worn_width). I also have multiple look up tables (such as aps_bare_width and aps_worn_width) which contain an ID column and a WEIGHTING column. The ID is recorded in the above columns of the aps_sections table. I need to sum the WEIGHTINGs of the columns in the aps_sections table (whereby the WEIGHTING value comes from the look up tables). I have successfully managed this using the below SELECT statement.
SELECT aps_sections.ogc_fid,
( aps_bare_width.weighting
+ aps_worn_width.weighting
+ aps_gradient.weighting
+ aps_braiding.weighting
+ aps_pigeon.weighting
+ aps_depth.weighting
+ aps_standing_water.weighting
+ aps_running_water.weighting
+ aps_roughness.weighting
+ aps_surface.weighting
+ aps_dynamic.weighting
+ aps_ex_cond.weighting
+ aps_promotion.weighting
+ aps_level_of_use.weighting) AS calc
FROM row_access.aps_sections,
row_access.aps_bare_width,
row_access.aps_worn_width,
row_access.aps_gradient,
row_access.aps_braiding,
row_access.aps_pigeon,
row_access.aps_depth,
row_access.aps_standing_water,
row_access.aps_running_water,
row_access.aps_roughness,
row_access.aps_surface,
row_access.aps_dynamic,
row_access.aps_ex_cond,
row_access.aps_promotion,
row_access.aps_level_of_use
WHERE aps_bare_width.fid = aps_sections.bare_width
AND aps_worn_width.fid = aps_sections.worn_width
AND aps_gradient.fid = aps_sections.gradient
AND aps_braiding.fid = aps_sections.braiding
AND aps_pigeon.fid = aps_sections.pigeon
AND aps_depth.fid = aps_sections.depth
AND aps_standing_water.fid = aps_sections.standing_water
AND aps_running_water.fid = aps_sections.running_water
AND aps_roughness.fid = aps_sections.roughness
AND aps_surface.fid = aps_sections.surface
AND aps_dynamic.fid = aps_sections.dynamic
AND aps_ex_cond.fid = aps_sections.ex_cond
AND aps_promotion.fid = aps_sections.promotion
AND aps_level_of_use.fid = aps_sections.level_of_use
What I now need to do is create a function that adds the calculated result to the physical_sn_priority column of the aps_sections table. My understanding so far is that my function should look similar to:
CREATE OR REPLACE FUNCTION row_access.aps_weightings()
RETURNS trigger AS
$BODY$
BEGIN
NEW.physical_sn_priority := ;
RETURN NEW;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION public.update_km()
OWNER TO postgres;
But I don't know what to put after NEW.physical_sn_priority :=. I am a beginner to SQL and to PostgreSQL so I would appreciate any guidance!
While Erwin is (as always) correct that a version would be helpful, I think your answer will be simplest with the SELECT ... INTO construction for PL/pgSQL. (Not the same as SELECT INTO that works like INSERT or CREATE TABLE.)
SELECT ( aps_bare_width.weighting
+ /* obvious deletia */
+ aps_level_of_use.weighting)
INTO NEW.physical_sn_priority
FROM row_access.aps_bare_width,
/* snip */,
row_access.aps_level_of_use
WHERE aps_bare_width.fid = NEW.bare_width
AND /* snip */
aps_level_of_use.fid = NEW.level_of_use;
RETURN NEW;
According to the documentation, the INTO can appear in several other places in the line; I find this simple to understand.
[EDIT]
While this works, on reflection, I think the schema should be revised.
CREATE TYPE weighted_item_t AS ENUM ('bare_width', /* ... */, 'level_of_use');
CREATE TABLE current_weights(item_type weighted_item_t, fid int, current_weight float);
/* key and checks omitted */
/* note, if item_type can be deduced from fid, we don't even need the enum */
CREATE TABLE sections_items(section_id int /* FK into aps_sections */,
item_type weighted_item_t, fid int);
Now the queries are going to collapse into simple sums. You need to insert records into section_items before aps_sections, which can be done with deferred constraints in a transaction with or without a stored procedure, depending on how you acquire the data and how much control you have over its format. If (and this is not clear, because it won't change on updates) you want the denormalized total, you can get it with
SELECT SUM(current_weight) INTO NEW.physical_sn_priority
FROM section_items NATURAL JOIN current_weights
WHERE NEW.section_id=section_items.section_id;
This will work out much better if additional weighted characteristics are added at some future date.
Simplified test case
You should present your question with less noise. This shorter query does the job just fine:
SELECT aps_sections.ogc_fid,
( aps_bare_width.weighting
+ aps_worn_width.weighting
+ aps_gradient.weighting) AS calc
FROM row_access.aps_sections,
row_access.aps_bare_width,
row_access.aps_worn_width,
row_access.aps_gradient,
WHERE aps_bare_width.fid = aps_sections.bare_width
AND aps_worn_width.fid = aps_sections.worn_width
AND aps_gradient.fid = aps_sections.gradient;
Answer
As #Andrew already advised, the golden way would be to simplify your schema.
If, for some reason, this is not possible, here is an alternative to simplify the addition of many columns (which may or may not be NULL), create this tiny, powerful function:
CREATE OR REPLACE FUNCTION f_sum(ANYARRAY)
RETURNS numeric LANGUAGE sql AS
'SELECT sum(i)::numeric FROM unnest($1) i';
Call:
SELECT f_sum('{2,NULL,7}'::int[])
It takes a polymorphic array type and returns numeric. Works for any number type that can be summed by sum(). Cast the result if needed.
Simplifies the syntax for summing lots of columns.
NULL values won't break your calculation because the aggregate function sum() ignores those.
In a trigger function this could be used like this:
NEW.physical_sn_priority := f_sum(ARRAY [
COALESCE(physical_sn_priority, 0) -- add to original value
,(SELECT weighting FROM aps_bare_width x WHERE x.fid = NEW.bare_width)
,(SELECT weighting FROM aps_worn_width x WHERE x.fid = NEW.worn_width)
,(SELECT weighting FROM aps_gradient x WHERE x.fid = NEW.gradient)
...
])
Since all your joined tables are only good for looking up a single field, and completely independent from each other, you can just as well use individual subqueries. I also went this way, because we do not know whether any of the subqueries might return NULL, in which case your original query or Andrew's version would result in no row / no assignment.
Assignment to NEW really only makes sense in a BEFORE trigger on the table aps_sections. This code works BEFORE INSERT (where the row cannot be found in the table yet!) as well as BEFORE UPDATE. You want to use the current values from NEW, not the old row version in the table.
Related
How to write an Open SQL statement with substring in the JOIN ON condition? [duplicate]
I have the following select statement in ABAP: SELECT munic~mandt VREFER BIS AB ZZELECDATE ZZCERTDATE CONSYEAR ZDIMO ZZONE_M ZZONE_T USAGE_M USAGE_T M2MC M2MT M2RET EXEMPTMCMT EXEMPRET CHARGEMCMT INTO corresponding fields of table GT_INSTMUNIC_F FROM ZCI00_INSTMUNIC AS MUNIC INNER JOIN EVER AS EV on MUNIC~POD = EV~VREFER(9). "where EV~BSTATUS = '14' or EV~BSTATUS = '32'. My problem with the above statement is that does not recognize the substring/offset operation on the 'ON' clause. If i remove the '(9) then it recognizes the field, otherwise it gives error: Field ev~refer is unknown. It is neither in one of the specified tables nor defined by a "DATA" statement. I have also tried doing something similar in the 'Where' clause, receiving a similar error: LOOP AT gt_instmunic. clear wa_gt_instmunic_f. wa_gt_instmunic_f-mandt = gt_instmunic-mandt. wa_gt_instmunic_f-bis = gt_instmunic-bis. wa_gt_instmunic_f-ab = gt_instmunic-ab. wa_gt_instmunic_f-zzelecdate = gt_instmunic-zzelecdate. wa_gt_instmunic_f-ZZCERTDATE = gt_instmunic-ZZCERTDATE. wa_gt_instmunic_f-CONSYEAR = gt_instmunic-CONSYEAR. wa_gt_instmunic_f-ZDIMO = gt_instmunic-ZDIMO. wa_gt_instmunic_f-ZZONE_M = gt_instmunic-ZZONE_M. wa_gt_instmunic_f-ZZONE_T = gt_instmunic-ZZONE_T. wa_gt_instmunic_f-USAGE_M = gt_instmunic-USAGE_M. wa_gt_instmunic_f-USAGE_T = gt_instmunic-USAGE_T. temp_pod = gt_instmunic-pod. SELECT vrefer FROM ever INTO wa_gt_instmunic_f-vrefer WHERE ( vrefer(9) LIKE temp_pod ). " PROBLEM WITH SUBSTRING "AND ( BSTATUS = '14' OR BSTATUS = '32' ). ENDSELECT. WRITE: / sy-dbcnt. WRITE: / 'wa is: ', wa_gt_instmunic_f. WRITE: / 'wa-ever is: ', wa_gt_instmunic_f-vrefer. APPEND wa_gt_instmunic_f TO gt_instmunic_f. WRITE: / wa_gt_instmunic_f-vrefer. ENDLOOP. itab_size = lines( gt_instmunic_f ). WRITE: / 'Internal table populated with', itab_size, ' lines'. The basic task i want to implement is to modify a specific field on one table, pulling values from another. They have a common field ( pod = vrefer(9) ). Thanks in advance for your time.
If you are on a late enough NetWeaver version, it works on 7.51, you can use the OpenSQL function LEFT or SUBSTRING. Your query would look something like: SELECT munic~mandt VREFER BIS AB ZZELECDATE ZZCERTDATE CONSYEAR ZDIMO ZZONE_M ZZONE_T USAGE_M USAGE_T M2MC M2MT M2RET EXEMPTMCMT EXEMPRET CHARGEMCMT FROM ZCI00_INSTMUNIC AS MUNIC INNER JOIN ever AS ev ON MUNIC~POD EQ LEFT( EV~VREFER, 9 ) INTO corresponding fields of table GT_INSTMUNIC_F. Note that the INTO clause needs to move to the end of the command as well.
field(9) is a subset operation that is processed by the ABAP environment and can not be translated into a database-level SQL statement (at least not at the moment, but I'd be surprised if it ever will be). Your best bet is either to select the datasets separately and merge them manually (if both are approximately equally large) or pre-select one and use a FAE/IN clause.
They have a common field ( pod = vrefer(9) ) This is a wrong assumption, because they both are not fields, but a field an other thing. If you really need to do that task through SQL, I'll suggest you to check native SQL sentences like SUBSTRING and check if you can manage to use them within an EXEC_SQL or (better) the CL_SQL* classes.
Postgres : Unable to pass the list of ids in IN clause in JDBC [duplicate]
What are the best workarounds for using a SQL IN clause with instances of java.sql.PreparedStatement, which is not supported for multiple values due to SQL injection attack security issues: One ? placeholder represents one value, rather than a list of values. Consider the following SQL statement: SELECT my_column FROM my_table where search_column IN (?) Using preparedStatement.setString( 1, "'A', 'B', 'C'" ); is essentially a non-working attempt at a workaround of the reasons for using ? in the first place. What workarounds are available?
An analysis of the various options available, and the pros and cons of each is available in Jeanne Boyarsky's Batching Select Statements in JDBC entry on JavaRanch Journal. The suggested options are: Prepare SELECT my_column FROM my_table WHERE search_column = ?, execute it for each value and UNION the results client-side. Requires only one prepared statement. Slow and painful. Prepare SELECT my_column FROM my_table WHERE search_column IN (?,?,?) and execute it. Requires one prepared statement per size-of-IN-list. Fast and obvious. Prepare SELECT my_column FROM my_table WHERE search_column = ? ; SELECT my_column FROM my_table WHERE search_column = ? ; ... and execute it. [Or use UNION ALL in place of those semicolons. --ed] Requires one prepared statement per size-of-IN-list. Stupidly slow, strictly worse than WHERE search_column IN (?,?,?), so I don't know why the blogger even suggested it. Use a stored procedure to construct the result set. Prepare N different size-of-IN-list queries; say, with 2, 10, and 50 values. To search for an IN-list with 6 different values, populate the size-10 query so that it looks like SELECT my_column FROM my_table WHERE search_column IN (1,2,3,4,5,6,6,6,6,6). Any decent server will optimize out the duplicate values before running the query. None of these options are ideal. The best option if you are using JDBC4 and a server that supports x = ANY(y), is to use PreparedStatement.setArray as described in Boris's anwser. There doesn't seem to be any way to make setArray work with IN-lists, though. Sometimes SQL statements are loaded at runtime (e.g., from a properties file) but require a variable number of parameters. In such cases, first define the query: query=SELECT * FROM table t WHERE t.column IN (?) Next, load the query. Then determine the number of parameters prior to running it. Once the parameter count is known, run: sql = any( sql, count ); For example: /** * Converts a SQL statement containing exactly one IN clause to an IN clause * using multiple comma-delimited parameters. * * #param sql The SQL statement string with one IN clause. * #param params The number of parameters the SQL statement requires. * #return The SQL statement with (?) replaced with multiple parameter * placeholders. */ public static String any(String sql, final int params) { // Create a comma-delimited list based on the number of parameters. final StringBuilder sb = new StringBuilder( String.join(", ", Collections.nCopies(possibleValue.size(), "?"))); // For more than 1 parameter, replace the single parameter with // multiple parameter placeholders. if (sb.length() > 1) { sql = sql.replace("(?)", "(" + sb + ")"); } // Return the modified comma-delimited list of parameters. return sql; } For certain databases where passing an array via the JDBC 4 specification is unsupported, this method can facilitate transforming the slow = ? into the faster IN (?) clause condition, which can then be expanded by calling the any method.
Solution for PostgreSQL: final PreparedStatement statement = connection.prepareStatement( "SELECT my_column FROM my_table where search_column = ANY (?)" ); final String[] values = getValues(); statement.setArray(1, connection.createArrayOf("text", values)); try (ResultSet rs = statement.executeQuery()) { while(rs.next()) { // do some... } } or final PreparedStatement statement = connection.prepareStatement( "SELECT my_column FROM my_table " + "where search_column IN (SELECT * FROM unnest(?))" ); final String[] values = getValues(); statement.setArray(1, connection.createArrayOf("text", values)); try (ResultSet rs = statement.executeQuery()) { while(rs.next()) { // do some... } }
No simple way AFAIK. If the target is to keep statement cache ratio high (i.e to not create a statement per every parameter count), you may do the following: create a statement with a few (e.g. 10) parameters: ... WHERE A IN (?,?,?,?,?,?,?,?,?,?) ... Bind all actuall parameters setString(1,"foo"); setString(2,"bar"); Bind the rest as NULL setNull(3,Types.VARCHAR) ... setNull(10,Types.VARCHAR) NULL never matches anything, so it gets optimized out by the SQL plan builder. The logic is easy to automate when you pass a List into a DAO function: while( i < param.size() ) { ps.setString(i+1,param.get(i)); i++; } while( i < MAX_PARAMS ) { ps.setNull(i+1,Types.VARCHAR); i++; }
You can use Collections.nCopies to generate a collection of placeholders and join them using String.join: List<String> params = getParams(); String placeHolders = String.join(",", Collections.nCopies(params.size(), "?")); String sql = "select * from your_table where some_column in (" + placeHolders + ")"; try ( Connection connection = getConnection(); PreparedStatement ps = connection.prepareStatement(sql)) { int i = 1; for (String param : params) { ps.setString(i++, param); } /* * Execute query/do stuff */ }
An unpleasant work-around, but certainly feasible is to use a nested query. Create a temporary table MYVALUES with a column in it. Insert your list of values into the MYVALUES table. Then execute select my_column from my_table where search_column in ( SELECT value FROM MYVALUES ) Ugly, but a viable alternative if your list of values is very large. This technique has the added advantage of potentially better query plans from the optimizer (check a page for multiple values, tablescan only once instead once per value, etc) may save on overhead if your database doesn't cache prepared statements. Your "INSERTS" would need to be done in batch and the MYVALUES table may need to be tweaked to have minimal locking or other high-overhead protections.
Limitations of the in() operator is the root of all evil. It works for trivial cases, and you can extend it with "automatic generation of the prepared statement" however it is always having its limits. if you're creating a statement with variable number of parameters, that will make an sql parse overhead at each call on many platforms, the number of parameters of in() operator are limited on all platforms, total SQL text size is limited, making impossible for sending down 2000 placeholders for the in params sending down bind variables of 1000-10k is not possible, as the JDBC driver is having its limitations The in() approach can be good enough for some cases, but not rocket proof :) The rocket-proof solution is to pass the arbitrary number of parameters in a separate call (by passing a clob of params, for example), and then have a view (or any other way) to represent them in SQL and use in your where criteria. A brute-force variant is here http://tkyte.blogspot.hu/2006/06/varying-in-lists.html However if you can use PL/SQL, this mess can become pretty neat. function getCustomers(in_customerIdList clob) return sys_refcursor is begin aux_in_list.parse(in_customerIdList); open res for select * from customer c, in_list v where c.customer_id=v.token; return res; end; Then you can pass arbitrary number of comma separated customer ids in the parameter, and: will get no parse delay, as the SQL for select is stable no pipelined functions complexity - it is just one query the SQL is using a simple join, instead of an IN operator, which is quite fast after all, it is a good rule of thumb of not hitting the database with any plain select or DML, since it is Oracle, which offers lightyears of more than MySQL or similar simple database engines. PL/SQL allows you to hide the storage model from your application domain model in an effective way. The trick here is: we need a call which accepts the long string, and store somewhere where the db session can access to it (e.g. simple package variable, or dbms_session.set_context) then we need a view which can parse this to rows and then you have a view which contains the ids you're querying, so all you need is a simple join to the table queried. The view looks like: create or replace view in_list as select trim( substr (txt, instr (txt, ',', 1, level ) + 1, instr (txt, ',', 1, level+1) - instr (txt, ',', 1, level) -1 ) ) as token from (select ','||aux_in_list.getpayload||',' txt from dual) connect by level <= length(aux_in_list.getpayload)-length(replace(aux_in_list.getpayload,',',''))+1 where aux_in_list.getpayload refers to the original input string. A possible approach would be to pass pl/sql arrays (supported by Oracle only), however you can't use those in pure SQL, therefore a conversion step is always needed. The conversion can not be done in SQL, so after all, passing a clob with all parameters in string and converting it witin a view is the most efficient solution.
Here's how I solved it in my own application. Ideally, you should use a StringBuilder instead of using + for Strings. String inParenthesis = "(?"; for(int i = 1;i < myList.size();i++) { inParenthesis += ", ?"; } inParenthesis += ")"; try(PreparedStatement statement = SQLite.connection.prepareStatement( String.format("UPDATE table SET value='WINNER' WHERE startTime=? AND name=? AND traderIdx=? AND someValue IN %s", inParenthesis))) { int x = 1; statement.setLong(x++, race.startTime); statement.setString(x++, race.name); statement.setInt(x++, traderIdx); for(String str : race.betFair.winners) { statement.setString(x++, str); } int effected = statement.executeUpdate(); } Using a variable like x above instead of concrete numbers helps a lot if you decide to change the query at a later time.
I've never tried it, but would .setArray() do what you're looking for? Update: Evidently not. setArray only seems to work with a java.sql.Array that comes from an ARRAY column that you've retrieved from a previous query, or a subquery with an ARRAY column.
My workaround is: create or replace type split_tbl as table of varchar(32767); / create or replace function split ( p_list varchar2, p_del varchar2 := ',' ) return split_tbl pipelined is l_idx pls_integer; l_list varchar2(32767) := p_list; l_value varchar2(32767); begin loop l_idx := instr(l_list,p_del); if l_idx > 0 then pipe row(substr(l_list,1,l_idx-1)); l_list := substr(l_list,l_idx+length(p_del)); else pipe row(l_list); exit; end if; end loop; return; end split; / Now you can use one variable to obtain some values in a table: select * from table(split('one,two,three')) one two three select * from TABLE1 where COL1 in (select * from table(split('value1,value2'))) value1 AAA value2 BBB So, the prepared statement could be: "select * from TABLE where COL in (select * from table(split(?)))" Regards, Javier Ibanez
I suppose you could (using basic string manipulation) generate the query string in the PreparedStatement to have a number of ?'s matching the number of items in your list. Of course if you're doing that you're just a step away from generating a giant chained OR in your query, but without having the right number of ? in the query string, I don't see how else you can work around this.
You could use setArray method as mentioned in this javadoc: PreparedStatement statement = connection.prepareStatement("Select * from emp where field in (?)"); Array array = statement.getConnection().createArrayOf("VARCHAR", new Object[]{"E1", "E2","E3"}); statement.setArray(1, array); ResultSet rs = statement.executeQuery();
Here's a complete solution in Java to create the prepared statement for you: /*usage: Util u = new Util(500); //500 items per bracket. String sqlBefore = "select * from myTable where ("; List<Integer> values = new ArrayList<Integer>(Arrays.asList(1,2,4,5)); string sqlAfter = ") and foo = 'bar'"; PreparedStatement ps = u.prepareStatements(sqlBefore, values, sqlAfter, connection, "someId"); */ import java.sql.Connection; import java.sql.PreparedStatement; import java.sql.SQLException; import java.util.ArrayList; import java.util.List; public class Util { private int numValuesInClause; public Util(int numValuesInClause) { super(); this.numValuesInClause = numValuesInClause; } public int getNumValuesInClause() { return numValuesInClause; } public void setNumValuesInClause(int numValuesInClause) { this.numValuesInClause = numValuesInClause; } /** Split a given list into a list of lists for the given size of numValuesInClause*/ public List<List<Integer>> splitList( List<Integer> values) { List<List<Integer>> newList = new ArrayList<List<Integer>>(); while (values.size() > numValuesInClause) { List<Integer> sublist = values.subList(0,numValuesInClause); List<Integer> values2 = values.subList(numValuesInClause, values.size()); values = values2; newList.add( sublist); } newList.add(values); return newList; } /** * Generates a series of split out in clause statements. * #param sqlBefore ""select * from dual where (" * #param values [1,2,3,4,5,6,7,8,9,10] * #param "sqlAfter ) and id = 5" * #return "select * from dual where (id in (1,2,3) or id in (4,5,6) or id in (7,8,9) or id in (10)" */ public String genInClauseSql(String sqlBefore, List<Integer> values, String sqlAfter, String identifier) { List<List<Integer>> newLists = splitList(values); String stmt = sqlBefore; /* now generate the in clause for each list */ int j = 0; /* keep track of list:newLists index */ for (List<Integer> list : newLists) { stmt = stmt + identifier +" in ("; StringBuilder innerBuilder = new StringBuilder(); for (int i = 0; i < list.size(); i++) { innerBuilder.append("?,"); } String inClause = innerBuilder.deleteCharAt( innerBuilder.length() - 1).toString(); stmt = stmt + inClause; stmt = stmt + ")"; if (++j < newLists.size()) { stmt = stmt + " OR "; } } stmt = stmt + sqlAfter; return stmt; } /** * Method to convert your SQL and a list of ID into a safe prepared * statements * * #throws SQLException */ public PreparedStatement prepareStatements(String sqlBefore, ArrayList<Integer> values, String sqlAfter, Connection c, String identifier) throws SQLException { /* First split our potentially big list into lots of lists */ String stmt = genInClauseSql(sqlBefore, values, sqlAfter, identifier); PreparedStatement ps = c.prepareStatement(stmt); int i = 1; for (int val : values) { ps.setInt(i++, val); } return ps; } }
Spring allows passing java.util.Lists to NamedParameterJdbcTemplate , which automates the generation of (?, ?, ?, ..., ?), as appropriate for the number of arguments. For Oracle, this blog posting discusses the use of oracle.sql.ARRAY (Connection.createArrayOf doesn't work with Oracle). For this you have to modify your SQL statement: SELECT my_column FROM my_table where search_column IN (select COLUMN_VALUE from table(?)) The oracle table function transforms the passed array into a table like value usable in the IN statement.
try using the instr function? select my_column from my_table where instr(?, ','||search_column||',') > 0 then ps.setString(1, ",A,B,C,"); Admittedly this is a bit of a dirty hack, but it does reduce the opportunities for sql injection. Works in oracle anyway.
Sormula supports SQL IN operator by allowing you to supply a java.util.Collection object as a parameter. It creates a prepared statement with a ? for each of the elements the collection. See Example 4 (SQL in example is a comment to clarify what is created but is not used by Sormula).
Generate the query string in the PreparedStatement to have a number of ?'s matching the number of items in your list. Here's an example: public void myQuery(List<String> items, int other) { ... String q4in = generateQsForIn(items.size()); String sql = "select * from stuff where foo in ( " + q4in + " ) and bar = ?"; PreparedStatement ps = connection.prepareStatement(sql); int i = 1; for (String item : items) { ps.setString(i++, item); } ps.setInt(i++, other); ResultSet rs = ps.executeQuery(); ... } private String generateQsForIn(int numQs) { String items = ""; for (int i = 0; i < numQs; i++) { if (i != 0) items += ", "; items += "?"; } return items; }
instead of using SELECT my_column FROM my_table where search_column IN (?) use the Sql Statement as select id, name from users where id in (?, ?, ?) and preparedStatement.setString( 1, 'A'); preparedStatement.setString( 2,'B'); preparedStatement.setString( 3, 'C'); or use a stored procedure this would be the best solution, since the sql statements will be compiled and stored in DataBase server
I came across a number of limitations related to prepared statement: The prepared statements are cached only inside the same session (Postgres), so it will really work only with connection pooling A lot of different prepared statements as proposed by #BalusC may cause the cache to overfill and previously cached statements will be dropped The query has to be optimized and use indices. Sounds obvious, however e.g. the ANY(ARRAY...) statement proposed by #Boris in one of the top answers cannot use indices and query will be slow despite caching The prepared statement caches the query plan as well and the actual values of any parameters specified in the statement are unavailable. Among the proposed solutions I would choose the one that doesn't decrease the query performance and makes the less number of queries. This will be the #4 (batching few queries) from the #Don link or specifying NULL values for unneeded '?' marks as proposed by #Vladimir Dyuzhev
SetArray is the best solution but its not available for many older drivers. The following workaround can be used in java8 String baseQuery ="SELECT my_column FROM my_table where search_column IN (%s)" String markersString = inputArray.stream().map(e -> "?").collect(joining(",")); String sqlQuery = String.format(baseSQL, markersString); //Now create Prepared Statement and use loop to Set entries int index=1; for (String input : inputArray) { preparedStatement.setString(index++, input); } This solution is better than other ugly while loop solutions where the query string is built by manual iterations
I just worked out a PostgreSQL-specific option for this. It's a bit of a hack, and comes with its own pros and cons and limitations, but it seems to work and isn't limited to a specific development language, platform, or PG driver. The trick of course is to find a way to pass an arbitrary length collection of values as a single parameter, and have the db recognize it as multiple values. The solution I have working is to construct a delimited string from the values in the collection, pass that string as a single parameter, and use string_to_array() with the requisite casting for PostgreSQL to properly make use of it. So if you want to search for "foo", "blah", and "abc", you might concatenate them together into a single string as: 'foo,blah,abc'. Here's the straight SQL: select column from table where search_column = any (string_to_array('foo,blah,abc', ',')::text[]); You would obviously change the explicit cast to whatever you wanted your resulting value array to be -- int, text, uuid, etc. And because the function is taking a single string value (or two I suppose, if you want to customize the delimiter as well), you can pass it as a parameter in a prepared statement: select column from table where search_column = any (string_to_array($1, ',')::text[]); This is even flexible enough to support things like LIKE comparisons: select column from table where search_column like any (string_to_array('foo%,blah%,abc%', ',')::text[]); Again, no question it's a hack, but it works and allows you to still use pre-compiled prepared statements that take *ahem* discrete parameters, with the accompanying security and (maybe) performance benefits. Is it advisable and actually performant? Naturally, it depends, as you've got string parsing and possibly casting going on before your query even runs. If you're expecting to send three, five, a few dozen values, sure, it's probably fine. A few thousand? Yeah, maybe not so much. YMMV, limitations and exclusions apply, no warranty express or implied. But it works.
No one else seems to have suggested using an off-the-shelf query builder yet, like jOOQ or QueryDSL or even Criteria Query that manage dynamic IN lists out of the box, possibly including the management of all edge cases that may arise, such as: Running into Oracle's maximum of 1000 elements per IN list (irrespective of the number of bind values) Running into any driver's maximum number of bind values, which I've documented in this answer Running into cursor cache contention problems because too many distinct SQL strings are "hard parsed" and execution plans cannot be cached anymore (jOOQ and since recently also Hibernate work around this by offering IN list padding) (Disclaimer: I work for the company behind jOOQ)
Just for completeness: So long as the set of values is not too large, you could also simply string-construct a statement like ... WHERE tab.col = ? OR tab.col = ? OR tab.col = ? which you could then pass to prepare(), and then use setXXX() in a loop to set all the values. This looks yucky, but many "big" commercial systems routinely do this kind of thing until they hit DB-specific limits, such as 32 KB (I think it is) for statements in Oracle. Of course you need to ensure that the set will never be unreasonably large, or do error trapping in the event that it is.
Following Adam's idea. Make your prepared statement sort of select my_column from my_table where search_column in (#) Create a String x and fill it with a number of "?,?,?" depending on your list of values Then just change the # in the query for your new String x an populate
There are different alternative approaches that we can use for IN clause in PreparedStatement. Using Single Queries - slowest performance and resource intensive Using StoredProcedure - Fastest but database specific Creating dynamic query for PreparedStatement - Good Performance but doesn't get benefit of caching and PreparedStatement is recompiled every time. Use NULL in PreparedStatement queries - Optimal performance, works great when you know the limit of IN clause arguments. If there is no limit, then you can execute queries in batch. Sample code snippet is; int i = 1; for(; i <=ids.length; i++){ ps.setInt(i, ids[i-1]); } //set null for remaining ones for(; i<=PARAM_SIZE;i++){ ps.setNull(i, java.sql.Types.INTEGER); } You can check more details about these alternative approaches here.
For some situations regexp might help. Here is an example I've checked on Oracle, and it works. select * from my_table where REGEXP_LIKE (search_column, 'value1|value2') But there is a number of drawbacks with it: Any column it applied should be converted to varchar/char, at least implicitly. Need to be careful with special characters. It can slow down performance - in my case IN version uses index and range scan, and REGEXP version do full scan.
After examining various solutions in different forums and not finding a good solution, I feel the below hack I came up with, is the easiest to follow and code: Example: Suppose you have multiple parameters to pass in the 'IN' clause. Just put a dummy String inside the 'IN' clause, say, "PARAM" do denote the list of parameters that will be coming in the place of this dummy String. select * from TABLE_A where ATTR IN (PARAM); You can collect all the parameters into a single String variable in your Java code. This can be done as follows: String param1 = "X"; String param2 = "Y"; String param1 = param1.append(",").append(param2); You can append all your parameters separated by commas into a single String variable, 'param1', in our case. After collecting all the parameters into a single String you can just replace the dummy text in your query, i.e., "PARAM" in this case, with the parameter String, i.e., param1. Here is what you need to do: String query = query.replaceFirst("PARAM",param1); where we have the value of query as query = "select * from TABLE_A where ATTR IN (PARAM)"; You can now execute your query using the executeQuery() method. Just make sure that you don't have the word "PARAM" in your query anywhere. You can use a combination of special characters and alphabets instead of the word "PARAM" in order to make sure that there is no possibility of such a word coming in the query. Hope you got the solution. Note: Though this is not a prepared query, it does the work that I wanted my code to do.
Just for completeness and because I did not see anyone else suggest it: Before implementing any of the complicated suggestions above consider if SQL injection is indeed a problem in your scenario. In many cases the value provided to IN (...) is a list of ids that have been generated in a way that you can be sure that no injection is possible... (e.g. the results of a previous select some_id from some_table where some_condition.) If that is the case you might just concatenate this value and not use the services or the prepared statement for it or use them for other parameters of this query. query="select f1,f2 from t1 where f3=? and f2 in (" + sListOfIds + ");";
PreparedStatement doesn't provide any good way to deal with SQL IN clause. Per http://www.javaranch.com/journal/200510/Journal200510.jsp#a2 "You can't substitute things that are meant to become part of the SQL statement. This is necessary because if the SQL itself can change, the driver can't precompile the statement. It also has the nice side effect of preventing SQL injection attacks." I ended up using following approach: String query = "SELECT my_column FROM my_table where search_column IN ($searchColumns)"; query = query.replace("$searchColumns", "'A', 'B', 'C'"); Statement stmt = connection.createStatement(); boolean hasResults = stmt.execute(query); do { if (hasResults) return stmt.getResultSet(); hasResults = stmt.getMoreResults(); } while (hasResults || stmt.getUpdateCount() != -1);
OK, so I couldn't remember exactly how (or where) I did this before so I came to stack overflow to quickly find the answer. I was surprised I couldn't. So, how I got around the IN problem a long time ago was with a statement like this: where myColumn in ( select regexp_substr(:myList,'[^,]+', 1, level) from dual connect by regexp_substr(:myList, '[^,]+', 1, level) is not null) set the myList parameter as a comma delimited string: A,B,C,D... Note: You have to set the parameter twice!
This is not the ideal practice, yet it's simple and works well for me most of the time. where ? like concat( "%|", TABLE_ID , "|%" ) Then you pass through ? the IDs in this way: |1|,|2|,|3|,...|
PostGIS 2.3 Split a line by points
I asked this question on gis.stackexchange ( but since my actual problem seems to be more a DB problem than GIS I am trying my luck here). Here is the question on gis.stackexchange : https://gis.stackexchange.com/questions/256535/postgis-2-3-splitting-multiline-by-points I have a trigger in which I a looping when inserting a new line to INSERT the set of splitted lines in my table, but for some reason I do not get the wanted result since in the example I only get two lines out of three. What a I doing wrong ? Here comes the code of the trigger function : CREATE OR REPLACE FUNCTION public.split_cable() RETURNS trigger AS $BODY$ DECLARE compte integer; DECLARE i integer := 2; BEGIN compte = (SELECT count(*) FROM boite WHERE st_intersects(boite.geom, new.geom)); WHILE i < compte LOOP WITH brs AS (SELECT row_number() over(), boite.geom FROM boite, cable2 WHERE st_intersects(boite.geom, new.geom) -- here the ORDER BY serve to get the "boite" objects in a specific order ORDER BY st_linelocatepoint(st_linemerge(new.geom),boite.geom)), brs2 AS (SELECT st_union(geom) AS geom FROM brs), cables AS (SELECT (st_dump(st_split(new.geom, brs2.geom))).geom FROM brs2) INSERT INTO cable2 (geom) VALUES ( SELECT st_multi(cables.geom) FROM cables WHERE st_startpoint(geom) = (SELECT geom FROM brs WHERE brs.row_number = i)); i = i + 1; END LOOP; new.geom = (WITH brs AS (SELECT row_number() over(), boite.geom FROM boite, cable2 WHERE st_intersects(boite.geom, new.geom) ORDER BY st_linelocatepoint(st_linemerge(new.geom),boite.geom)), brs2 AS (SELECT st_union(geom) as geom from brs), cables AS (SELECT (st_dump(st_split(new.geom, brs2.geom))).geom FROM brs2) SELECT st_multi(cables.geom) FROM cables WHERE st_startpoint(geom) = (SELECT geom FROM brs WHERE brs.row_number = 1)); RETURN new; END $BODY$ LANGUAGE plpgsql;
This is a relatively complex query and has a lot of moving parts. my recommendation for debugging the query involves multiple ideas: Consider splitting the function into smaller functions, that are easier to test, and then compose the function from a set of parts you know for sure work as you need them to. export a set of intermediate results to an intermediate table, you you can visualise the intermediate result-sets easily using a graphical tool and can better assess where the data went wrong. is is possible that the combination of ST_ functions you are using don't create the geometries you think they create, one way to rule this out is by visualising the results of geographical function combinations, like st_dump(st_split(...))) or st_dump(st_split(...)) for example. perhaps this check: st_startpoint(geom) = (SELECT geom FROM brs WHERE brs.row_number = i)) could be made by checking "points near" and not "exact point", maybe the points are very near, as in centimeters near, making them essentially "the same point", but not actually be the exact point. this is just an assumption though. Consider sharing more data with StackOverflow! like a small dataset or example so we can actually run the code! :)
Strategy to alter database synonym during normal production operation
Currently I have a scenario that involves switching a synonym definition after the completion of a scheduled job. The job will create a table with an identifier of even or odd to correspond with the hour being even or odd. What we are currently doing is this: odd_job: create foo_odd ... replace foo_syn as foo_odd and even_job: create foo_even ... replace foo_syn as foo_even What is happening is that during normal production the foo_syn is in a locked state. So what we are looking for is a production capable way of swapping synonym definitions. The question is how can we swap a synonym definition in a production level system with minimum user interruption in Oracle 10g? From the comments Does foo_syn have any dependent objects? No foo_syn is nothing more than a pointer to a table that I generate. That is there are no procedures that need to be recompiled for this switch. That sounds like a really strange thing to do. Can you explain a bit what that switch is for/how it is used? Sure. We have an application that interfaces with the database, the SQL that is executed from Java (business logic queries) has a reference to foo_syn. Because of the dynamic nature of the data it is a guarantee that the hourly swap will give new results that are important as we try to get closer to real time. Prior to this it was a once a day and be happy with it type scenario. The reasoning for the swap is I do not want dynamic SQL (in terms of table names) to be a part of my application queries. So therefore the database does a switch on the newer data set without changing the name of the synonym that is referenced as part of my application.
If using dynamic SQL is distasteful to you (and I'll quickly point out that in my experience dynamic SQL has never proved to be a performance issue, but YMMV) then a UNION query might be what you're looking for - something like SELECT * FROM EVEN_DATA_TABLE WHERE TO_NUMBER(TO_CHAR(SYSDATE, 'HH')) IN (0, 2, 4, 6, 8, 10, 12) UNION ALL SELECT * FROM ODD_DATA_TABLE WHERE TO_NUMBER(TO_CHAR(SYSDATE, 'HH')) IN (1, 3, 5, 7, 9, 11) This also eliminates the need to have a periodic job to change the synonym as it's driven off of SYSDATE. This makes the assumption that the columns in EVEN_DATA_TABLE and ODD_DATA_TABLE are the same. Share and enjoy.
The solution that we came up with is as follows: 1) Define a function that will return which set of tables you should be looking at: create or replace function which_synonym return varchar2 as to_return varchar2(4) := NULL; is_valid number :=- 1; current_time number := to_number(to_char(sysdate,'HH')); is_odd boolean := FALSE; BEGIN if = mod(current_time,2) -- it is an even time slot then select success into is_valid from success_table where run='EVEN'; else select success into is_valid from success_table where run='ODD'; end if; if is_valid=0 and is_odd=TRUE then to_Return ='ODD'; else to_return='EVEN'; end if; Return to_return; END which_synonym; De Morgan's laws omitted for conciseness. 2) Configure the application procedures to take advantage of this flipping: a) Tokenize enumerated sql strings with a sequence that you want to match on: select * from foo_&&& b) write the function that will replace this sequence: public String whichSynonym(String sql) { if(null==sql || "".equals(sql.trim())) { throw new IllegalArgumentException("Cannot process null or empty sql"); } String oddEven = ""; //removed boilerplate PreparedStatement statement = conn.prepareStatement("Select which_synonym from dual"); statement.execute(); ResultSet results = statement.getResults(); while(results.next()) { oddEven=results.getString(1); } return sql.replace("&&&",oddEven); }
Delphi query parameter usage when all values is also an option
I have a tquery (going thru BDE or BDE emulating component) that has been used to select either a single record or all records. Traditionally this has been done as such: select * from clients where (clientid = :clientid or :clientid = -1) And then they would put a -1 in the field when they wanted the query to return all values. Going through this code though, I have discovered that when they have done this the query does not use proper indexing for the table and only does a natural read. Is there a best practices method for achieving this? Perhaps a way to tell a parameter to return all values, or must the script be modified to remove the where clause entirely when all values are desired? Edit: This is Delphi 7, by the way (And going against Firebird 1.5 sorry for leaving that out)
As you use deprecated BDE, that may be one more reason to migrate from BDE to 3d party solutions. AnyDAC (UniDAC, probably others too. Most are commercial libraries) has macros, which allow to dynamically change a SQL command text, depending on the macro values. So, your query may be written: ADQuery1.SQL.Text := 'select * from clients {IF &clientid} where clientid = &clientid {FI}'; if clientid >= 0 then // to get a single record ADQuery1.Macros[0].AsInteger := clientid else // to get all records ADQuery1.Macros[0].Clear; ADQuery1.Open;
For the queries with "optional" parameters I always use ISNULL (MSSQL, or NVL Oracle), ie. SELECT * FROM clients WHERE ISNULL(:clientid, clientid) = clientid Setting the parameter to NULL then selects all records. You also have to take care of NULL values in the table fields because NULL <> NULL. This you can overcome with a slight modification: SELECT * FROM clients WHERE COALESCE(:clientid, clientid, -1) = ISNULL(clientid, -1)
I would use this: SELECT * FROM CLIENTS WHERE clientid = :clientid or :clientid IS NULL
Using two queries is best: if (clientid <> -1) then begin DBComp.SQL.Text := 'select * from clients where (clientid = :clientid)'; DBComp.ParamByName('clientid').Value := clientid; end else begin DBComp.SQL.Text := 'select * from clients'; end; DBComp.Open; ... Alternatively: DBComp.SQL.BeginUpdate; try DBComp.SQL.Clear; DBComp.SQL.Add('select * from clients'); if (clientid <> -1) then DBComp.SQL.Add('where (clientid = :clientid)'); finally DBComp.SQL.EndUpdate; end; if (clientid <> -1) then DBComp.ParamByName('clientid').Value := clientid; DBComp.Open; ...
Remy's answer may be re-formulated as single query. It may be better, if you gonna prepare it once and then re-open multiple times. select * from clients where (clientid = :clientid) and (:clientid is not null) UNION ALL select * from clients where (:clientid is null) This just aggregates two distinct queries (with same results vector) together. And condition just turns one of those off. Using would be like that: DBComp.Prepare. ... DBComp.Close; DBComp.ParamByName('clientid').Value := clientid; DBComp.Open; ... DBComp.Close; DBComp.ParamByName('clientid').Clear; DBComp.Open; However this query would rely on SQL Server optimizer capability to extract query invariant (:clientid is [not] null) and enable/disable query completely. But well, your original query depends upon that too. Why still use obsolete FB 1.5 ? Won't FB 2.5.2 work better there ? I think your original query is formulated poorly. select * from clients where (:clientid = -1) or ((clientid = :clientid) and (:clientid <> -1)) would probably be easier on SQL Server optimizer. Yet i think FB could do better job there. Try to download later FB, and run your query in it, using IDEs like IBExpert or FlameRobin. Re-arranging parenthesis and changing -1 to NULL are obvious ideas to try. Using BDE is fragile now. It is not very fast, limiting in datatypes and connectivity (no FB/IB Events for example). And would have all sorts of compatibility problems with Vista/Win7 and Win64. If FB/IB is your server of choice, consider switching to some modern component set: (FLOSS) Universal Interbase by http://uib.sf.net (RIP all Delphi pages of http://Progdigy.com ) (FLOSS) ZeosLib DBO by http://zeos.firmos.at/ (propr) FIB+ by http://DevRace.com (propr) IB Objects by http://IBobjects.com (propr) AnyDAC by http://da-soft.com - sold out to Embarcadero, not-avail for D7 (propr) IB-DAC/UniDAC http://DevArt.com Also it would be good thing to show the table and indices definition and selectivity of those indices.