The Django generating sql queries with function calls in where clause like below, when using Django queries for search or any other purposes.
SELECT COUNT(*) AS "_count"
FROM "products_product"
INNER JOIN "products_productmeta" ON ("products_product"."meta_id" = "products_productmeta"."id")
WHERE ((NOT ("products_product"."archived" = true) AND
"products_product"."owner_id" = 1281 AND
UPPER(UNACCENT("products_product"."sku")::text) LIKE '%' ||
UPPER(REPLACE(REPLACE(REPLACE((UNACCENT('octavi')), '\', '\'), '%', '\%'), '', '_')) ||
'%') OR
(NOT ("productsproduct"."archived" = true) AND
"products_product"."owner_id" = 1281 AND
UPPER(UNACCENT("products_productmeta"."name")::text) LIKE '%' ||
UPPER(REPLACE(REPLACE(REPLACE((UNACCENT('octavi')), '\', '\'), '%', '\%'), '', '_')) ||
'%')
)
Response time is taking too long with these type of queries because of whole data sets are passing through the function calls.
Is there any way to avoid those function calls such as UPPER, REPLACE etc... in where clause of sql generated by django?
Is writing raw sql queries the only option?
EDIT: Django query example with SQL string output added:
Product.objects.filter(meta__name__icontains="foo").query.sql_with_params()
The simple Django query above is generating SQL string below:
('SELECT
"products_product"."id",
....
"products_product"."is_primary"
FROM "products_product"
INNER JOIN "products_productmeta" ON ("products_product"."meta_id" = "products_productmeta"."id")
WHERE (NOT ("products_product"."archived" = %s) AND
UPPER("products_productmeta"."name"::text) LIKE UPPER(%s))',
(True, '%foo%'))
As you can see; Django adding function calls to where clause. I want to remove that function calls. Complex queries are taking too long time.
Related
I am trying to implement fuzzy text search with PostgreSQL on a Nodejs/Express server. I've got the query working using the "ILIKE" operator but it is very important that this query be optimised and I read that using the similarity operator "%" here would force use of the GIN index created on the title column using gin_trgm_ops. But for some reason, the query does not return any values using the "%" operator. I've tried escaping the string then concatenation which received some syntax errors. Any pointers would be appreciated.
This version is not working:
var query = ['SELECT title, user FROM reviews']
query.push("WHERE (title % $1) AND NOT (user = $2) AND ((approved_date IS NULL) OR (approved_date < now()))")
query.push('ORDER BY similarity(title,$1) DESC LIMIT 99')
This version using ILIKE is working:
var query = ['SELECT title, user FROM reviews']
query.push("WHERE title ILIKE '%' || $1 || '%' AND NOT (user = $2) AND ((approved_date IS NULL) OR (approved_date < now()))")
query.push('ORDER BY similarity(title,$1) DESC LIMIT 99')
Also, am I to understand from the documentation that using the distance operator <-> will not work properly here because I am using a GIN index rather than GIST? Thanks.
I am getting array from front end to perform filters according that inside the SQL query.
I want to apply a LIKE filter on the array. How to add an array inside LIKE function?
I am using Angular with Html as front end and Node as back end.
Array being passed in from the front end:
[ "Sports", "Life", "Relationship", ...]
SQL query is :
SELECT *
FROM Skills
WHERE Description LIKE ('%Sports%')
SELECT *
FROM Skills
WHERE Description LIKE ('%Life%')
SELECT *
FROM Skills
WHERE Description LIKE ('%Relationship%')
But I am getting an array from the front end - how to create a query for this?
In SQL Server 2017 you can use OPENJSON to consume the JSON string as-is:
SELECT *
FROM skills
WHERE EXISTS (
SELECT 1
FROM OPENJSON('["Sports", "Life", "Relationship"]', '$') AS j
WHERE skills.description LIKE '%' + j.value + '%'
)
Demo on db<>fiddle
As an example, for SQL Server 2016+ and STRING_SPLIT():
DECLARE #Str NVARCHAR(100) = N'mast;mode'
SELECT name FROM sys.databases sd
INNER JOIN STRING_SPLIT(#Str, N';') val ON sd.name LIKE N'%' + val.value + N'%'
-- returns:
name
------
master
model
Worth to mention that input data to be strictly controlled, since such way can lead to SQL Injection attack
As the alternative and more safer and simpler approach: SQL can be generated on an app side this way:
Select * from Skills
WHERE (
Description Like '%Sports%'
OR Description Like '%Life%'
OR Description Like '%Life%'
)
A simple map()-call on the words array will allow you to generate the corresponding queries, which you can then execute (with or without joining them first into a single string).
Demo:
var words = ["Sports", "Life", "Relationship"];
var template = "Select * From Skills Where Description Like ('%{0}%')";
var queries = words.map(word => template.replace('{0}', word));
var combinedQuery = queries.join("\r\n");
console.log(queries);
console.log(combinedQuery);
I want to query my psql database from my rails app using the example query string:
select * from venues where upper(regexp_replace(postcode, ' ', '')) = '#{postcode}' or name = '#{name}'
There are 2 aspects to this query:
the first is to compare against a manipulated value in the database (upper and regexp_replace) which I can do within an active record where method
the second is to provide the or condition which appears to require the use of ARel
I would appreciate some help in joining these together.
See squeel, it can handle OR queries and functions in a pretty human friendly way: https://github.com/activerecord-hackery/squeel & http://railscasts.com/episodes/354-squeel
for example:
[6] pry(main)> Page.where{(upper(regexp_replace(postcode, ' ', '')) == 'foo') | (name == 'bar')}.to_sql
=> "SELECT \"pages\".* FROM \"pages\" WHERE upper(regexp_replace(\"pages\".\"postcode\", ' ', '')) = 'foo'"
alternative is to code the query directly:
scope :funny_postcode_raw, lambda{ |postcode, name| where("upper(regexp_replace(postcode, ' ', '')) = ? or name = ?", postcode, name) }
I don't suggest going the Arel path, not worth it 99.9% of the time
NOTE: the OR operator for squeel is |
For eg. i have:
Input:
ArrayList<Integer> idlist
String nameText
String q = "DELETE FROM myTable AS a WHERE a.name LIKE '%' || :name || '%' OR a.id IN (:idlist)";
Query query = entityManager.createQuery(q);
query.setParameter("name", nameText);
query.setParameter("idlist", idlist);
query.executeUpdate();
Would the above snippet of code susceptible to sql injections?
I have one more question is there a diffrence between the way above query is built and if we form query by criterias:
CriteriaBuilder cb = em.getCriteriaBuilder();
CriteriaQuery<Country> q = cb.createQuery(Country.class);
Root<Country> c = q.from(Country.class);
q.select(c);
Is there any advantage of one over another?
No, it wouldn't. That's the whole point of parameterized queries.
There is no chance for SQL injection with prepared statements.
But if the values of parameter are entered by users, the expression LIKE % || :name || % can be misused by entering only one letter and deleting all the records that contain that letter anywhere in the a.name column
Alright, so I have a query that looks like this:
SELECT
`orders`.*,
GROUP_CONCAT(
CONCAT(
`menu_items`.`name`,
' ($',
FORMAT(`menu_items`.`price`,2),
')'
) SEPARATOR '<br>'
) as `items`,
SUM(`menu_items`.`price`) as `additional`,
`children`.`first_name`,
`children`.`last_name`,
`organizations`.`base_price`
FROM
`orders`, `order_items`, `menu_items`, `children`, `organizations`
WHERE
`order_items`.`menu_item_id` = `menu_items`.`id` AND
`order_items`.`order_id` = `orders`.`id` AND
`orders`.`added_by` = {$user_id} AND
`orders`.`date` > '{$cutoff}' AND
`children`.`id` = `orders`.`child_id` AND
`organizations`.`id` = `children`.`organization_id`
GROUP BY
`orders`.`id`
I know it's a monstrosity and that some people will die before not using explicit joins. Ignoring that, however, what I wish to do is to only use the CONCAT inside the GROUP_CONCAT if the menu_items.price is greater than 0, otherwise only return menu_items.name. I have had, however, no success trying to throw an IF in there. I've read the manual but all the ways that I've tried aren't working and I'm pretty sure I'm missing something on the whole conditional statements thing.
Have you tried using something like this?
CASE WHEN 'menu_items'.'price' = 0 THEN 'menu.items'.'name' ELSE CONCAT (etc) END
Replacing the CONCAT statement of course.
Something like this should work (but I didn't test it, sorry):
GROUP_CONCAT(
CONCAT(
`menu_items`.`name`,
IF(`menu_items`.`price` > 0, -- <condition>
CONCAT(' ($', FORMAT(`menu_items`.`price`,2), ')'), -- <true-expr>
'' -- <false-expr>
)
)
SEPARATOR '<br>'
) as `items`,
The IF() function is really simple:
IF( <condition>, <true-expr>, <false-expr> )
The function has three arguments: the first is <condition>. If the condition evaluates to true, the function returns the result of <true-expr>. Else the function returns the result of <false-expr>.
Things get harder to get right when you use really long, multi-line expressions that contain parentheses and commas and so on. You just have to do it carefully. I suggest starting with more simple expressions and then build up.