Use Ruby To Convert Any Query To Count Query - sql

Question

In my application I use PG to execute queries defined by the user in-app.
require 'pg'
database = PG.connect(*credentials)
query = 'select id, created_at from users where id % 2 = 0'
database.connection.exec(query)
Part of the application requires fetching a count before running the actual query so I use regex to convert the query to a count query. (Assume LIMIT and ORDER BY are not allowed)
query = 'select id, created_at from users where id % 2 = 0'
query.gsub!(%r{(?<=SELECT)[^\/]+(?=FROM)}, ' count(*) ')
count = database.exec(query).first['count'].to_i
But if the query includes CTE's and/or sub-queries...
query = 'with new_table as (select id from users where id % 2 = 0)
select created_at, name from users where id in (select * from new_table)'
the above regex doesn't work, and I haven't been able to figure out another regex based solution.
Using SQL, Ruby, or REGEX, how could I convert any query a read-only db user could perform into a count query WITHOUT wrapping the query in its own CTE or just running the query and counting the results?
More simply, given a query, how can one get the row count for that query without actually running the full query?
Any engineers at Looker, PeriscopeData, or Mode should have this one in the bag :-)

Answer 1

Modifying SQL queries with a regex is a non-starter for all the reasons you don't try to modify XML with a regex: you need something which understands the grammar. What you're looking for is a SQL Query Builder.
A SQL Query Builder is sort of like an ORM without the ORM. You use it to write SQL queries using method calls, not strings, but you don't have to tell it what all your tables and columns are like an ORM, nor do you have to make classes for all your tables. It just makes SQL queries.
Your query is held as an object and only turned into SQL when it's time to communicate with the database. If you want to modify the query, you do it with method calls and regenerate the SQL. So you can add where clauses and group bys and limits and add more rows to select and join tables and, yes, count.
They also will often smooth over SQL incompatibilities for you, so the same code can run on MySQL or SQLite or Postgresql or Oracle.
A good non-Ruby one is Knex.js in Javascript. I'm having a hard time finding a pure SQL Query Builder for Ruby. The ones I've found (ActiveRecord, Sequel, squeel, and ARel) are all ORMs and require you to set up classes and schemas and all that. The only one I could find for Ruby that isn't an ORM is qdsl.

Answer 2

Easy way: create a new query from your existing one by just surrounding it to convert it into a subquery:
require 'pg'
database = PG.connect(*credentials)
query = 'select id, created_at from users where id % 2 = 0'
# Create a `count` query, based on the existing one.
# The original query must NOT end with ';'
count_query = 'SELECT count(*) AS count FROM (' + query + ') AS q0'
database.connection.exec(count_query)
# Follow on
database.connection.exec(query)
This may not be the most efficient way of getting the count, but I think it is the simplest and (probably) the less error-prone. It assumes that the original query is well-formed, and that it does not call functions with side-effects [a practice that should be reserved for very few use-cases].
Assuming the users table is similar to:
CREATE TABLE users AS
SELECT * FROM
(VALUES
(1::integer, 'name1'::text, now()::timestamp),
(2, 'name2', now() - interval '1 hour'),
(3, 'name3', now() - interval '2 hours'),
(4, 'name4', now() - interval '2 hours')
) x(id, name, created_at) ;
This solution works with WITH clases, because the following SQL query is legal:
SELECT count(*) FROM
(
with new_table as (select id from users where id % 2 = 0)
select created_at, name from users where id in (select * from new_table)
) AS q0 ;
... and returns 2 as expected.