TypeORM: column must appear in the GROUP BY clause or be used in an aggregate function - sql

I'm trying to work out why the following raw SQL works perfectly with PostGres, whereas the SQL generated via TypeORM does not.
Running this works:
SELECT symbol, MAX(created_at) AS created_at
FROM update_history
WHERE exchange = 'NYSE'
AND data_type = 'companyRecord'
GROUP BY symbol
ORDER BY created_at ASC
Example: https://dbfiddle.uk/yocl-rBq
Whereas, the TypeORM sql generated by the following:
const result = this.repository
.createQueryBuilder('h1')
.select(['MAX(h1.createdAt) AS created_at', 'h1.symbol'])
.where('h1.exchange = :exchange', { exchange })
.andWhere('h1.dataType = :dataType', { dataType })
.groupBy('h1.symbol')
.orderBy({ created_at: 'ASC' })
.take(limit)
.getMany();
/* Produces this:
SELECT "h1"."symbol" AS "h1_symbol",
MAX("h1"."created_at") AS created_at
FROM "update_history" "h1"
WHERE "h1"."exchange" = 'NYSE'
AND "h1"."data_type" = 'companyRecord'
GROUP BY "h1"."symbol"
ORDER BY created_at ASC LIMIT 5000
*/
Example (second select box): https://dbfiddle.uk/yocl-rBq
Returns the following error:
QueryFailedError: column "h1.id" must appear in the GROUP BY clause or be used in an aggregate function
If I add h1.id to the GROUP BY clause, the query no longer returns the correct result set, as it is effectively different.
As you can see from the DBFiddle links, both generated SQL queries work outside of TypeORM.
What am I missing here?

You're using getMany() which always adds id field in the select. The query that you've pasted is most probably has been logged before adding getMany(). Try setting logging: true in your ormconfig.js and you'll see the exact query being fired.
You should use getRawMany() and you'll be able to get the exact query and also the desired result.

Related

Prevent Oracle error ORA-18043 in select statement

I'm storing in a table the logs of an API in my application. In one of the columns, I store the raw Json sent in the HTTP request.
I've been tasked to create a page in my application dedicated to easily explore every entered logs, with filters, sorting, etc.
One of the sorting needed is on the date that was indicated in the Json body of the HTTP call. I've managed to do so using the Oracle's Json API :
SELECT *
FROM FUNDING_REQUEST f
ORDER BY TO_TIMESTAMP_TZ(JSON_VALUE(
f.REQUEST_CONTENTS,
'$.leasing_information.consumer_request_date_time'
), 'YYYY/MM/DD"T"HH24:MI:SS.FFTZH:TZM') ASC
This works either if $.leasing_information.consumer_request_date_time is defined or not but I have an issue when the value is wrongly formatted. In one of my test, I sent this to my API :
{
[...],
"leasing_information": {
"consumer_request_date_time": "2021-25-09T12:30:00.000+02:00",
[...]
}
}
There is no 25th month, and my SQL query now returns the following error :
ORA-01843: not a valid month.
I would like to handle this value as NULL rather than returning an error, but it seems like the TO_TIMESTAMP_TZ DEFAULT clause does not really work the way I want it to. Doing this also returns an error :
SELECT *
FROM FUNDING_REQUEST f
ORDER BY TO_TIMESTAMP_TZ(JSON_VALUE(
f.REQUEST_CONTENTS,
'$.leasing_information.consumer_request_date_time'
) DEFAULT NULL ON CONVERSION ERROR, 'YYYY/MM/DD"T"HH24:MI:SS.FFTZH:TZM') ASC
ORA-00932:inconsistent datatypes ; expected : - ; got : TIMESTAMP WITH TIME ZONE
I would also like to avoid using a PL/SQL function if possible, how can I prevent this request from returning an error ?
You don't need to extract a string and then convert that to a timestamp; you can do that within the json_value(), and return null if that errors - at least from version 12c Release 2 onwards:
SELECT *
FROM FUNDING_REQUEST f
ORDER BY JSON_VALUE(
f.REQUEST_CONTENTS,
'$.leasing_information.consumer_request_date_time'
RETURNING TIMESTAMP WITH TIME ZONE
NULL ON ERROR
) ASC
db<>fiddle, including showing what the string and timestamp extractions evaluate too (i.e. a real timestamp value, or null).

Add array of other records from the same table to each record

My project is a Latin language learning app. My DB has all the words I'm teaching, in the table 'words'. It has the lemma (the main form of the word), along with the definition and other information the user needs to learn.
I show one word at a time for them to guess/remember what it means. The correct word is shown along with some wrong words, like:
What does Romanus mean? Greek - /Roman/ - Phoenician - barbarian
What does domus mean? /house/ - horse - wall - senator
The wrong options are randomly drawn from the same table, and must be from the same part of speech (adjective, noun...) as the correct word; but I am only interested in their lemma. My return value looks like this (some properties omitted):
[
{ lemma: 'Romanus', definition: 'Roman', options: ['Greek', 'Phoenician', 'barbarian'] },
{ lemma: 'domus', definition: 'house', options: ['horse', 'wall', 'senator'] }
]
What I am looking for is a more efficient way of doing it than my current approach, which runs a new query for each word:
// All the necessary requires are here
class Word extends Model {
static async fetch() {
const words = await this.findAll({
limit: 10,
order: [Sequelize.literal('RANDOM()')],
attributes: ['lemma', 'definition'], // also a few other columns I need
});
const wordsWithOptions = await Promise.all(words.map(this.addOptions.bind(this)));
return wordsWithOptions;
}
static async addOptions(word) {
const options = await this.findAll({
order: [Sequelize.literal('RANDOM()')],
limit: 3,
attributes: ['lemma'],
where: {
partOfSpeech: word.dataValues.partOfSpeech,
lemma: { [Op.not]: word.dataValues.lemma },
},
});
return { ...word.dataValues, options: options.map((row) => row.dataValues.lemma) };
}
}
So, is there a way I can do this with raw SQL? How about Sequelize? One thing that still helps me is to give a name to what I'm trying to do, so that I can Google it.
EDIT: I have tried the following and at least got somewhere:
const words = await this.findAll({
limit: 10,
order: [Sequelize.literal('RANDOM()')],
attributes: {
include: [[sequelize.literal(`(
SELECT lemma FROM words AS options
WHERE "partOfSpeech" = "options"."partOfSpeech"
ORDER BY RANDOM() LIMIT 1
)`), 'options']],
},
});
Now, there are two problems with this. First, I only get one option, when I need three; but if the query has LIMIT 3, I get: SequelizeDatabaseError: more than one row returned by a subquery used as an expression.
The second error is that while the code above does return something, it always gives the same word as an option! I thought to remedy that with WHERE "partOfSpeech" = "options"."partOfSpeech", but then I get SequelizeDatabaseError: invalid reference to FROM-clause entry for table "words".
So, how do I tell PostgreSQL "for each row in the result, add a column with an array of three lemmas, WHERE existingRow.partOfSpeech = wordToGoInTheArray.partOfSpeech?"
Revised
Well that seems like a different question and perhaps should be posted that way, but...
The main technique remains the same. JOIN instead of sub-select. The difference being generating the list of lemmas for then piping then into the initial query. In a single this can get nasty.
As single statement (actually this turned out not to be too bad):
select w.lemma, w.defination, string_to_array(string_agg(o.defination,','), ',') as options
from words w
join lateral
(select defination
from words o
where o.part_of_speech = w.part_of_speech
and o.lemma != w.lemma
order by random()
limit 3
) o on 1=1
where w.lemma in( select lemma
from words
order by random()
limit 4 --<<< replace with parameter
)
group by w.lemma, w.defination;
The other approach build a small SQL function to randomly select a specified number of lemmas. This selection is the piped into the (renamed) function previous fiddle.
create or replace
function exam_lemma_definition_options(lemma_array_in text[])
returns table (lemma text
,definition text
,option text[]
)
language sql strict
as $$
select w.lemma, w.definition, string_to_array(string_agg(o.definition,','), ',') as options
from words w
join lateral
(select definition
from words o
where o.part_of_speech = w.part_of_speech
and o.lemma != w.lemma
order by random()
limit 3
) o on 1=1
where w.lemma = any(lemma_array_in)
group by w.lemma, w.definition;
$$;
create or replace
function exam_lemmas(num_of_lemmas integer)
returns text[]
language sql
strict
as $$
select string_to_array(string_agg(lemma,','),',')
from (select lemma
from words
order by random()
limit num_of_lemmas
) ll
$$;
Using this approach your calling code reduces to a needs a single SQL statement:
select *
from exam_lemma_definition_options(exam_lemmas(4))
order by lemma;
This permits you to specify the numbers of lemmas to select (in this case 4) limited only by the number of rows in Words table. See revised fiddle.
Original
Instead of using a sub-select to get the option words just JOIN.
select w.lemma, w.definition, string_to_array(string_agg(o.definition,','), ',') as options
from words w
join lateral
(select definition
from words o
where o.part_of_speech = w.part_of_speech
and o.lemma != w.lemma
order by random()
limit 3
) o on 1=1
where w.lemma = any(array['Romanus', 'domus'])
group by w.lemma, w.definition;
See fiddle. Obviously this will not necessary produce the same options as your questions provides due to random() selection. But it will get matching parts of speech. I will leave translation to your source language to you; or you can use the function option and reduce your SQL to a simple "select *".

cannot group with eloquent with PostreSQL

Here's my Eloquent query:
$visits = Visit::orderBy('date', 'desc')->groupBy('user_id')->get(['date', 'user_id']);
But posgreSQL is refusing the query, telling me:
SQLSTATE[42803]: Grouping error: 7 ERROR: column "visits.date" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: select "date", "user_id" from "visits" group by...
the same stuff works on MySQL when I disable ONLY_FULLY_GROUP_BY
what can I do to make it work? It would be great if I didn't have to edit configs, just the code.
This selects the latest date for each user_id:
$sub = Visit::select('user_id', DB::raw('max("date") "date"'))->groupBy('user_id');
$sql = '(' . $sub->toSql() . ') as "sub"';
$visits = Visit::join(DB::raw($sql), function($join) {
$join->on('visits.user_id', 'sub.user_id')
->on('visits.date', 'sub.date');
})->orderBy('visits.date', 'desc')->get(['visits.date', 'visits.user_id']);
If there are multiple visits for a user_id and date combination, the query returns all of them. Removing the duplicates is possible, but makes the query more complex.
It's easier to remove them afterwards:
$visits = $visits->unique('user_id');
Using postgre sql you could use distinct on user_id and order your date column to pick latest date per user_id, In laravel you could raw expression
$visits = Visit::select(DB::raw('distinct on (user_id)'), 'date')
->orderBy('date', 'desc')
->get();
Demo

OrientDB using LET values in subQuery

How can you use a LET temporary variable inside the Where clause in an OrientDB SQL subQuery.
Here is the context in wich I'm trying to use it.
select *, $t.d from Currency
let $t = (select createdDate.asLong() as d from 13:1)
where createdDate.asLong() >= $t.d and #rid <> #13:1
order by createdDate ASC
The validation in the where statement for the dates does not work. The subQuery actually works on its own. The Query works as well when replacing $t.d with the result from the subQuery.
The $t.d is an array so you are comparing something like createdDate.asLong() >= [1234599]
You have to do this: createdDate.asLong() >= $t[0].d

How to count the number of rows with a date from a certain year in CodeIgniter?

I have the following query.
$query = $this->db->query('SELECT COUNT(*) FROM iplog.persons WHERE begin_date LIKE '2014%'');
I need to count the number of columns with a begin_date in the year 2014.
When I run this script I'm getting an error:
Parse error: syntax error, unexpected '2014' (T_LNUMBER) in C:\xampp\htdocs\iPlog2\application\controllers\stat.php on line 12
I was trying to change my CI script to
$query = $this->db->query('SELECT COUNT(*) FROM iplog.persons WHERE begin_date LIKE "2014%"');
but it caused an error.
You mean, count ROWS:
So for that, just count the number of rows you have based on a condition:
$year = '2014'
$this->db->from('iplog');
$this->db->like('begin_date', $year);
$query = $this->db->get();
$rowcount = $query->num_rows();
First, you have a simple typo regarding the use of single quotes. Your complete sql string should be double quoted so that your value-quoting can be single quoted.
Second, you are using inappropriate query logic. When you want to make a comparison on a DATE or DATETIME type column, you should NEVER be using LIKE. There are specific MYSQL functions dedicated to handling these types. In your case, you should be using YEAR() to isolate the year component of your begin_date values.
Resource: https://www.w3resource.com/mysql/date-and-time-functions/mysql-year-function.php
You could write the raw query like this: (COUNT(*) and COUNT(1) are equivalent)
$count = $this->db
->query("SELECT COUNT(1) FROM persons WHERE YEAR(begin_date) = 2014")
->row()
->COUNT;
Or if you want to employ Codeigniter methods to build the query:
$count = $this->db
->where("YEAR(begin_date) = 2014")
->count_all_results("persons");
You could return all of the values in all of the rows that qualify, but that would mean asking the database for values that you have no intention of using -- this is not best practice. I do not recommend the following:
$count = $this->db
->get_where('persons', 'YEAR(begin_date) = 2014')
->num_rows();
For this reason, you should not be generating a fully populated result set then calling num_rows() or count() when you have no intention of using the values in the result set.
Replace quotes like this :
$query = $this->db->query("SELECT COUNT(*) FROM iplog.persons WHERE begin_date LIKE '2014%'");
Double quote your entire query, then simple quote your LIKE criteria.