Optimize the Query in PostgreSQL - Takes too long - sql

Made this view to see all the 'socios' turned their status_contrato to 301 in the last 30 days
The thing is the query takes around 30 secs, and I have to get down to 20 secs max
CREATE OR REPLACE VIEW public.socios_desligados_inadimplencis_ultimos_30_dias_v2
AS SELECT tb.nome,
tb.cpf,
tb.email,
tb.plano,
tb.status_contrato,
CASE
WHEN ad.dt_inicial IS NULL THEN tb.dt_inicial
ELSE ad.dt_inicial
END AS data_admissao,
tb.dthr_desligamento,
tb.total_parcelas_pagas
FROM ( SELECT p.idpessoa,
p.nome,
p.cpf,
p.email,
pl.dscplano AS plano,
ea.dscestado_ativacao AS status_contrato,
eal.ins_dthr AS dthr_desligamento,
cr.valor_total AS total_parcelas_pagas,
con.dt_inicial
FROM base_pessoa p
JOIN ( SELECT gs_contrato.idpessoa,
gs_contrato.idcontrato,
gs_contrato.idplano,
gs_contrato.idestado_ativacao,
gs_contrato.dt_inicial
FROM gs_contrato
WHERE gs_contrato.idestado_ativacao::text = '301'::text AND gs_contrato.alt_dthr >= (CURRENT_DATE - 30)) con ON p.idpessoa::text = con.idpessoa::text
JOIN gs_plano pl ON con.idplano::text = pl.idplano::text
JOIN gs_estado_ativacao ea ON con.idestado_ativacao::text = ea.idestado_ativacao::text
JOIN ( SELECT row_number() OVER (PARTITION BY lg.idcontrato ORDER BY lg.ins_dthr DESC) AS linha,
lg.idcontrato,
lg.idestado_ativacao,
lg.ins_dthr
FROM gs_estado_ativacao_l lg
JOIN gs_contrato con_1 ON con_1.idcontrato::text = lg.idcontrato::text
WHERE lg.idestado_ativacao::text = '301'::text AND con_1.idestado_ativacao::text = '301'::text AND con_1.alt_dthr >= (CURRENT_DATE - 30) AND lg.ins_dthr >= (CURRENT_DATE - 30)) eal ON con.idcontrato::text = eal.idcontrato::text AND eal.linha = 1
JOIN ( SELECT ccr.idcontrato,
sum(ccr.valor - ccr.valor_desconto + ccr.valor_juros + ccr.valor_multa + ccr.valor_adesao + ccr.valor_acrescimo) AS valor_total
FROM cx_conta_receber ccr
JOIN gs_contrato c ON c.idcontrato::text = ccr.idcontrato::text
WHERE ccr.status::text = 'PA'::text AND c.idestado_ativacao::text = '301'::text AND c.alt_dthr >= (CURRENT_DATE - 30)
GROUP BY ccr.idcontrato) cr ON cr.idcontrato::text = con.idcontrato::text) tb
JOIN data_adesao ad ON tb.idpessoa::text = ad.idpessoa::text;

I checked your query. I write some recommendations for performance.
Recommended using materialized CTE (with as materialized) instead of subqueries.
Maybe you have used subqueries or table-joining to get a few values. You can use return-table functions instead at this.
Check your indexes. Fields are used on the conditions (where or join), on the order by or group by commands must be indexed.
Not recommended use casting fields or set functions to fields in the conditions. In this time DB can not use indexes. If you need set functions to fields in the conditions, then recommended creating expression index (function-based index).
Your query has some cast types:
WHERE gs_contrato.idestado_ativacao::text = '301'::text
AND gs_contrato.alt_dthr >= (CURRENT_DATE - 30)) con ON
p.idpessoa::text = con.idpessoa::text
JOIN gs_plano pl
ON con.idplano::text = pl.idplano::text
JOIN gs_estado_ativacao ea
ON con.idestado_ativacao::text = ea.idestado_ativacao::text
WHERE lg.idestado_ativacao::text = '301'::text
AND con_1.idestado_ativacao::text = '301'::text
AND con_1.alt_dthr >= (CURRENT_DATE - 30)
AND lg.ins_dthr >= (CURRENT_DATE - 30)) eal ON con.idcontrato::text = eal.idcontrato::text AND eal.linha = 1
If you need cast types, so use the same cast types at creating indexes. Expression index.
And the latest and most important recommendation is that use explain analyze for this query to get execute plan of DB.

Related

SQL Poor Performance on join query

i'm experiencing really poor performance with this query, probably because of my poor experience with join.
The following query have to run on a DB2 database.
SELECT A.n_ann_ord
|| A.n_prg_ord AS numeroOrdine,
DATE(B.d_ins) AS dataInserimentoEGO,
DATE(A.d_ord) AS dataInserimento,
C.n_lot_prd AS lotto,
E.t_rag_soc AS consorzio,
D.t_rag_soc AS cliente,
b.c_obu_new AS obuid,
j.c_tar AS targa,
j.c_naz AS nazione,
C.c_pos AS postazione,
CASE
WHEN B.f_ric_obu_sat = 'S' THEN 'Satellitare'
ELSE 'Non Satellitare'
END AS tipoTitolo,
F.t_des AS statoOrdine,
CASE
WHEN (SELECT 1
FROM evda.tetsau_sos_tit_mul X
WHERE x.d_ord = a.d_ord
FETCH first ROW only) = 1 THEN 'S'
ELSE 'N'
END AS SOSTITUZIONE,
DATE(G.d_ape_pro) AS aperturaLotto,
DATE(G.d_chiu_pro) AS chiusuraLotto,
DATE(G.d_com_con) AS confezionamento,
DATE(C.d_spe_spe) AS spedizione,
C.c_ide_spe AS trackingNumber,
DATE(H.d_acq_con) AS consegna
FROM evda.tetsvi_ord A
JOIN evda.tets3t_int_srv B
ON A.c_ctp_cli = B.c_ctp_cli
AND A.d_ord = B.d_ord
JOIN evda.tetski_int_con_ord C
ON A.c_ctp_cli = C.c_ctp_cli
AND A.d_ord = C.d_ord
JOIN evda.tets25_ctp D
ON C.c_ctp_cli = D.c_ctp
JOIN evda.tets25_ctp E
ON C.c_ctp_con = E.c_ctp
JOIN evda.tetsvk_sta_ord F
ON A.c_sta_ord = F.c_sta_ord
JOIN evda.tetsvp_pre_tit I
ON a.c_ctp_cli = i.c_ctp_cli
AND a.d_ord = i.d_ord
JOIN evda.tetsmj_tit_pre_est J
ON a.c_ctp_cli = j.c_ctp_cli
AND a.d_ord = j.d_ord
AND b.n_prg_tit_pre = j.n_prg
LEFT JOIN evda.tetskk_lot_prd G
ON C.n_lot_prd = G.n_lot_prd
LEFT JOIN evda.tets3u_int_srv_spe H
ON B.d_ins = H.d_ins
WHERE DATE(a.d_ord) <= CURRENT_DATE
AND DATE(a.d_ord) >= CURRENT_DATE - 30 DAYS
GROUP BY A.n_ann_ord
|| A.n_prg_ord,
DATE(B.d_ins),
DATE(A.d_ord),
C.n_lot_prd,
E.t_rag_soc,
D.t_rag_soc,
b.c_obu_new,
j.c_tar,
j.c_naz,
C.c_pos,
B.f_ric_obu_sat,
F.t_des,
a.d_ord,
DATE(G.d_ape_pro),
DATE(G.d_chiu_pro),
DATE(G.d_com_con),
DATE(C.d_spe_spe),
C.c_ide_spe,
DATE(H.d_acq_con)
Avoid using functions on table columns in where clause. Also, check whether "group by" is necessary as suggested by #jarlh
If d_ord is timestamp column, change where clause to
a.d_ord < timestamp( CURRENT_DATE + 1 days)
AND a.d_ord >= timestamp( CURRENT_DATE - 30 DAYS )

Rails - how to convert output of find_by_sql (array) to an ActiveRecord relation?

I am in a need of running a complex SQL query and because I didn't know how to construct the query using ActiveRecod, I had to use a raw SQL query by using find_by_sql:
scope :get_cars, -> do
find_by_sql('select * from
(SELECT cars.*,
manufacturer.company_name AS manufacturer_company_name,
services.a_num AS service_a_num,
(SELECT car_documents.file_url FROM car_documents
WHERE car_documents.car_id = cars.id AND car_documents.doc_type = 1 LIMIT 1) AS doc1_file_url,
(SELECT car_documents.file_s3_url FROM car_documents
WHERE car_documents.cart_id = cars.id AND car_documents.doc_type = 3 LIMIT 1) AS file_inv,
(SELECT car_data.demand_lvl FROM car_data
WHERE car_data.car_id = cars.id) AS demand_lvl,
(SELECT car_logs.invoice FROM car_logs
WHERE car_logs.car_id = cars.id AND car_logs.invoice = 1 LIMIT 1) AS invoice_sent
FROM "cars"
INNER JOIN "services" ON "services"."id" = "cars"."service_id"
LEFT JOIN manufacturers ON manufacturers.id = cars.manufacturer_id
WHERE (cars.status_id != 6
AND cars.delivery_date < NOW() - INTERVAL \'15 days\'
) ORDER BY cars.pickup_date ASC) t
Where doc1_file_url IS NULL OR file_inv IS NULL OR invoice_sent IS NULL')
end
The output is an array. How do I convert this array into an ActiveRecord object? Or possibly, is there any workaround?
I think, You can use the from() method from the Active Record interface. You can use a subquery as a table for a SQL select statement with help of from() method. check the below example. In this way, you will get the active record object.
subquery = Setting.limit(10)
Setting.from("(#{subquery.to_sql}) settings")
In your case,
subquery = "(SELECT cars.*,
manufacturer.company_name AS manufacturer_company_name,
services.a_num AS service_a_num,
(SELECT car_documents.file_url FROM car_documents
WHERE car_documents.car_id = cars.id AND car_documents.doc_type = 1 LIMIT 1) AS doc1_file_url,
(SELECT car_documents.file_s3_url FROM car_documents
WHERE car_documents.cart_id = cars.id AND car_documents.doc_type = 3 LIMIT 1) AS file_inv,
(SELECT car_data.demand_lvl FROM car_data
WHERE car_data.car_id = cars.id) AS demand_lvl,
(SELECT car_logs.invoice FROM car_logs
WHERE car_logs.car_id = cars.id AND car_logs.invoice = 1 LIMIT 1) AS invoice_sent
FROM "cars"
INNER JOIN "services" ON "services"."id" = "cars"."service_id"
LEFT JOIN manufacturers ON manufacturers.id = cars.manufacturer_id
WHERE (cars.status_id != 6
AND cars.delivery_date < NOW() - INTERVAL \'15 days\'
) ORDER BY cars.pickup_date ASC)"

How can i optimize this oracle query?

i'm working with oracle 11g and we have a problem, well, this query takes forever to execute, the main table tbl_inc has about 17 million records, is there any way i can improve this query?
i can't add indexes, i don't have privileges for that.
SELECT count(*) FROM TBL_INC INC LEFT JOIN TBL_PDS P ON INC.SID = P.TRX
INNER JOIN TBL_ASTS ASTS ON ASTS.CODE = INC.CODE AND ASTS.MIT
= INC.MIT AND ASTS.OPE_PROD = 3
WHERE (INC.INC_DATE -1) >= to_date('29/10/20', 'DD/MM/YY')
AND INC.INC_DATE - 1 <= to_date('05/11/20', 'DD/MM/YY')
AND INC.OPE = 50 AND SUBSTR(INC.CARD_NMBR, 1, 6) = 123456
AND INC.MIT='05' AND INC.CODE='00';
thanks
First, I would rewrite this so the where clause does not have expressions on the columns:
SELECT count(*)
FROM TBL_INC INC LEFT JOIN
TBL_PDS P
ON INC.SID = P.TRX INNER JOIN
TBL_ASTS ASTS
ON ASTS.CODE = INC.CODE AND
ASTS.MIT = INC.MIT AND
ASTS.OPE_PROD = 3
WHERE INC.INC_DATE >= to_date('29/10/20', 'DD/MM/YY') + interval '1 day' AND
AND INC.INC_DATE - 1 <= to_date('05/11/20', 'DD/MM/YY') + interval '1 day' AND
INC.OPE = 50 AND
INC.CARD_NMBR LIKE '123456%' AND
INC.MIT = '05' AND INC.CODE = '00';
Then for this query, you want an index on: TBL_INC(MIT, CODE, OPE, INC_DATE, CARD_NMBR). I am guessing that you have indexes on the JOIN keys used in the other tables, but those would be TBL_PDS(TRX) and TBL_ASTS(CODE< MIT, OPE_PROD).
It is a very good practice to reduce the number of the rows from the join, rather that apply a 'where' condition to the end
SELECT count(1) FROM TBL_INC INC
INNER JOIN TBL_ASTS ASTS
ON ASTS.CODE = INC.CODE
AND ASTS.MIT = INC.MIT
AND ASTS.OPE_PROD = 3
AND INC.OPE = 50
AND SUBSTR(INC.CARD_NMBR, 1, 6) = 123456
AND INC.MIT='05'
AND INC.CODE='00'
AND (INC.INC_DATE -1) >= to_date('29/10/20', 'DD/MM/YY')
AND INC.INC_DATE - 1 <= to_date('05/11/20', 'DD/MM/YY')
LEFT JOIN
TBL_PDS P
ON INC.SID = P.TRX

SQL - SELECT subquery AS BIT value for EXIST check

I have a problem.
I'm trying to get a BIT value to check whether a person has entered the building last night between 10pm to midnight. When I run the subquery code by itself, it gives me the results I need. As I'm working with SSRS2008 I need for all the results to be in the same stored procedure.
So the problem is, it gives me the bit values somewhat right, for the ones that are obviously false, it gives false, for the ones that are obviously true, it gives true. But for the ones in the middle (the day shift, who leave at 23) it gives the results somewhat random..
Does anyone have a clue?
SELECT DISTINCT TOP 200
Events.LoggedTime,
PTUsers.Name,
PTDoors.PTDoorsID,
PTUsers.AccessLevel,
CAST(CASE
WHEN EXISTS (SELECT Events.LoggedTime
FROM Events
INNER JOIN PTUsers AS PTUsers_1 ON Events.GlobalIndex1 = PTUsers.GlobalRecord
INNER JOIN PTDoors AS PTDoors_1 ON Events.RecordIndex2 + 1 = PTDoors.Address
WHERE (DATEPART(day, Events.LoggedTime) = DATEPART(day, GETDATE() - 1))
AND (DATEPART(hour, Events.LoggedTime) IN (22, 23))
AND (PTDoors_1.PTDoorsID = 14)) THEN 1 ELSE 0 END AS BIT) AS Night
FROM
Events
INNER JOIN
PTUsers ON Events.GlobalIndex1 = PTUsers.GlobalRecord
INNER JOIN
PTDoors ON Events.RecordIndex2 + 1 = PTDoors.Address
WHERE
(PTUsers.Panel = 0)
AND (PTDoors.Panel = 0)
AND (PTDoors.PTDoorsID = 14)
AND (DATEPART(day, Events.LoggedTime) = DATEPART(day, GETDATE()) - 1)
AND (PTUsers.AccessLevel IN (3))
ORDER BY
Events.LoggedTime DESC
#lrd i did the corrections you suggested,
thanks for pointing out the table aliases :)
i removed the cast, so now i get the BIT column. but now the problem is that i get all "1"'s as results.
What baffles me is the that subquery works as it should as a query on it's own. it goes back a day, and displays the entries on that door in the given timeframe.
now i'm trying to compare that information for the same person, meaning - i see a person in the list, arriving yesterday at 6am, check if that person also arrived the day before that between 22 & midnight and return a bit value to display that.
SELECT DISTINCT TOP 200 Events.LoggedTime, PTUsers.Name, PTDoors.PTDoorsID, PTUsers.AccessLevel, CASE WHEN EXISTS
(SELECT Events.LoggedTime
FROM Events INNER JOIN
PTUsers AS PTUsers_1 ON Events.GlobalIndex1 = PTUsers_1.GlobalRecord INNER JOIN
PTDoors AS PTDoors_1 ON Events.RecordIndex2 + 1 = PTDoors_1.Address
WHERE (DATEPART(day, Events.LoggedTime) = DATEPART(day, GETDATE() - 1)) AND (DATEPART(hour, Events.LoggedTime) IN (22, 23)) AND
(PTDoors_1.PTDoorsID = 14)) THEN 1 ELSE 0 END AS BIT
FROM Events INNER JOIN
PTUsers ON Events.GlobalIndex1 = PTUsers.GlobalRecord INNER JOIN
PTDoors ON Events.RecordIndex2 + 1 = PTDoors.Address
WHERE (PTUsers.Panel = 0) AND (PTDoors.Panel = 0) AND (PTDoors.PTDoorsID = 14) AND (DATEPART(day, Events.LoggedTime) = DATEPART(day, GETDATE())
- 1) AND (PTUsers.AccessLevel IN (3))
ORDER BY Events.LoggedTime DESC
I don't think you need a CAST because you are explicitly defining Night as a BIT By setting the result to EXISTS(), which is a bit. I removed the query that was incorrect.
I see your problem. You are not using the correct table alias for your join constraint in your subquery.
It should be:
INNER JOIN PTUsers AS PTUsers_1 ON Events.GlobalIndex1 = PTUsers_1.GlobalRecord
INNER JOIN PTDoors AS PTDoors_1 ON Events.RecordIndex2 + 1 = PTDoors_1.Address
Also,check and make sure you are using the correct values in your join. I would change the following for a test.
FROM Events Events_2
INNER JOIN PTUsers AS PTUsers_1 ON Events_2.GlobalIndex1 = PTUsers_1.GlobalRecord
INNER JOIN PTDoors AS PTDoors_1 ON Events_2.RecordIndex2 + 1 = PTDoors_1.Address

How to use min() and max() in an efficient way?

I have an sql query where I check if a value is between a max and a min value of a table. I've now implement this as follows:
SELECT spectrum_id, feature_table_id
FROM 'spectrum', 'feature'
WHERE `spectrum`.msrun_msrun_id = 1
AND `feature`.msrun_msrun_id = 1
AND (SELECT min(rt) FROM `convexhull` WHERE `convexhull`.feature_feature_table_id = `feature`.feature_table_id) <= scan_start_time
AND scan_start_time <= (SELECT max(rt) FROM `convexhull` WHERE 'convexhull'.feature_feature_table_id = 'feature'.feature_table_id)
AND (SELECT min(mz) FROM `convexhull` WHERE `convexhull`.feature_feature_table_id = `feature`.feature_table_id) <= base_peak_mz
AND base_peak_mz <= (SELECT max(mz) FROM `convexhull` WHERE `convexhull`.feature_feature_table_id = `feature`feature_table_id)
This is running very slowly, because I'm selecting from convexhull 4 times every time I run this query, so I tried to improve it using an inner join:
SELECT spectrum_id, feature_table_id
FROM 'spectrum', 'feature'
INNER JOIN `convexhull` ON `convexhull`.feature_feature_table_id = `feature`.feature_table_id
WHERE `spectrum`.msrun_msrun_id = ? "+
AND `feature`.msrun_msrun_id = ? "+
AND min(`convexhull`.rt) <= scan_start_time "+
AND scan_start_time <= max(`convexhull`.rt) "+
AND min(`convexhull`.mz) <= base_peak_mz "+
AND base_peak_mz <= max(`convexhull`.mz)", spectrumFeature_InputValues)
However, the min() and max() statements can only be used after a select statement. How can I make the first query more efficient, so that I can get the min and max rt and mz without having to do 4 queries?
EDIT: had a few more mins and looked again and realised all the data comes from that one table so something like this should work
SELECT
spectrum_id
,feature_table_id
FROM
spectrum AS s
INNER JOIN feature AS f
on f.msrun_msrun_id = s.msrun_msrun_id
INNER JOIN (select
feature_feature_table_id
,min(rt) AS rtMin
,max(rt) AS rtMax
,min(mz) AS mzMin
,max(mz) as mzMax
FROM
convexhull
GROUP BY
feature_feature_table_id
) AS t
ON t.feature_feature_table_id = f.feature_table_id
WHERE
s.msrun_msrun_id = 1
AND s.scan_start_time >= t.rtMin
AND s.scan_start_time <= t.rtMax
AND base_peak_mz >= t.mxMin
AND base_peak_mz <= t.mzMax
I think you want to select from the convexhull table and group by feature_feature_table_id getting the min and max rt within that grouping.
you can then wrap that select in brackets give it a name (as t) and join to it.
Hope this is enought to get you on the road.. if not create a sample schema here: http://sqlfiddle.com/
and put in your query and i can modify it.
as a side note, I think you wan to join these tables on a particular field rather than select from both with a where clause compare:
SELECT spectrum_id, feature_table_id
FROM 'spectrum', 'feature'
WHERE `spectrum`.msrun_msrun_id = 1
AND `feature`.msrun_msrun_id = 1
and:
SELECT
spectrum_id
,feature_table_id
FROM
spectrum AS s
INNER JOIN feature AS f
on f.msrun_msrun_id = s.msrun_msrun_id
WHERE
s.msrun_msrun_id = 1
If i have got something wrong there let me know.