I am using sql server 2008 r2 and I have two database, which is one have 11.000 record and another is just 3000 record, when i do run this query
SELECT Right(rtrim(tbltransac.No_Faktur),6) as NoUrut,
tbltransac.No_Faktur,
tbltransac.No_FakturP,
tbltransac.Kd_Plg,
Tblcust.Nm_Plg,
GRANDTOTAL AS Total_Faktur,
tbltransac.Nm_Pajak,
tbltransac.Tgl_Faktur,
tbltransac.Tgl_FakturP,
tbltransac.Total_Distribusi
FROM Tblcust
INNER JOIN ViewGrandtotal AS tbltransac ON Tblcust.Kd_Plg = tbltransac.Kd_Plg
WHERE tbltransac.Kd_Trn = 'J'
and year(tbltransac.tgl_faktur)=2015
And ISNULL(tbltransac.No_OPJ,'') <> 'SHOP'
Order by Right(rtrim(tbltransac.No_Faktur),6) Desc
It takes me 1 minute 30 sec in my server (I query it using sql management tool) that have 3000 record but it only took 3 sec to do a query in my another server which is have 11000 record, whats wring with my database?
I've already tried to backup and restore my 3000 record database and restore it in my 11000 record server, it's faster.. took 30 sec to do a query, but it's still annoying if i compare to my 11000 record server. They are in the same spec
How this happend? what i should check? i check on event viewer, resource monitor or sql management log, i couldn't find any error or blocked connection. There is no wrong routing too..
Please help... It just happen a week ago, before this it was fine, and I haven't touch the server more than a month...
as already mentioned before, you have three issues in your query.
Just as an example, change the query to this one:
SELECT Right(rtrim(tbltransac.No_Faktur),6) as NoUrut,
tbltransac.No_Faktur,
tbltransac.No_FakturP,
tbltransac.Kd_Plg,
Tblcust.Nm_Plg,
GRANDTOTAL AS Total_Faktur,
tbltransac.Nm_Pajak,
tbltransac.Tgl_Faktur,
tbltransac.Tgl_FakturP,
tbltransac.Total_Distribusi
FROM Tblcust
INNER JOIN ViewGrandtotal AS tbltransac ON Tblcust.Kd_Plg = tbltransac.Kd_Plg
WHERE tbltransac.Kd_Trn = 'J'
and tbltransac.tgl_faktur BETWEEN '20150101' AND '20151231'
And tbltransac.No_OPJ <> 'SHOP'
Order by NoUrut Desc --Only if you need a sorted output in the datalayer
Another idea, if your viewGrandTotal is quite large, could be an pre-filtering of this table before you join it. Sometimes SQL Server doesn't get a good plan which needs some lovely touch to get him in the right direction.
Maybe this one:
SELECT Right(rtrim(vgt.No_Faktur),6) as NoUrut,
vgt.No_Faktur,
vgt.No_FakturP,
vgt.Kd_Plg,
tc.Nm_Plg,
vgt.Total_Faktur,
vgt.Nm_Pajak,
vgt.Tgl_Faktur,
vgt.Tgl_FakturP,
vgt.Total_Distribusi
FROM (SELECT Kd_Plg, Nm_Plg FROM Tblcust GROUP BY Kd_Plg, Nm_Plg) as tc -- Pre-Filter on just the needed columns and distinctive.
INNER JOIN (
-- Pre filter viewGrandTotal
SELECT DISTINCT vgt.No_Faktur, vgt.No_Faktur, vgt.No_FakturP, vgt.Kd_Plg, vgt.GRANDTOTAL AS Total_Faktur, vgt.Nm_Pajak,
vgt.Tgl_Faktur, vgt.Tgl_FakturP, vgt.Total_Distribusi
FROM ViewGrandtotal AS vgt
WHERE tbltransac.Kd_Trn = 'J'
and tbltransac.tgl_faktur BETWEEN '20150101' AND '20151231'
And tbltransac.No_OPJ <> 'SHOP'
) as vgt
ON tc.Kd_Plg = vgt.Kd_Plg
Order by NoUrut Desc --Only if you need a sorted output in the datalayer
The pre filtering could increase the generation of a better plan.
Another issue could be just the multi-threading. Maybe your query get a parallel plan as it reaches the cost threshold because of the 11.000 rows. The other query just hits a normal plan due to his lower rows. You can take a look at the generated plans by including the actual execution plan inside your SSMS Query.
Maybe you can compare those plans to get a clue. If this doesn't help, you can post them here to get some feedback from me.
I hope this helps. Not quite easy to give you good hints without knowing table structures, table sizes, performance counters, etc. :-)
Best regards,
Ionic
Note: first of all you should avoid any function in Where clause like this one
year(tbltransac.tgl_faktur)=2015
Here Aaron Bertrand how to work with date in Where clause
"In order to make best possible use of indexes, and to avoid capturing too few or too many rows, the best possible way to achieve the above query is ":
SELECT COUNT(*)
FROM dbo.SomeLogTable
WHERE DateColumn >= '20091011'
AND DateColumn < '20091012';
And i cant understand your logic in this piece of code but this is bad part of your query too
ISNULL(tbltransac.No_OPJ,'') <> 'SHOP'
Actually Null <> "Shop" in this case, so Why are you replace it to ""?
Thanks and good luck
Here is some recommendations:
year(tbltransac.tgl_faktur)=2015 replace this with tbltransac.tgl_faktur >= '20150101' and tbltransac.tgl_faktur < '20160101'
ISNULL(tbltransac.No_OPJ,'') <> 'SHOP' replace this with tbltransac.No_OPJ <> 'SHOP' because NULL <> 'SHOP'.
Order by Right(rtrim(tbltransac.No_Faktur),6) Desc remove this, because ordering should be done in presentation layer rather then in data layer.
Read about SARG arguments and predicates:
What makes a SQL statement sargable?
To write an appropriate SARG, you must ensure that a column that has
an index on it appears in the predicate alone, not as a function
parameter. SARGs must take the form of column inclusive_operator
or inclusive_operator column. The column name is alone
on one side of the expression, and the constant or calculated value
appears on the other side. Inclusive operators include the operators
=, >, <, =>, <=, BETWEEN, and LIKE. However, the LIKE operator is inclusive only if you do not use a wildcard % or _ at the beginning of
the string you are comparing the column to
Related
i am trying to update a colleague's MS Access application (with vb-code). I am rather experienced in writing SQL queries but i am not able to solve the following problem.
The query i am looking to fix uses a pass-through-query's result and a local MS Access table and joins them togehter in the where-clause (i tried using the normal way with ON but it seems this doesn't work when there is a pass through query involved). I have little experience with joining tables in the where-clause but is there such a thing that i cant use certain columns (of both tables) in the where-clause when joining tables in the where-clause? -> When i use a filter criteria such as columnA <> 'somerandomtext' (which is always satisfied, just to point out the problem) the query result is empty. When i delete the latter criteria in the where clause, the query returns results (although too many because i cant filter them accordingly).
Furthermore: I checked the pass-through-query, the results are correct. I checked the MS-Access table, the data in the table is correct. Therefore, i think i might be doing something wrong in the query where i join the two mentioned above.
THIS QUERY WORKS AS INTENDED AND RETURNS RESULTS:
SELECT t.tr_id, t.ser_num, t.contrgnt_id, t.pos_ekey, t.sernum AS cmdty, format(t.vol,""##,###,###.00"") AS volume, t.unit_def, t.value_date,
format(t.coup,""##,###,###.00"") AS fixprice,
format(s.calcvarprice,""##,###,###.00"") AS marketprice,
format(s.calcamount,""##,###,###.00"") AS payamount,
format(s.settlevarprice, ""##,###,##0.00"") as settleprice,
format(s.settleamount, ""##,###,##0.00"") as settleamount, s.sync
FROM pms_trans AS t, settledata AS s
WHERE t.tr_id=s.tr_id And t.is_booked='N' And t.value_date>='01.01.2021' And t.value_date<='01.04.2021'
THIS QUERY SOMEHOW RETURNS 0 RESULTS:
SELECT t.tr_id, t.ser_num, t.contrgnt_id, t.pos_ekey, t.sernum AS cmdty, format(t.vol,""##,###,###.00"") AS volume, t.unit_def, t.value_date,
format(t.coup,""##,###,###.00"") AS fixprice,
format(s.calcvarprice,""##,###,###.00"") AS marketprice,
format(s.calcamount,""##,###,###.00"") AS payamount,
format(s.settlevarprice, ""##,###,##0.00"") as settleprice,
format(s.settleamount, ""##,###,##0.00"") as settleamount, s.sync
FROM pms_trans AS t, settledata AS s
WHERE t.tr_id=s.tr_id And t.is_booked='N' And t.pos_ekey <> 'BGGS' And t.value_date>='01.01.2021' And t.value_date<='01.04.2021'
As mentioned before, i suspect that there are some limitations when joining via where-clause (although i didnt find sufficient information online).
Best Regards and thank you in advance,
Peter
First, apply the correct syntax for the date expressions:
WHERE t.tr_id=s.tr_id And t.is_booked='N' And t.pos_ekey <> 'BGGS' And t.value_date >= #2021-01-01# And t.value_date <= #2021-04-01#
Next, double-check the values for pos_ekey. For example, try to apply the filter, <> 'BGGS', directly on the field in table view.
Im learning how to improve some queries I have, for example I saw that using EXISTS over IN does better, so I did the following modification but im not 100% sure if im getting the same results I expect by using EXISTS, so far it gives me the same results when I execute with different periods but im still doubting, could some one clear for me if it does better performance EXISTS and also if its doing the same?
VARIABLE periodo STRING;
SELECT MO.STRMOVANOMES,
C.STRCLINOMBRE,
MO.STRCLINIT,
MO.STROBLOBLIGSARC,
MO.NUMPROCODIGO,
MO.NUMMOVTIPOCREDITO,
MO.NUMMOVTIPOGARANTIA,
MO.STROBLMODALIDAD,
MO.NUMMOVCALIFICACION,
MO.NUMMOVVLRCAPCREDITO,
MO.NUMMOVVLRINTCREDI,
MO.NUMMOVVLRCAPOTRO
FROM TBLMOVOBLIGACIONES MO,
TBLCLIENTES C
WHERE MO.STRMOVANOMES = :periodo
AND C.STRCLITIPOID ='N'
AND MO.NUMMOVTIPOCREDITO = 2
AND MO.NUMPROCODIGO IN (1,3,4,5,6,7,10,15,16,18,24,29,32,38,40,43,44,45,49,51,54,55,56,70,71,72,73,74,75,76,77,78,81,82,83,84,85)--ALL APPLICATIONS
AND C.STRCLINIT=MO.STRCLINIT
AND SUBSTR(MO.STRCLINIT,1,9) >= 600000000
AND SUBSTR(MO.STRCLINIT,1,9) <= 999999999;
-- USING EXISTS
SELECT MO.STRMOVANOMES,
C.STRCLINOMBRE,
MO.STRCLINIT,
MO.STROBLOBLIGSARC,
MO.NUMPROCODIGO,
MO.NUMMOVTIPOCREDITO,
MO.NUMMOVTIPOGARANTIA,
MO.STROBLMODALIDAD,
MO.NUMMOVCALIFICACION,
MO.NUMMOVVLRCAPCREDITO,
MO.NUMMOVVLRINTCREDI,
MO.NUMMOVVLRCAPOTRO
FROM TBLMOVOBLIGACIONES MO,
TBLCLIENTES C
WHERE MO.STRMOVANOMES = :periodo
AND C.STRCLITIPOID ='N'
AND MO.NUMMOVTIPOCREDITO = 2
AND EXISTS (SELECT NUMPROCODIGO FROM TBLMOVOBLIGACIONES)--ALL APPLICATIONS
AND C.STRCLINIT=MO.STRCLINIT
AND SUBSTR(MO.STRCLINIT,1,9) >= 600000000
AND SUBSTR(MO.STRCLINIT,1,9) <= 999999999;
First, some general info
In theory, the IN clause is to be used when there is a need to check some value against a list of values, so the DB has to loop through the list looking for a value given. Whereas EXIST is to let db to perform a quick query (which is db designed for) in order to check if there is at least one row.
If the values list you're checking against is in a table, it is better to use EXISTS in general. That was some theory.
In practice though, starting from version 10 (which is very old one) Oracle builds same exec plans for IN and EXISTS queries, here is a good explanation if you need some more experienced guy.
To check whether it affects query execution, check explain plans (google for it for your dev-tool) and compare overall "costs" (cost comparison it's not always a good thing to relay on, it is okay for such case though)
Now, back to your query.
The EXISTS in the way you are using it will always return true whenever table TBLMOVOBLIGACIONES has at least one row. Not sure this is what you were looking for.
I believe you need to use it somehow in this manner
AND EXISTS (SELECT 1 FROM TBLMOVOBLIGACIONES WHERE NUMPROCODIGO = MO.NUMPROCODIGO)
Thus it will check whether there is at least one record connected to the data you selected on previous step.
Next step I see here is the EXISTS clause can be easily converted to a regular table join, so instead of choosing between IN or EXISTS you might write the following
FROM TBLMOVOBLIGACIONES MO,
TBLCLIENTES C,
TBLMOVOBLIGACIONES M
WHERE MO.STRMOVANOMES = :periodo
AND C.STRCLITIPOID ='N'
AND MO.NUMMOVTIPOCREDITO = 2
AND M.NUMPROCODIGO = MO.NUMPROCODIGO--ALL APPLICATIONS
AND C.STRCLINIT=MO.STRCLINIT
AND SUBSTR(MO.STRCLINIT,1,9) >= 600000000
AND SUBSTR(MO.STRCLINIT,1,9) <= 999999999;
It is worth doing because joins are always better option allowing database to vary table joins ordering to achieve better performance.
So I am honestly a little puzzled by this!
I have a query that returns a set of transactions that contain both repair costs and an odometer reading at the time of repair on the master level. To get an accurate Cost per mile reading I need to do a subquery to get both the first meter reading between a starting date and an end date, and an ending meter.
(select top 1 wf2.ro_num
from wotrans wotr2
left join wofile wf2
on wotr2.rop_ro_num = wf2.ro_num
and wotr2.rop_fac = wf2.ro_fac
where wotr.rop_veh_num = wotr2.rop_veh_num
and wotr.rop_veh_facility = wotr2.rop_veh_facility
AND ((#sdate = '01/01/1900 00:00:00' and wotr2.rop_tran_date = 0)
OR ([dbo].[udf_RTA_ConvertDateInt](#sdate) <= wotr2.rop_tran_date
AND [dbo].[udf_RTA_ConvertDateInt](#edate) >= wotr2.rop_tran_date))
order by wotr2.rop_tran_date asc) as highMeter
The reason I have the tables aliased as xx2 is because those tables are also used in the main query, and I don't want these to interact with each other except to pull the correct vehicle number and facility.
Basically when I run the main query it returns a value that is not correct; it returns the one that is second(keep in mind that the first and second have the same date.) But when I take the subquery and just copy and paste it into it's own query and run it, it returns the correct value.
I do have a work around for this, but I am just curious as to why this happening. I have searched quite a bit and found not much(other than the fact that people don't like order bys in subqueries). Talking to one of my friends that also does quite a bit of SQL scripting, it looks to us as if the subquery is ordering differently than the subquery by itsself when you have multiple values that are the same for the order by(i.e. 10 dates of 08/05/2016).
Any ideas would be helpful!
Like I said I have a work around that works in this one case, but don't know yet if it will work on a larger dataset.
Let me know if you want more code.
I have the following FireBird query:
update hrs h
set h.plan_week_id=
(select first 1 c.plan_week_id from calendar c
where c.calendar_id=h.calendar_id)
where coalesce(h.calendar_id,0) <> 0
(Intention: For records in hrs with a (non-zero) calendar_id
take calendar.plan_week_id and put it in hrs.plan_week_id)
The trick to select the first record in Oracle is to use WHERE ROWNUM=1, and if understand correctly I do not have to use ROWNUM in a separate outer query because I 'only' match ROWNUM=1 - thanks SO for suggesting Questions that may already have your answer ;-)
This would make it
update hrs h
set h.plan_week_id=
(select c.plan_week_id from calendar c
where (c.calendar_id=h.calendar_id) and (rownum=1))
where coalesce(h.calendar_id,0) <> 0
I'm actually using the 'first record' together with the selection of only one field to guarantee that I get one value back which can be put into h.plan_week_id.
Question: Will the above query work under Oracle as intended?
Right now, I do not have a filled Oracle DB at hand to run the query on.
Like Nicholas Krasnov said, you can test it in SQL Fiddle.
But if you ever find yourself about to use where rownum = 1 in a subquery, alarm bells should go off, because in 90% of the cases you are doing something wrong. Very rarely will you need a random value. Only when all selected values are the same, a rownum = 1 is valid.
In this case I expect calendar_id to be a primary key in calendar. Therefor each record in hrs can only have 1 plan_week_id selected per record. So the where rownum = 1 is not required.
And to answer your question: Yes, it will run just fine. Though the brackets around each where clause are also not required and in fact only confusing (me).
I'm working with a non-profit that is mapping out solar potential in the US. Needless to say, we have a ridiculously large PostgreSQL 9 database. Running a query like the one shown below is speedy until the order by line is uncommented, in which case the same query takes forever to run (185 ms without sorting compared to 25 minutes with). What steps should be taken to ensure this and other queries run in a more manageable and reasonable amount of time?
select A.s_oid, A.s_id, A.area_acre, A.power_peak, A.nearby_city, A.solar_total
from global_site A cross join na_utility_line B
where (A.power_peak between 1.0 AND 100.0)
and A.area_acre >= 500
and A.solar_avg >= 5.0
AND A.pc_num <= 1000
and (A.fips_level1 = '06' AND A.fips_country = 'US' AND A.fips_level2 = '025')
and B.volt_mn_kv >= 69
and B.fips_code like '%US06%'
and B.status = 'active'
and ST_within(ST_Centroid(A.wkb_geometry), ST_Buffer((B.wkb_geometry), 1000))
--order by A.area_acre
offset 0 limit 11;
The sort is not the problem - in fact the CPU and memory cost of the sort is close to zero since Postgres has Top-N sort where the result set is scanned while keeping up to date a small sort buffer holding only the Top-N rows.
select count(*) from (1 million row table) -- 0.17 s
select * from (1 million row table) order by x limit 10; -- 0.18 s
select * from (1 million row table) order by x; -- 1.80 s
So you see the Top-10 sorting only adds 10 ms to a dumb fast count(*) versus a lot longer for a real sort. That's a very neat feature, I use it a lot.
OK now without EXPLAIN ANALYZE it's impossible to be sure, but my feeling is that the real problem is the cross join. Basically you're filtering the rows in both tables using :
where (A.power_peak between 1.0 AND 100.0)
and A.area_acre >= 500
and A.solar_avg >= 5.0
AND A.pc_num <= 1000
and (A.fips_level1 = '06' AND A.fips_country = 'US' AND A.fips_level2 = '025')
and B.volt_mn_kv >= 69
and B.fips_code like '%US06%'
and B.status = 'active'
OK. I don't know how many rows are selected in both tables (only EXPLAIN ANALYZE would tell), but it's probably significant. Knowing those numbers would help.
Then we got the worst case CROSS JOIN condition ever :
and ST_within(ST_Centroid(A.wkb_geometry), ST_Buffer((B.wkb_geometry), 1000))
This means all rows of A are matched against all rows of B (so, this expression is going to be evaluated a large number of times), using a bunch of pretty complex, slow, and cpu-intensive functions.
Of course it's horribly slow !
When you remove the ORDER BY, postgres just comes up (by chance ?) with a bunch of matching rows right at the start, outputs those, and stops since the LIMIT is reached.
Here's a little example :
Tables a and b are identical and contain 1000 rows, and a column of type BOX.
select * from a cross join b where (a.b && b.b) --- 0.28 s
Here 1000000 box overlap (operator &&) tests are completed in 0.28s. The test data set is generated so that the result set contains only 1000 rows.
create index a_b on a using gist(b);
create index b_b on a using gist(b);
select * from a cross join b where (a.b && b.b) --- 0.01 s
Here the index is used to optimize the cross join, and speed is ridiculous.
You need to optimize that geometry matching.
add columns which will cache :
ST_Centroid(A.wkb_geometry)
ST_Buffer((B.wkb_geometry), 1000)
There is NO POINT in recomputing those slow functions a million times during your CROSS JOIN, so store the results in a column. Use a trigger to keep them up to date.
add columns of type BOX which will cache :
Bounding Box of ST_Centroid(A.wkb_geometry)
Bounding Box of ST_Buffer((B.wkb_geometry), 1000)
add gist indexes on the BOXes
add a Box overlap test (using the && operator) which will use the index
keep your ST_Within which will act as a final filter on the rows that pass
Maybe you can just index the ST_Centroid and ST_Buffer columns... and use an (indexed) "contains" operator, see here :
http://www.postgresql.org/docs/8.2/static/functions-geometry.html
I would suggest creating an index on area_acre. You may want to take a look at the following: http://www.postgresql.org/docs/9.0/static/sql-createindex.html
I would recommend doing this sort of thing off of peak hours though because this can be somewhat intensive with a large amount of data. One thing you will have to look at as well with indexes is rebuilding them on a schedule to ensure performance over time. Again this schedule should be outside of peak hours.
You may want to take a look at this article from a fellow SO'er and his experience with database slowdowns over time with indexes: Why does PostgresQL query performance drop over time, but restored when rebuilding index
If the A.area_acre field is not indexed that may slow it down. You can run the query with EXPLAIN to see what it is doing during execution.
First off I would look at creating indexes , ensure your db is being vacuumed, increase the shared buffers for your db install, work_mem settings.
First thing to look at is whether you have an index on the field you're ordering by. If not, adding one will dramatically improve performance. I don't know postgresql that well but something similar to:
CREATE INDEX area_acre ON global_site(area_acre)
As noted in other replies, the indexing process is intensive when working with a large data set, so do this during off-peak.
I am not familiar with the PostgreSQL optimizations, but it sounds like what is happening when the query is run with the ORDER BY clause is that the entire result set is created, then it is sorted, and then the top 11 rows are taken from that sorted result. Without the ORDER BY, the query engine can just generate the first 11 rows in whatever order it pleases and then it's done.
Having an index on the area_acre field very possibly may not help for the sorting (ORDER BY) depending on how the result set is built. It could, in theory, be used to generate the result set by traversing the global_site table using an index on area_acre; in that case, the results would be generated in the desired order (and it could stop after generating 11 rows in the result). If it does not generate the results in that order (and it seems like it may not be), then that index will not help in sorting the results.
One thing you might try is to remove the "CROSS JOIN" from the query. I doubt that this will make a difference, but it's worth a test. Because a WHERE clause is involved joining the two tables (via ST_WITHIN), I believe the result is the same as an inner join. It is possible that the use of the CROSS JOIN syntax is causing the optimizer to make an undesirable choice.
Otherwise (aside from making sure indexes exist for fields that are being filtered), you could play a bit of a guessing game with the query. One condition that stands out is the area_acre >= 500. This means that the query engine is considering all rows that meet that condition. But then only the first 11 rows are taken. You could try changing it to area_acre >= 500 and area_acre <= somevalue. The somevalue is the guessing part that would need adjustment to make sure you get at least 11 rows. This, however, seems like a pretty cheesy thing to do, so I mention it with some reticence.
Have you considered creating Expression based indexes for the benefit of the hairier joins and where conditions?