Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
In PostgreSQL I created a set of 8 table based functions originally but the query was taking 7 seconds, so I recreated as 8 temporary tables, the query is very heavy on leads / lags and using sub queries to find the next code that isn't NULL etc. Along with an unpivot on a large dataset.
Apart from joins this is the only processing required, I'm wondering why what is the exact same code runs so much faster when passing parameters through functions.
The original reason I did this was because using the functions meant the processing had to be done twice as the functions would refer to each other
To give some idea of the structure it goes:
cycle_1_load
cycle_2_dump
cycle_3_unpivot
cycle_4_truck_union
cycle_5_truck_convert
cycle_6_excav_union
cycle_7_excav_convert
cycle_8_add_states
using the temp tables I thought this would speed up the query greatly but it actually made it 3x as slow. Is there some performance hit with temp tables in PostgreSQL?
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I'm writing an app where users pass quizzes. So my purpose is to show quizzes which user didn't tackle before. For this reason I'm using SELECT id, name, problem FROM quizzes WHERE id NOT IN (...).
Imagine that there will be thousands of ids and quizzes.
Is it ok? How does it scale? Probably I need to redesign something / using DB appropriate for that or use another technique to achieve my purpose?
If you have a fixed list, then it should be fine.
If you have a subquery, then I strongly encourage not exists:
from foo f
where not exists (select 1 from <bar> b where b.quiz_id = f.quiz_id)
I recommend this based on the semantics of not exists versus not in. not exists handles NULL values more intuitively.
That said, with appropriate indexes, in most databases, not exists also often has the better performance.
You should consider there are limits to the SQL statements length imposed by each database engine. Though I haven't tested those limits, a 1k values in an IN operator should still work well for most databases, I would think that if you scale it up to 10k or more it could reach some databases limits and your statements will crash.
I would suggest rethinking this solution unless you can verify the worst possible case (with maximum parameters) still works well.
Usually a subquery can do the job, instead of manually sending 1k parameters or assembling a big SQL statement by concatenating strings.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
Which one is better, filtering in query itself [SQL] or extracting the large data and then filtering while processing each row [nodejs]?
It will generally be more efficient in terms of processing and network traffic to put checks in SQL itself.
Consider a table with a million rows, and only one row satisfies the check.
If the check is put at the query-level, the database will return only one row. The database can optimize the query execution to do the operation efficiently.
If the check is put at the application-level, then all million rows will be returned, and each one needs to be checked to find the one relevant row. This will increase the network traffic and likely be slower than if done at the database-level.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have been given an 800 lines SQL Query which is taking around 20 hours to fetch around 400 million records.
There are 13 tables which are partitioned by month.
The tables have records ranging from 10k to 400 million in each partition.
The tables are indexed on primary keys.
The query uses many inline views and outer joins and a few group by functions.
DBAs say we cannot add more indexes as it would slow down the performance since it is an OLTP system.
I have been asked to convert the query logic to pl/sql and then populate a table in chunks.Then do a select * from that table.
My end result should be a query which can be fed to my application.
So even after I use pl/sql to populate a table in chunks,ultimately I need to fetch the data from that table as a query.
My question is, since pl/sql would require select and insert both, are there any chances pl/sql can be faster than sql?
Are there any cases where pl/sql is faster for any result which is achievable by sql?
I will be happy to provide more information if the given info doesn't suffice.
Implementing it as a stored procedure could be faster because the SQL will already be parsed and compiled when the procedure is created. However, given the volume of data you are describing its unclear if this will make a significant difference. All you can do is try it and see.
I think you really need to identify where the performance problem is; where the time is being spent. For example (and I have seen examples of this many times), the majority of the time might be in fetching to 400M rows to whatever the "client" is. In that case, re-writing the query or as PL/SQL will make no difference.
Anyway, once you can enumerate the problem, you have a better chance of getting sound answers, rather than guesses...
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I am making a attendance system , i have a database which is i created and it has a tables from jan-dec (12 tables) and each table has this kind of column (Jan1,jan2//feb1,feb2//mar1,mar2... etc ) i know it is not a good practice tho i'm not familiar with sql, I would like to ask if how would i be able to make a much lesser tables/columns ? and it will based on my datepicker in my vb.net program?
Delve deeper into relational database design (take the link as a first step).
One thing is to create just one table, and have in it a column of type DATE or DATETIME to denote the date. Additional columns would have related data that is linked to the date. That would simplify your table structure greatly. From 12 tables with approx. 30 colums, to just one table with one column + columns with related information.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I have a lot of databases with relatively large amounts of columns ranging from 5 to 300. Each table has at least 50,000 rows in it.
What is the most effective way to store this data? Presently the data has just been dumped into an indexed sql database.
It was suggested to me to create 3 columns as follows.
Column Name, Column category, Row ID, Row Data.
example data would be
Male, 25-40, 145897, 365
Would this be faster? Would this be slower? Is there better ways to store such large and bulky databases?
I will almost never be updating or changing data. It simply be outputted to a 'datatables' dynamic table where it will be sorted, limited and ect. The category column will be used to break up the columns on the table.
Normalize your db!
I have struggled with this "theory" for a long time and experience has proven that if you can normalize data across multiple tables it is better and performance will not suffer.
Do Not try to put all the data in one row with hundreds of columns. Not because of performance but because of development ease.
Learn More here