I have currently written SQL to bring back data in a select statement without using ORDER BY. From what I have read, the selection seems to be random? As I am trying to run tests on data that should be updated once selected, is there a way to 'predict' which entries will be returned by the select statement (and therefore should be modified)? My select statement is:
SELECT x.FIELD_A,
x.FIELD_B,
x.FIELD_C,
y.FIELD_D,
y.FIELD_E,
y.FIELD_F
FROM TABLE_X x
LEFT JOIN TABLE_Y Y ON x.FIELD_A = y.FIELD_G
WHERE y.FIELD_G in ('xxx', 'yyy', 'zzz', 'www')
AND y.FIELD_E = 'vvvv'
AND y.FIELD_G IS NULL
AND y.FIELD_H IS NULL
AND y.FIED_I IS NULL
AND x.FIELD_J IN ('uuu', 'ttt', 'sss', 'rrrr')
AND y.FIELD_K_DATE > ADD_MONTHS(SYSDATE, -12)
AND ROWNUM <= 20
How does Oracle SQL order rows when there is no ORDER BY?
It doesn't order them at all.
From a classic Tom Kyte article, and quoting his own book:
You should think of a heap organized table as a big unordered collection of rows. These rows will come out in a seemingly random order, and depending on other options being used (parallel query, different optimizer modes and so on), they may come out in a different order with the same query. Do not ever count on the order of rows from a query unless you have an ORDER BY statement on your query!
If you run your query several times today you may see the same results in the same order, but might not. You could run it a hundred times and perhaps get the same result each time, and think that means that's what it will always return. But if you run it again tomorrow it may or may not be the same. If you run it weeks or months or years in the future it still could be the same as you see today, but is increasingly likely not to be as other things change - including data volumes, statistics gathering, database upgrades or even potentially patching, and so on.
Without an order-by clause you are at the mercy of the optimiser and things even that can't control, potentially even down to the order blocks of data are read from disk.
So no, you can't predict which entries will be returned by your query. You will get 20 rows but which ones you get and what order those are in is non-deterministic.
If you need a predictable result you need to order the results before apply the rownum filter (or using fetch first).
Related
I am bit new to SQL, I want to write query with TOP clause and order by clause.
So, for returning all the records I write below query
select PatientName,PlanDate as Date,* from OPLMLA21..Exams order
by PlanDate desc
And I need top few elements from same query, so I modified the query to
select top(5) PatientName,PlanDate as Date,* from OPLMLA21..Exams
order by PlanDate desc
In my understanding it will give the top 5 results from the previous query, but I see ambiguity there. I have attached the screen shot of query results .
May be my understanding is wrong, I read a lot but not able to understand this please help me out.
I stated this in a comment, however, to repeat that:
TOP (5) doesn't give the "top results" of the prior query though, no. It gives the top (first) rows from the dataset defined in the query it in is. If there are multiple rows that have the same "rank", then the row(s) returned for that rank are arbitrary. So, for example, for your query if you have 100 rows all with the same value for PlanDate, what 5 rows you get are completely arbitrary and could be different (including the order they are in) every time you run said query.
What I mean by arbitrary is that, effectively, SQL Server is free to choose whatever rows, of those applicable, are returned. Sometimes this might be the same everytime you run the query, but this by luck more than anything. As your database gets larger, you have more users querying the data, you involve joins, things like locks, indexes, parrallelism, etc all will effect the "order" that SQL Server is processing said data, and will effect an ambigious TOP clause.
Take the example data below:
ID | SomeDate
---|---------
1 |2020-01-01
2 |2020-01-01
3 |2020-01-01
4 |2020-01-01
5 |2020-01-01
6 |2020-01-02
Now, what would you expect if I ran a TOP (2) against that table with an ORDER BY clause of SomeDate DESC. Well, certainly, you'd expect the "last" row (with an ID of 6) to be returned, but what about the next row? The other 5 rows all have the same value for SomeDate. Perhaps, because your under the impression that data in a table is pre-sorted, you might expect the row with a value of 5 for ID. What if I told you that there was a CLUSTERED INDEX on ID ASC; that might well end up meaning that the row with a value of 1 is returned. What if there is also an index on SomeDate DESC?
What if the table was 10,000 of rows in size, and you also have a JOIN to another table, which also has a CLUSTERED INDEX, and some user is performing a query with some specific row locking on in while you run your query? What would you expect then?
Without your ORDER BY being specific enough to ensure that each row has a distinct ordering position, SQL Server will return other rows in an arbitrary order and when mixed with a TOP means the "top" rows will also be arbitrary.
Side note: I note in your image (of what appears to be SSMS), your "dates" are in the format yyyyMMdd. This strongly implies that you are storing a date value as a varchar or int type. This is a design flaw and needs to be fixed. There are 6 date and time data types, and 5 of them are far superior to using a string and numerical data type to storing the data.
I am trying to filter between two dates on a SQL server from a PHP process.
This is the query:
select *
from webStocks
where FECHAMODIFICADO between '2020-06-03 17:16:02' and '2020-06-04 17:16:03'
ORDER BY webStocks.FECHAMODIFICADO DESC
This is the result:
The result is not what I want. In the table I have the following information and it should be the result.
What am I doing wrong in the query?:(
I'd try to make sure the date column actually contains 'timestamp' data type.
If it doesn't, the following code should fix it:
SELECT *
FROM webStocks
where CAST(FECHAMODIFICADO AS timestamp) BETWEEN '2020-06-03 17:16:02' AND '2020-06-04 17:16:03'
ORDER BY webStocks.FECHAMODIFICADO DESC
You can see more information about this kind of statements here.
(this solution is valid mostly for MySQL, but will probably work with either CAST or CONVERT statement with other SQL servers).
SQL tables represent unordered sets. That means that when you run a query with no ORDER BY, the results can be in any order -- and even in different orders on different runs.
In this case, you have an ORDER BY. But the column has duplicates. The same principle applies: rows with the same key value can be in any order -- and even in different orders on different runs.
So, you need to add a new key to capture the order that you want. It is not obvious from your data. But the results would at least be stable if you used:
ORDER BY webStocks.FECHAMODIFICADO DESC, CodeArticulo
It is also odd that your WHERE clause includes very specific times. But the data in these rows is all occurring at midnight. Usually midnight is not such an active time, if the time stamps represent human behavior.
Through R I connect to a remotely held database. The issue I have is my hardware isn't so great and the dataset contains tens of millions of rows with about 10 columns per table. When I run the below code, at the df step, I get a "Not enough RAM" error from R:
library(DatabaseConnector)
conn <- connect(connectionDetails)
df <- querySql(conn,"SELECT * FROM Table1")
What I thought about doing was splitting the tables into two parts any filter/analyse/combine as needed going forward. I think because I use the conn JDBC conection I have to use SQL syntax to make it work. With SQL, I start with the below code:
df <- querySql(conn,"SELECT TOP 5000000 FROM Table1")
And then where I get stuck is how do I create a second dataframe starting with n - 5000000 rows and ending at the final row, retrieved from Table1.
I'm open to suggestions but I think there are two potential answers to this question. The first is to work within the querySql to get it working. The second is to use an R function other than querySql (no idea what this would look like). I'm limited to R due to work environment.
The SQL statement
SELECT TOP 5000000 * from Table1
is not doing what you think it's doing.
Relational tables are conceptually unordered.
A relation is defined as a set of n-tuples. In both mathematics and the relational database model, a set is an unordered collection of unique, non-duplicated items, although some DBMSs impose an order to their data.
Selecting from a table produces a result-set. Result-sets are also conceptually unordered unless and until you explicitly specify an order for them, which is generally done using an order by clause.
When you use a top (or limit, depending on the DBMS) clause to reduce the number of records to be returned by a query (let's call these the "returned records") below the number of records that could be returned by that query (let's call these the "selected records") and if you have not specified an order by clause, then it is conceptually unpredictable and random which of the selected records will be chosen as the returned records.
Since you have not specified an order by clause in your query, you are effectively getting 5,000,000 unpredictable and random records from your table. Every single time you run the query you might get a different set of 5,000,000 records (conceptually, at least).
Therefore, it doesn't make sense to ask about how to get a second result-set "starting with n - 5000000 and ending at the final row". There is no n, and there is no final row. The choice of returned records was not deterministic, and the DBMS does not remember such choices of past queries. The only conceivable way such information could be incorporated into a subsequent query would be to explicitly include it in the SQL, such as by using a not in condition on an id column and embedding id values from the first query as literals, or doing some kind of negative join, again, involving the embedding of id values as literals. But obviously that's unreasonable.
There are two possible solutions here.
1: order by with limit and offset
Take a look at the PostgreSQL documentation on limit and offset. First, just to reinforce the point about lack of order, take note of the following paragraphs:
When using LIMIT, it is important to use an ORDER BY clause that constrains the result rows into a unique order. Otherwise you will get an unpredictable subset of the query's rows. You might be asking for the tenth through twentieth rows, but tenth through twentieth in what ordering? The ordering is unknown, unless you specified ORDER BY.
The query optimizer takes LIMIT into account when generating query plans, so you are very likely to get different plans (yielding different row orders) depending on what you give for LIMIT and OFFSET. Thus, using different LIMIT/OFFSET values to select different subsets of a query result will give inconsistent results unless you enforce a predictable result ordering with ORDER BY. This is not a bug; it is an inherent consequence of the fact that SQL does not promise to deliver the results of a query in any particular order unless ORDER BY is used to constrain the order.
Now, this solution requires that you specify an order by clause that fully orders the result-set. An order by clause that only partially orders the result-set will not be enough, since it will still leave room for some unpredictability and randomness.
Once you have the order by clause, you can then repeat the query with the same limit value and increasing offset values.
Something like this:
select * from table1 order by id1, id2, ... limit 5000000 offset 0;
select * from table1 order by id1, id2, ... limit 5000000 offset 5000000;
select * from table1 order by id1, id2, ... limit 5000000 offset 10000000;
...
2: synthesize a numbering column and filter on it
It is possible to add a column to the select clause which will provide a full order for the result-set. By wrapping this SQL in a subquery, you can then filter on the new column and thereby achieve your own pagination of the data. In fact, this solution is potentially slightly more powerful, since you could theoretically select discontinuous subsets of records, although I've never seen anyone actually do that.
To compute the ordering column, you can use the row_number() partition function.
Importantly, you will still have to specify id columns by which to order the partition. This is unavoidable under any conceivable solution; there always must be some deterministic, predictable record order to guide stateless paging through data.
Something like this:
select * from (select *, row_number() over (id1, id2, ...) rn from table1) t1 where rn>0 and rn<=5000000;
select * from (select *, row_number() over (id1, id2, ...) rn from table1) t1 where rn>5000000 and rn<=10000000;
select * from (select *, row_number() over (id1, id2, ...) rn from table1) t1 where rn>10000000 and rn<=15000000;
...
Obviously, this solution is more complicated and verbose than the previous one. And the previous solution might allow for performance optimizations not possible under the more manual approach of partitioning and filtering. Hence I would recommend the previous solution.
My above discussion focuses on PostgreSQL, but other DBMSs should provide equivalent features. For example, for SQL Server, see Equivalent of LIMIT and OFFSET for SQL Server?, which shows an example of the synthetic numbering solution, and also indicates that (at least as of SQL Server 2012) you can use OFFSET {offset} ROWS and FETCH NEXT {limit} ROWS ONLY to achieve limit/offset functionality.
Will the following two SQL statements always produce the same result set?
1. SELECT * FROM MyTable where Status='0' order by StartTime asc limit 10
2. SELECT * FROM (SELECT * FROM MyTable where Status='0' order by StartTime asc) limit 10
Yes, but ordering subqueries is probably a bad habit to get into. You could feasibly add a further ORDER BY outside the subquery in your second example, e.g.
SELECT *
FROM (SELECT *
FROM Test
ORDER BY ID ASC
) AS A
ORDER BY ID DESC
LIMIT 10;
SQLite still performs the ORDER BY on the inner query, before sorting them again in the outer query. A needless waste of resources.
I've done an SQL Fiddle to demonstrate so you can view the execution plans for each.
No. First because the StartTime column may not have UNIQUE constraint. So, even the first query may not always produce the same result - with itself!
Second, even if there are never two rows with same StartTime, the answer is still negative.
The first statement will always order on StartTime and produce the first 10 rows. The second query may produce the same result set but only with a primitive optimizer that doesn't understand that the ORDER BY in the subquery is redundant. And only if the execution plan includes this ordering phase.
The SQLite query optimizer may (at the moment) not be very bright and do just that (no idea really, we'll have to check the source code of SQLite*). So, it may appear that the two queries are producing identical results all the time. Still, it's not a good idea to count on it. You never know what changes will be made in a future version of SQLite.
I think it's not good practice to use LIMIT without ORDER BY, in any DBMS. It may work now, but you never know how long these queries will be used by the application. And you may not be around when SQLite is upgraded or the DBMS is changed.
(*) #Gareth's link provides the execution plan which suggests that current SQLite code is dumb enough to execute the redundant ordering.
I have this postgres sql query:
select * from stats where athlete_id = 5
That will return, as you'd imagine, all the stats for each athlete of id 5
So far so normal.
However I just noticed that for athlete_id above 110, it returns the stats rows in the opposite order.
So for athlete_id = 110 the 'id' column comes out like this which is what i'm used to:
2325
2401
2482
2537
2592
2647
...
etc. Each stats table row in order by id.
Then if you select 111, it comes out like this:
5652
5610
5569
5528
5487
5437
5387
5336
...
How is that even possible? This is all doing the queries inside the pgadmin interface. There's no additional where clauses, only the one I've stated, i.e. WHERE athlete_id = 111
What? I've been making all sorts of code changes but what on earth would cause this within pgaadmin /pgsql ?
What happened between 110 & 111? Lots and lots of refactoring and code changes inside rails, but no actual direct SQL manipulation. Yes I realise the answer might be in there, but I don't know how to look, as this is not in the rails app - this is pure SQL via pgadmin, so something must have been done in postgres - I need to understand what that might be or I'm at a loss to debug.
Any ideas? In rails, things like:
Athlete.stats.last.score return the 'first' row instead of the 'last' row, completely screwing up the application, but only for athletes with an id above 110!
Totally confused!
If you don't specify an ORDER BY clause, there is no guarantee as to what order the rows will be returned in. They could be ascending order, descending order, an order that has no apparent logic to it, and potentially even a different order each time you query (though that is unlikely to happen in practice).
The specific order you get back is due to implementation details of the steps in the query plan. When you change the parameters it can cause PostgreSQL to prefer one query plan over another. For example, it may decide to use a table scan for one query, but an index for the other. It's hard to know exactly the reason in your specific case, but looking at the results of EXPLAIN might give some hints.
To fix your query to return the rows in the order you expect, add an ORDER BY clause:
SELECT *
FROM stats
WHERE athlete_id = 5
ORDER BY id
Unless you specify an ORDER BY clause on your query, you cannot assume/expect that SQL will return the data to you in any certain order.