We are using Legacy SQL on a specific request. We can't use standard SQL for some internal reasons.
We would like to paginate our results, because we have a lots of rows. Like that :
SELECT ... FROM ... LIMIT 10000 30000 // In standard SQL
But in Legacy SQL Offset don't exists. So how to do the same job ?
Edit :
I don't want to order. I want to paginate. For example get 1000 rows after skipping 2000 rows. A simple LIMIT clause with an offset, like in traditional SQL Database or like in BigQuery Standard SQL.
To do this, I want to use Big Query Legacy SQL.
The pagination you talking about is done via tabledata.list API
Based on your question and follow-up comments - It might be the way for you to go. Even though it does not involve querying. Just API or related method in client of your choice.
pageToken parameter allows you to page result
Btw, another benefit of this approach - it is free of charge
If you still need to do pagination via query - you option is using ROW_NUMBER()
In this case - you can prepare your data in temp table with below query
SELECT <needed fields>, ROW_NUMBER() OVER() num
FROM `project.dataset.table`
Then, you can page it using num
SELECT <needed fields>
FROM `project.dataset.temp`
WHERE num BETWEEN 10000 AND 30000
Related
When making a Bigquery query like for example:
Select Campaign FROM TABLE WHERE Campaign CONTAINS 'buy' GROUP BY Campaign IGNORE CASE LIMIT 100
The IGNORE CASE clause is not working when used with LIMIT clause.
Some time ago it did work.
Is this a Bigquery fault or something changed?
Thanks a lot
Ramiro
A couple of things here:
Legacy SQL expects IGNORE CASE to appear at the end of the query, so you need to use LIMIT 100 IGNORE CASE instead of IGNORE CASE LIMIT 100
The BigQuery team advises using standard SQL instead of legacy SQL if you're working on new queries, since it tends to have better error messages, better performance, etc. and it's where we're focusing our efforts going forward. You may also be interested in the migration guide.
If you want to use standard SQL for your query, you could do:
Select LOWER(Campaign) AS Campaign
FROM TABLE
WHERE LOWER(Campaign) LIKE '%buy%'
GROUP BY LOWER(Campaign)
LIMIT 100
In my app there's a following relation: Page hasMany Paragraphs and I need to create a query that returns all pages where the number of paragraphs is less then limit. The problem is that the pages are created in another app approximately 2 per second and the paragraphs table contains more then 2 million rows. All standard grails approaches, like dynamic finders and criteria queries just hang as they create very not optimal sql. In the database console the following query does the job:
select * from (
select a.id, count(b.page_id) count from page a
left join paragraph b ON a.id = b.page_id
group by 1) sub
WHERE sub.count <= 10 LIMIT 1000
And I couln't translate this query into HQL. I know there's groovy sql available, but it's rows method returns a List of GroovyResult, not list of domain classes. Is there a better approach to the issue?
If a query gets too complicated I tend to do something like this:
def results = new Sql(dataSource).rows(SQL)*.id*.asType(Integer).colect(DomainClass.&get)
I know it doesn't look too great and you'd probable get no kudos for it but it gets the job done.
However it if you'd like to use something more expressive you could give a try to JOOQ (http://www.jooq.org/)
I'm writing my own SQLIteBrowser and I have one final problem, which apparently is quite often discussed on the web, but it doesn't seem to have a good general solution.
So currently I store the SQL which the user entered. Whenever I need to fetch rows, I execute the SQL by adding "Limit n, m` at the end of the SQL.
For normal SQLs, which I mostly use, this seems good enough. However if I want to use limit myself in the query, this will obviously give an error, because the resulting sql can look like this:
select * from table limit 30 limit 1,100
which is obviously wrong. Is there some better way to do this?
My idea was, that I could scan the SQL and check if there is a limit clause already used and then ignore it. Of course it's not as simlpe as that, because if I have an sql like this:
select * from a where a.b = ( select x from z limit 1)
it obviously should still apply my limit in such a case, so I could scan the string from the end and look if there is a limit somehwere. My question now is, how feasable this is. As I don't know who the SQL parser works, I'm not sure if LIMIT has to be at the end of SQL or if there can be other commands at the end as well.
I tested it with order byand group by and I get SQL errors if limit is not at the end, so my assumption seems to be true.
I found now a much better solution which is quite simple and doesn't require me to parse the SQL.
The user can enter an arbitrary sql. The result is loaded into a table. Since we don't want to load the whole result at once, as this can return millions of records, only N records are retriueved. When the user scroll to the bottom of the table the next N items are fetched and loaded into the table.
The solution is, to wrapt the SQL into an outer sql with my page size limits.
select * from (arbitrary UserSQL) limit PageSize, CurrentOffset
I tested it with SQLs I regularly use, and this seem to work quite nicely and is also fast enough for my purpose.
However, I don't know wether SQLite has a mechanism to fetch the new rows faster, or if the sql has to be rerun every time. In that case it might not be a good solution fo rrealy complex queries with a long response time.
I'm using firebird database and it does not seem to have ROWNUM or ROW_NUMBER() like sql server does so I cannot follow the procedure explained here.
I have a query whose result dataset is bigger than what my system memory can accommodate. I'm trying load the dataset in smaller chunks. I have a series of generic queries which I cannot modify and they could be anything. I know I can do
select first 5000 * from
(-my actual query here-)
to get the first 5000 records. But how can I go about getting the next 5000 records.
Thanks
Since FireBird 2.0 ROWS syntax is supported, ie you would use
select * from tab ROWS m TO n
I suggest you download FB's language reference update where it is documented.
In firebird you use Select First ? Skip ? to specific how many, and what your offset is.
What's the equivalent of mysql Limit in ms access. TOP is not sufficient since I'm going to use it for pagination.
Thanks
There isn't one. Your best bet is to add an ID column as a primary key (if you don't already have one) and chunk output by looping through:
SELECT * FROM table
WHERE id >= offset AND id <= offset + chunk_size - 1
until you get all the rows.
Curiously, there are a few references in Microsoft documentation to a LIMIT TO nn ROWS syntax for the Access Database Engine:
ACC2002: Setting ANSI 92 Compatibility in a Database Does Not Allow DISTINCT Keyword in Aggregate Functions
About ANSI SQL query mode (MDB)
However, actual testing seems to confirm that this syntax has never existed in a release version of the Access Database Engine. Perhaps this is one of those features that the SQL Server team wanted to put into Jet 4.0 but were ordered to rollback by the Windows team? Whatever, it seem we must simply put it down to a bad documentation error that Microsoft won't take the time to correct :(
If you need to do pagination on the server** side then I suggest you consider a more capable, modern SQL product with better documentation ;)
** conceptually, that is: the Access Database Engine is not a server DBMS.
Since it doesn't appear that you have any type of sequencial unique key number for these rows, you'll need to create a ranking column: How to Rank Records Within a Query
You need to determine how many rows at a time you will return N = (10, 25,100).
You need to keep track of what "page" the user is on and the values of the first and last rank.
Then when you make the call for the next page it is either the next N rows that are > or < the first and last ranks (depending if the users is going to the previous or next page.).
I'm sure there is a way to calculate the last page, first page, etc.
Only way to achive paging SQL similar to Limit statement by using TOP keywords is as follows:
First Step:
sql = "select top "&LESS_COUNT&" * from (SELECT top "&(PAGE_COUNT*getPage)&" * FROM (SELECT "&COLUMNS&" FROM "&TABLENAME&") AS TBL "&getWhere&getOrderby("asc")&") as TBL "&getOrderby("desc")
Second step:
sql = "SELECT TOP "&PAGE_COUNT&" * FROM (" & sql & ") as TBL "&getOrderby("asc")
To summarize; you should re-order and make the results upside down for 3 times.
port your project to PHP & MySQL. Better support for these type of actions and queries and much much better online documentation. As a 16 year veteran DB developer, I have grown to dispise MS Access and MS SQL with a passion unmatched by anything else. This is due exclusively to their lack of support and documentation.