Today I notice another strange behaviour of BigQuery.
I run UDF standard SQL in the BQ web ui:
CREATE TEMPORARY FUNCTION ...
INSERT INTO projectid.dataset.inserttable...
All seems good, the result of the UDF SQL are inserted in the insert table correct, I can tell from "Number of rows". But the table size is not correct, still keep the table size before run the insert query. Furthermore, I found all the inserted rows are gone after 1 hour later.
Some more info I found, when run a "DETELE FROM insert table true" or "SELECT ...", then the deleted number of rows and table size seems correct with the inserted data. But just can not preview the insert table correctly in the WEB UI.
Then I am guessing the "Detail" or "Preview" info of the table has time delay? May I know do you have any idea about this behaviour?
The preview may have a delay, so SELECT * FROM YourTable; will give the most up-to-date results, or you can use COUNT(*) just to verify that the number of rows is correct. You can think of it as being similar to streaming, if you have tried that, where some rows may be in the streaming buffer for a while before they make it into regular storage.
Related
I am trying to fasten up a SQL Server report regarding the IBM OS/400 operating system for my sales department.
A colleague of mine (which left the company) did this report and used a ton of sub selects.
The report usually takes about 30 min to process and often just fails to be displayed. I already tried to cut out some tables/rows in hopes of fastening up the process without success (all is needed by the sales department).
It works over all relevant data (orders, customers, articles, our order at the manufacturer, the manufacturer and so on). Any ideas?
I can't index it, due to the OS/400 system; guess it would be a new programming task for our contractor which leads to costs.
Can I use some clever joins? or somehow reduce the amount of subselects?
Are you using 4 part names in your query? That's probably your problem...
From SQL server...
-- Pull all rows from the table(s) back to MS SQL server and do the where locally on the MS SQL server
select * from LINKEDSVR.MYIBMI.MYLIB.MYTBL where locnbr = '00335';
-- Sends the statement to IBM i server for processing, only results are returned..
select * from openquery(LINKEDSVR, 'select * from MYTBL where locnbr = ''00335''');
Try running the subselects first, sending the output of each to its own table.
Update statistics on the tables. Then run the rest of the SQL, replacing what were originally subselects with the tables created in the first step.
Treat multiple layers of nesting the same way: each layer is its own insert into another table.
I've found that query optimizers have a hard time with complex SQL. Breaking-out the subqueries into separate steps often resolves this.
Between runs my preference is to leave the data intact as a reference in case debugging is needed, then truncate the tables as the first step of a run.
Responding to eraser's comments
Assuming your original query takes this general form:
select [columns] from
(-- subquery
select [columns] from TableA
) as Subquery
from TableB
where mainquery_where_clause
Re-write:
-- Create a table to handle results for your subquery:
Create Table A ;
-- Update the data distribution statistics:
update stats (TableA) ;
-- Now run the subquery:
insert into SubQTable select [columns] from TableA
-- Now run the re-written main query:
Select [columns]
from TableA, TableB
where TableA.joincol = TableB.joincol
and mainquery_where_clause ;
I noticed some syntax issues with the SQL you posted. Looks like something got left out. But the principle of my answer remains the same. Please note that applying my suggestion may not help, as there are potentially many variables to your scenario; you mentioned subqueries, so I chose to address that.
Halfer's suggestion is a great one: edit your original question, adding the SQL code, and putting it in the "{}" supplied by the text editing tool.
I strongly suggest that you obtain the SQL execution plan and post the results.
I am writing this post because I have encountered something truly weird with an SQL statement I am trying to make.
Context:
I am developing an app which uses JPA in the backend to persist / retrieve objects to/from a postgres database.
Problem:
During some tests I have noticed that when a particular user adds entries in the database and later I try to fetch them by his facebook id, the result is an empty list, even though the entries are there in the database. Doing a select statement on the database returns no rows. This does not happen with other users.
I have noticed that the mentioned user's facebook id is slightly longer then others. I do not know if and how this affects this situation.
Interesting part:
When during debugging I created an entry not programmatically, but manually with a SQL INSERT statement directly on the database (marked red on the 1st screenshot), I could fetch the data by facebook id both in my app and with a select statement.
Do you have any ideas what is going on here?
Please check the screenshots:
result of select * from table:
result of select * from table where user_facebook_id = 10215905779020408 :
Please help,
Thanks
please help with a query on Oracle. I'm using SQLPlus (but with SQLDeveloper is the same) to accomplish a simple query like:
select count(*) from ARCHIT_D_CC where (TYP_ID=22 OR TYP_ID=23) and SUBTM like '%SEP%' and CONS=1234
This is a very simple query that works perfect until I'll execute it on a big table that contains tons of data. After a few minutes I got:
ERROR at line 1: ORA-08103: object no longer exists
This because the database is partitioned and due to large ammount of data in the table and before my query finishes, oracle BT mechanism rotates the table partitions. That's why I got the message.
Now, is there a way to avoid this error? Maybe specify the partition or something like that. As already wrote, in other table with less data, it works perfect.
Thanks
Lucas
I do have a table in the database, which supposed to have more than 1k rows. The DB is Postgress. I use the following command:
select count(*) from icdten; it returns 1000 which is wrong
and also
select * from icdten;
returns the first 1000 rows, which is wrong I want to have all of them. Googling didnt find, or maybe I was googling the wrong thing.
EDIT1: I use PgAdmin, maybe it is PgAdmin issue.. I just did not find that option looking through the interface. It supposed to have 14k rows.
Possible limit is set in PgAdmin. Please look at Options/Query tab of PgAdmin like described in http://www.pgadmin.org/docs/1.4/options-tab3.html (http://www.pgadmin.org/docs/1.16/options-tab4.html for version 1.16 you mentioned):
Maximum rows to retrieve can be set to 1000 (default is 100, people usually change this value), and also "Count rows if estimated less than" may affect to COUNT function (it can use count from table statistics instead of real table count, so try to set temporary this value to a big number)
Here is how to troubleshoot this issue:
First try from psql. While select count(*) should give you the same on both, maybe there is something going on. Also there is no limit there, so you can:
\o testfile
\i select * from icdten
\q (exits psql)
wc -l testfile
If that still shows 1000 rows then you probably have 1000 rows in that table, start making sure you are connected to the right db, querying the table you think you are, etc.
EXPLAIN ANALYSE
May be helpful in that case.
This is not an error. It's simply a default parameter set in some versions of PostgreSQL.
You can change the parameter by going to the PostgreSQL's installation directory and navigating to pgAdmin 4\web. The directory for version 12 looks like this:
C:\Program Files\PostgreSQL\12\pgAdmin 4\web
Open config.py with a text editor and search for ON_DEMAND_RECORD_COUNT. It is initially set to 1000.
##########################################################################
# Number of records to fetch in one batch in query tool when query result
# set is large.
##########################################################################
ON_DEMAND_RECORD_COUNT = 1000
You can comment this line out but it could result in other errors. I would suggest to change it to a large value like 10,000,000. If you can't save the file after the modification you need to copy the file somewhere else, make the changes and save, then copy back to the original folder and replace with the original file.
In PL/SQL Developer v7.1.x, is there way way to ignore large data types in queries or the "Query Data" feature. For example: If you right click on table FOO, and select "Query Data" this will execute a SELECT * FROM FOO. If that table contains BLOB data the query will take a while to complete and temporarily lock up the application. This is especially problematic when querying remote databases (for obvious reasons).
I would like a way to tell PL/SQL Developer not to retrieve large data by default. I know there is a way to limit the ResultSet size but this doesn't do what I am looking for.
I could just select each column I wanted ignoring certain ones but then I couldn't use the "Query Data" feature.
Thanks.
No, the Query Data feature does one thing and one thing only - queries all the data.
What you might find useful is that you can drag the name of a table or view from the Browser into a SQL Window, choose "Select" from the menu that pops up, and it will generate a SELECT statement on the table with all the column names included - but does not execute the query straight away. You can then edit it however you like (e.g. comment out the LOB columns) before you run it.
I know that Toad has something like that built in, but I'm not aware of a PL/SQL Developer option that disables BLOBS.
The option you are left with, for now, is to simply select all the columns individually and truncate the blob.
ie:
select foo, bar, trunc(baz,100) from foo where ...
Create a View that doesn't contain the blob column or whatever columns you don't routinely want to look at.