I am trying to download data from bigquery table which has 3 million records. I get the error
"response too large to return, try will allow_large_results = true"
I tried with the below command:
df = bq.Query('SELECT * FROM [Test.results]', allow_large_results = True).to_dataframe()
Any help would be greatly appreciated.
The way to retrieve result of query that is expected to be bigger than ~128MB is to issue query insert job api with destination table and allow large result flag. After result is stored in that table you can retrieve it using tabledata.list job. Of course than you can delete that [intermediate] table
Hope you can identify respective syntax in client you are using
This is quite old, but for those who land here, the way to do it is:
from google.cloud import bigquery
...
client = bigquery.Client()
job_config = bigquery.job.QueryJobConfig(allow_large_results=True)
q = client.query("""SELECT * FROM [Test.results]""", job_config=job_config)
r = q.result()
df = r.to_dataframe()
From the docs here.
I've been trying to search for a similar query on stackoverflow to modify to get what I need but I can't seem to get it right. I hope someone here can help.
I have two tables located in two different databases. Both databases are configured on the same server. Table 1 called 'DiscreteLive' and located in Database 'Runtime'. Table 2 is called 'v_DiscreteHistory' and located in Database 'WWALMDB'.
They have the following fields
DiscreteLive
Tagname (type String)
Value (type Integer --> can only ever be 1 or 0)
'v_DiscreteHistory'
Tagname (type String)
Value (type String --> can only ever be true of false)
EventStamp (type datetime)
Description (type String)
The 'DiscreteLive' table can only ever have one unique tagname line. An external software will overwrite to each tagname's corresponding Value field. It's basically showing the live values of the system. Example shown below. For example, you would never find Device1.Commfail twice in this table.
Device1.Commfail
Device1.Auto
Device1.Man
Device2.Commfail
Device2.Auto
Device2.Man
Device3.Commfail
Device3.Auto
Device3.Man
Device4.commfail
Device4.Auto
Device4.Man
Device5.Commfail
Device5.Auto
Device5.Man
etc.
The 'v_DiscreteHistory' table is the history of the specific tag. There can be multiple entries of the same tag along with its Description and EventStamp (Time the even happened).
What I'm trying to do is to filter out the 'DiscreteLive' table to show only the tag values where the tagname is a %.CommFail and the Value is 1. Then I would like to take the result of that initial query and attach the latest EventStamp and Description for those tags in the initial query from 'v_DiscreteHistory'.
Not sure if this can be done. Let me know if you need more clarification.
SQL Server: This might be what you are looking for
SELECT
D.TagName
,D.Value
,V.Tagname
,V.Value,V.EventStamp
,V.Description
,MAX(V.EventStamp) AS 'LatestDate'
FROM
SERVER.Runtime.dbo.DiscreteHistory D
INNER JOIN
SERVER.WWALMDB.dbo.v_DiscreteHistory V
ON
D.Tagname = V.Tagname
WHERE
D.Value = 1
AND
V.Value LIKE '%.CommFail'
GROUP BY
D.TagName
,D.Value
,V.Tagname
,V.Value,V.EventStamp
,V.Description
So here is my project using MS Access 2010,
I have developed 2 queries to select 2 different reading periods. These queries are called CycleStart and CycleEnd. When I run these 2 queries individually I get expected output results. these 2 queries pull data from tables with a couple lookup fields in them. So the lookup fields use other tables where there are only 2 columns. The next step I use SQL to create a UNION ALL query to bring these 2 cycle queries together for reporting purposes. The problem I run into is that my resulting Union query does not output the same information as the 2 individual cycle queries.
Now the specific issues. My cycle queries have a couple lookup fields referencing another table. For example the Read_Cycle field comes for a table(Read_Cycles) and only has 2 columns, the unique identifer assigned by Access and the Read_Cycle column with the data I enter. When I run the cycle queries the field for Read_Cycle returns the Read_Cycle data as expected, but the union query does not. So here is some structure of my project:
Read_Cycles Table
|ID Col1 | |Cycle_ID Col2|
1 Spring
2 Fall
3 Winter
The data tables behind the CycleStart and the CycleEnd have fields that are lookup values referencing the above described Read_Cycles table.
Query CycleStart and CycleEnd return Spring or fall or winter, which ever value is associated with the record, correctly.
however, the problem I have is that the Union SQL Query returns the ID instead of the value, so instead of getting Fall, I get the 2.
Here is my UNION ALL SQL........
SELECT "CycleEnd" AS source,
[CycleEnd].[Recloser_SerialNo],
[CycleEnd].[Read_Date],
[CycleEnd].[3_Phase_Reading],
[CycleEnd].[A_Phase_Reading],
[CycleEnd].[B_Phase_Reading],
[CycleEnd].[C_Phase_Reading],
[CycleEnd].[Read_Cycle],
[CycleEnd].[PoleNo],
[CycleEnd].[Substation],
[CycleEnd].[Feeder],
[CycleEnd].[Feeder_Description],
[CycleEnd].[Recloser_Location]
FROM [CycleEnd]
UNION ALL
SELECT "CycleStart" AS source,
[CycleStart].[Recloser_SerialNo],
[CycleStart].[Read_Date],
[CycleStart].[3_Phase_Reading] * - 1,
[CycleStart].[A_Phase_Reading] * - 1,
[CycleStart].[B_Phase_Reading] * - 1,
[CycleStart].[C_Phase_Reading] * - 1,
[CycleStart].[Read_Cycle],
[CycleStart].[PoleNo],
[CycleStart].[Substation],
[CycleStart].[Feeder],
[CycleStart].[Feeder_Description],
[CycleStart].[Recloser_Location]
FROM [CycleStart];
All other fields are coming across just fine and as expected, I have narrowed it down to only fields that are a lookup in the original tables.
Any help would be greatly appreciated. Also my SQL experience is really limited so example code would help greatly.
UPDATE:
here is the sql from the CycleEnd that works. I got this by building the query then changing to the SQL view...
SELECT Recloser_Readings.Recloser_SerialNo,
Recloser_Readings.Read_Date,
Recloser_Readings.[3_Phase_Reading],
Recloser_Readings.A_Phase_Reading,
Recloser_Readings.B_Phase_Reading,
Recloser_Readings.C_Phase_Reading,
Recloser_Locations.PoleNo,
Recloser_Locations.Substation,
Recloser_Locations.Feeder,
Recloser_Locations.Feeder_Description,
Recloser_Locations.Recloser_Location,
Recloser_Readings.Read_Cycle
FROM (
Recloser_Inventory LEFT JOIN Recloser_Locations
ON Recloser_Inventory.PoleNo = Recloser_Locations.PoleNo
)
RIGHT JOIN Recloser_Readings
ON Recloser_Inventory.Serial_No = Recloser_Readings.Recloser_SerialNo
WHERE (((Recloser_Readings.Read_Cycle) = "8"));
UPDATE#2
I noticed I grabbed the wrong code that references the Read_Cycles table. Here it is...
SELECT Read_Cycles.Cycle_ID, Read_Cycles.ID
FROM Read_Cycles
ORDER BY Read_Cycles.Cycle_ID DESC;
UPDATE : SYNTAX ERROR FROM THE FOLLOWING CODE!!
SELECT "CycleEnd" as source,
[CycleEnd].[Recloser_SerialNo],
[CycleEnd].[Read_Date],
[CycleEnd].[3_Phase_Reading],
[CycleEnd].[A_Phase_Reading],
[CycleEnd].[B_Phase_Reading],
[CycleEnd].[C_Phase_Reading],
[CycleEnd].[Read_Cycle],
[CycleEnd].[PoleNo],
[CycleEnd].[Substation],
[CycleEnd].[Feeder],
[CycleEnd].[Feeder_Description],
[CycleEnd].[Recloser_Location]
FROM [CycleEnd] JOIN [Read_Cycles] ON [CycleEnd].[Read_Cycle] = [Read_Cycles].[ID]
UNION ALL SELECT "CycleStart" as source,
[CycleStart].[Recloser_SerialNo],
[CycleStart].[Read_Date],
[CycleStart].[3_Phase_Reading]*-1,
[CycleStart].[A_Phase_Reading]*-1,
[CycleStart].[B_Phase_Reading]*-1,
[CycleStart].[C_Phase_Reading]*-1,
[CycleStart].[Read_Cycle],
[CycleStart].[PoleNo],
[CycleStart].[Substation],
[CycleStart].[Feeder],
[CycleStart].[Feeder_Description],
[CycleStart].[Recloser_Location]
FROM [CycleStart] JOIN [Read_Cycles] ON [CycleStart].[Read_Cycle] = [Read_Cycles].[ID];
I've multiple tables and try to update some tables based on matching translation in master tables. I could update the table based on the first available translation, but I would like to do it based on the most frequent one. I found quite a lot of samples on how to achieve that in simple queries, but can't have it work in this complex query below.
In brief I just need to take the most frequent translation [target] from [EC3_800_FR_M] and copy it to [EC3_800_FR], instead of taking the first occurence in the table.
UPDATE [ec3_800_fr]
SET [ec3_800_fr].[target] = (SELECT TOP 1
[ec3_800_fr_m].[target]
FROM [ec3_800_fr_m]
WHERE (
[ec3_800_fr].[enus] = [ec3_800_fr_m].[enus] AND
[ec3_800_fr].[length] = [ec3_800_fr_m].[length] AND
[ec3_800_fr].[key6_domainname] = [ec3_800_fr_m].[key6_domainname] AND
[ec3_800_fr_m].[status] > 1
)
);
who can help me???
merci - thank you - Dank u - Danke
BR
lolo
I'm having a problem with the result obtained on a select in my sqlite.
I already have a database fed, and I'm doing queries in my application in adobe air. In my table I have 6 columns:
id | name | email | CITY_ID | state_id | phone
When I do a select of the entire table, it returns me an array of objects.
result[30].id = 30;
result [30]. name = John;
result [30]. email = john#xxx.com;
result [30]. city_id = 1352;
result [30]. state_id = 352;
result [30]. phone = xxxxxxxxx;
All information came right, but the id value is incorrect ( correct is not 30 ) . It seems to me that i'm getting the numerical order and not getting the id column value.
Has anyone had this problem?
UPDATE
My query is:
_selectStat = new SQLStatement();
_selectStat.addEventListener( SQLEvent.RESULT, onDataLoaded );
_selectStat.addEventListener( SQLErrorEvent.ERROR, onSqlError );
_selectStat.sqlConnection = _connection;
var sql:String = 'SELECT * FROM "main"."agencia"';
_selectStat.text = sql;
_selectStat.execute();
I'm not familiar with Adobe development or sqlite, so I'm speculating here. If 'id' is a database property, then you may need to indicate that you want the column 'id', and not the property 'id'. There should be a way to do this with the adobe application syntax or the sqlite SQL syntax. In mssql, brackets [] are used for this, so it would be [id] to indicate the column 'id' and not the property. There should be something similar for your environment.