My bigquery table has a start time and an finish time. I want to add number 1 to the start time, but I want to do it as much as the end time. How can I do that?
this is my bigquery table and I want to show this table to below table.
For example
MY bigquery
Results
----> Result
Related
I have a couple of years of data on a big query partitioned table by day. I want to replace the data of just the last 30 days; however, when I use the create or replace table function on bigquery, it replaces the entire table with just the new dates partitions. Is there any way to update only those partitions without losing the previous data?
I have several tables that contain sensordata that consist of 2 colums:
timestamp | value
I want a second table that adds a new line when a new value comes into one of the above tables, that takes the last xx values, calculates the median, and adds this line.
How is this best done in BQ / Scheduled query? I know how to calculate a median, just hardest is to do the append 'on schedule'.
For append data into a table in a scheduled query, you require to set the Append to table option in the Destination table write preference when the scheduled query is configured in the by BigQuery UI:
I am trying to write sql query for the following scenario. It would be great if can get any help on it.
Scenario :
I have a table(emp) which has three columns id,time and value. There is no primary key in the table. time is a date column holding date and timestamp.
Table will be started to update from afternoon onwards. So table will have yesterday data in it from morning to afternoon. Before inserting current data in it all the yesterday's rows will be cleared and table will be start to be updated dynamically from noon till evening .But I need to run the query running from morning onwards and my query should not fetch yesterday's data. So from morning to afternoon I should ideally wait for data to come and should not fetch any rows and start to fetch once the current date data was inserted into it.
I need to run the query for every five minutes and when I run the query I should get all the latest rows in the table so that whenever there is a update in the table those rows will be fetched .
For example when the table is updated from 1 PM onwards. I should get all the rows when i start the first query and after five minutes when i once again run the same query at 1.05 i should get all the rows inserted between 1PM and 1.05PM.
My idea :
Select max(time) from emp;
At the start of the day I should check the max time in the table and it will be definitely yesterday date so I will set today date(2018-07-14 00:00:00) in a local variable or if it's today's date then that value will be stored in the local variable.
I can also do the same in the above query by comparing it with sysdate like below query but not sure about the performance as I saw it took time by comparing with all the rows I guess. See the modified above query below
select max(time) from emp where time = sysdate;
After getting the max time from the table , will have it one variable say lastquerytime, then query the table which has rows greater than this time stamp so that we can fetch all the latest rows for every five minutes.
Select id,time,value from emp where time > lastquerytime;
So the idea is getting all the rows and check the maximum timestamp in it and query the table next time with rows having timestamp greater than this max timestamp. Like this need to do the same for every five minutes till the end of the day.
Now I am using two queries to achieve this scenario.
Any suggestions for better approach and queries to write for this scenario will help me a lot.
You should use something like to get data from yesterday:
SELECT id,time,value FROM emp WHERE time BETWEEN TRUNC(SYSDATE - 1) AND TRUNC(SYSDATE) - 1/86400
If I understand correctly, you want all rows on the maximum date of the table.
If so:
select e.*
from emp
where time >= (select trunc(max(time)) from emp);
If you want the results based on clock-time, then you would use trunc(sysdate) instead.
I have some basic SQL experience but am seeking to do some data reconciliation on an Oracle database, hence the query i'm writing will be in PL/SQL.
Having run a performance test, I would like to find out how many records were inserted, and how many were updated over a period time (just over 2 hours) in total in the schema.
Is there a simple way of writing a query that does the following.
- Displays a count for records inserted
- Displays a count for records updated
- Alongside the Table name
So in summary, the table would be 3 rows, and my time period is as follows.
between to_date('16-01-2015 13:00:00','DD-MM-YYYY HH24:MI:SS')
and to_date('16-01-2015 15:30:00','DD-MM-YYYY HH24:MI:SS')
Any suggestions or help would be much appreciated.
My question is about table partitioning in SQL Server 2008.
I have a program that loads data into a table every 10 mins or so. Approx 40 million rows per day.
The data is bcp'ed into the table and needs to be able to be loaded very quickly.
I would like to partition this table based on the date the data is inserted into the table. Each partition would contain the data loaded in one particular day.
The table should hold the last 50 days of data, so every night I need to drop any partitions older than 50 days.
I would like to have a process that aggregates data loaded into the current partition every hour into some aggregation tables. The summary will only ever run on the latest partition (since all other partitions will already be summarised) so it is important it is partitioned on insert_date.
Generally when querying the data, the insert date is specified (or multiple insert dates). The detailed data is queried by drilling down from the summarised data and as this is summarised based on insert date, the insert date is always specified when querying the detailed data in the partitioned table.
Can I create a default column in the table "Insert_date" that gets a value of Getdate() and then partition on this somehow?
OR
I can create a column in the table "insert_date" and put a hard coded value of today's date.
What would the partition function look like?
Would seperate tables and a partitioned view be better suited?
I have tried both, and even though I think partition tables are cooler. But after trying to teach how to maintain the code afterwards it just wasten't justified. In that scenario we used a hard coded field date field that was in the insert statement.
Now I use different tables ( 31 days / 31 tables ) + aggrigation table and there is an ugly union all query that joins togeather the monthly data.
Advantage. Super timple sql, and simple c# code for bcp and nobody has complained about complexity.
But if you have the infrastructure and a gaggle of .net / sql gurus I would choose the partitioning strategy.