If I am inserting rows into BQ, using a custom partiton column, the rows are placed in the right date partition. Great.
If I subsequently issue a DML Update statement to a bunch of row, updating the timestamp value of the column used for partitioning, will these rows be re-partitioned based on the new value just update to? Essentially 'moving' its partition?
Thanks
Yes, it does
Even more - using an UPDATE statement you can modify the _PARTITIONTIME pseudo column
Related
Table info:
I want to add new column and calculated the different of the alarmTime column with this code:
ALTER TABLE [DIALinkDataCenter].[dbo].[DIAL_deviceHistoryAlarm]
ADD dif AS (DATEDIFF(HOUR, LAG((alarmTime)) OVER (ORDER BY (alarmTime)), (alarmTime)));
How to add the calculation on the table? Because always there's error like this:
Windowed functions can only appear in the SELECT or ORDER BY clauses.
You are using the syntax for a generated virtual column that shows a calculated value (ADD columnname AS expression).
This, however, only works on values found in the same row. You cannot have a generated column that looks at other rows.
If you consider now to create a normal column and fill it with calculated values, this is something you shouldn't do. Don't store values redundantly. You can always get the difference in an ad-hoc query. If you store this redundantly instead, you will have to consider this in every insert, update, and delete. And if at some time you find rows where the difference doesn't match the time values, which column holds the correct value then and which the incorrect one? alarmtime or dif? You won't be able to tell.
What you can do instead is create a view for convenience:
create view v_dial_devicehistoryalarm as
select
dha.*,
datediff(hour, lag(alarmtime) over (order by alarmtime), alarmtime) as dif
from dial_devicehistoryalarm dha;
Demo: https://dbfiddle.uk/?rdbms=sqlserver_2019&fiddle=b7f9b5eef33e72955c7f135952ef55b5
Remember though, that your view will probably read and sort the whole table everytime you access it. If you query only a certain time range, it will be faster hence to calculate the differences in your query instead.
Hey guys, maybe this is a basic SQL qn. Say I have this very simple table, I need to run a simple sql statement to return a result like this:
Basically, the its to dedup Name based on it's row's Value column, whichever is larger should stay.
Thanks!
Framing the problem correctly would help you figure it out.
"Deduplication" suggests altering the table - starting with a state with duplicates, ending with a state without them. Usually done in three steps (getting the rows without duplicates into temp table, removing original table, renaming temp table).
"Removing rows with duplicated column values" also suggests alteration of data and derails train of thought.
What you do want is to get the entire table, and in cases where the columns you care about have multiple values attached get the highest one. One could say... group by columns you care about? And attach them to the highest value, a maximum value?
select id,name,max(value) from table group by id,name
I've been trying to add multiple partition columns, to a BigQuery table, but it seems to only take one field, even if I add multiple partition fields in the query parameters.
I'm partitioning by date time and integer range.
It only takes the later of the pair to create partitions and ignores the first partition field.
Any ideas, would be appreciated?
BigQuery only supports partitioning on one column. If you want to partition on multiple columns, you can consider partitioning+clustering. The table can be clustered on up to 4 columns.
I use coalesce to combine the columns and partition the new field created from coalesce, worked for my purpose.
I have a table that is partitioned by the timestamp contained in column X. However, during ingestion this value might be NULL, and only later on will be filled with an UPDATE.
Will BigQuery move the record to the right partition after the UPDATE?
Yes, if you execute an UPDATE statement and set the partitioning column to have different timestamps, BigQuery will move the associated rows into the appropriate partitions.
I am relatively new to SQL, so I had a question about insertion.
I have a table of data that I need to import above the existing content of another table. For example, the table I am bringing in has 100 rows, and the table I'm bringing the data into has 100.
I need to make the table I am bringing new data into have 200 rows, and have the first 100 rows blank (so I can update those rows with my new content).
Is there an easy way to do that that I am just missing? Thanks for your help!!
Consider that the database is just a data store. How it's ordered should be up to the client or the caller. Usually the best means of this is with the ORDER BY clause when SELECTing.
So I'd suggest not worrying about how the RDBMS is storing the data, but how it's being extracted.
Likely there's a column or attribute that you're focused on keeping/maintaining order. Perhaps it's a date or number? Consider using that column in your ORDER BY, and remember you can use more than one column in your ordering.
We shouldn't rely on how the data is stored for presentation later on.
/* use SQLite's current_time to save when these records were created*/
INSERT INTO MyTable (Foo, Bar, CreatedOn)
SELECT Foo, Bar, current_time
FROM OtherTable