Query is related to list partition. I have 600 tables and each of them stored in list partition. Question is - can i drop partition without losing data?
If your partition has data you will lose them.
If your partition is empty you will not have any problem.
Regards.
Related
Problem statement:
I need to insert/update a few columns in a big query table that is partitioned by date.So basically I need to do the necessary changes for each partitioned date (done by day).
(its the sessions table that is created automatically by linking the GA View to BQ so I haven't done the partition manually but its automatically taken care by google).
query reference from google_docs
my query:
I also tried the below :
Can anyone help me here ? sorry I am a bit naive with BQ.
You are trying to insert into a wildcard table, a meta-table that is actually composed of multiple tables. Wildcard table is read only and cannot be inserted into.
As Hua said, ga_sessions_* is not a partitioned table, but represents many tables, each with a different suffix.
You probably want to do this then:
INSERT INTO `p.d.ga_sessions_20191125` (visitNumber, visitId)
SELECT 1, 1574
For streaming inserts, I want to use a template table (with user id suffix) which is itself a Partitioned table. This way I can make my tables smaller than just using Partitioned Tables and hence make my queries more cost-effective. Also my query cost per user stays constant irrespective of the number of users in my system. As per the documentation at https://cloud.google.com/bigquery/streaming-data-into-bigquery:-
To create smaller sets of data by date, use time-partitioned tables. To create smaller tables that are not date-based, use template tables and BigQuery creates the tables for you.
It sounds as if it can either be a time-partitioned table OR a template table. Can it not be both? If not, is there another architecture that I should look into?
One more concern regarding my above proposed architecture is the 4000 limit that I saw on https://cloud.google.com/bigquery/docs/partitioned-tables . Does it mean that my partitioned table can't cover more than 4000 days? Will I have to delete old partitions in this case or will the last partition keep storing any subsequent streamed data?
You should look into Clustered Tables on partitioned tables.
With that you can have ONE table with all users in it, partitioned by time, and clustered by user_id as you would use in a template table.
Introduction to Clustered Tables
When you create a clustered table in BigQuery, the table data is automatically organized based on the contents of one or more columns in the table’s schema. The columns you specify are used to colocate related data. When you cluster a table using multiple columns, the order of columns you specify is important. The order of the specified columns determines the sort order of the data.
Clustering can improve the performance of certain types of queries such as queries that use filter clauses and queries that aggregate data. When data is written to a clustered table by a query job or a load job, BigQuery sorts the data using the values in the clustering columns. These values are used to organize the data into multiple blocks in BigQuery storage. When you submit a query containing a clause that filters data based on the clustering columns, BigQuery uses the sorted blocks to eliminate scans of unnecessary data.
Similarly, when you submit a query that aggregates data based on the values in the clustering columns, performance is improved because the sorted blocks colocate rows with similar values.
Clustered table pricing
When you create and use clustered tables in BigQuery, your charges are based on how much data is stored in the tables and on the queries you run against the data. Clustered tables help you to reduce query costs by pruning data so it is not processed by the query.
In dynamic partitioning in Hive, suppose we want to partition a column which is there in the middle of the table, we must be creating a new table and then reordering the columns to get the column which is to be partitioned in the last.
Is it really fine if we do this on the cluster where we get huge data?
I was wondering if anyone knows if there is a way to sort the data that already resides in a database. That is, I want to sort what is there but NOT retrieve it in in query.
I am asking because I have a list of things in this database, that I would like to add to in future and would like to order it once I've added them.
So what I mean is, I would like to not have to download all the data; sort it; then put it back onto the database.
Thanks in advance.
If there is only one column in your table then this is fine and you can simply sort the data in that table.
But if there are more than one columns in your table then it would be dependent on the other columns as well(ie, you need to specify which column you are looking to sort.) Also if there is a primary key attached to the table then its not possible as primary key would be in ascending order by default. In that case you can only have it while selecting the data from the table.
(My suggestion is to sort the data while selecting your records from the table as that would be easy and will have less risk)
EDIT:
To make my point straight and clear, the best way to achieve what you are trying to simply use ORDER BY ASC or ORDER BY DESC while you are selecting the data from your table.
If you are creating a new table then you can create index organized table to ensure that the data is stored ordered by index.
I was wondering if anyone knows if there is a way to sort the data that already resides in a database.
Normal relational tables, called as heap-organized tables, store rows in any order i.e. unsorted. You sort the rows if required only when you fetch them, not when you store them. And the only way to guarantee the sorting while retrieving the rows is to use an ORDER BY clause.
I have a table with lots of columns, and I'd like to add two more (date and time) to the front of the existing table.
There is no data in the table right now, but I'm wondering what the best way is get the table in the format I need it.
I could just drop the table and create a new one with the correct configuration, but I'm wondering if there is a better way?
This is currently not possible. You have to drop and recreate the table.
Theoretically you could add the column, drop and re-add all other columns, but that's hardly practical.
It's an ongoing discussion and an open TODO-item of the Postgres project to allow reordering of columns. But a lot of dependencies and related considerations make that hard.
Quoting the Postgres project's ToDo List:
Allow column display reordering by recording a display, storage, and
permanent id for every column?
Contrary to what some believe, the order of columns in a table is not irrelevant, for multiple reasons.
The default order is used for statements like INSERT without column definition lists.
Or SELECT *, which returns columns in the predefined order.
The composite type of the table uses the same order of columns.
The order of columns is relevant for storage optimization (padding and alignment matter). More:
Calculating and saving space in PostgreSQL
People may be confusing this with the order of rows, which in undefined in a table.
In relational databases the order of columns in a table is irrelevant
Create a view that shows you the columns in the order you want
If you still want to, drop the table and recreate it