Is there a way to increase INFORMATION_SCHEMA default Data Retention past 180 Days?
I have a requirement to report on User Query Trends for a large amount of time, greater than 180 days.
Any and all feedback and comments are welcome :)
Related
I have noticed that my big query database was configured to clear data more than 60 days old. The setting has been changed on the dataset for Default table expiry = Never but came with the warning "existing tables will not be affected"
What does this mean for future data preservation? It looks as though each day is its own table, so the 60 day expiry will only stop being a problem 60 days from now?
any clarification on what the wording of this warning means and how it will affect our data preservation in big query would be great
thanks
Aaron
If you have already created tables in that dataset, then these table still have 60 days expiry. You need to clear this setting on each table individually. Otherwise all data older than 60 days will keep deleting every day.
If your system creates new table everyday, then yes, the expiry will stop being a problem after 60 days, because new tables will have no expiry. All old tables will be emptied though.
I am using the free bigquery sandbox to generate some custom metrics based on my analytics data. I have read in the documentation that the expiration time of table in free account is 60 days. What does this expiration time means ? What will exactly happen after 60 days. All my datas will be lost ? How can i increase the expiration time in this case ? Should i need to pay for it ? If yes, what will be the cost ?
According to the documentation:
The BigQuery sandbox gives you free access to the power of BigQuery
subject to the sandbox's limits. The sandbox allows you to use the web
UI in the Cloud Console without providing a credit card. You can use
the sandbox without creating a billing account or enabling billing for
your project.
In addition, according to the limits :
All datasets have the default table expiration time and the default
partition expiration set to 60 days. Any tables, views, or partitions
in partitioned tables automatically expire after 60 days.
You can edit this expiration date if your data is exported to BigQuery but, in order to do that, you have to upgrade the project's plan to use it (if needed). Then you would be billed by the amount of bytes processed, you can check the billing options here.
Thus, within BigQuery you can edit the expiration date. In BigQuery, you go to Project > Dataset > Table > Details > click in the pencil next to the table's name and set expiration date to never or select a date. As follows:
I have a basic question relating to Cloud storage pricing. I see many clould service providers mentioning charges, for example "$1 per 25 GB per month". Let us consider two cases -
If I am storing 25 GB every day but deleting data of previous day then I will eventually have 25 GB storage on last day of the month. In this case my charges can be $1.
If I am storing 25 GB every day and deleting data of previous day, but I wrote 25 * 30 GB amount of data entire month. So even if I was cleaning the data my total storage amounts to 750 GB and so my charges should be 750/25 * 1 = $30.
What will be my cost at the end of the month?
Storage charges only apply for the duration of time that data existed, which is your 1st case.
Put differently, storing one 50 GB object for 10 days costs the same as storing two 25 GB objects for 5 days each (or one 500 GB object for 1 day, for that matter).
See also the second point here about prorated storage charges.
In your 2nd case, "wrote 25 * 30 GB" actually describes network ingress -- which is free (see here under "General network usage")
The documentation for Google Custom Search JSON/ATOM API is unclear regarding pricing for additional queries. It states:
The API provides 100 search queries per day for free. If you need
more, you may sign up for billing in the API Console. Additional
requests cost $5 per 1000 queries, up to 10k queries per day.
For those that use that API in excess of the initial free 100, does the $5/1000 additional queries reset each day, or does that number roll over for subsequent days?
For instance, if I have a total number of queries on 3 consecutive days of 110, 120, and 130, will the account be billed $5 each day for the 10, 20, and 30 extra queries? Or will I be billed $5 the first day and by the end of the 3rd day I'll still have a bucket of 940 additional queries left to use for future overages?
In case anyone is also still looking for an answer to this question, (as I was a day ago), it turns out that for the Google Custom Search API the billing for $5/1000 queries is prorated.
Google Cloud Support emailed me the following:
With the Google Custom Search, you have a free 100 search queries per day which resets daily. But for example if you exceeded the daily limit, you will be charged $5 per 1,000 queries. In case that you go over the 100 queries per day and did not reach the 1,000 queries, it will be prorated from the $5. This resets daily as well. Please take note that 10k queries is the maximum daily.
I then clarified with them the following example, which they agreed was correct.
An example of making an average of 180 JSON API queries/day:
100 queries/day are free
80 queries/day are charged at $5 * (80/1000) = $0.40/day
Monthly it would be $12.00 (40cents * 30)
I'm wondering if anyone knows whether the long-term-storage pricing (https://cloud.google.com/bigquery/pricing#long-term-storage) applies to GA daily export (https://support.google.com/analytics/answer/3437719?hl=en) data in BigQuery?
Since these tables cannot be modified, does it mean there will always be a discount 90 days after a daily GA table is available in BigQuery?
As Mikhail said in a comment, the answer is yes: The long term storage discount will be applied on the GA tables, as in any other unchanged table you store in BigQuery for more than 90 days (https://cloud.google.com/bigquery/pricing#long-term-storage).