I have a GA 360 view that gets decent amount of traffic daily and i want to export the hit level data (using GA_client_id) to the aws s3 server. the limitation here is that GA API allows only 10000 records a day. someone suggested that if we put the GA_client_id in a custom dimension, the limit would not apply. is it true? please let me know if there is another solution to export more than 10000 records for a single view per day. please note that this will be a single query that will auto run daily at a specific time.
Thank you so much in advance.
10000 records a day.
Correction 10000 requests per day per view. A request response can include millions of records (rows)
Related
I am using the free bigquery sandbox to generate some custom metrics based on my analytics data. I have read in the documentation that the expiration time of table in free account is 60 days. What does this expiration time means ? What will exactly happen after 60 days. All my datas will be lost ? How can i increase the expiration time in this case ? Should i need to pay for it ? If yes, what will be the cost ?
According to the documentation:
The BigQuery sandbox gives you free access to the power of BigQuery
subject to the sandbox's limits. The sandbox allows you to use the web
UI in the Cloud Console without providing a credit card. You can use
the sandbox without creating a billing account or enabling billing for
your project.
In addition, according to the limits :
All datasets have the default table expiration time and the default
partition expiration set to 60 days. Any tables, views, or partitions
in partitioned tables automatically expire after 60 days.
You can edit this expiration date if your data is exported to BigQuery but, in order to do that, you have to upgrade the project's plan to use it (if needed). Then you would be billed by the amount of bytes processed, you can check the billing options here.
Thus, within BigQuery you can edit the expiration date. In BigQuery, you go to Project > Dataset > Table > Details > click in the pencil next to the table's name and set expiration date to never or select a date. As follows:
Is it possible to set a date range for requestReport?
POST /v2/sp/{recordType}/report
{
"segment": {segment},
"reportDate": {reportDate}, <-- here
"metrics": {metrics}
}
Or do I need to make 30 request in order to get results for a month? Maybe snapshots can help?
Yes, you have to do multiple requests (one for each day) and merge them. The Advertising API currently does not offer an option to request data for a date range.
Snapshots can maybe help you to get data which has no click data for the requested date(s).
Yes, you do. What I do is run it daily using a cron job and load the data into a database, and query the database. Keep in mind certain metrics, like sales, don't finalize until after the attribution window e.g. 14 days.
The documentation for Google Custom Search JSON/ATOM API is unclear regarding pricing for additional queries. It states:
The API provides 100 search queries per day for free. If you need
more, you may sign up for billing in the API Console. Additional
requests cost $5 per 1000 queries, up to 10k queries per day.
For those that use that API in excess of the initial free 100, does the $5/1000 additional queries reset each day, or does that number roll over for subsequent days?
For instance, if I have a total number of queries on 3 consecutive days of 110, 120, and 130, will the account be billed $5 each day for the 10, 20, and 30 extra queries? Or will I be billed $5 the first day and by the end of the 3rd day I'll still have a bucket of 940 additional queries left to use for future overages?
In case anyone is also still looking for an answer to this question, (as I was a day ago), it turns out that for the Google Custom Search API the billing for $5/1000 queries is prorated.
Google Cloud Support emailed me the following:
With the Google Custom Search, you have a free 100 search queries per day which resets daily. But for example if you exceeded the daily limit, you will be charged $5 per 1,000 queries. In case that you go over the 100 queries per day and did not reach the 1,000 queries, it will be prorated from the $5. This resets daily as well. Please take note that 10k queries is the maximum daily.
I then clarified with them the following example, which they agreed was correct.
An example of making an average of 180 JSON API queries/day:
100 queries/day are free
80 queries/day are charged at $5 * (80/1000) = $0.40/day
Monthly it would be $12.00 (40cents * 30)
I am an analytics student and wanted to work on a live dashboard. With the help of below API end point, I was able to build a sample dashboard. But the total rows being returned is just 1000. The dataset actually has almost 50k rows
https://data.hartford.gov/resource/889t-nwfu.json
But I want the complete dataset as an input. How should i go about it ?
The Socrata dev api has a default limit of 1000 records. If you want more, you need to set the limit higher and probably implement some paging:
http://dev.socrata.com/docs/paging.html
For those of you who use the Amazon Product Advertising API, what experience have you had with running into their throttle? Supposedly, the limit is set at 1 request per second, is that your experience?
I want my site to grow to be nation-wide, but I'm concerned about its capability to make all the Amazon API requests without getting throttled. We cache all the responses for 24 hours, and also throttle our own users who make too many searches within a short period.
Should I be concerned? Any suggestions?
I believe they have changed it. Per this link:
https://forums.aws.amazon.com/message.jspa?messageID=199771
Hourly request limit per account = 2,000 + 500 * [Average associate revenue driven per day over the past 30 days period]/24 to a maximum of 25,000 requests per hour.
Here is the latest on request limits that I could find, effective Sept 3rd, 2012.
If your application is trying to submit requests that exceed the
maximum request limit for your account, you may receive error messages
from Product Advertising API. The request limit for each account is
calculated based on revenue performance. Each account used to access
the Product Advertising API is allowed an initial usage limit of 1
request per second. Each account will receive an additional 1 request
per second (up to a maximum of 10 requests per second) for every
$4,600 of shipped item revenue driven per hour in a trailing 30-day
period.
https://affiliate-program.amazon.com/gp/advertising/api/detail/faq.html
They have updated their guidelines, you now have more requests when you sell more items.
Effective 23-Jan-2019, the request limit for each account is calculated based on revenue performance attributed to calls to the
Product Advertising API (PA API) during the last 30 days.
Each account used for Product Advertising API is allowed an initial
usage limit of 8640 requests per day (TPD) subject to a maximum of 1
request per second (TPS). Your account will receive an additional 1
TPD for every 5 cents or 1 TPS (up to a maximum of 10) for every $4320
of shipped item revenue generated via the use of Product Advertising
API for shipments in the last 30 days.
Source: https://docs.aws.amazon.com/AWSECommerceService/latest/DG/TroubleshootingApplications.html
Amazon enforces limits on how many calls you can make per hour and per second.
You can increase the former by following the sanctioned route (increase commission revenue) or by privately petitioning Amazon with a valid reason. When whitelisted, your limit will go up to 25,000 calls per hour, which is more than good enough for the vast majority of projects I can think of.
The latter limit is murkier and enforced depending on the type of query you make. My interpretation is that it is meant to keep serial crawlers who do batch item lookups in check. If you are simply doing keyword searches etc., I would not worry so much about it. Otherwise, the solution is to distribute your calls across multiple IPs.
One other point to keep in mind if you are querying multiple locales is to use separate accounts per locale. Some locales are grouped and will count to the same call quota. European Amazons, for instance, form such a pool.