What kind of service should I use to measure how hot a topic is on Twitter, and how hot it has been in the past?
I thought about:
The Twitter API (https://dev.twitter.com/rest/reference/get/search/tweets) that lets me run searches up to 100 tweets. So in this case I have to make multiple calls to determine how many tweets there are. Is that correct?
TweetReach, that gives reports like this: https://tweetreach.com/reports/16000571, but the cheapest plan is at 300$/month.
With the Twitter API, you have a few options, but none of them may be exactly what you want, and none of them can go back very far into the past. You would have to either compile that information yourself, or use an external service like the one you mentioned.
Using the search API, you can only get results from the past 7 days, and are limited to 100 tweets per request. You can also set result_type to popular to get the most popular tweets about that search term. Twitter does have rate limits, but the ones for search are relatively high. You can use 180 requests every 15 minutes for any user you have authenticated, plus 450 requests every 15 minutes for the app itself (completely separate from the user requests). So if you only use app requests, you can get 45,000 tweets every 15 minutes.
If you don't need to search for specific terms, you can get trending topics in different areas using trends. The available areas can be retrieved using trends/available. Searching for trends also gives you the tweet_volume of each trend over the past 24 hours. If you check the trends every 24 hours and save the volumes, you can build up histories of trending topics.
Another option is using the streaming api. This only gives you current tweets, but you can use track to only get results for a set of terms, which you can then analyze.
Any external service, like TweetReach, will probably either cost you money or strictly limit the amount you can do with it unless you pay.
I'm the Social Media Manager for Union Metrics (we make TweetReach and lots of other things) and I just wanted to let you know that our free snapshots are built on the Search API which gives it those restrictions you've already discussed above, while our full snapshot reports can grab up to 1500 tweets for $20.
We do have more comprehensive Twitter analytics which I think you've already looked at, and those do backfill 30 days before tracking going forward. However you might have missed our new product Echo, which allows for a full, interactive search of the entire Twitter archive (you can see it in action here https://unionmetrics.com/product/echo-twitter-archive-search/) and is available through our Social Suite.
I understand if you don't have a large budget, and I completely understand the dilemma of cost of your time to build what you need vs. budget restrictions. Hope this helps at least let you know what else we offer!
Sarah A. Parker
Social Media Manager | Union Metrics
Fine Makers of TweetReach, The Union Metrics Social Suite, and more
Related
I have a project where I need to have the API quota increased significantly from the 10,000 daily hits, and I think this is being processed by Google as part of a YouTube API Services Compliance Review.
However, I have not had any response in over a week and the delay is putting the project at risk of a delayed launch and additional costs.
Does anyone know if this is normal and if there is a way to expedite the review, or speak to someone? Even pay for a higher tier of support?
Thanks in advance.
If you’ve filled the audit form https://support.google.com/youtube/contact/yt_api_form?hl=en properly, you should get a response within two weeks (Google reviews thousands of these, among other things to prevent abuse this is one of the processes that isn’t fully automated).
I recommend if your in a rush since your paying for credits you might as well open a second account and load balance between two or even three accounts; in your code you can create counters and swap before capping out the 24 hour term; not sure what data you’re looking to extract but depends on what data you may be able to even use other services to supplement.
They will get back to you about your application; just requires massive patience.
The problem: Running into Google Sheets API read/write quota limits. Specifically, the read/write requests per 100 seconds and read/write requests per 100 seconds per user quotas.
Some background:
For the past few months I've been developing a web app for students and staff in our school district which uses a Google spreadsheet as the database. Each school in our district was assigned a different Google spreadsheet, and a service account was created to make read and write calls to these spreadsheets on behalf of the web app.
We started with one school of approximately 1000 students, but it has now expanded to two other schools with a total user load of around 4000. Due to the nature of a school day schedule, we started hitting our quota limit (per 100 sec & per 100 sec per user) since almost everyone uses the app at the same time.
I found the usage limits guide for the Google sheets API, and as per the instructions I created a billing account, and linked the associated service account project to it. I then went to the quotas section in the developers console and applied for a higher quota. This involved filling out a Google form which asked "How much quota do you need? Express in number of API queries per day." Again, queries per day is not the problem, rather it's the number of queries per 100 seconds and per user (service account). After a couple of weeks our limit was increased to 2500 read/write requests per 100 seconds and 500 read/write requests per 100 seconds per user. The billing account was not charged, and after a little searching, I realized this was a free increase. This bump in our quota limit helped, but it's still going to be an issue because our district wants to add more schools in the future.
Here's what I need to know:
1) [ESSENTIAL QUESTION] Does Google have an upper limit or maximum to the number of read/write requests a single service account/user/IP can make within the 100 second time frame, and if so what is it?
2) If it is possible to go beyond our current quota limit (2500/500), is there another way of requesting/applying for the increase. Once again we have a billing account established for the project and are willing to pay for the service.
I've been pulling (what's left of) my hair out trying to find definitive answers to my questions. This post came close to what I was looking for, and I even did some of the things the OP suggested, but I just need a direct answer to my "essential" question.
Couple more things.
I understand that Google Charts Visualization doesn't have a quota limitation, and I'd consider using it however, for privacy reasons I can't have the spreadsheet keys exposed in plain javascript. Are there other options here?
Also, one might suggest creating multiple service accounts, but I'd rather avoid this if possible.
Thank you for your help. I'm very much a novice and I greatly appreciate your time and expertise.
To answer your questions:
1) [ESSENTIAL QUESTION] Does Google have an upper limit or maximum to the number of read/write requests a single service account/user/IP can make within the 100 second time frame, and if so what is it?
*The provided documentation only stated that Google Sheets API has a limit of 500 requests per 100 seconds per project, and 100 requests per 100 seconds per user. Check this post for additional information.*
2) If it is possible to go beyond our current quota limit (2500/500), is there another way of requesting/applying for the increase. Once again we have a billing account established for the project and are willing to pay for the service.
AFAIK, you can request for a higher quota limit and the Google Engineers may grant the request as long as you are making a reasonable request.
Also, you may check this thread for additional tips:
You can use spreadsheets.get to read the entire spreadsheet in a single call, rather than 1 call per request. Alternately, you
can use spreadsheets.values.batchGet to read multiple different
ranges in a single call, if all you need are the values.
The Drive API offers "push notifications", so you can get notified when changes occur and react to those, instead of polling for
them. The latency of the notifications is a little on the slow side,
but it gets the job done.
According to this link, it says that JSON Custom Search API provides 100 search queries per day for free. If you need more, you may sign up for billing in the API Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day.
I have been using the JSON Customsearch API which is https://www.googleapis.com/customsearch/v1 for quite a while now. Yesterday, I fired around 360 requests. So does it mean that since I crossed 100 search queries, I'll be charged for it?
The reason that I asked this was because my Dashboard doesn't show any charges yet.
Yes, you need to pay it.
According to my experience, the charge information on Dashboard do not refresh all the time.
I'm trying to find a way to get historical speed data for a certain road in the UK to calculate its average speed per time of day AND the maximum speed driven by a any driver on the road between a period time. Any pointers how to do this from Waze? Thanks
I'm afraid Waze doesn't expose that data (understandably, as it is their core business). This excerpt from the help page should say enough:
Please note: Waze does not share any historical data with partners.
If you work for a local government or organisation, you could consider joining the Connected Citizens Program. As a partner you are able to get a data feed for a certain route and you're allowed to store that data to get historical data (as detailed on the Waze Partners Help site).
While I'm not certain about the legal status of doing this without being a partner, you could probably also start building your own historical dataset based on what Waze provides as average speed on a segment by periodically looking up the data returned when you plan a route on the Waze Live Map.
Routing requests are sent to https://www.waze.com/row-RoutingManager/routingRequest?... (see the network console of your browser for more details), but this requires some additional work managing CSRF and session cookies and providing the proper referral header. While not impossible, it's not too easy to pull off.
The response of such a routing request contains the instructions you see on the live map, but also includes things like the length of each specific segment on the route (distance), its average speed without realtime data (crossTimeWithoutRealTime) and its average speed with realtime data (crossTime). It's also possible to request the average speed for a certain time in the day, but this tends to be somewhat unreliable data.
The maximum speed is something you won't be able to find in Waze's data though, I'm afraid. I'm not even certain Waze stores that information as those statistical outliers generally aren't that interesting for navigational instructions. You could try to contact Waze for more information if you're doing a scientific study, but don't get your hopes up too much in that case as they have a small team that is constantly overwhelmed by the amount of questions they receive.
I need the historical numbers of followers for a set of 60 Twitter users. Unfortunately the official Twitter API only returns the most current follower count. I know that there is probably no one-size-fit all and fool-proof way to get an accurate follower growth graph for all users in my list. However, would there be a good and logical way to estimate or deduce this based on other information available through Twitter API (e.g. number of retweets, likes)? I read somewhere that retweet rate grows proportionally with number of followers.
Any help or advice would be much appreciated. Thanks!
I think any approximation would be based on some criteria that would most probably not be applicable to all users.
From my point of view, I would start recording the followers from now on in your database and build a graph starting from today and not from the past.
There are services that allow you to buy or access historical data. But only if they have been tracking the account already. For large accounts, it is more likely, but for smaller accounts they probably haven't been tracking them.
https://twittercounter.com/pages/buy-stats/apple
I haven't used these services but I assume they are accurate.