I have seen that there are various APIs and various tools that allow you to see the most visited pages of the Wikimedia projects such as Wikipedia, but all these services have a limit, they do not allow to show more than 1000 pages, while I would like to have the list of 5000-10000(or more) most visited pages in order of traffic.
these are all the services that I checked and with which I found this limit:
https://en.wikipedia.org/w/api.php?action=help&modules=query%2Bmostviewed
https://stats.wikimedia.org/#/en.wikipedia.org/reading/top-viewed-articles/normal|table|last-month|~total|monthly
https://pageviews.toolforge.org/topviews/?project=en.wikipedia.org&platform=all-access&date=last-month&excludes=
https://wikimedia.org/api/rest_v1/#/Pageviews%20data
I have also found services like https://quarry.wmflabs.org/ or
https://query.wikidata.org/ where you can run a query, technically perhaps through this service you could but I don't know the query to be performed to show the pages with most visits.
I also found an interesting article here: https://www.reddit.com/r/bigquery/comments/3dg9le/analyzing_50_billion_wikipedia_pageviews_in_5/ where it is explained that it is possible to use Google's BigQuery but it is an external service and before using it I wanted to know if it existed a simpler method.
If the REST API doesn't suit your purpose, you'd need to parse the raw data yourself. That's because all the tools you've linked just consume the REST API.
The raw data are available at https://dumps.wikimedia.org/other/pageviews/. There are two groups of files there. One starts with pageviews-, which lists the number of views of individual pages, the second starts with projectviews-, which lists the number of views of individual projects.
For your target, you need the pageviews ones. Download the files for your timespan, and then analyze them using a script.
The file is space-separated. Each row represents one page that was visited in that hour. First column represents the project (en is English Wikipedia, for instance), second is the page title (spaces are represented by underscores) and then there are total pageviews.
The technical documentation is available at https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Pageviews.
Related
Is it possible to create a Kapow Robot that can search Google for the Operating hours of the Businesses from our list/database and update the timings if changes are made?
Please share if there are any other more efficient ways than the KAPOW robot that can be implemented with minimal effort and cost-effectiveness.
That's what the Google Places API is there for. While you could in theory just open Google Maps in a Load Page action, enter the query string and then parse the results, I would advise against it. Here's why:
The API will be faster, returning results in a structured manner (JSON)
Kapow has actions for calling RESTful services and parsing/modifying JSON
Google does not like robots parsing their pages, and most likely will lock you out (i.e. present you with Captchas sooner or later)
If you decide to go for the API, here's what you should do:
Get your API key first, see this page for details: https://developers.google.com/places/web-service/get-api-key. Note that the free plan allows for 1,000 requests within a 24-hours limit (https://developers.google.com/places/web-service/usage)
Maintain the place ids for all the businesses you'd like to query regularly, and update your list.
For each place, retrieve the details as described in the API documentation. The opening hours will be within the JSON response: https://developers.google.com/places/web-service/details
Update your list. I'd recommend using a definite type in Kapow for that, and using the actions Store in Database and Query Database. In case you need the data elsewhere, you may create additional robots (e.g. for Excel files, sending data per email, et cetera).
We've been working successfully with the AdWords API (Version: 201708 -
Google Ads Python Client Library) for a good while building internal reports for our application. Until, that is, we hit placements…
I define placements as anywhere an AdWords ad is shown. The placement might be a domain, page, ad unit, app you name it! Placements are a very broad definition.
For our app to work for placements we need to join API spend data with activity on our website.
To do this we are running AdWords API reports and then collecting session data using AdWords ValueTrack parameters.
The ValueTrack parameters are easy enough, as there seems to be only 1 option: {placement}.
However, it's on the API where things get interesting, the API has numerous options for getting placement data. For example:
https://developers.google.com/adwords/api/docs/reference/v201708/AdGroupCriterionService.MobileApplication
https://developers.google.com/adwords/api/docs/appendix/reports/url-performance-report
https://developers.google.com/adwords/api/docs/appendix/reports/placement-performance-report#criteria
https://developers.google.com/adwords/api/docs/appendix/reports/automatic-placements-performance-report#domain
https://developers.google.com/adwords/api/docs/reference/v201708/AdGroupCriterionService
After spending some time going back and forth on the various options, and burning lots of dev time, we've come to the conclusion that there must be some best practice advice out there for joining placement data from the API and ValueTrack. One that works for all types of placements, including:
Websites
Apps
AdSense
Blogspot
AMP
An example of where we are running into a matching problem is "10060.android.com.nytimes.android.adsenseformobileapps.com"... this is a placement we see coming in from ValueTrack but has no match in any of our spend reports. (In fact there are many many adsenseformobileapps.com traffic sources for which there are no spend items).
Also seeing strings like "mobileapp::2-com.mobilesrepublic.appy". These show up on our spend side but only appear in our ValueTrack around 10% of the time. Some match. The vast majority don't.
A definitive workflow on this would be SO useful for ourselves and no doubt other users…
Thanks!
According to https://developers.google.com/adwords/api/docs/guides/valuetrack-mapping
the incoming ValueTrack placement should map to the following report fields:
PlacementPerformanceReport.Criteria
CriteriaPerformanceReport.Criteria
AutomaticPlacementsPerformanceReport.DisplayName
In addition to this I have also found this report useful:
UrlPlacementPerformanceReport.Domain and .Url
But I have found it is not so clear in practice. For one thing each of these reports return a slightly different subset of results..and none of these subsets exactly match the ValueTrack data set.
Here are the exceptions I have found:
subdomains
ValueTrack placements have urls with www on them... some of the time. None of the other reports do, so you will either have to strip www from ValueTrack or add www to your report data in order to match them. But be careful, other subdomains are preserved (like edition.cnn.com) and not all urls have a subdomain, so you can't just strip all the subdomains from Valuetrack and you can't just add www to all the urls in the reports. What I have found actually matches the best is the url field from the UrlPlacementPerformanceReport... but for this field you just need to strip everything after the / to get a best case matching subset. To use the other reports you would need to strip all the subdomain information from ValueTrack and sum the totals from those records. This means you would lose potentially useful data such as differences between espn.com, scores.espn.com, insider.espn.com and games.espn.com. Using the UrlPlacementPerformanceReport.url is the only way to preserve that info.
mobileapp::
ValueTrack reports on mobileapp:: placements. Many of the reports return these values too but I have found that each report just gives a subset of the whole. In particular the CriteriaPerformanceReport.Criteria report gives you many mobileapp:: values that none of the other reports do, but the other reports give you at least a few values that the CriteriaPerformanceReport doesn't. To be complete you would have to take a Union of the mobileapps: returned by the criteria performance report and another report such as the UrlPlacementPerformanceReport.url.
anonymous.google
ValueTrack provides sudomains to anonymous.google that look like a8122ac7e5da8e49.anonymous.google. If you want to match this information to your spend the only report that has this detail is UrlPlacementPerformanceReport.url.
adsenseformobileapps.com
ValueTrack provides detailed domains such as 1.iphone.com.localtvllc.fox2.adsenseformobileapps.com. None of the adwords reports can match this. The best you can get is a single summation record for the entire adsenseformobileapps.com group.
I see that search engines are prominently capable of finding pages chronologically for forum websites and the like, offering the option to show the results for the last 24 hours, last week, last month, last year, etc.
I understand that these sites need to be continuously crawled to provide those updates, but I have technical doubts about what structure, tags or whatever I need to do to achieve it for my website.
I see that at the client side (which is also the side search engines are at) content appears basically as static data, already processed by the server, so the question is:
If I have a website for which I update and add content constantly to the index page to make it easily visible, and for which I even add links, times and dates as text for the new pages, why don't these updates show at all in search engines?
Do I need to add XML/RSS feeds, or what else?
How do forums and sites with heavy updates with a chronological mark achieve the capability to allow search engines to list results separated by hours, days, etc.?
What specific set of tags and overall structure do I need to add for this feature?
I also see that search engines, mainly Googlebot, usually take a minimum of 3 days to crawl those new pages, but still, they aren't organized persistently (or at all) in a chronological way in search results.
I am not using any forum, blog or other kind of web publishing software, just raw HTML and PHP written by hand, and the minimum I mentioned above, of pointing to new documents from the index page of the website along with a description.
Do I need to add XML/RSS feeds, or what else?
Yes. Atom or one of the RSS formats (or several formats at the same time, so you could offer Atom and RSS).
Search engines will know about new blog posts, microblog post, forum threads, forum thread answers etc., because they subscribe to the feed. So sometimes you'll notice that a page is indexed by a search engines only minutes after it was published. But for smaller sites, search engines probably don't check for updates every few minutes, instead it might take even days until a new page is indexed.
A sitemap might help, too.
Is this API simply for searching your website only, or can any standard google search (even advanced search features) be submitted to it? I understand there is a limit of 100 per day, I am just curious if it can be invoked from say your own machine as the code samples and introduction indicate its intended use is for displaying results on your website. I want to search outside of a given domain and scrape standard google results for any given search. This will not be an ajax call.
My current understanding:
You're limited to 100/day only if you don't pay.
You do have to specify domains, but some tlds are fine (eg: .uk)
There's a limit to 100 search results for any given search query (ten pages of up to ten responses)
It can be invoked from your own machine.
For our web app, which will use Amazon's API as a basis for some of the site's main interactions, we required the ability to do a generalized search of Amazon's products and return results based on relevancy. The expectation was that their API would work exactly like their actual site's search.
Unfortunately it does not. For instance, querying "joy of cooking" does not return a link to the famous cook book, but to some food processor. Contrarily, on the actual site, one would see the book isn't just first, but it and any derivations occupy the top 5 or so results.
Is there a way of getting this level of relevancy search from Amazon's API without specifying a node to browse through? We need to be able to search everything at once, and the API seems very limited on parameter sets.
The answer is that, if you use "All" as your sorting basis, rather than "Blended", you will get results that are inline with Amazon's own product search. Older docs don't seem to account for this discrepency, but testing both methods has shown "All" to be the preferred product sorting method.
http://docs.amazonwebservices.com/AWSECommerceService/2010-11-01/DG/
Pagesearch under "SearchIndex: All"
You don't get any item sorting options with this method, but if all you want is "most relevant" results, this is the preferred method.