We are adding CSE to our large website.
A primary requirement is that time sensitive data, such as an event, must be removed from the CSE results soon after the event has passed. Our initial strategy is to use page map data to set a boolean flag on each of these pages for inclusion in the CSE results. While this implementation works well, at more than 14k pages, the cost for Googles Demand Indexing on a weekly basis is unrealistic. Without the On-Demand indexing, some pages remain un-indexed weeks after their addition to the sitemap.
Are there any other strategies to work around the indexing cost?
One alternative I can think of is Linked CSE. It is a CSE where you host the annotations (urls associated with labels, which decide whether url should be displayed or not) lives on your own server.
Because you are not pushing the annotations to Google, you don't incur indexing cost.
https://developers.google.com/custom-search/docs/linked_cse
Related
We've been working successfully with the AdWords API (Version: 201708 -
Google Ads Python Client Library) for a good while building internal reports for our application. Until, that is, we hit placements…
I define placements as anywhere an AdWords ad is shown. The placement might be a domain, page, ad unit, app you name it! Placements are a very broad definition.
For our app to work for placements we need to join API spend data with activity on our website.
To do this we are running AdWords API reports and then collecting session data using AdWords ValueTrack parameters.
The ValueTrack parameters are easy enough, as there seems to be only 1 option: {placement}.
However, it's on the API where things get interesting, the API has numerous options for getting placement data. For example:
https://developers.google.com/adwords/api/docs/reference/v201708/AdGroupCriterionService.MobileApplication
https://developers.google.com/adwords/api/docs/appendix/reports/url-performance-report
https://developers.google.com/adwords/api/docs/appendix/reports/placement-performance-report#criteria
https://developers.google.com/adwords/api/docs/appendix/reports/automatic-placements-performance-report#domain
https://developers.google.com/adwords/api/docs/reference/v201708/AdGroupCriterionService
After spending some time going back and forth on the various options, and burning lots of dev time, we've come to the conclusion that there must be some best practice advice out there for joining placement data from the API and ValueTrack. One that works for all types of placements, including:
Websites
Apps
AdSense
Blogspot
AMP
An example of where we are running into a matching problem is "10060.android.com.nytimes.android.adsenseformobileapps.com"... this is a placement we see coming in from ValueTrack but has no match in any of our spend reports. (In fact there are many many adsenseformobileapps.com traffic sources for which there are no spend items).
Also seeing strings like "mobileapp::2-com.mobilesrepublic.appy". These show up on our spend side but only appear in our ValueTrack around 10% of the time. Some match. The vast majority don't.
A definitive workflow on this would be SO useful for ourselves and no doubt other users…
Thanks!
According to https://developers.google.com/adwords/api/docs/guides/valuetrack-mapping
the incoming ValueTrack placement should map to the following report fields:
PlacementPerformanceReport.Criteria
CriteriaPerformanceReport.Criteria
AutomaticPlacementsPerformanceReport.DisplayName
In addition to this I have also found this report useful:
UrlPlacementPerformanceReport.Domain and .Url
But I have found it is not so clear in practice. For one thing each of these reports return a slightly different subset of results..and none of these subsets exactly match the ValueTrack data set.
Here are the exceptions I have found:
subdomains
ValueTrack placements have urls with www on them... some of the time. None of the other reports do, so you will either have to strip www from ValueTrack or add www to your report data in order to match them. But be careful, other subdomains are preserved (like edition.cnn.com) and not all urls have a subdomain, so you can't just strip all the subdomains from Valuetrack and you can't just add www to all the urls in the reports. What I have found actually matches the best is the url field from the UrlPlacementPerformanceReport... but for this field you just need to strip everything after the / to get a best case matching subset. To use the other reports you would need to strip all the subdomain information from ValueTrack and sum the totals from those records. This means you would lose potentially useful data such as differences between espn.com, scores.espn.com, insider.espn.com and games.espn.com. Using the UrlPlacementPerformanceReport.url is the only way to preserve that info.
mobileapp::
ValueTrack reports on mobileapp:: placements. Many of the reports return these values too but I have found that each report just gives a subset of the whole. In particular the CriteriaPerformanceReport.Criteria report gives you many mobileapp:: values that none of the other reports do, but the other reports give you at least a few values that the CriteriaPerformanceReport doesn't. To be complete you would have to take a Union of the mobileapps: returned by the criteria performance report and another report such as the UrlPlacementPerformanceReport.url.
anonymous.google
ValueTrack provides sudomains to anonymous.google that look like a8122ac7e5da8e49.anonymous.google. If you want to match this information to your spend the only report that has this detail is UrlPlacementPerformanceReport.url.
adsenseformobileapps.com
ValueTrack provides detailed domains such as 1.iphone.com.localtvllc.fox2.adsenseformobileapps.com. None of the adwords reports can match this. The best you can get is a single summation record for the entire adsenseformobileapps.com group.
I have sort of a problem how to use Google Analytics properly with Boostrap.
My page has 3 level deep subpages and the last subpage has it's own subdomain. In GA I see I can use max. 50 tracking codes within one service. What if I need more than that?
You are limited to 50 properties not 50 pages. Each property can track many pages and (up to 10 million hits a month for the free version) and events.
Typically you would use the same property code on all pages on the same site so you can see all that data together (though with option to drill down).
You would only use a new property code for a new site (though your subdomain might qualify for that if you want to track it separately).
So the two questions you want to ask yourself are:
Do you want to be able to report on two pages together? E.g. To see that your site gets 10,000 hits and 20% are for this page and 5% are for that page. Or people start at this page and then go to that page and then on to this page. If so it should be the same analytics property.
Do different people need to see these page stats? And is it a problem if they do? If so put as a separate property so you can permission separately.
It sounds like these are part of the same site so I'd be veering towards tracking them together on same property.
On a different note you should set one page as the main version (with a rel canonical tag) and redirect other version to that page to avoid confusing search engines thinking you have duplicated content. Do you have a reason for having the same content on two different addresses? It can cause SEO and other problems.
I see that search engines are prominently capable of finding pages chronologically for forum websites and the like, offering the option to show the results for the last 24 hours, last week, last month, last year, etc.
I understand that these sites need to be continuously crawled to provide those updates, but I have technical doubts about what structure, tags or whatever I need to do to achieve it for my website.
I see that at the client side (which is also the side search engines are at) content appears basically as static data, already processed by the server, so the question is:
If I have a website for which I update and add content constantly to the index page to make it easily visible, and for which I even add links, times and dates as text for the new pages, why don't these updates show at all in search engines?
Do I need to add XML/RSS feeds, or what else?
How do forums and sites with heavy updates with a chronological mark achieve the capability to allow search engines to list results separated by hours, days, etc.?
What specific set of tags and overall structure do I need to add for this feature?
I also see that search engines, mainly Googlebot, usually take a minimum of 3 days to crawl those new pages, but still, they aren't organized persistently (or at all) in a chronological way in search results.
I am not using any forum, blog or other kind of web publishing software, just raw HTML and PHP written by hand, and the minimum I mentioned above, of pointing to new documents from the index page of the website along with a description.
Do I need to add XML/RSS feeds, or what else?
Yes. Atom or one of the RSS formats (or several formats at the same time, so you could offer Atom and RSS).
Search engines will know about new blog posts, microblog post, forum threads, forum thread answers etc., because they subscribe to the feed. So sometimes you'll notice that a page is indexed by a search engines only minutes after it was published. But for smaller sites, search engines probably don't check for updates every few minutes, instead it might take even days until a new page is indexed.
A sitemap might help, too.
For our web app, which will use Amazon's API as a basis for some of the site's main interactions, we required the ability to do a generalized search of Amazon's products and return results based on relevancy. The expectation was that their API would work exactly like their actual site's search.
Unfortunately it does not. For instance, querying "joy of cooking" does not return a link to the famous cook book, but to some food processor. Contrarily, on the actual site, one would see the book isn't just first, but it and any derivations occupy the top 5 or so results.
Is there a way of getting this level of relevancy search from Amazon's API without specifying a node to browse through? We need to be able to search everything at once, and the API seems very limited on parameter sets.
The answer is that, if you use "All" as your sorting basis, rather than "Blended", you will get results that are inline with Amazon's own product search. Older docs don't seem to account for this discrepency, but testing both methods has shown "All" to be the preferred product sorting method.
http://docs.amazonwebservices.com/AWSECommerceService/2010-11-01/DG/
Pagesearch under "SearchIndex: All"
You don't get any item sorting options with this method, but if all you want is "most relevant" results, this is the preferred method.
We have a web application that has a defect and is updating all pages Last Modified Header to the date of the last publish. We are in the process of fixing the defect, but we wanted to know if this defect might impact our SE results for this site.
Basically each time a page on the site get's updated all pages updates the last modified date even if the page has not been updated.
Is there any possibility of the Search Engine detecting the site as spam, since all pages are changing too often? -- Theory
It's unlikely to change much, since all the search engines will notice that your content hasn't actually changed. They will crawl at a rate commensurate with the observed rate of content change, more or less regardless of what you tell them, and small changes like that won't be marked as content changes in the index.
Changing the last modified date too often will NOT have a negative impact with the big 3.
The only way you can affect crawl rate via meta data (and sitemap.xml) is to reduce it. The reason for this is indicators that increase ranking/indexing are too easily abused. However, reducing spider rate is still an option for the resource conscious webmaster.