We bought a new domain from HugeDomains.com before a month and made it live last week.
Before we move live, the advertisement published by HugeDomains.com got cached in search engines.
Now we need to remove that cached URL from all search engines.
Following is the Pattern of URL got cached, it's just a query string getting passed
http://www.example.com/?fp=ah1QKL6n%2FlECnlCZX2M7prGsvtbv8ddXendjKdEvTBtzHaEkYE%2BEk37MD1iDIPnimmKBVn7jZKj%2BPGqRUxNQzA%3D%3D&prvtof=ytNnOdijWVo6UL0CLJYkUNs043cNT%2BNtJQ5d5VD69Ac%3D&poru=RLg1S8TlJRc59ObVEdjqkbBOZjhk%2FIf%2BH8W1DtjVOk5VRbieT62uHl%2FGfuWk4d%2FnOfDQwYDvqLza3nG76SMxZA%3D%3D&
I have used Disallow in Robots.txt to remove that but its not working, following will be the code
Disallow: /*?fp=
Disallow:
/?fp=ah1QKL6n%2FlECnlCZX2M7prGsvtbv8ddXendjKdEvTBtzHaEkYE%2BEk37MD1iDIPnimmKBVn7jZKj%2BPGqRUxNQzA%3D%3D&prvtof=ytNnOdijWVo6UL0CLJYkUNs043cNT%2BNtJQ5d5VD69Ac%3D&poru=RLg1S8TlJRc59ObVEdjqkbBOZjhk%2FIf%2BH8W1DtjVOk5VRbieT62uHl%2FGfuWk4d%2FnOfDQwYDvqLza3nG76SMxZA%3D%3D&
I even enabled a 302 Redirect for this query string fp= to my home page
Please let me know a way to resolve this.
Thanks in advance.
I wouldn't do this with robots.txt.
Just wait. I think the most search engines will recognize that your website is new so they will crawl it again in near future.
Otherwise you can create a google-webmaster account and send your url to google to crawl it again.
EDIT: You're also able to disallow url-parameter in webmaster tools.
Robots.txt disallow should do it, but another good way is to return a 410 Gone result, then google will stop indexing it since it'll see this page has disappeared.
Edit
Looks like I was wrong about Robots.txt, but right about 410 Gone response:
Reference
You have to do a 301 permanent redirect for Google to drop old indexed page. If you do 302, Google will try to crawl that url once in a while as it is temporary. Ignoring query parameters does not help in clearing the cache, it just sends signal saying the url with query param is same as the one without it. I guess that is not what you want. My suggestion would be to do a 301 permanent redirect if you encounter query param fb.
Right now i doubt google handles 404 and 410 lot differently. So you can do a 410 as well.
Google webmaster can help you in removing outdated/ cache content from Google search results
Copy your domain Cached URL
Browse to https://www.google.com/webmasters/tools/removals
Follow Request instructions.
The cache can be removed in a few numbers of hours. Google search engine crawls to new/current URL contents.
Related
I have added 301 redirect on my website by mistake (because I was doing maintenance). Now lots of people can't get back to my website, because they are still redirected to other page - eventhough I removed redirection (even deleted htaccess). As much as I searched around it's because htaccess (or 301 redirect) is cached in users browser and I wasn't able to find any solution for this. Is there any way to fix this, I can't just loose hundreds of visitors because of something like this?
This page explains what is going on in good detail:
301 Redirects: The Horror That Cannot Be Uncached
Basically, modern browsers cache the redirect response for 301 for some indeterminate amount of time and will not make an updated request to your old web page to refresh it. Users can manually clear the cache and, because it is a cache, data can be purged if the browser needs more space for other data (like other redirects).
This SuperUser question resolves the caching issue from the client's end:
How can I make Chrome stop caching redirects?
One interesting answer is:
//superuser.com/a/660522/178910
In this answer, the user points out that the browser treats http://example.com/ and http://example.com/? as two different URLs. You could go to the "new" site and setup an HTTP 302 redirect pointing back to the original page with a ? on the end and it should load. If they original page already had a query as part of the URL, you can simple add an & to the end to achieve the same result.
It's not perfect -- it is a different URL after all -- but at least they'll be able to view your old site.
Note that your web application may try to redirect empty queries or invalid queries back to a "clean" page, which you may have to disable to get the intended result.
UPDATE
One other option is to put a redirect from the new site back to the old site (make this a 302 or 307 redirect to avoid the 301 problem you're currently having). From my testing, Chrome will remove the old redirect when it does this. It may throw a "redirect loop" error, but only once. I was unable to reproduce the cached redirect problem at all with the latest version of Firefox. Other browsers' behavior is probably going to be inconsistent.
I am new to this so I will try to explain myself clearly.
I am doing my 301 redirect from a custom 404 page. Now I got it working my question is more regarding how google would treat this. Cause we going to a 404 page would google just record it as a 40 page or would it actually record the 301? As i said I am new to this and have looked through google to try and find an answer to this.
Anyway any help or comment would be greatly appriciated. thanks in advance
Best practice in this case could be:
If the page doesn't exist, but we have new one, with highly similar content, we can make 301 redirection, simply saying: "Moved permanently", which is instructing Google to actually take new URL on account and prioritize it.
If the page doesn't exist, and we actually have no idea why someone could type this link, as this URL never existed and is just wrong, then we serve 404 "Not Found". It simply means that the URL is wrong, and someone (or some other website) has fooled you to follow this link. You shouldn't automatically redirect user from this page, but place a link to the homepage instead, so user can choose his action.
If the page doesn't exist, and we know that we had this page, but it doesn't exist, and it will not exist in the future as well (we has simply decided that we will no longer have this page), then serve 410 "Gone" page, with a link to homepage as well, and let user decide.
HTTP codes, are not just a theory, it's a standard we should use. I noticed, that many 404 pages are served without correct HTTP response code, which only suggests that there is a poor development behind it.
More about HTTP response codes here: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
From my understanding, a 301 redirect is the best way to retain "link juice" and should be used if the 404 page is referencing has a lot of external links, has substantial traffic, etc.
Sending a generic 404 page straight to the home page is not ideal, as it may confuse the user. Allowing the 404 keeps the page from being repeatedly indexed and crawled by search engines.
Read more about it here: http://moz.com/learn/seo/http-status-codes.
it is not ok to redirect 404 page to another. it's better to correct it and show the old page. if it's impossible you should show 404 page and put some helpful links in it.
if you want to redirect to the correct one it's ok but the best way is to show display original page regardless of duplication. but you must use rel canonical to tell search engines where is the correct version on the page.
https://support.google.com/webmasters/answer/139394?hl=en
I recently suffered a hack on a number of websites which were hosted on the same server. I've identified and removed the source of the hack, and used Patrick Altoft's smart Google Alerts idea to monitor for further attempts.
I've then logged into Google webmaster tools, asked to be re-evaluated post hack, and I've also re-submitted site maps to speed up a re-crawl.
However I would like to remove the infected url's from Google, and was thinking the best way to speed up this process would be to use .htaccess to return a 404 error, whenever a page with a specific string variable appeared.
Is this possible with a .htaccess file, or is there a better course of action to take?
You can see the damage done here.
Thanks for any help and suggestions.
404 will work, but is possible not the best solution. A better solution would be 301: moved permanently, or 410: gone.
A 404 tells you that a page is missing, but not why. Google may keep these urls for a while to investigate later whether they exist again. By using 301 or 410, you explicitly tell Google that that url is not going to be fixed.
410 is the better option, but I'm not sure if this is possible from htaccess, athough you could 301 to a php-file that returns a 410 header.
Addition: Here's an article about redirecting using the '410, Gone' header with .htaccess. http://diveintomark.org/archives/2003/03/27/http_error_410_gone
Yep, give them 404/410/301 status code, then Google will remove them in a day or two. I've done that before. It will take way too long for Google to renew its cache with 200 status code.
We have #1 and #2 spots we would like to keep, but because of the way things were jumbled we have to migrate to a new domain.
We do not want the new domain to be penalized for duplicate content, we want it to naturally take the spot on Google.
How do we tell Google our page has moved?
301 redirects are the recommended way to do this according to Google themselves. I tend to perform 301's using a .htaccess file (a few different methods here) but it can also be done using PHP like this:
header('Location: '.$newlocation, true, 301)
What seengee user has answered is perfect.
Have a look to Google Webmaster Tools too; there's a specific option for Address change.
Change of address
If you're planning to move your site to a new domain, use the Change of Address tool to tell Google about your new URL. This will help us update our index faster and smooth the transition for your users.
For best results, follow these steps:
Set up the new site
Review our guidelines for moving your site to a new domain. Set up your content on your new domain, then make sure all internal links point to the new domain.
Redirect all traffic from the old site
Use a 301 redirect to permanently redirect the pages on your old site to your new site. This tells users and search engines that your site has permanently moved. Ask webmasters to update their links to point to your new domain and make sure incoming links to your old site are redirected correctly using the 301 redirects.
Add your new site to Webmaster Tools
Make sure you have added and verified your new domain.
Tell us the URL of your new domain
Try this: https://www.google.com/webmasters/tools add both domains there and then, go Change of address.
Http 301 Request is used for Permanent Redirect the old website's content,URL and Links to New Website URL. Google Never Penalize the New Website for Duplicate content.
what about "Keyword" you use in meta for performing in SERP result. After redirection how you manage them.
position before Redirection , after redirection have same OR not.
AS SEO purpose don't make Redirection withour proper plan. First make proper optimization of "B" page before "A" redirect ( 301, 302 ) "B" page.
Dont make 302 Redirect , its spammy after 3 months, why within 20 Days SERP catche show "B" page instead of "A" page.
in the end : Before Redirection - Optimize the Destination URl - for Benefit SEOand business
Have a Nice Day ... Thanks
Google itself detect your web page if you implemented the 301, 302 or 404 redirection. But if you want to tell google that you have moved your page then:
Go to google webmaster tool
In Crawl section there is an option of Fetch as Google.
Just enter your url and check the fetch status, you will find that google will index your new url.
in google webmaster tools click on change of site address.
firstly you need to put content on new website where you want to redirect the website.
redirect usingold website by using HTTP 301 code and after verify in google webmaster tools.
Do 301 redirects. This will tell the search engines that the pages have moved and where they are now. This also associates the old URL with the new URL for Google which means all of your old incoming links will now be redirects to your new pages. Also use Google Webmaster Tools to submit XML sitemap.
In both Google and Bing Webmaster tools, you can notify a change of address. I will warn you, however, that when changing domains, you're not going to keep your current rankings.
Submit a change of address notice in Webmaster tools
Do 1:1 page redirecting (redirect all other pages that can't be mapped to a similar page to your homepage)
Submit an up-to-date sitemap
This should help:
https://support.google.com/webmasters/answer/83106?hl=en
http://moz.com/blog/achieving-an-seo-friendly-domain-migration-the-infographic
A few days ago we replaced our web site with an updated version. The original site's content was migrated to http://backup.example.com. Search engines do not know about the old site, and I do not want them to know.
While we were in the process of updating our site, Google crawled the old version.
Now when using Google to search for our web site, we get results for both the new and old sites (e.g., http://www.example.com and http://backup.example.com).
Here are my questions:
Can I update the backup site content with the new content? Then we can get rid all of old content. My concern is that Google will lower our page ranking due to duplicate content.
If I prevent the old site from being accessed, how long will it take for the information to clear out of Google's search results?
Can I use google disallow to block Google from the old web site.
You should probably put a robots.txt file in your backup site and tell robots not to crawl it at all. Google will obey the restrictions though not all crawlers will. You might want to check out the options available to you at Google's WebMaster Central. Ask Google and see if they will remove the errant links for you from their data.
you can always use robot.txt on backup.* site to disallow google to index it.
More info here: link text
Are the URL formats consistent enough between the backup and current site that you could redirect a given page on the backup site to its equivalent on the current one? If so you could do so, having the backup site send 301 Permanent Redirects to each of the equivalent pages on the site you actually want indexed. The redirecting pages should drop out of the index (after how much time, I do not know).
If not, definitely look into robots.txt as Zepplock mentioned. After setting the robots.txt you can expedite removal from Google's index with their Webmaster Tools
Also you can make a rule in your scripts to redirect with header 301 each page to new one
Robots.txt is a good suggestion but...Google doesn't always listen. Yea, that's right, they don't always listen.
So, disallow all spiders but....also put this in your header
<meta name="robots" content="noindex, nofollow, noarchive" />
It's better to be safe than sorry. Meta commands are like yelling at Google "I DONT WANT YOU TO DO THIS TO THIS PAGE". :)
Do both, save yourself some pain. :)
I suggest you to either add no index meta tag in all old page or just disallow by robots.txt. Best way to just blocked the by robots.txt. One thing more add the sitemap in new site and submit it in webmaster that improve your new website indexing.
Password protect your webpages or directories that you don't want web spiders to crawl/index by putting password protecting code in the .htaccess file (if present in your website's root directory on the server or create a new one and upload it).
The web spiders will never know that password and hence won't be able to index the protected directories or web pages.
you can block any particular urls in webmasters check once...even you can block using robots.txt....remove sitemap for your old backup site and put noindex no follow tag for all of your old backup pages...i too handled this situation for one of my client............