Three months ago I published my small personal website (~10 pages), submitted the URL to Google, and a few days later Googlebot showed up. Over the course of the last couple of weeks, Googlebot visits my website approximately twice a week and crawls maybe every other page.
Ever since Googlebot first crawled my website, whenever I run a search for site:example.com Google returns only my homepage. (Interestingly, so does Bing, so maybe the problem isn't specific to Google.)
I built the website with CodeIgniter mainly to familiarize myself with it. It's really simple, only a couple of pages about me and my projects. I am not using any black-hat SEO techniques, JavaScript, or anything like that.
What could be possible reasons why Googlebot would crawl my pages but not index them?
EDIT:
I do have a Webmaster Tools account. There are no crawl errors, internal links are listed, but listed keywords come only from my homepage.
Create XML site map and "tell" to Google about it in Google Webmaster tool
Ensure your 10 pages have distinguished content - since search engine can eliminate if content looks identical
Ensure links from home page lead to all another 9 pages. If so place link in
Related
Google is showing www.example.com/myPage as a search result.
I do not want this /myPage to be indexed by google, so a robots.txt was placed in the page.
How long will it take to stop being showed in google?
I know that people can still visit it if they have the URL, but my aim is just to remove it from google's search results.
My knowledge in SEO is little, and I feel the answer may vary depending on the site traffic and other SEO-related factors, but speaking in general terms, how long would this take?
Crawls are based on many factors such as PageRank, links to a page, and crawling constraints such as the number of parameters in a URL. Any number of factors can affect the crawl frequency of individual sites.
The crawl process is algorithmic; computer programs determine which sites to crawl, how often, and how many pages to fetch from each site. They don't accept payment to crawl a site more frequently. For tips on maintaining a crawler-friendly website, please visit the Webmaster Guidelines.
I would suggest you to use the google webmaster tool for your SEO this will help you to see that when Google last crawled your website also there are many SEO options that will help you to index your site better.
There is also an option in Webmaster to ask Google to crawl your site again telling Google bots to re crawl your site as the content on your site has changed.
This link might help you understand better.Also to get an overview of Webmaster setup and features visit this link
I have a site that I submitted to google, did some of the webmaster tools things for (sitemap, preferred domain etc) and set up analytics for about a week ago.
Yesterday i made a pile of changes to the body text, and to the meta and title tags text.
Should I re-submit the site to google, and how long will it take for me to see any results from my experiments!
Many thanks
Google may have not crawled your website yet. It may take 2 or 3 weeks.
From google support: Google's spiders regularly crawl the web to rebuild our index. Crawls are based on many factors such as PageRank, links to a page, and crawling constraints such as the number of parameters in a URL. Any number of factors can affect the crawl frequency of individual sites.
see also: https://support.google.com/webmasters/answer/70897?hl=en
You will know it did when you start seeing statistics in your webmaster tools. It's more of a trial and error here.
Implemented Google site search in our company website. We need to automate the google indexing for our website.
Suppose like our customers are updated the forum. We need to show the up to updated forum information in our forum search ?
Is there any option in google API or any other API please help me ?
You can use an XML sitemap. This will tell the search engines where your content is so they can find it and crawl it. Keep in mind there is no way to make the search engines crawl your site when you want them to. They will crawl on a schedule they determine to be right for your site. (You can set a crawl rate in Google Webmaster Tools but that rate is relative to what crawl rate Google already has set for you. Setting it to fastest will not speed up heir crawl rate)).
Unfortunately, Google will only crawl your site when it feels like it. It is based on many variables to determine how often this occurs (i.e. site ranking, standards compliance, and so on). The sitemap XML is a helpful way to help Google determine what parts of your site to index, however if you don't have one Google will find it by crawling links on other parts of your page and updating its index if the page changes.
The more visitors you get and the more often your site's links appear on other sites will make Google index more frequently.
To start, I'd suggest http://validator.w3.org/ to validate your site and make sure you get it as close to possible to no errors. This makes it easier for Google to index your site because it can find the information it expects without having to crawl over invalid markup. Also, chances are, if a site validates with a very small amount of errors, it is more credible than one containing many errors. It tells the search engine that you update your site to ensure most all browsers can use it and that it is accessible.
Also validating your site gives you some bragging rights over those who don't meet W3 standards :)
Hope this helps!
I have created a sitemap for my site and it complies with the protocol set by http://www.sitemaps.org/
Google has been told about this sitemap via webmaster tools. It has tracked all the urls within the sitemap (500+ urls) but has only indexed 1 of them. The last time google downloaded the sitemap was on the 21st of Oct 2009.
When I do a google search for site:url it picks up 2500+ results.
Google says it can crawl the site.
Does anyone have any ideas as to why only 1 url is actually indexed?
Cheers,
James
First off, make sure Google hasn't been forbidden from those pages using robots.txt, etc. Also make sure those URLs are correct. :)
Second, Google doesn't just take your sitemap at face value. It uses other factors, such as inbound links, etc, to determine whether it wants to crawl all of the pages in your sitemap. The sitemap then serves mostly as a hint more than anything else (it helps Google know when pages are updated more quickly, for example). Get high-quality, relevant, useful links (inbound and outbound) and your site should start getting indexed.
Your two statements seem to contradict one another.
but has only indexed 1 of them.
and
When I do a google search for site:url it picks up 2500+ results
bdonlan is correct in their logic (robot.txt and Google's lack of trust for sitemaps) but I think the issue is what you "think" is true about your site.
That is, Google Webmaster Tools says you only have 1 page indexed but site:yoursite.com shows 2.5k.
Google Webmaster Tools aren't very accurate. They are nice but they are buggy and MIGHT help you learn about issues about your site. Trust the site: command. Your in Google's index if you search site:yoursite.com and you see more than 1 result.
I'd trust site:yoursite.com. You have 2.5k pages in Google, indexed and search-able.
So, now optimize those pages and see the traffic flow. :D
Sidenote: Google can crawl any site, flash, javascript, etc.
What are other ways of making your website searchable by Google, other than submitting the link directly to Google.
Submitting links to yahoo is a breeze, gets crawled for a day or two... Google though takes a while...
Thanks...
if you add a link to your website on a website that's already indexed by google, google will follow that and reach your site without you needing to submit to their page. it's actually not recommended to submit your site to their page because then you're put at the end of the queue. but if you have a link on a page that google indexes in the next minute, it will get to you much faster. more links on many pages with higher ranking the better. cheers
Add your site to DMOZ.org, and encourage everyone you know to link to your site. The more places that link to your site, the more likely it'll get indexed sooner (and more fully), and the better it will rank.
Also, if your site is very large, it is not unreasonable to sign up for their webmaster tools and submit a sitemap index. This is especially effective for fast ranking, and showing up in obscure search results, but it will not help you rank for difficult terms.
Also note that if your site was visited by googlebot,
it doesn't necessarily end up in the google index.
Use this link to check:
http://www.google.com/webmasters/tools/sitestatus