TIPS and TRICKS
Google Inclusion
Q. My website has never been included in the Google search index.
A. Google is a mechanized search engine, which employs robots known as 'spiders' to crawl the web on a daily basis and find sites for inclusion in the Google index. Your best chance of getting your site spidered by Google is to increase your Link Popularity. You can also review Google's Basics to Submitting Your Site to Google. You should also make good use of the current tools that Google has available in Webmaster Central.
Reasons your site may not be included:
1. Your pages are dynamically generated. While Google can index dynamically generated pages, sites serving dynamic content can easily become overwhelmed and crash. Google limits the amount of dynamic pages that they'll index.
2. You employ doorway pages (commonly used by software like Web Position Gold). Google does not encourage the use of doorway pages. Why? Simple, they want their search results to point users to content pages, not to doorways or splash screens. Google feels that "doorway pages" clutter up their index and serve the visitor no purpose as it forces them an extra click to get to the content they want.
3. Your page uses frames. Google supports frames to the extent that it can. Frames tend to cause problems with search engines, bookmarks, emailing links and so on, because frames do not fit the conceptual model of the web (every page corresponds to a single URL). If a user's query matches the site as a whole, Google returns the frame set. If a user's query matches an individual page on the site, Google returns that page. That individual page is not displayed in a frame, because there may be no frame set corresponding to that page.
If you are concerned with the description of your site as seen by search engines, please read "Search Engines and Frames". It describes the use of the 'NoFrames' tag, which is used to provide alternative content. Instead of providing alternative content, you use wording such as "This site requires the use of frames" or "Upgrade your browser", then you are excluding both search engines and people who use browsers with frames turned off. (For example, audio web browsers, such as those used in automobiles and by the visually impaired, typically do not deal with frames, which are a visual mechanism.) You can read about NoFrames in the HTML standard.
4. Google does not index all of my pages. Why?
Although Google has more than 25 billion web pages in their index (actual number is not known), they cannot guarantee to crawl and include all the pages on a particular site. However, Google is always working to increase the number of pages they crawl and include in their next index update.
If your site's internal link structure does not provide a path to all your pages, Google's robot may not see all the pages on your site. Google follows links from one page to the next, so pages that are not linked through either text navigation or a complete Site Map they may not be spidered. This is also a good reason to use the Webmaster Tools
Q. My web pages used to be listed and now they aren't.
A. This can be caused by many different issues. Some of them include:
1. Changes from one index to the next. Each time Google updates their database of web pages (this happens everyday and is part of what is called "Everflux"), their index shifts: they find new sites, remove some sites, and site rankings change. If your site was dropped from Google and you have not made major changes to it in the last month, they will likely pick it up again in their next deep crawl. It's possible your site was simply inaccessible when their robots attempted to crawl it. Check the results in your Google Sitemaps account for likely problems, as it could be just something minor.
You may want to check and see if the number of other sites linking to your URL has decreased. This is the single biggest factor in determining what sites Google indexes, as they find most pages when their robots crawl the web and jump from page to page via hyperlinks. To find out who links to your site, simply type "link:www.yourdomain.com" in the search box at Google. Note: Make sure you substitute your actual domain name for the "your domain" in the example. Also, the search should be performed without quotes.
It is also possible your rank decreased because other sites were found and assigned a higher rank. You can be assured that no one at Google has manually adjusted the results to boost the ranking of another site. Google's order of results is automatically determined by several factors, including their PageRank algorithm. Please see "Why Use Google" for more information on how this works.
2. Multiple indices
Google updates their index every day. This is part of their "continuous update" they went to during the summer of 2003. If you happen to enter the same query repeatedly while they are in the process of posting the index at their various data centers, it might seem like you are seeing inconsistent results from Google. What is actually happening is that you are seeing a result from an 'old' version of the index one time and a result from a 'new' version the next. Due to the size of their index, they cannot simultaneously post a new index at all of their data centers, which may result in this behavior for a short period of time.
3. Other reasons
If your page does not appear at all, here are some other possible explanations.
1. Your site may not have been reachable when Google attempted to crawl it because of network or hosting problems. When this happens, Google will retry multiple times, but if the site cannot be crawled, it will not be listed in the current index.
A technical glitch on Google's side may have caused them to 'miss' your site. In crawling more than 25 billion pages every few weeks, their system experiences hiccups from time to time. Please be patient with Google during this period, as we are not able to modify our index by hand to add sites missed in this way.
The contents of your page or the links pointing to your page changed significantly and you no longer have a sufficiently high PageRank, or your page had low PageRank to begin with and a small change caused you to be dropped from the Google index.
Your page was manually removed because it did not conform to the quality standards necessary to assign accurate PageRank. Google will not comment on the individual reasons a page was removed and they do not offer an exhaustive list of practices that can cause removal.
However, certain actions such as cloaking, writing text that can be seen by search engines but not by users, or setting up pages/links with the sole purpose of fooling search engines may result in permanent removal from their index.
If you believe your site may fall into this category, you might try 'cleaning up' the page and sending a re-inclusion request to help@google.com, or going through your Google Sitemap's account, which is the preferred method.
