Why is my website not Indexed by Search Engines?

Normally, It takes between 4 days and 4 weeks for your brand new website to be crawled and indexed by Google. This range, however, is fairly broad and has been challenged by those who claim to have indexed sites in less than 4 days. Even though Google’s inimitable search engine works on an algorithm, the eternal math that’s happening behind the scenes can’t produce a single, solid answer for us. Still, a guideline of 4 days to 1 month gives most webmasters a small amount of comfort while they wait to see where their page will appear in the search results pages.

However, there are ways you need to do to ensure that your site is indexable:

Your Site is Indexed Under a www- or Non-www Domain

Technically www is a subdomain. Thus, http://example.com is not the same as http://www.example.com. Make sure you add both sites to your GWT account to ensure they are both indexed. Be sure to set your preferred domain, but verify ownership of both.

Google Hasn’t Found Your Site Yet

This is usually a problem with new sites. Give it a few days (at least), but if Google still hasn’t indexed your site, make sure your sitemap is uploaded and working properly. If you haven’t created or submitted a sitemap, this could be your problem. You should also request Google crawl and fetch your site. Here is Google’s instructions on how to do that.

The Site or Page(s) are Blocked With robots.txt

Another problem is your developer or editor has blocked the site using robots.txt. This is an easy fix. Just remove the entry from the robots.txt, and your site will be reappear in the index. Read more about robots.txt here.

You Don’t Have a sitemap.xml

Every website should have a sitemap.xml, which is a simple list of directions that Google should follow to index your site. You can read about Google’s Sitemap policy, and create one pretty easily.

If you are experiencing indexation issues on any portion of your site, I recommend that you revise and resubmit your sitemap.xml just to make sure.

You Have Crawl Errors

In some cases, Google will not index some pages on your site because it can’t crawl them. Even though it can’t crawl them, it can still see them.

To identify these crawl errors, go to Google Webmaster Tools → Select your site, → Click on “Crawl” → Click on “Crawl Errors”. If you have any errors, i.e., unindexed pages, you will see them in the list of “Top 1,000 pages with errors.”

You Have Lots of Duplicate Content

Too much duplicate content on a site can confuse search engines and make them give up on indexing your site. If multiple URLs on your site are returning the exact same content, then you have a duplicate content issue on your site. To correct this problem, pick the page you want to keep and 301 the rest.

It sometimes makes sense to canonicalize pages, but be careful. Some sites have reported that a confused canonicalization issue has prevented indexation.

You’ve Turned On Your Privacy Settings

You may have accidentally kept the privacy settings on.

The Site is Blocked by .htaccess

Your .htaccess file is part of your website’s existence on the server, which allows it to be available on the world-wide web. The .htaccess file is written in Apache. Although .htacess is handy and useful, it can be used to block crawlers and prevent indexation.

The Site Has NOINDEX in the Meta Tag

Another way of saying “no” to the robots, and thus not having any indexation, is to have noindex meta tags. It often looks like this:

You Have AJAX/JavaScript Issues

Google does index JavaScript and AJAX. But these languages are not as easily indexable as HTML. So, if you are incorrectly configuring your AJAX pages and JavaScript execution, Google will not index the page.

Your Site is Slow to Load

Google doesn’t like it if your site takes an eternity to load. If the crawler encounters interminable load times, it will likely not index the site at all.

You Have Hosting Down Times

If the crawlers can’t access your site, they won’t index it. This is obvious enough, but why does it happen? Check your connectivity. If your host has frequent outage, it could be that the site isn’t getting crawled. Time to go shopping for a new host.

You Got Deinedexed

This one is really bad.

If you got hit with a manual penalty and removed from the index, you probably already know about it. If you have a site with a shady history (that you don’t know about) it could be that a lurking manual penalty is preventing indexation.

If your site has dropped from the index, you’re going to have to work very hard to get it back in.

Additional Notes:

It is always better to remember that:

- The indexed query system is not 100% accurate.

- There is a small percentage of sites that would never show up as indexed.

- If the site is new (new domain) there may be a long interval for it to be indexed in search engines, some may even take months.

- If the website appears in a regular domain search, please note that our query interval is 15 days.